Title: Non-Rigid Shape Registration via Deep Functional Maps Prior
Venue: NeurIPS 2023
Accepted: yes
Keywords: shape registration; functional maps; unsupervised learning
Abstract: In this paper, we propose a learning-based framework for non-rigid shape registra- tion without correspondence supervision. Traditional shape registration techniques typically rely on correspondences induced by extrinsic proximity, therefore can fail in the presence of large intrinsic deformations. Spectral mapping methods overcome this challenge by embedding shapes into, geometric or learned, high- dimensional spaces, where shapes are easier to align. However, due to the dependency on abstract, non-linear embedding schemes, the latter can be vulnerable with respect to perturbed or alien input. In light of this, our framework takes the best of both worlds. Namely, we deform source mesh towards the target point cloud, guided by correspondences induced by high-dimensional embeddings learned from deep functional maps (DFM). In particular, the correspondences are dynamically updated according to the intermediate registrations and filtered by consistency prior, which prominently robustify the overall pipeline. Moreover, in order to alleviate the requirement of extrinsically aligned input, we train an orientation regressor on a set of aligned synthetic shapes independent of the training shapes for DFM. Empirical results show that, with as few as dozens of training shapes of limited variability, our pipeline achieves state-of-the-art results on several benchmarks of non-rigid point cloud matching, but also delivers high-quality correspondences between unseen challenging shape pairs that undergo both significant extrinsic and intrinsic defor- mations, in which case neither traditional registration methods nor intrinsic methods work. The code is available at https://github.com/rqhuang88/DFR.

Non-Rigid Shape Registration via Deep Functional Maps Prior

Abstract

In this paper, we propose a learning-based framework for non-rigid shape registration without correspondence supervision . Traditional shape registration techniques typically rely on correspondences induced by extrinsic proximity, therefore can fail in the presence of large intrinsic deformations. Spectral mapping methods overcome this challenge by embedding shapes into, geometric or learned, highdimensional spaces, where shapes are easier to align. However, due to the dependency on abstract, non-linear embedding schemes, the latter can be vulnerable with respect to perturbed or alien input. In light of this, our framework takes the best of both worlds. Namely, we deform source mesh towards the target point cloud, guided by correspondences induced by high-dimensional embeddings learned from deep functional maps (DFM). In particular, the correspondences are dynamically updated according to the intermediate registrations and filtered by consistency prior, which prominently robustify the overall pipeline. Moreover, in order to alleviate the requirement of extrinsically aligned input, we train an orientation regressor on a set of aligned synthetic shapes independent of the training shapes for DFM. Empirical results show that, with as few as dozens of training shapes of limited variability, our pipeline achieves state-of-the-art results on several benchmarks of non-rigid point cloud matching, but also delivers high-quality correspondences between unseen challenging shape pairs that undergo both significant extrinsic and intrinsic deformations, in which case neither traditional registration methods nor intrinsic methods work. The code is available at https://github.com/rqhuang88/DFR .

1 Introduction

Estimating correspondences between deformable shapes is a fundamental task in computer vision and graphics, with a significant impact on an array of applications including robotic vision [50], animation [40], 3D reconstruction [58], to name a few. In this paper, we tackle the challenging task of estimating correspondences between unstructured point clouds sampled from surfaces undergoing significant non-rigid deformations.

In fact, this task has attracted increasing interest [59; 26; 35; 24; 8] in shape matching community. These methods all follow a data-driven approach to learn embedding schemes, which project point clouds into certain high-dimensional spaces. By lifting to high-dimensional spaces, the non-rigid deformations are better characterized than in the ambient space, R 3 . The optimal transformations and therefore the correspondences are then estimated in the embedded spaces. Despite the great progresses, such approaches suffer from the following limitations: 1) Their performance on unseen shapes is largely unwarranted; 2) The learned high-dimensional embeddings lack intuitive geometric meaning, and the induced correspondences are difficult to evaluate and analyze without ground-truth labels.

Registration

Template

NDP

Ours

Target

Onthe other hand, the above limitations are less of a concern from the perspective of shape registration, either axiomatic [1; 55] or learning-based [31]. Unlike the former, the latter deform a source shape to non-rigidly align with a target explicitly in the ambient space R 3 . In consequence, the mapping quality can be directly related to the alignment quality. Given a deformed source shape and a target, one can evaluate both qualitatively (by visual comparison) and quantitatively (by computing Chamfer distance), and, more importantly, further optimize the registration outcome via the correspondencelabel-free quantitative metric. However, shape registration itself does not serve as a direct rescue to our problem of interest, as the previous approaches typically rely on the premise that the undergoing non-rigid deformation can be approximated by a set of local, small-to-moderate, rigid deformations, which severely hinders their performance in the presence of large deformations (see Fig. 1) and heterogeneity.

Motivated by the above observations, we propose a framework to take the best of classic shape registration and learning-based embedding techniques. In a nutshell, we leverage the estimated correspondence from the latter to guide shape deformation via the former iteratively. Our key insight is to enforce similarity between the deformed source and target in both ambient space and learned high-dimensional space . Intuitively, in the presence of large deformation, the learned correspondence is more reliable than that obtained by proximity in the ambient space. On the other hand, properly deforming source mesh to target point cloud increases spatial similarity in the ambient space, leading to higher similarity in the embedded space and therefore more accurate correspondences. In the end, we can compute correspondences between raw point clouds with a shared source mesh as a hub. As will be shown in Sec. 4, our method allows for choosing source shape during inference, which can be independent of the training data.

While being conceptually straightforward, we strive to improve the performance, robustness and practical utility of our pipeline by introducing several tailored designs. First of all, the key component of our pipeline is an embedding scheme for accurately and efficiently estimating correspondences between the deformed source shape and the target point cloud . In particular, to take advantage of Deep Functional Maps (DFM) [29; 7; 49], the current state-of-the-art approach on matching triangular meshes , we pre-train an unsupervised DFM [49] on meshes as a teacher net and then learn a pointbased feature extractor ( i.e. , an embedding scheme for points) as a student net on the corresponding vertex sets with a natural self-supervision. Unlike the approach taken in [8], which heavily relies on DiffusionNet [47] to extract intricate structural details (e.g., Laplace-Beltrami operators) from both mesh and point cloud inputs, the teacher-student paradigm allows us to utilize a more streamlined backbone - DGCNN [53]. Secondly, in contrast to prior works that either implicitly [31; 55] or explicitly [46; 8; 24] require rigidly aligned shapes for input/initialization, or demand dense correspondence labels [35]. We train an orientation regressor on a large set of synthetic, rigidly aligned shapes to automatically pre-process input point clouds in arbitrary orientations. Last but not least, we propose a dynamic correspondence updating scheme, bijectivity-based correspondence filtering, two-stage registration in our pipeline. We defer the technical details to Sec. 3.

Overall, our framework enjoys the following properties: 1) Due to the hybrid nature of our framework, it can handle point clouds undergoing significant deformation and/or heterogeneity; 2) As our feature extractor is self-supervised by some deep functional maps network, our framework is free of correspondence annotation throughout; 3) Our core operation is performed in the ambient space, enabling more efficient, faithful, and straightforward analysis of registration/mapping results than the

TransMatch

Ours

SSMSM

Source

purely learning-based approaches; 4) Based on a data-driven orientation regressor, we achieve an automatic pipeline for estimating correspondences between deformable point clouds.

As shown in Fig. 2, trained on the SCAPE [2] dataset, our framework generalizes well and outperforms the competing methods by a large margin in the challenging SHREC07-H dataset, which presents prominent variability in both extrinsic orientation and intrinsic geometry. We conduct a set of experiments to verify the effectiveness of our pipeline. We highlight that it achieves state-of-the-art results in matching non-rigid point clouds in both near-isometric and heterogeneous shape collections. More remarkably, it generalizes well despite the distinctiveness between the training set and test set. Moreover, our method, being point-based, is more robust with respect to topological perturbations within input shapes than the mesh-based baselines.

2 Related Works

Non-rigid Shape Registration: Unlike the rigid counterpart, non-rigidly aligning shapes is more challenging because of the perplexity of deformation models. In general, axiomatic approaches [1] assume the deformation of interest can be approximated by local, small-to-moderate, rigid deformations, therefore suffer from large intrinsic deformations. In fact, recent advances such as [60; 28; 55] mainly focus on improving the efficiency and robustness regarding noises and outliers. On the other hand, there exists as well a trend of incorporating deep learning techniques [6; 5; 21]. Among them, NDP [31] is similar to our method in the sense that the non-learned version follows an optimizationbased approach based on some neural encoding. However, its shape registration is purely guided by proximity in the ambient space, hindering its performance in our problem of interest. Perhaps the most relevant approach with our method along this line is TransMatch [51], which follows a supervised learning scheme and learns a transformer to predict directly 3D flows between input point clouds. As demonstrated in Fig. 2 and further in Sec. 4, as a supervised method, TransMatch generalizes poorly to unseen shapes.

Non-rigid Point Cloud Matching: Unlike the above, several recent approaches [26; 30; 59] directly establish correspondence between a pair of point clouds. In general, these methods embed point clouds into a canonical feature space, and then estimate correspondences via the respective proximity. Leapard [30] learns such feature extractor under the supervision of ground-truth correspondences between partial point clouds with potentially low overlaps. On the other hand, DPC [26] and CorrnetNet3D [59] leverage point cloud reconstruction as proxy task to learn embeddings without correspondence label. Since intrinsic information is not explicitly formulated in these methods, they can suffer from significant intrinsic deformations and often generalize poorly to unseen shapes.

Deep Functional Maps: Stemming from the seminal work of functional maps [39] and a series of follow-ups [38; 22; 44; 37; 23], spectral methods have achieved significant progress in axiomatic non-rigid shape matching problem, and consequentially laid solid foundation for the more recent advances on Deep Functional Maps (DFM), which is pioneered by Litany et al. [32]. Unlike axiomatic functional maps frameworks that aim at optimizing for correspondences represented in spectral basis (so-called functional maps) with hand-crafted features as prior . DFM takes an inverse

Stage1

Stage II

Registration

Registration

A

F

IlsT

IlsT

Fs

Vs*

S

V

A

F

T

Orientation

PointFeature

High-dimensional

Coordinates

A

Regressor

F

Extractor

featureembedding

asfeature

Permutationmatrix

viewpoint and aims to search for the optimal features, such that the induced functional maps satisfy certain structural priors as well as possible. Recent findings along this direction empirically suggest that the structural priors suffice to train an effective feature extractor, without any correspondence supervision [12; 29; 7; 3]. However, because of the heavy dependence of eigenbasis of LaplaceBeltrami operators, DFM is primarily designed for shapes represented by triangle meshes and can suffer notable performance drop when applied to point clouds without adaptation [8].

In fact, inspired by the success of DFM, several approaches [24; 8] have been proposed to leverage intrinsic geometry information carried by meshes in the training of feature extractors tailored for nonstructural point clouds. Among these, NIE [24] and SSMSM [8] are closely related to our approach. As all three works attempt to train an effective, intrinsic-geometry-aware, per-point feature extractor by leveraging meshes during training. However, unlike the former two, our method incorporates optimization-based shape registration techniques, which effectively alleviates the limitation of purely relying on an abstract high-dimensional embedding scheme ( i.e. , the learned feature extractor).

Combination of Extrinsic and Intrinsic Methods: Unlike the above, our method operates in both extrinsic and intrinsic spaces. From this point of view, the most relevant work is Smooth Shells [14], which proposes to perform shape registration under a mixed representation of shapes, including eigenbasis, coordinates, and normal. Though achieving considerable high-quality correspondence on standard benchmarks, it heavily relies on the availability of mesh representation, but also is sensitive to the original extrinsic alignment of input shapes. These limitations are inherited by Deep Shells [16] - the learning version of the former. Similarly, NeuralMorph [15] jointly performs interpolation and correspondence estimation between a pair of shapes, which inject the intrinsic information by imposing geodesic preservation loss. As a result, it is non-trivial to lift the requirement of mesh input.

3 Methodology

Given a pair of shapes S , T , our target is to deform S to non-rigidly align with T . We assume that S is represented as a triangle mesh so that we can effectively regularize the deformed shape by preserving local intrinsic geometry. On the other hand, we require no structural information on T and generally assume it to be a point cloud . Our pipeline can also be extended to compute correspondences between two raw point clouds T 1 , T 2 . To this end, we fix a template mesh S , perform respective shape registration between S and the target point clouds, and finally compute the map by composition T 12 = T s 2 ◦ T 1 s .

Our pipeline is shown in Fig. 3, which consists of three main components: 1) An orientation regressor, A , for extrinsically aligning input shapes, either mesh or point cloud; 2) A point feature extractor, F ,

trained under deep functional maps scheme; 3) A registration module that iteratively optimizes for deformations non-rigidly aligning S with T . In particular, it takes the rigidly aligned shapes from 1) as input, and leverages 2) to update correspondences during optimization.

Though our pipeline leverages a pre-trained feature extractor as registration prior, we highlight that neither the source nor the target is necessarily within or close to the respective training set. We provide throughout experimental evidence in Sec. 4 showing the generalizability of our pipeline.

3.1 Orientation Regressor

Our first objective is to align input point clouds with arbitrary orientations into a canonical frame. We take a data-driven approach by training an orientation regressor [9] on 5000 synthetic SURREAL shapes from [52], which are implicitly aligned to a canonical frame by the corresponding generative codes. We refer readers to Supplementary Materials for more details.

3.2 Point Feature Extractor

In order to efficiently and accurately estimate correspondences between deformed source shape and the target point cloud during registration, our next goal is to train a feature extractor for point clouds, which is intrinsic-geometry aware. The authors of [8] propose a multi-modal feature extractor based on DiffusionNet [47], which can process point clouds with an extra step of graph Laplacian construction [48]. Though it has demonstrated satisfying performance accommodating point cloud representation in [8], the explicit graph Laplacian construction is computationally heavy. Hence, we adopt the modified DGCNN proposed in [24] as our backbone, which is lightweight and robust regarding the sampling density and sizes of point clouds.

In particular, our training follows a teacher-student paradigm. Namely, we first train a deep functional maps (DFM) network, G , on a collection of meshes. Then we train a DFM network F on the corresponding vertex sets, with an extra self-supervision according to the inclusion map between meshes and their vertex sets. In other words, the feature produced by F is point-wisely aligned with that produced by G .

Training DFM on Meshes: Our training scheme in principle follows that of [49], which delivers both accurate map estimation and excellent generalization performance.

We take a pair of shapes S 1 , S 2 , and compute the leading k eigenfunctions of the Laplace-Beltrami operator on each shape. The eigenfunctions are stored as matrices Φ i ∈ R n i × k , i = 1 , 2 .

We compute G 1 = G ( S 1 ) , G 2 = G ( S 2 ) , projected into the spaces spanned by Φ 1 , Φ 2 respectively, and leverage the differential FMReg module proposed in [13] to estimate the functional maps C 12 , C 21 between the two shapes, then by regularizing the structure of C 12 , C 21 , say:

3) To enhance G , we estimate the correspondences between S 1 , S 2 via the proximity among the rows of G 1 , G 2 respectively. Namely, we compute

where δ ij = ∥ G 1 ( i, :) -G 2 ( j, :) ∥ and α is the temperature parameter, which is increased during training [49]. Then Π 12 is the soft map between the two shapes. Ideally, it should be consistent with the functional maps estimated in Eqn.(1), giving rise to the following loss:

where † denotes to the pseudo-inverse. To sum up, G ∗ = arg min E DFM ( G ) , which is defined as:

Training DFM on Point Clouds: Now we are ready for training F with a modified DGCNN [24] as backbone. We let F 1 = F ( S 1 ) , F 2 = F ( S 2 ) . On top of E DFM in Eqn. (4), we further enforce the

extracted feature from F to be aligned with that from G ∗ via PointInfoNCE loss [54]:

where ⟨ . ⟩ refers to the dot products, γ is the temperature parameter, and n i is the number of points of S i . So the final training loss is defined as:

which is optimized on all pairs of shapes within the training set.

3.3 Shape Registration

Now we are ready to formulate our registration module. We first describe the deformation graph construction in our pipeline. Then we formulate the involved energy functions and finally describe the iterative optimization algorithm. During registration, we denote the deformed source model by S k = {V k , E} , V k = { v k i | i = 1 , . . . , N } , where k indicates the iteration index and v k i is the position of the i -th vertex at iteration k . The target point cloud is denoted by T = { u j | j = 1 , . . . , M } .

Deformation Graph: Following[20], we reduce the vertex number on S to H = [ N/ 2] with QSlim algorithm [17]. Then an embedded deformation graph DG is parameterized with axis angles Θ ∈ R H × 3 and translations ∆ ∈ R H × 3 : X = { Θ , ∆ } . The vertex displacements are then computed from the corresponding deformation nodes. For a given X k , we can compute displaced vertices via:

Dynamic correspondence update: Thanks to our point-based feature extractor F , we can freely update correspondences between deforming source mesh and target point clouds. In practice, for the sake of efficiency, we update every 100 iterations in optimization (see Alg. 1).

Correspondence Filtering via Bijectivity: One key step in shape registration is to update correspondences between the deforming source and the target. It is then critical to reject erroneous corresponding points to prevent error accumulation over iterations. In particular, we propose a filter based on the well-known bijectivity prior. Given S k and T , we first compute maps represented as permutation matrices Π ST , Π T S , either based on the learned feature or the coordinates. For v k i ∈ S k , we compute the geodesic on S k between v k i and its image under permutation Π ST Π T S , then reject all v k i if the distance is above some threshold. For the sake of efficiency, we pre-compute the geodesic matrix of S and approximate that of S k with it. Finally, we denote the filtered set of correspondences by C k = { ( v k i 1 , u j 1 ) , ( v k i 2 , u j 2 ) · · · } .

In the following, we introduce the energy terms regarding our registration pipeline.

Correspondence Term measures the distance of filtered correspondences between S k and T , given as:

Chamfer Distance Term has been widely used [31; 30] to measure the extrinsic distance between S k and T :

As-rigid-as-possible Term reflects the deviation of estimated local surface deformations from rigid transformations. We follow [20; 27] and define it as:

Algorithm 1: Shape registration pipeline.

Input: Source mesh S = {V , E} and target point cloud T ;Trained point feature extractor F Output: X ∗ converging to a local minimum of E total ; Deformed source model {V ∗ , E} ; Correspondence Π ∗ , Π ∗ between S and T .

Here, g ∈ R H × 3 are the original positions of the nodes in the deformation graph DG , and ψ ( h ) is the 1-ring neighborhood of the h -th deformation node. R ( · ) is Rodrigues' rotation formula that computes a rotation matrix from an axis-angle representation and α is the weight of the smooth rotation regularization term.

Total cost function: The total cost function E total combines the above terms with the weighting factors λ cd , λ corr , and λ arap to balance them:

Now we are ready to describe our algorithm for minimizing E total , which is shown in Alg. 1.

Two-stage Registration: Finally, we observe that solely depending on learned features to infer correspondence is sub-optimal. At the converging point, the deformed source shape is often at the right pose but has deficiencies in shape style. To this end, we perform a second-stage registration, based on the coordinates of deforming source and target. As shown in Sec. 4.2, such a design is clearly beneficial.

4 Experiments

Datasets: We evaluate our method and several state-of-the-art techniques for estimating correspondences between deformable shapes on an array of benchmarks as follows: FAUST_r: The remeshed version of FAUST dataset [4], which consists of 100 human shapes (10 individuals performing the same 10 actions). We split the first 80 as training shapes and the rest as testing shapes; SCAPE_r: The remeshed version of SCAPE dataset [2], which consists 71 human shapes (same individual in varying poses). We split the first 51 as training shapes and the rest as testing shapes; SHREC19_r: The remehsed version of SHREC19 dataset [36], which consists of 44 shapes of different identities and poses. We use it solely in test, and follow the test pair list provided by [36]; SHREC07-H: A subset of SHREC07 dataset [18], which consists of 20 heterogeneous human shapes of the varying number of vertices. We use it solely in test, and use the accompanied sparse landmark annotations to quantify all the pairwise maps among them; DT4D-H: A dataset proposed in [34], which consists of 10 categories of heterogeneous humanoid shapes. We use it solely in testing, and evaluating the inter-class maps split in [29]; TOPKIDS: This is a challenging dataset [25] consisting of 26 shapes of a kid in different poses, which manifest significant topological perturbations in meshes.

Baselines: We compare our method with an array of competitive baselines, including axiomatic shape registration methods: Smooth Shells [14], NDP [31], AMM [55]; learning-based registration

methods: 3D-CODED [19], Deep Shells [16], TransMatch [51], SyNoRiM [21]; deep functional maps frameworks: SURFMNet [45], WSupFMNet [46], GeomFMaps [13], DiffFMaps [35], NIE [24], ConsistFMaps [7], SSMSM [8]. According to input requirements, we put those relying on pure mesh input on the top, and the rest, suitable for point clouds, in the bottom of both tables.

Train/Test Cases: Throughout this section, we consider a highly challenging scenario: for each learning-based method, we train two models respectively with FAUST_r and SCAPE_r dataset, and then we run tests in a range of test cases including FAUST_r, SCAPE_r, SHREC19_r, SHREC07-H, DT4D-H and TOPKIDS. The test pairs of each case have been described above. In the following, A/B means we train on dataset A and test on dataset B .

Datasets Alignment: There exist extrinsically aligned versions for all the involved datasets but SHREC07-H. We equally feed in aligned datasets to all the baselines when available. On the other hand, the original version of SHREC07-H manifests significant variability of orientations. For the sake of fairness and simplicity, we apply our orientation regressor to it and provide baselines for our automatically aligned data. Note that, all the aligned datasets, as well as the synthetic dataset on which we train our orientation regressor, roughly share the same canonical orientation, which is defined by the SMPL [33] model. Finally, we always apply automatic alignment before inputting shapes into our pipeline, whatever the original orientation they are.

Choice of Source Shape: As mentioned at the beginning of Sec. 3, we compute correspondences between two raw input point clouds by associating them via a common source mesh S . In Tab. 1 and Tab. 2, we indicate our choice of source mesh by Oursname , where name indicates the origin of the source mesh. For simplicity, we fix the source mesh from each dataset and visualize them in the appendix. On the other hand, when implementing axiomatic shape registration methods, we only consider deforming the same template we pick from FAUST_r (resp. SCAPE_r) to every test point cloud in the test set of FAUST_r (resp. SCAPE_r), and establish correspondence by map composition, in the same manner as ours.

Metric: Though we primarily focus on matching point clouds, we adopt the commonly used geodesic error normalized by the square root of the total area of the mesh, to evaluate all methods, either for mesh or point cloud.

Hyper-Parameters: We remark that all the hyper-parameters are fixed for all experiments in this section. In particular, we settle them by performing a grid search with respect to the weights used in the final optimization to seek for the combination that leads to the the best registration results (quantitatively in terms of Chamfer distance and qualitatively by visual inspection) on a few training shapes. We provide more detailed discussion and ablation of the choice of hyper-parameters in the appendix.

4.1 Experimental Results

Near-isometric Benchmarks: As shown in Fig. 1, in the presence of large deformation between the template and the target, NDP[31] and AMM[55] fail completely while our method delivers highquality deformation. Moreover, as illustrated in Table 1, our method achieves the best performance in 5 out of 6 test cases. Remarkably, in the two most challenging tasks, FAUST_r/SHREC19_r and FAUST_r/SHREC19_r, our method indeed outperforms all of the baselines, including the state-ofthe-art methods that take meshes as input. Regarding point-based methods, SSMSM [8] performs well in the standard case and outperforms ours in FASUT_r/FAUST_r, but generalizes poorly to unseen shapes. Another important observation is that our method manifests robustness with respect to the choice of template shapes. In fact, the above observations remain true no matter which template we select.

Non-isometric Benchmarks: We stress test our method on challenging non-isometric datasets including SHREC07-H and DT4D-H. We emphasize that these test shapes are unseen during training.

1) SHREC07-H contains 20 heterogeneous human shapes, whose number of vertices ranges from 3000 to 15000 . Moreover, there exists some topological noise ( e.g. , the hands of the rightmost shape in Fig. 2 is attached to the thigh in mesh representation). As shown in Fig. 2, SSMSM [8] barely delivers reasonable results, which might be due to the sensitivity of graph Laplacian construction on point clouds. Topological noise, on the other hand, degrades mesh-based methods like Deep Shells [16]. Meanwhile, as shown in Tab. 2, our method achieves a performance improvement of

over 40% compared to the previous SOTA approaches ( 8 . 5 vs. 15.3; 5 . 9 vs. 13.4), which again confirms the robustness of our approach. 2) Regarding DT4D-H, We follow the test setting of AttentiveFMaps [29], and only consider the more challenging inter-class mapping. Interestingly, our method outperforms the state-of-the-art point cloud-based approach by approximately 30% and exceeds the performance of mesh-based state-of-the-art methods by over 70%. Furthermore, even in the worst case, training on FAUST_r and using SCAPE template, our error is still the lowest compared to external baselines.

Topologically Perturbed Benchmark: The shapes in TOPKIDS present various adhesions hand/arm adhered to abdomen/thigh (see Fig. 6 in the appendix). Given the ground-truth maps from each topologically perturbed shape to the reference shape, we evaluate all 25 maps (excluding the trivial one from reference to itself) with models trained on FAUST_r and SCAPE_r respectively. Since the orientation of the TOPKIDS dataset is not originally agreeable with the aligned version of the two training sets, we align the input with our orientation regressor and feed them into the baselines depending on extrinsic information [16; 35; 24; 8]. Regarding the template shape, we adopt the reference shape as our source mesh in registration. We report the quantitative comparisons in Tab. 3. It is obvious that topological noise poses great challenges for methods based on pure intrinsic information [13; 7; 29], while the counterparts perform relatively well. Especially, among the point-based methods, our method outperforms the latest SOTA method by a large margin, namely, over 40% relative error reduction ( 7 . 1 vs. 12 . 3 ). We refer readers to the appendix for a qualitative comparison, which agrees with the quantitative results above.

4.2 Ablation Study

We report ablation studies in Tab. 4, in which we verify the effectiveness of each core design formulated in Sec. 3. Throughout this part, we train on SCAPE_r and test on SHREC07-H. In the registration stage, we use the SCAPE_r template as the source mesh and deform it to each point cloud from SHREC07-H. It is evident that each ablated module contributes to the final performance. In particular, both stages of registration play an equally important role in our framework.

Finally, to validate the superiority of our registration scheme, we report in Tab. 5 the average errors of the initial maps computed by our point-based DFM, and that of output maps of Ours, NDP [31], AMM[55], which are all based on the former. It is evident that, across three different test sets, our method consistently improves the initial maps, while NDP and AMM can even lead to deteriorated maps than the initial input.

5 Conclusion and Limitation

In this paper, we propose a novel learning-based shape registration framework without correspondence supervision. Our framework incorporates learning-based shape matching and optimization-based shape registration. Based on the former, our framework can perform registration between shapes undergoing significant intrinsic deformations; On the other hand, thanks to the latter, our method exhibit superior generalizability over the learning-based competitors. Apart from several designs tailored for our intuitive pipeline, we also introduce a data-driven solution to facilitate the burden of extrinsically aligning non-rigid points. We verify our framework through a series of challenging tests, in which it shows superior performance but also remarkable robustness.

Limitation & Future Work We identify two main limitations of our method: 1) Our optimization procedure involves iteratively updating correspondences between the intermediate point clouds and the target one, leaving sufficient room for improvement for efficiency; 2) Our pipeline is designed for full shapes, while it is applicable to partial targets, the registration quality is sub-optimal. In the future, we also plan to incorporate more existing techniques ( e.g. , various regularization terms) from shape registration to further enhance the overall performance.

Acknowledgement The authors thank anonymous reviewers for their constructive comments and suggestions. This work was supported in part by the National Natural Science Foundation of China under contract No. 62171256, 62331006, in part by Shenzhen Key Laboratory of next-generation interactive media innovative technology (No. ZDSYS20210623092001004).

References

Brian Amberg, Sami Romdhani, and Thomas Vetter. Optimal step nonrigid icp algorithms for surface registration. 2007.

Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. SCAPE: Shape Completion and Animation of People. 2005.

Souhaib Attaiki and Maks Ovsjanikov. Understanding and improving features learned in deep functional maps. In CVPR , 2023.

Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. FAUST: Dataset and evaluation for 3D mesh registration. In CVPR , 2014.

Aljaz Bozic, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, and Matthias Nießner. Neural non-rigid tracking. In NeurIPS , volume 33, pages 18727-18737, 2020.

Aljaz Bozic, Michael Zollhofer, Christian Theobalt, and Matthias Nießner. Deepdeform: Learning non-rigid rgb-d reconstruction with semi-supervised data. In CVPR , pages 7002-7012, 2020.

Dongliang Cao and Florian Bernard. Unsupervised deep multi-shape matching. In ECCV , 2022.

Dongliang Cao and Florian Bernard. Self-supervised learning for multimodal non-rigid shape matching. In CVPR , 2023.

Jiayi Chen, Yingda Yin, Tolga Birdal, Baoquan Chen, Leonidas J Guibas, and He Wang. Projective manifold gradient layer for deep rotation regression. In CVPR , 2022.

Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas. Vector neurons: a general framework for so(3)-equivariant networks. In ICCV , 2021.

Jiacheng Deng, Chuxin Wang, Jiahao Lu, Jianfeng He, Tianzhu Zhang, Jiyang Yu, and Zhe Zhang. Se-ornet: Self-ensembling orientation-aware network for unsupervised point cloud shape correspondence. arXiv preprint arXiv:2304.05395 , 2023.

Nicolas Donati, Etienne Corman, and Maks Ovsjanikov. Deep orientation-aware functional maps: Tackling symmetry issues in shape matching. In CVPR , pages 742-751, 2022.

Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov. Deep geometric functional maps: Robust feature learning for shape correspondence. In CVPR , 2020.

Marvin Eisenberger, Zorah Lahner, and Daniel Cremers. Smooth shells: Multi-scale shape registration with functional maps. In CVPR , 2020.

Marvin Eisenberger, David Novotny, Gael Kerchenbaum, Patrick Labatut, Natalia Neverova, Daniel Cremers, and Andrea Vedaldi. Neuromorph: Unsupervised shape interpolation and correspondence in one go. In CVPR , 2021.

Marvin Eisenberger, Aysim Toker, Laura Leal-Taixé, and Daniel Cremers. Deep shells: Unsupervised shape correspondence with optimal transport. In NeurIPS , 2020.

Michael Garland and Paul S Heckbert. Surface simplification using quadric error metrics. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques , pages 209-216, 1997.

Daniela Giorgi, Silvia Biasotti, and Laura Paraboschi. Shape retrieval contest 2007: Watertight models track. SHREC competition , 2007.

Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. 3d-coded: 3d correspondences by deep deformation. In ECCV , 2018.

Chen Guo, Xu Chen, Jie Song, and Otmar Hilliges. Human performance capture from monocular video in the wild. In 3DV , 2021.

Jiahui Huang, Tolga Birdal, Zan Gojcic, Leonidas J. Guibas, and Shi-Min Hu. Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-1, 2022.

Ruqi Huang and Maks Ovsjanikov. Adjoint map representation for shape analysis and matching. In Computer Graphics Forum , volume 36, pages 151-163. Wiley Online Library, 2017.

Ruqi Huang, Jing Ren, Peter Wonka, and Maks Ovsjanikov. Consistent zoomout: Efficient spectral map synchronization. In Computer Graphics Forum , volume 39, pages 265-278. Wiley Online Library, 2020.

Puhua Jiang, Mingze Sun, and Ruqi Huang. Neural intrinsic embedding for non-rigid point matching. In

CVPR , 2023.

Zorah Lähner, Emanuele Rodolà, Michael M Bronstein, Daniel Cremers, Oliver Burghard, Luca Cosmo, Andreas Dieckmann, Reinhard Klein, and Yusuf Sahillioglu. Shrec'16: Matching of deformable shapes with topological noise. Proc. 3DOR , 2(10.2312), 2016.

Itai Lang, Dvir Ginzburg, Shai Avidan, and Dan Raviv. Dpc: Unsupervised deep point correspondence via cross and self construction. In 2021 International Conference on 3D Vision (3DV) , pages 1442-1451. IEEE, 2021.

Zohar Levi and Craig Gotsman. Smooth rotation enhanced as-rigid-as-possible mesh animation. IEEE transactions on visualization and computer graphics , 21(2):264-277, 2014.

Kun Li, Jingyu Yang, Yu-Kun Lai, and Daoliang Guo. Robust non-rigid registration with reweighted position and transformation sparsity. IEEE transactions on visualization and computer graphics , 25(6):22552269, 2018.

Lei Li, Nicolas Donati, and Maks Ovsjanikov. Learning multi-resolution functional maps with spectral attention for robust shape matching. In NeurIPS , 2022.

Yang Li and Tatsuya Harada. Lepard: Learning partial point cloud matching in rigid and deformable scenes. In CVPR , 2022.

Yang Li and Tatsuya Harada. Non-rigid point cloud registration with neural deformation pyramid. In NeurIPS , 2022.

Or Litany, Tal Remez, Emanuele Rodolà, Alexander M. Bronstein, and Michael M. Bronstein. Deep functional maps: Structured prediction for dense shape correspondence. In ICCV , 2017.

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. TOG , 34(6):248:1-248:16, 2015.

Robin Magnet, Jing Ren, Olga Sorkine-Hornung, and Maks Ovsjanikov. Smooth non-rigid shape matching via effective dirichlet energy optimization. In 3DV , 2022.

Riccardo Marin, Marie-Julie Rakotosaona, Simone Melzi, and Maks Ovsjanikov. Correspondence learning via linearly-invariant embedding. Advances in Neural Information Processing Systems , 33:1608-1620, 2020.

S. Melzi, R. Marin, E. Rodolà, U. Castellani, J. Ren, A. Poulenard, P. Wonka, and M. Ovsjanikov. Matching Humans with Different Connectivity. In Eurographics Workshop on 3D Object Retrieval , 2019.

Simone Melzi, Jing Ren, Emanuele Rodolà, Peter Wonka, and Maks Ovsjanikov. Zoomout: Spectral upsampling for efficient shape correspondence. Proc. SIGGRAPH Asia , 2019.

Dorian Nogneng and Maks Ovsjanikov. Informative descriptor preservation via commutativity for shape matching. Computer Graphics Forum , 36(2):259-267, 2017.

Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas. Functional Maps: A Flexible Representation of Maps Between Shapes. ACM Transactions on Graphics (TOG) , 31(4):30, 2012.

Gianluca Paravati, Fabrizio Lamberti, Valentina Gatteschi, Claudio Demartini, and Paolo Montuschi. Point cloud-based automatic assessment of 3d computer animation courseworks. IEEE Transactions on Learning Technologies , 10(4):532-543, 2016.

Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov. Effective rotation-invariant point cnn with spherical harmonics kernels. In 3DV , 2019.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , 2017.

Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurIPS , 2017.

Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovsjanikov. Continuous and orientation-preserving correspondences via functional maps. ACM Trans. Graph. , 37(6):248:1-248:16, Dec. 2018.

Jean-Michel Roufosse, Abhishek Sharma, and Maks Ovsjanikov. Unsupervised deep learning for structured shape matching. In ICCV , 2019.

Abhishek Sharma and Maks Ovsjanikov. Weakly supervised deep functional maps for shape matching. In NeurIPS , 2020.

Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic learning on surfaces. ACM Transactions on Graphics , 2022.

Nicholas Sharp and Keenan Crane. A laplacian for nonmanifold triangle meshes. Computer Graphics Forum , 2020.

Mingze Sun, Shiwei Mao, Puhua Jiang, Maks Ovsjanikov, and Ruqi Huang. Spatially and spectrally consistent deep functional maps. In ICCV , 2023.

Carlos Sánchez-Belenguer, Simone Ceriani, Pierluigi Taddei, Erik Wolfart, and Vítor Sequeira. Global matching of point clouds for scan registration and loop detection. Robotics and Autonomous Systems , 123:103324, 2020.

Giovanni Trappolini, Luca Cosmo, Luca Moschella, Riccardo Marin, Simone Melzi, and Emanuele Rodolà. Shape registration in the time of transformers. Advances in Neural Information Processing Systems , 34:5731-5744, 2021.

Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR , 2017.

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG) , 2019.

Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In ECCV , 2020.

Yuxin Yao, Bailin Deng, Weiwei Xu, and Juyong Zhang. Fast and robust non-rigid registration using accelerated majorization-minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Hao Yu, Ji Hou, Zheng Qin, Mahdi Saleh, Ivan Shugurov, Kai Wang, Benjamin Busam, and Slobodan Ilic. Riga: Rotation-invariant and globally-aware descriptors for point cloud registration. arXiv preprint arXiv:2209.13252 , 2022.

Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, and Slobodan Ilic. Rotationinvariant transformer for point cloud matching. arXiv preprint arXiv:2303.08231 , 2023.

Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, and Yebin Liu. Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor. In CVPR , 2018.

Yiming Zeng, Yue Qian, Zhiyu Zhu, Junhui Hou, Hui Yuan, and Ying He. Corrnet3d: Unsupervised end-to-end learning of dense correspondence for 3d point clouds. In CVPR , pages 6052-6061, 2021.

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Fast global registration. ECCV , 2016.

In this appendix, we provide more technical details and experimental results. In Sec. A, we provide 1) A detailed description of our orientation regressor in Sec. A.1; 2) More details on the registration method presented in Alg. 1 of the main text in Sec. A.3; 3) Implementation details of the whole pipeline in Sec. A.4.

In the second part, we provide more experimental results. We first provide more qualitative results in Sec. B.1, as well as quantitative results on training on large dataset and on animal dataset in Sec. B.2; Then we demonstrate the robustness of our method with respect to several perturbations in Sec. B.3; In Sec. B.4, we showcase the scalability of our method in matching real scans. Finally, we report a run-time decomposition and analysis in Sec. B.5.

Note that we also include a short video showcasing the registration process of our pipeline in the Supplemental Material.

A Technical Details

A.1 Orientation Regressor

Orientation plays an important role in point cloud processing. Classic point-based feature extractors [42; 43; 53] are not rotation-invariant/equivariant by design and, therefore are limited. A common practice is to leverage data augmentation [42]. However, such can only robustify the model under small perturbations, while falling short of handling arbitrary rotations in SO (3) . On the other hand, recent advances in designing rotation-invariant/equivariant feature extractor [41; 10] seem to shed light on alleviating alignment requirements in non-rigid point matching. However, we empirically observe that such a design generally degrades the expressive power of the feature extractor, leading to inferior performance of the trained DFM.

This issue also attracts attention from the community of shape matching. In [11], the rotation prediction, instead of directly estimating the transformation, is formulated as a classification problem. However, this method can only address the rotation around a fixed axis and exhibits relatively low accuracy. Other works [56; 57] transform the input point cloud into handcrafted features. However, it should be noted that this approach often increases the computational complexity due to the additional feature extraction and processing steps involved.

In this paper, we propose a data-driven solution to this problem, accompanied by a registration module that is robust with the estimated alignment (see Tab.3 in the main text). As mentioned in the main text, we adopt the regressor model proposed in [9]. We use the synthetic SURREAL shapes [52], which are implicitly aligned by the corresponding generative codes to train our orientation regressor. We follow [9] and use PointNet++ [43] as the feature extractor. As we assume to be given full shape, either mesh or point cloud, for each input cloud, we translate it so its mass lies at the origin point. Then we estimate the orientation deviation, R , of it from the orientation implicitly defined by the SURREAL dataset. Finally, we apply the inverse rotation R T on the input to obtain the aligned shape. We emphasize again that all the shapes, either train or test, go through this alignment in our pipeline.

A.2 Deep Functional Maps (DFM)

We illustrate the DFM, both for meshes and point clouds in Fig. 4.

A.3 Registration Algorithm

We provide a more detailed version of our registration algorithm in Alg. 2. given a source mesh S , target point cloud T and a trained point feature extractor F , we first initialize the deformation graph DG with QSlim method [17] and set X (0) with identity transformations. After that, we compute the target point feature F T . During registration, we first deform the source model with a deformation graph, then we update the correspondence by computing the nearest neighbor between the source feature and target feature in stage-I or the source point cloud and target point cloud in stage-II. After we compute the set of filtered correspondences C k , we update the total cost function E total with the ADMMoptimizer. Finally, we follow [31] to implement an early stop strategy in both stages.

D

G

A1

sharedweights

FMreg

1

G

G2

A2

L(C1,C2)

π

(a)

Φ

V1

F1

E1

sharedweights

FMreg

F

F2

E2

L(C1,C2)

Φ+

π

G2

(b)

Algorithm 2: Shape registration pipeline. Input: Source mesh S = {V , E} and target point cloud T ;Trained point feature extractor F Output: X ∗ converging to a local minimum of E total ; Deformed source model {V ∗ , E} ; Correspondence Π ∗ ST , Π ∗ T S between S and T . (0) ;k = 0;

A.4 Implementation Details

We implement our framework in PyTorch. We use modified DGCNN [24] the backbone of our feature extractor. The output feature dimension is set to 128. In the training related to deep functional maps, we always use the first 50 eigenfunctions of Laplace-Beltrami operators. For training the DFM network, in Eqn.(6) of the main text, we empirically set λ bij = 1.0, λ orth = 1.0, λ align = 1e-4, λ NCE = 1.0. We train our feature extractor with the Adam optimizer with a learning rate equal to 2e-3. The batch size is chosen to be 4 for all datasets.

Regarding the registration optimization, in Eqn.(12) of the main text, we empirically set λ cd = 0.01, λ corr = 1.0, λ arap = 20 in Stage-I and λ cd = 1.0, λ corr = 0.01, λ arap = 1 in Stage-II. For α in Eqn.(9) of the main text, we set α = 0 . 2 . Note that the roles of λ cd and λ corr are reversed from Stage-I to Stage-II. In fact, in Stage-I, the initial shapes are likely to differ by large deformation, therefore it is more reliable to estimate correspondences via the learned embeddings. Once Stage-I converges, we empirically observe that the deformed source shape admits a similar pose with the target (see the video in Supplemental Material), therefore it is reasonable to further deform the source by extrinsic embeddings, i.e. , Chamfer distance on the coordinates. We stress that both losses are critical in both stages and perform the following ablation experiment on setting λ cd and λ corr to be zero independently in the two stages. Tab. 6 reports the scores. We find that most of the time, turning off leads to worse results, especially when the test shapes are heterogeneous (see, e.g., SHREC07-H).

B Experiments

B.1 Further Qualitative Results

Visualization of Registration Results In this section, we show more qualitative results of registration regarding different templates and different test shapes in Fig 5. Our method is capable of registering shapes with significant deformations across different templates.

Source

Att.FMaps

SSMSM

Ours

GT

Qualitative Comparison in TOPKIDS: In Fig. 6, we qualitatively visualize maps obtained by different methods tested in the TOPKIDS benchmark. It is obvious that our results outperform all the competing methods. Especially, a purely intrinsic method, AttentiveFMaps seriously fails due to the topological perturbation within the mesh structure.

Failure Cases: On the other hand, we also identify some failure cases, which are shown in Fig. 7. The two cases both suffer prominent heterogeneity between the template and the target. In the left case, where an ideal reference is given, we adapt it in registration instead of enforcing the hard highly non-isometric deformation with templates from SCAPE_r or FAUST_r (see Sec. B.3). While in the right case, it is not obvious how to choose an easier template for establishing correspondences between the two shapes, we, therefore, settle down at the setting described in Sec.4.2. It is worth noting that, though, we can easily identify that the problematic regions, i.e., head, and hands from the visualization, which is unavailable for methods only return maps.

Templates: In the Fig. 8 we visualize the templates used in different experiments. We select a pose close to the T-pose from the training sets of SCAPE_r, DT4D-H, and FAUST_r, respectively, as our templates.

B.2 Further Quantitative Results

Trained on Large-Scale Dataset: We have followed the setting of SSMSM [8] to train our feature extractor on the SURREAL 5k dataset [f] and to test it on the FAUST_r, SCAPE_r, and SHREC19_r datasets. We select one training shape from SURREAL 5k as the template in registration. As shown in Table 7, our method outperforms the competing non-rigid point cloud matching methods by a noticeable margin. We also refer the readers to Table 3 in [8] for a complete table including methods utilizing mesh input, and we highlight that our method is even comparable with the latter. In particular, ours vs. SOTA of mesh-based techniques over the three test datasets: 3.2 vs. 2.3, 2.5 vs. 3.3, 4.6 vs. 4.7. That is, we achieved the best overall result in two of the three test datasets.

Trained on Animal Dataset: We also perform experiments on the remeshed SMAL dataset. We first randomly generate 5000 shapes with SMAL model to train the alignment module. Then we train a DFM with the remeshed SMAL dataset. The template shape is 'dog_06' from the training set. The quantitative results are reported in Table 8. Remarkably, our method achieves more than a 40% performance improvement than the second-best baselines.

B.3 Robustness

Robustness with respect to Input Perturbation: We evaluate the robustness of different methods with respect to noise and orientation perturbations in Tab. 9. In the former, we perturb the input point clouds with Gaussian noise. In the latter, we randomly rotate the input point clouds within ± 30 degrees along x, y, z -axis. Thanks to our optimization-based approach and orientation regressor, in both tests we achieved the best accuracy, but also the lowest standard deviation over three runs of experiments.

Robustness with respect to Missing Parts: In Tab.1 of the main text, we report shape matching results on SHREC19_r dataset, where the 40 th shape is commonly excluded in the test phase of prior works [29; 8]. The main reason is that this shape is incomplete, consisting of two main connected components (see the abdomen of the source shape in Fig. 9). Therefore, for intrinsic methods depending on mesh eigenbasis, such disconnectedness makes spectral alignment extremely difficult, if not impossible. On the other hand, extrinsic methods show a more robust outcome. In particular, we show in Fig. 9 that our framework can reasonably deform the template mesh (from SCAPE_r) to

the incomplete point cloud with a significantly isolated portion, which leads to superior matching results compared to the baselines.

Ourregistration

Source

Att.FMaps

SSMSM

Ours

GT

Robustness of Orientation Regressor: To demonstrate that our learned orientation regressor (see Sec.3.1 of the main text), we visualize its outcome on shapes from SHREC07-H, which manifests notable diversity in sampling density, size, and initial orientation.

Aligned

Raw

B.4 Scalability

Our approach is scalable w.r.t input size. Thanks to the fact that we non-rigidly align shapes in R 3 , we can in theory freely down- and up-sample both the template mesh and the target point clouds. Note this is non-trivial for methods based on mesh or graph Laplacian, as deducing dense maps with landmark correspondences over graph structures is a difficult task on its own. In Fig. 11, we show the matching results on the real scans from the FAUST challenge, each of which consists of around 160,000 vertices. In contrast, [8] can handle at most 50,000 vertices without running out of 32G memory on a V100 GPU. We visualize its matching results on the subsampled (to 40,000 vertices) point clouds for comparison.

W

SSMSM

Ours

Source (40k verts)

Target (40k verts)

Source (160k verts)

Target (200k & 180k verts)

B.5 Run-time Analysis

In this part, we analyze the time cost of our optimization-based registration module. In particular, we separate it into two main parts, initialization, and iterative optimization. For the former, we in general fix a source mesh as a template, and then we construct a deformation graph on it as described in Sec.3.2 in the main text. Since the deformation graph can be pre-computed and cached for further use of the same template, we put more emphasis on the latter.

In particular, we perform registration between the template shape from SCAPE_r and the 20 test shapes therein and compute the average run-time of each core step in the optimization. As illustrated in Tab. 10, our method achieves an average optimization time of approximately 13.4 seconds per shape, with about 90% of the total time being dedicated to Stage-I. Analysis depicted in Fig. 12 reveals that the most time-consuming operation is the ADMM backward step, while the combined time cost of our correspondence filtering and updating operations accounts for less than 3% of the overall execution time. Though the reported run-time is not quite satisfying, we emphasize that to evaluate maps across n point clouds, we only need to register them to the same template shape ( O ( n ) ), which is different from the pair-wise ( O ( n 2 ) ) mapping and post-processing approaches [7].

2%

14%

Correspondence Filtering

Mesh deforming

18%

Energy Computing

Correspondence Updating

65%

1%

ADMMBackward

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe method proposes an unsupervised pipeline to solve 3D shape-to-shape registration. Firstly, the two shapes are aligned by a pre-trained orientation regressor. Then, a soft-correspondence is obtained by a Point Feature extractor, optimized using a Deep Functional Maps schema, and considering several unsupervised regularizations (e.g, bijectivity, orthogonality). Finally, the source shape is iteratively deformed using a Deformation graph (using Chamfer distance, the learned correspondence, and as-rigid-as-possible), and the correspondence is updated every 100 iterations. The method is tested on humanoid datasets (FAUST_r, SCAPE_r, SHREC19_r, SHREC07-H, DT4D-H),  and extensively compared against many other approaches, both  supervised and unsupervised. obtaining interesting results.\n\n== FINAL RATING ==\n\nIn the discussion phase, authors provided much further evidence and clarified my concerns. I notice, however, that while a majority is leaning toward acceptance, there is not a consensus among the reviewers. Going through the reviewers that assigned negative scores, I do not find significant ground to change my rating. In particular, Reviewer xFDL mainly criticizes the novelty of the work but does not point to other works to support this claim. While I can understand that the work does not seem particular novel in its component (and the general principle of registration + correspondence in feature embedding can be found in previous works (e.g., more SmoothShell, DeepShell than DPC, while both of them are designed for meshes), I do not find itself a proper ground of rejection (as also reported in the Reviewing Guidelines; also combinations of existing techniques are valuable). The experimental evaluation is appreciable, the performance convincing, and I do not see this as incremental w.r.t. any other previous work. I find this an interesting contribution to a research field that counts only a limited amount of work. \nInstead, I see the detailed review and discussion of Reviewer iB8H, and I think it contains many valuable observations that could improve the paper. However, summarizing the reported criticisms seems to be: \n\nA) novelty w.r.t to DPC; which in the last comment looks solved, or at least significantly tuned down \n\nB) general motivation of the work; while I see that the underlying principle and positioning of the work in literature can be improved, overall, I do not see other works that perform similarly, and I struggle to see this as a \"follow-up\" of a specific methodology to consider this an incremental contribution of something in particular. Also, the obtained results seem to be already a reasonable justification for the proposed approach (since I think we all agree that the paper effort is beyond engineering work, and hence, they communicate a promising research direction)\n\nC) Other details (e.g., clarifying the role of hyper-parameters, missing citations, rephrasing); that I think make sense, but they can be easily addressed in the camera ready, and I do not consider a sufficient ground for rejection.\n\nFor these reasons, I lean toward acceptance. I suggest authors incorporate the suggestions (especially about suggested experiments and paper positioning in literature), and I wish them the best of luck with their work.\n\nSTRENGTHS:\n1) Not many methods are available to solve shape-to-shape correspondence in an unsupervised way; the proposed approach smartly combines existing techniques and well-established methods to obtain good results.\n\n2) The proposed approach outperforms existing direct competitors, even some supervise approaches. I am sure it would be of interest for the shape-matching community.\n\n3) The paper is well presented, clear and direct, I enjoyed the reading, and I had no problem understanding the main components. I am sure the method can be re-implemented with limited effort.\n\nWEAKNESSES:\n1) The general applicative context of the method is unclear. From the introduction and the pipeline figure, I had the impression that the method is designed to obtain correspondence between point cloud. However, the method is about registering a mesh to a point cloud, which can be extended to the point cloud case, using the source mesh as a bridge. This, however, is not properly tested: while other approaches also show noisy point clouds with different levels of noise (e.g., [30]), in this case, only point clouds sampled from meshes are reported, with a gaussian noise that is not detailed nor in the main manuscript or in the supp.mat. . Given the nature of the used feature extractor (which I expect to be quite sensitive to real noise and clutter), I suggest including more details on the considered Gaussian noise, and test on real raw pointclouds (which could also be another failure case, but would point the reader to interesting future directions)\n\n2) While I appreciate the simplicity of the proposed approach and the combination, I struggle to see a general message or insight. Also, I do not see the method as directly applicable since it is trained only on a few shapes, and we do not know how it would scale on larger datasets (e.g., AMASS), and I guess it would fail to solve registrations in real contexts (e.g., noisy scans), since it would ruin the learning on the underlying surface. It is also tested only on a domain (humans) with a significant amount of labelled data. So, without these elements, I wonder if the paper could be of significant impact or if it is just a carefully designed pipeline that will be just another matching method without much influence in the field. I suggest discussing the main message, analysing the scalability of the approach, and testing on domains in which labelling data are much more complicated to obtain (e.g., chairs, which also would make stronger the claim about topological noise in the supplementary)\n\nQUESTIONS:\n1) In Table 1 some results are not reported. Why?\n2) The orientation regressor module resembles the input transform module of PointNet. Wouldn't a rotation augmentation for the feature extractor work (and remove the need for a further module)? Another possible alternative is given by the method proposed in [A].\n3) Given that overall the pipeline assumes some degrees of bijectivity, how would it perform in the presence of a significantly different number of vertices between the two shapes (e.g., by a factor of x100)? Even considering two complete shapes which have a bijectivity on the surface, the losses are defined vertex wise, and might lead to a degradation of the regularization impact.\n\nMinor:\n1) I see that reference [30] is incorrect in the bibliography, and [31] has names compressed. I suggest double-checking the references.\n2) Relevant references not discussed: [B], [C], [D]\n\n[A]: Adjoint Rigid Transform Network: Task-conditioned Alignment of 3D Shapes, Zhou et al., 3DV 2022 \\\n[B]: NeuroMorph: Unsupervised Shape Interpolation and Correspondence in One Go, Eisenberger et al., ECCV 2021 \\\n[C]: NCP: Neural Correspondence Prior for Effective Unsupervised Shape Matching, Attiki et al., NeurIPS 2022 \\\n[D]: 3d-coded: 3d correspondences by deep deformation, Groueix et al., ECCV 2018\n\nLIMITATIONS:\nLimitations are discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper an unsupervised non-rigid shape registration method is proposed. The proposed method combines intrinsic spectral mapping (i.e. based on the deep functional map framework) together with  extrinsic deformable shape registration (i.e. a deformation graph) to enable unsupervised 3D deformable shape matching. In many challenging benchmark datasets, the proposed method demonstrates competitive matching performance, better cross-dataset generalisation ability, and robustness against noise and rotation of input shapes.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow. The main contribution and methodology are well illustrated.\n2. The paper integrates shape matching and shape registration into the same framework. The shape matching part is based on the deep functional map framework to obtain point-wise correspondences. The shape registration part is based on the deformation graph to non-rigid align two shapes to refine the final correspondences.   \n3. In order to enable matching shapes with different orientation, the paper proposes an orientation regressor to align shapes into a canonical frame.\n\nWEAKNESSES:\n1. The novelty of the method is limited. The proposed method consists of three components (rotation regressor, feature extractor, shape registration) and each component is derived from prior works without large modifications.\n2. The interconnection of the components in the method is missing. The first stage is to train a rotation regressor to align shapes into the canonical frame. The second stage is to train a feature extractor to obtain correspondences. The third stage is to optimise the deformation graph to align two shapes. Every stage is somehow separated during the training stage, while it is desired to see the connection of shape matching and shape registration like Deep Shells or NeuroMorph.\n\nQUESTIONS:\n1. Since the proposed method is based on test-time optimisation, what is the convergence speed of the proposed method? \n2. Since the deep functional map can be fully intrinsic (i.e. with intrinsic input features), what if we use it to obtain the initial correspondences and rigidly align the shapes into the canonical frame without using rotation regressor?\n\nLIMITATIONS:\n1. The proposed method is tailored to complete shape matching, so it cannot achieve desirable matching results for partial shapes.\n2. The proposed method is based on iterative optimisation to align two shapes, so the runtime is slower than other learning-based methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a method for corresponding a 3D triangle mesh to a point cloud of a similar (possibly articulated) shape.  The two-stage process first corresponds the two shapes in a high-dimensional feature space, then corresponds them again using geometric features while deforming the source closer to the target.\n\nSTRENGTHS:\nThe approach makes sense, and the results seem strong compared to previous work, both qualitatively and quantitatively.\n\nWEAKNESSES:\nIn general, the presentation could be improved.\n\nThere should be a figure analogous to Figure 1, but showing qualitative correspondences for cases where train/test datasets do match. Only showing severe failures of competing methods on very disparate shapes does not convey the full picture.\n\nThe supplementary video showing the progress of the correspondence algorithm is very helpful. It would be even more informative if the mesh was texture mapped with the checkerboard as in Figure 1.\n\nEquations, Figures, Tables need more thorough descriptions.\n\n  Equation 1:\n          Even though the text says that one can optimize the feature function F when it introduces equation 1, it is unclear how the function F is optimized. If not including an equation involving F, you could at least point to a specific equation number in [11].\n  Equation 2: Parameter alpha should be defined after this equation, rather than after equation 4.\n  Equation 3: What does the cross symbol represent?\n  Equation 4: Parameter n2 is not used in the expression. Should the denominator summation go up to n2?\n  Table 1 & 2:\n          What are the two sections separated by a horizontal line (they each have a bold set of numbers)?\n          Train/Test column contains no values.\n  Table 2:  What is \"Ours-CRYPTO\", this acronym is not discussed/referenced in the text?\n  Table 3: What is \"Ideal PC\"?\n\nAlgorithm should also initialize \"Flag\" (presumably to Stage-I).\n\nQUESTIONS:\nCould this approach be implemented without learned functions? The intro mentions that spectral mapping can be geometric (rather than learned), and the second stage is akin to non-rigid ICP.\n\nWould it make sense to run Stage-I again after Stage-II, and repeat both stages a few times?\n\nLIMITATIONS:\nThe method requires a triangle mesh for the source shape, which is not discussed in the limitations section. The exposition would be stronger if the authors took the source shape as a point cloud and ran some automatic meshing on it for their experiments.\n\nAnother limitation that the authors mention is that the method only works on \"full shapes\". I assume this means that the whole source and target surfaces are adequately sampled. It would help if the authors suggested some ideas on how one would go about lifting this requirement.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an unsupervised framework for non-rigid shape registration. The proposed method deforms source mesh towards the target point cloud, guided by correspondences induced by high-dimensional embeddings learned from deep functional maps. Empirical results show that the proposed method achieves state-of-the-art results on several benchmarks for non-rigid point cloud matching.\n\nSTRENGTHS:\n1. The overall writing is fluent;\n\n2. The proposed method outperforms state-of-the-art approaches on several benchmarks;\n\nWEAKNESSES:\n1. The organization can be further improved (the organization of Methodology does not follow Fig. 1), which makes the paper sometimes hard to follow (probably due to the complexity introduced by assembling 3 stages for tackling the problem);\n\n2. The novelty of this paper would be a major concern. It seems the proposed algorithm in this paper simply assembles three stages, each with an existing method. The orientation regressor is from [9]. The feature extractor is the modified DGCNN proposed in [22]. The non-rigid registration mainly follows [18].\n\n3. The submission and main paper have different titles. Please fix it.\n\nQUESTIONS:\n1. Why is the proposed method unsupervised? It seems the orientation regressor requires the ground-truth pose to the canonical space. And the PointInFoNCE loss (Eq. 4) also requires correspondence labels? Or the authors call the generalization from one dataset to the other datasets the unsupervised learning?\n\n2. How good is the orientation regressor? Can it deal with large translations? e.g., the translation introduced by a walking human. From my experiences, this kind of pose regressor can only overfit to a specific model and hard to generalize. Also, it is usually not very accurate (correct me if I was wrong);\n\n3. Why all the models are called \"pre-trained\"? Should it be generalizing a model trained on the same task to new data?\n\nLIMITATIONS:\nLimitations have been discussed in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of non-rigid shape matching. The problem is first decomposed into learning (rigid) orientation of shapes and   then learning shape matching on aligned shapes. For the later, it proposes to learn it with a  sequential pipeline, consisting of various modules, that is also optimized in a two stage manner.  the main idea is to facilitate learning by similarity in both ambient space (R^3) as well as high dimensional learned feature space. To this end, it trains a DGCNN feature extractor based on combination of several loss functions (DFM prior loss, ARAP loss, Chamfer/cosine loss). The method is validated on  isometric as well non isometric benchmarks.\n\nPost Rebuttal:\n\nThe rebuttal provided additional experiments to support some of its claim as mentioned below. I lean towards rejection since the work is a sequential concatenation of existing individual modules (also noted by reviewer CNuX and xFDL ) without any conceptual justification to why those modules need to be combined e.g. why do we need to combine a DFM and ARAP loss. The rebuttal also contains several unsubstantiated statements detailed below. Combining two lines of research is not a technical contribution or contribution to community if there no conceptual justification given. Majority of the introduction section needs to be rewritten and contextualised *correctly* wrt prior work. Besides, submission had several hyper-parameters whose values were missing and there is no mention of model sensitivity towards these hyper-parameters even though the approach is unsupervised.\n\nSTRENGTHS:\n- Experimental results on various near isometric benchmark look promising (Table 1) and compares extensively with existing work.\n\n- the paper identifies the generalization problem of some embedding based approach [30,8] to unseen shape data.\n\nWEAKNESSES:\n- Presentation: Section 1 contains several factually incorrect or unsupported claims. Moreover, the motivation of the work is misplaced since their is no validation of it later in the experimental section. Please support all the intuitive claims either with a prior reference or by explicitly mentioning results from this paper. \n\na)  why do we need to decompose the shape matching problem into learning an alignment first and then matching aligned shapes? Is it \n because [8], [22] do so or is there a scientific justification behind it as shown empirically in rigid shape matching or non-rigid shape \n matching with DFM. e.g. it is shown to provide an extrinsic supervision that helps to disambiguate symmetry issues in DFM. Please motivate the problem/solution accordingly.\n\nb)  Line 46 < prior work with resulting high dimensional embedding lack intuitive geometric meaning> Since this is used as one of 3 \nreasons behind this paper/formulation, please demonstrate this on an example where in contrast, this paper obtains an intuitive \nembedding with geometric meaning.\n\nc)  Line 54-57: Please provide a reference to support these claim or prove it later with visualization and examples in experimental section.\n\nd) [30] requires shapes to be pre-aligned at train or test time. This is not true. You can train and test [30] without such alignment and it makes no such assumption on input requirements in the paper[30].\n\n\n-  Novelty: the main conceptual idea/key insight (Line 52) in this work is to enforce similarity in both ambient space as well as learned feature space. DPC [b] proposed the exact same idea with DGCNN, cosine/chamfer loss in the most simplest possible way for non-rigid shape matching. The idea is already well known that others have even built upon it for non-rigid shape matching e.g. [a]. So claiming this as a key insight is not justified IMO.\n\n-  Missing references and comparison of a very similar work[b]:  authors should also compare their work with DPC on their benchmarks. this will show what gains (if any) are brought by DFM prior or ARAP loss.\n\na. Learning Canonical Embeddings for Unsupervised Shape Correspondence with Locally Linear Transformations\nb. DPC: Unsupervised deep point correspondence via cross and self construction, 3DV 2021.\n\n-  formulation without a principled approach: the network consisting of (DGCNN, DFM prior, ARAP loss, chamfer loss, pointinfo loss etc) combines modules from different frameworks without any principled reason. e.g. DFM prior from DFM literature, DGCNN feature extractor, chamfer & cosine loss from DPC etc. in a two stage optimization procedure. why do we need a DFM prior when we are deforming a source and a target shape with a ARAP loss ? \n\n-  Too many hyperparameters in an unsupervised approach : The paper should mention in the main body how many hyperparameters  overall does this approach has and how were their values chosen? I count more than 10. Moreover, how is it justified to choose different values for the same hyperparameter  (weighing scalars) in an algorithm and call the resulting approach *unsupervised*?  there is a two order magnitude of difference in hyperparameter values ($\\lambda_{cd}$ and $\\lambda_{corr}$) between different runs of the algorithm (Stage 1 and Stage 2).\n\nQUESTIONS:\n- for Non-isometric, SMAL is considered the main benchmark and there is already a large literature that benchmark their results on SMAL. why ignore such standard benchmark?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}