SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Puhua Jiang $^{1,2\#}$  Mingze Sun $^{1\#}$  Ruqi Huang $^{1*}$
1. Tsinghua Shenzhen International Graduate School, China 2. Peng Cheng Lab, China
In this paper, we propose a learning-based framework for non-rigid shape registration without correspondence supervision. Traditional shape registration techniques typically rely on correspondences induced by extrinsic proximity, therefore can fail in the presence of large intrinsic deformations. Spectral mapping methods overcome this challenge by embedding shapes into, geometric or learned, high-dimensional spaces, where shapes are easier to align. However, due to the dependency on abstract, non-linear embedding schemes, the latter can be vulnerable with respect to perturbed or alien input. In light of this, our framework takes the best of both worlds. Namely, we deform source mesh towards the target point cloud, guided by correspondences induced by high-dimensional embeddings learned from deep functional maps (DFM). In particular, the correspondences are dynamically updated according to the intermediate registrations and filtered by consistency prior, which prominently robustify the overall pipeline. Moreover, in order to alleviate the requirement of extrinsically aligned input, we train an orientation regressor on a set of aligned synthetic shapes independent of the training shapes for DFM. Empirical results show that, with as few as dozens of training shapes of limited variability, our pipeline achieves state-of-the-art results on several benchmarks of non-rigid point cloud matching, but also delivers high-quality correspondences between unseen challenging shape pairs that undergo both significant extrinsic and intrinsic deformations, in which case neither traditional registration methods nor intrinsic methods work. The code is available at https://github.com/rqhuang88/DFR.
Estimating correspondences between deformable shapes is a fundamental task in computer vision and graphics, with a significant impact on an array of applications including robotic vision [50], animation [40], 3D reconstruction [58], to name a few. In this paper, we tackle the challenging task of estimating correspondences between unstructured point clouds sampled from surfaces undergoing significant non-rigid deformations.
In fact, this task has attracted increasing interest [59; 26; 35; 24; 8] in shape matching community. These methods all follow a data-driven approach to learn embedding schemes, which project point clouds into certain high-dimensional spaces. By lifting to high-dimensional spaces, the non-rigid deformations are better characterized than in the ambient space,  $\mathbb{R}^3$ . The optimal transformations and therefore the correspondences are then estimated in the embedded spaces. Despite the great progresses, such approaches suffer from the following limitations: 1) Their performance on unseen shapes is largely unwarranted; 2) The learned high-dimensional embeddings lack intuitive geometric meaning, and the induced correspondences are difficult to evaluate and analyze without ground-truth labels.
*# indicates equal contribution. Corresponding author: ruqihuang@sz.tsinghua.edu.cn
Figure 1: Shape registration methods like NDP [31] and AMM [55] estimate intermediate correspondences via extrinsic proximity, therefore suffering from large intrinsic deformations. In contrast, by incorporating a DFM-based point feature extractor, our method successfully deforms a FAUST template (the left-most mesh) to another individual of a different pose (the right-most point cloud).
On the other hand, the above limitations are less of a concern from the perspective of shape registration, either axiomatic [1; 55] or learning-based [31]. Unlike the former, the latter deform a source shape to non-rigidly align with a target explicitly in the ambient space  $\mathbb{R}^3$ . In consequence, the mapping quality can be directly related to the alignment quality. Given a deformed source shape and a target, one can evaluate both qualitatively (by visual comparison) and quantitatively (by computing Chamfer distance), and, more importantly, further optimize the registration outcome via the correspondence-label-free quantitative metric. However, shape registration itself does not serve as a direct rescue to our problem of interest, as the previous approaches typically rely on the premise that the undergoing non-rigid deformation can be approximated by a set of local, small-to-moderate, rigid deformations, which severely hinders their performance in the presence of large deformations (see Fig. 1) and heterogeneity.
Motivated by the above observations, we propose a framework to take the best of classic shape registration and learning-based embedding techniques. In a nutshell, we leverage the estimated correspondence from the latter to guide shape deformation via the former iteratively. Our key insight is to enforce similarity between the deformed source and target in both ambient space and learned high-dimensional space. Intuitively, in the presence of large deformation, the learned correspondence is more reliable than that obtained by proximity in the ambient space. On the other hand, properly deforming source mesh to target point cloud increases spatial similarity in the ambient space, leading to higher similarity in the embedded space and therefore more accurate correspondences. In the end, we can compute correspondences between raw point clouds with a shared source mesh as a hub. As will be shown in Sec. 4, our method allows for choosing source shape during inference, which can be independent of the training data.
While being conceptually straightforward, we strive to improve the performance, robustness and practical utility of our pipeline by introducing several tailored designs. First of all, the key component of our pipeline is an embedding scheme for accurately and efficiently estimating correspondences between the deformed source shape and the target point cloud. In particular, to take advantage of Deep Functional Maps (DFM) [29; 7; 49], the current state-of-the-art approach on matching triangular meshes, we pre-train an unsupervised DFM [49] on meshes as a teacher net and then learn a point-based feature extractor (i.e., an embedding scheme for points) as a student net on the corresponding vertex sets with a natural self-supervision. Unlike the approach taken in [8], which heavily relies on DiffusionNet [47] to extract intricate structural details (e.g., Laplace-Beltrami operators) from both mesh and point cloud inputs, the teacher-student paradigm allows us to utilize a more streamlined backbone - DGCNN [53]. Secondly, in contrast to prior works that either implicitly [31; 55] or explicitly [46; 8; 24] require rigidly aligned shapes for input/initialization, or demand dense correspondence labels [35]. We train an orientation regressor on a large set of synthetic, rigidly aligned shapes to automatically pre-process input point clouds in arbitrary orientations. Last but not least, we propose a dynamic correspondence updating scheme, bijectivity-based correspondence filtering, two-stage registration in our pipeline. We defer the technical details to Sec. 3.
Overall, our framework enjoys the following properties: 1) Due to the hybrid nature of our framework, it can handle point clouds undergoing significant deformation and/or heterogeneity; 2) As our feature extractor is self-supervised by some deep functional maps network, our framework is free of correspondence annotation throughout; 3) Our core operation is performed in the ambient space, enabling more efficient, faithful, and straightforward analysis of registration/mapping results than the
Figure 2: We estimate correspondences between heterogeneous shapes from SHREC'07 with four learning-based methods, all trained on the SCAPE_r dataset. Our method outperforms the competing methods by a large margin. Remarkably, our method manages to deform a SCAPE template shape to heterogeneous shapes, as indicated by the blue shapes.
purely learning-based approaches; 4) Based on a data-driven orientation regressor, we achieve an automatic pipeline for estimating correspondences between deformable point clouds.
As shown in Fig. 2, trained on the SCAPE [2] dataset, our framework generalizes well and outperforms the competing methods by a large margin in the challenging SHREC07-H dataset, which presents prominent variability in both extrinsic orientation and intrinsic geometry. We conduct a set of experiments to verify the effectiveness of our pipeline. We highlight that it achieves state-of-the-art results in matching non-rigid point clouds in both near-isometric and heterogeneous shape collections. More remarkably, it generalizes well despite the distinctiveness between the training set and test set. Moreover, our method, being point-based, is more robust with respect to topological perturbations within input shapes than the mesh-based baselines.
Non-rigid Shape Registration: Unlike the rigid counterpart, non-rigidly aligning shapes is more challenging because of the perplexity of deformation models. In general, axiomatic approaches [1] assume the deformation of interest can be approximated by local, small-to-moderate, rigid deformations, therefore suffer from large intrinsic deformations. In fact, recent advances such as [60; 28; 55] mainly focus on improving the efficiency and robustness regarding noises and outliers. On the other hand, there exists as well a trend of incorporating deep learning techniques [6; 5; 21]. Among them, NDP [31] is similar to our method in the sense that the non-learned version follows an optimization-based approach based on some neural encoding. However, its shape registration is purely guided by proximity in the ambient space, hindering its performance in our problem of interest. Perhaps the most relevant approach with our method along this line is TransMatch [51], which follows a supervised learning scheme and learns a transformer to predict directly 3D flows between input point clouds. As demonstrated in Fig. 2 and further in Sec. 4, as a supervised method, TransMatch generalizes poorly to unseen shapes.
Non-rigid Point Cloud Matching: Unlike the above, several recent approaches [26; 30; 59] directly establish correspondence between a pair of point clouds. In general, these methods embed point clouds into a canonical feature space, and then estimate correspondences via the respective proximity. Leapard [30] learns such feature extractor under the supervision of ground-truth correspondences between partial point clouds with potentially low overlaps. On the other hand, DPC [26] and CorrnetNet3D [59] leverage point cloud reconstruction as proxy task to learn embeddings without correspondence label. Since intrinsic information is not explicitly formulated in these methods, they can suffer from significant intrinsic deformations and often generalize poorly to unseen shapes.
Deep Functional Maps: Stemming from the seminal work of functional maps [39] and a series of follow-ups [38; 22; 44; 37; 23], spectral methods have achieved significant progress in axiomatic non-rigid shape matching problem, and consequentially laid solid foundation for the more recent advances on Deep Functional Maps (DFM), which is pioneered by Litany et al. [32]. Unlike axiomatic functional maps frameworks that aim at optimizing for correspondences represented in spectral basis (so-called functional maps) with hand-crafted features as prior. DFM takes an inverse
Figure 3: The schematic illustration of our pipeline. A is a pre-trained orientation regressor for aligning input shapes. Then a pre-trained feature extractor  $\mathbf{F}$  embeds them into a high-dimensional canonical space. During the iterative optimization procedure of registration, correspondences are dynamically updated according to learned features (Stage-I) and coordinates (Stage-II) of the intermediate shapes. See more details in the text.
viewpoint and aims to search for the optimal features, such that the induced functional maps satisfy certain structural priors as well as possible. Recent findings along this direction empirically suggest that the structural priors suffice to train an effective feature extractor, without any correspondence supervision [12; 29; 7; 3]. However, because of the heavy dependence of eigenbasis of Laplace-Beltrami operators, DFM is primarily designed for shapes represented by triangle meshes and can suffer notable performance drop when applied to point clouds without adaptation [8].
In fact, inspired by the success of DFM, several approaches [24; 8] have been proposed to leverage intrinsic geometry information carried by meshes in the training of feature extractors tailored for non-structural point clouds. Among these, NIE [24] and SSMSM [8] are closely related to our approach. As all three works attempt to train an effective, intrinsic-geometry-aware, per-point feature extractor by leveraging meshes during training. However, unlike the former two, our method incorporates optimization-based shape registration techniques, which effectively alleviates the limitation of purely relying on an abstract high-dimensional embedding scheme (i.e., the learned feature extractor).
Combination of Extrinsic and Intrinsic Methods: Unlike the above, our method operates in both extrinsic and intrinsic spaces. From this point of view, the most relevant work is Smooth Shells [14], which proposes to perform shape registration under a mixed representation of shapes, including eigenbasis, coordinates, and normal. Though achieving considerable high-quality correspondence on standard benchmarks, it heavily relies on the availability of mesh representation, but also is sensitive to the original extrinsic alignment of input shapes. These limitations are inherited by Deep Shells [16] – the learning version of the former. Similarly, NeuralMorph [15] jointly performs interpolation and correspondence estimation between a pair of shapes, which inject the intrinsic information by imposing geodesic preservation loss. As a result, it is non-trivial to lift the requirement of mesh input.
Given a pair of shapes  $S, T$ , our target is to deform  $S$  to non-rigidly align with  $T$ . We assume that  $S$  is represented as a triangle mesh so that we can effectively regularize the deformed shape by preserving local intrinsic geometry. On the other hand, we require no structural information on  $T$  and generally assume it to be a point cloud. Our pipeline can also be extended to compute correspondences between two raw point clouds  $T_1, T_2$ . To this end, we fix a template mesh  $S$ , perform respective shape registration between  $S$  and the target point clouds, and finally compute the map by composition  $T_{12} = T_{s2} \circ T_{1s}$ .
Our pipeline is shown in Fig. 3, which consists of three main components: 1) An orientation regressor,  $\mathbf{A}$ , for extrinsically aligning input shapes, either mesh or point cloud; 2) A point feature extractor,  $\mathbf{F}$ ,
trained under deep functional maps scheme; 3) A registration module that iteratively optimizes for deformations non-rigidly aligning  $S$  with  $\mathcal{T}$ . In particular, it takes the rigidly aligned shapes from 1) as input, and leverages 2) to update correspondences during optimization.
Though our pipeline leverages a pre-trained feature extractor as registration prior, we highlight that neither the source nor the target is necessarily within or close to the respective training set. We provide throughout experimental evidence in Sec. 4 showing the generalizability of our pipeline.
3.1 Orientation Regressor
Our first objective is to align input point clouds with arbitrary orientations into a canonical frame. We take a data-driven approach by training an orientation regressor [9] on 5000 synthetic SURREAL shapes from [52], which are implicitly aligned to a canonical frame by the corresponding generative codes. We refer readers to Supplementary Materials for more details.
3.2 Point Feature Extractor
In order to efficiently and accurately estimate correspondences between deformed source shape and the target point cloud during registration, our next goal is to train a feature extractor for point clouds, which is intrinsic-geometry aware. The authors of [8] propose a multi-modal feature extractor based on DiffusionNet [47], which can process point clouds with an extra step of graph Laplacian construction [48]. Though it has demonstrated satisfying performance accommodating point cloud representation in [8], the explicit graph Laplacian construction is computationally heavy. Hence, we adopt the modified DGCNN proposed in [24] as our backbone, which is lightweight and robust regarding the sampling density and sizes of point clouds.
In particular, our training follows a teacher-student paradigm. Namely, we first train a deep functional maps (DFM) network,  $\mathbf{G}$ , on a collection of meshes. Then we train a DFM network  $\mathbf{F}$  on the corresponding vertex sets, with an extra self-supervision according to the inclusion map between meshes and their vertex sets. In other words, the feature produced by  $\mathbf{F}$  is point-wisely aligned with that produced by  $\mathbf{G}$ .
Training DFM on Meshes: Our training scheme in principle follows that of [49], which delivers both accurate map estimation and excellent generalization performance.
1) We take a pair of shapes  $S_{1}, S_{2}$ , and compute the leading  $k$  eigenfunctions of the Laplace-Beltrami operator on each shape. The eigenfunctions are stored as matrices  $\Phi_{i} \in \mathbb{R}^{n_{i} \times k}$ ,  $i = 1, 2$ .
2) We compute  $G_{1} = \mathbf{G}(S_{1}), G_{2} = \mathbf{G}(S_{2})$ , projected into the spaces spanned by  $\Phi_1, \Phi_2$  respectively, and leverage the differential FMReg module proposed in [13] to estimate the functional maps  $C_{12}, C_{21}$  between the two shapes, then by regularizing the structure of  $C_{12}, C_{21}$ , say:
$$
E _ {\mathbf {b i j}} (\mathbf {G}) = \left\| C _ {1 2} C _ {2 1} - I \right\| _ {F} ^ {2}, E _ {\mathbf {o r t h o}} (\mathbf {G}) = \left\| C _ {1 2} C _ {1 2} ^ {T} - I \right\| + \left\| C _ {2 1} C _ {2 1} ^ {T} - I \right\|. \tag {1}
$$
3) To enhance  $\mathbf{G}$ , we estimate the correspondences between  $S_{1}, S_{2}$  via the proximity among the rows of  $G_{1}, G_{2}$  respectively. Namely, we compute
$$
\Pi_ {1 2} (i, j) = \frac {\exp (- \alpha \delta_ {i j})}{\sum_ {j ^ {\prime}} \exp (- \alpha \delta_ {i j ^ {\prime}})}, \forall i \in [ | \mathcal {S} _ {1} | ], j \in [ | \mathcal {S} _ {2} | ], \tag {2}
$$
where  $\delta_{ij} = \| G_1(i,:) - G_2(j,:)\|$  and  $\alpha$  is the temperature parameter, which is increased during training [49]. Then  $\Pi_{12}$  is the soft map between the two shapes. Ideally, it should be consistent with the functional maps estimated in Eqn.(1), giving rise to the following loss:
$$
E _ {\text {a l i g n}} (\mathbf {G}) = \left\| C _ {1 2} - \Phi_ {2} ^ {\dagger} \Pi_ {2 1} \Phi_ {1} \right\| + \left\| C _ {2 1} - \Phi_ {1} ^ {\dagger} \Pi_ {1 2} \Phi_ {2} \right\|, \tag {3}
$$
where  $\dagger$  denotes to the pseudo-inverse. To sum up,  $\mathbf{G}^{*} = \arg \min E_{\mathrm{DFM}}(\mathbf{G})$ , which is defined as:
$$
E _ {\mathrm {D F M}} (\mathbf {G}) = \lambda_ {\mathrm {b i j}} E _ {\mathrm {b i j}} (\mathbf {G}) + \lambda_ {\mathrm {o r t h}} E _ {\mathrm {o r t h}} (\mathbf {G}) + \lambda_ {\mathrm {a l i g n}} E _ {\mathrm {a l i g n}} (\mathbf {G}). \tag {4}
$$
Training DFM on Point Clouds: Now we are ready for training  $\mathbf{F}$  with a modified DGCNN [24] as backbone. We let  $F_{1} = \mathbf{F}(S_{1}), F_{2} = \mathbf{F}(S_{2})$ . On top of  $E_{\mathrm{DFM}}$  in Eqn. (4), we further enforce the
extracted feature from  $\mathbf{F}$  to be aligned with that from  $\mathbf{G}^*$  via PointInfoNCE loss [54]:
$$
E _ {\mathrm {N C E}} (\mathbf {F}, \mathbf {G} ^ {*}) = - \sum_ {i = 1} ^ {n _ {1}} \log \frac {\exp \left(\left\langle F _ {1} (i , :) , G _ {1} ^ {*} (i , :) \right\rangle / \gamma\right)}{\sum_ {j = 1} ^ {n _ {1}} \exp \left(\left\langle F _ {1} (i , :) , G _ {1} ^ {*} (j , :) \right\rangle / \gamma\right)} - \sum_ {i = 1} ^ {n _ {2}} \log \frac {\exp \left(\left\langle F _ {2} (i , :) , G _ {2} ^ {*} (i , :) \right\rangle / \gamma\right)}{\sum_ {j = 1} ^ {n _ {2}} \exp \left(\left\langle F _ {2} (i , :) , G _ {2} ^ {*} (j , :) \right\rangle / \gamma\right)}, \tag {5}
$$
where  $\langle .\rangle$  refers to the dot products,  $\gamma$  is the temperature parameter, and  $n_i$  is the number of points of  $S_{i}$ . So the final training loss is defined as:
$$
E (\mathbf {F}) = E _ {\mathrm {D F M}} (\mathbf {F}) + \lambda_ {\mathrm {N C E}} E _ {\mathrm {N C E}} \left(\mathbf {F}, \mathbf {G} ^ {*}\right), \tag {6}
$$
which is optimized on all pairs of shapes within the training set.
3.3 Shape Registration
Now we are ready to formulate our registration module. We first describe the deformation graph construction in our pipeline. Then we formulate the involved energy functions and finally describe the iterative optimization algorithm. During registration, we denote the deformed source model by  $\mathcal{S}^k = \{\mathcal{V}^k,\mathcal{E}\} ,\mathcal{V}^k = \left\{v_i^k\mid i = 1,\dots ,N\right\}$ , where  $k$  indicates the iteration index and  $v_{i}^{k}$  is the position of the  $i$ -th vertex at iteration  $k$ . The target point cloud is denoted by  $\mathcal{T} = \{u_j\mid j = 1,\ldots ,M\}$ .
Deformation Graph: Following[20], we reduce the vertex number on  $S$  to  $H = [N/2]$  with QSlim algorithm [17]. Then an embedded deformation graph  $\mathcal{DG}$  is parameterized with axis angles  $\Theta \in \mathbb{R}^{H \times 3}$  and translations  $\Delta \in \mathbb{R}^{H \times 3}$ :  $\mathbf{X} = \{\Theta, \Delta\}$ . The vertex displacements are then computed from the corresponding deformation nodes. For a given  $\mathbf{X}^k$ , we can compute displaced vertices via:
$$
\mathcal {V} ^ {k + 1} = \mathcal {D G} \left(\mathbf {X} ^ {k}, \mathcal {V} ^ {k}\right). \tag {7}
$$
Dynamic correspondence update: Thanks to our point-based feature extractor  $\mathbf{F}$ , we can freely update correspondences between deforming source mesh and target point clouds. In practice, for the sake of efficiency, we update every 100 iterations in optimization (see Alg. 1).
Correspondence Filtering via Bijectivity: One key step in shape registration is to update correspondences between the deforming source and the target. It is then critical to reject erroneous corresponding points to prevent error accumulation over iterations. In particular, we propose a filter based on the well-known bijectivity prior. Given  $S^k$  and  $\mathcal{T}$ , we first compute maps represented as permutation matrices  $\Pi_{\mathcal{S}\mathcal{T}}$ ,  $\Pi_{\mathcal{T}\mathcal{S}}$ , either based on the learned feature or the coordinates. For  $v_i^k \in S^k$ , we compute the geodesic on  $S^k$  between  $v_i^k$  and its image under permutation  $\Pi_{\mathcal{S}\mathcal{T}} \Pi_{\mathcal{T}\mathcal{S}}$ , then reject all  $v_i^k$  if the distance is above some threshold. For the sake of efficiency, we pre-compute the geodesic matrix of  $S$  and approximate that of  $S^k$  with it. Finally, we denote the filtered set of correspondences by  $\mathcal{C}^k = \{(v_{i_1}^k, u_{j_1}), (v_{i_2}^k, u_{j_2}) \cdots \}$ .
In the following, we introduce the energy terms regarding our registration pipeline.
Correspondence Term measures the distance of filtered correspondences between  $S^k$  and  $\mathcal{T}$ , given as:
$$
E _ {\text {c o r r}} = \frac {1}{\left| \mathcal {C} ^ {k} \right|} \sum_ {\left(v _ {i} ^ {k}, u _ {j}\right) \in \mathcal {C} ^ {k}} \left\| v _ {i} ^ {k} - u _ {j} \right\| _ {2} ^ {2}. \tag {8}
$$
Chamfer Distance Term has been widely used [31; 30] to measure the extrinsic distance between  $S^k$  and  $\mathcal{T}$ :
$$
E _ {\mathbf {c d}} = \frac {1}{N} \sum_ {i \in [ N ]} \min  _ {j \in [ M ]} \left\| v _ {i} ^ {k} - u _ {j} \right\| _ {2} ^ {2} + \frac {1}{M} \sum_ {j \in [ M ]} \min  _ {i \in [ N ]} \left\| v _ {i} ^ {k} - u _ {j} \right\| _ {2} ^ {2} \tag {9}
$$
As-rigid-as-possible Term reflects the deviation of estimated local surface deformations from rigid transformations. We follow [20; 27] and define it as: