SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The fmap loss,  $\mathcal{L}_{\mathrm{fmap}}$ , is used to regularize the structural properties of the predicted fmaps, such as bijectivity and orthogonality [28]:  $\mathcal{L}_{\mathrm{fmap}} = \| \mathbf{C}_{ij} \mathbf{C}_{ji} - I_k\|_F^2 + \| \mathbf{C}_{ij}^\top \mathbf{C}_{ij} - I_k\|_F^2$ . The bijectivity constraint ensures that the map from  $S_i$  through  $S_j$  and back to  $S_i$  is the identity map, while the orthogonality constraint enforces local area preservation of the map.
The cycle loss,  $\mathcal{L}_{\mathrm{cycle}}$ , enforces cyclic consistency between predicted p2p maps:  $\mathcal{L}_{\mathrm{cycle}} = \|\hat{\Pi}_{12}\hat{\Pi}_{21}\mathbf{S}_1 - \mathbf{S}_1\|_F^2$ . Lastly, the primo energy,  $\mathcal{L}_{\mathrm{primo}}$ , is employed to encourage the decoder to yield a coherent shape and to ensure the deformation appears smooth and natural, facilitating the training process. It is calculated as the energy  $E$  from Eq. (3).
In this section, we evaluate our method on different challenging shape matching scenarios.
Our experimental setup spans two tasks and involves four distinct datasets referenced in previous literature, incorporating both human and animal representations.
For near-isometric shape matching, we employed several standard benchmark datasets, namely, FAUST [2], SCAPE [76], and SHREC'19 [77]. To enhance the complexity, we selected the remeshed versions of these datasets, as suggested in [29, 25, 48], instead of their original forms. The FAUST
Table 1: Quantitative Results on Near-Isometric Benchmarks. The best and second-best results are highlighted. Our method comes out on top among axiomatic methods and matches or even surpasses some trained methods.
METHOD	FAUST	SCAPE	SHREC
Axiom	Ini + BCICP [19]	6.4	11	-
Ini + ZoomOut [18]	6.1	7.5	-
Smooth Shells [55]	2.5	4.7	12.2
DiscreteOp [43]	5.6	13.1	-
Sup	3D-CODED [34]	2.5	31	-
FMNet [24]	11	17	-
GeoFMNet [25]	3.1	4.4	9.9
DiffGeoFMNet [64]	2.6	2.9	8.5
TransMatch [80]	2.7	18.3	14.5
Unsup	Unsup. FMNet [53]	10.0	16.0	-
SurFMNet [28]	15.0	12.0	30.1
Cyclic FMaps [54]	14.8	12.7	36.5
WSupFMNet [29]	3.3	7.3	11.3
Deep Shells [35]	1.7	2.5	21.1
NeuroMorph [36]	8.5	29.9	26.3
SNK (Ours)	1.8	4.7	5.8
dataset contains 100 shapes, depicting 10 individuals in 10 different poses each, with our evaluation focused on the last 20 shapes. The SCAPE dataset comprises 71 distinct poses of a single person, and we based our assessment on the final 20 shapes. The SHREC dataset, with its notable variations in mesh connectivity and poses, presents a more significant challenge. It consists of 44 shapes and a total of 430 evaluation pairs. For unsupervised methods, we used the oriented versions of the datasets as detailed in [29]. Moreover, for training methods, we showcase their optimal performance on SHREC using a model trained either on FAUST or SCAPE.
In the context of non-isometric shape matching, we conducted our experiments using the SMAL animal-based dataset [78, 79], hereinafter referred to as SMAL. This dataset includes 50 distinct, non-rigid 3D mesh models of organic forms. We divided these into two equal sets of 25 for training and testing, ensuring that the specific animals and their poses used for testing were not seen during the training phase.
5.2 Near-Isometric Shape Matching
In this section, we evaluate our method's performance in near-isometric, non-rigid shape matching. We draw comparisons with several contemporary approaches in the field of non-rigid shape matching. These can be broadly divided into three categories. The first category is axiomatic approaches and it includes BCICP [48], ZoomOut [18], Smooth Shells [55], and DiscreteOp [43]. The second category is supervised learning approaches and it comprises methods such as 3D-CODED [34], FMNet [24], GeomFmaps [25], DiffGeoFMNet [64], and TransMatch [80]. Finally the third category is unsupervised learning approaches and it includes Unsupervised FMNet [53], SURFMNet [28], Cyclic FMaps [81], Weakly Supervised FMNet [29], DeepShells [35], and NeuroMorph [36].
To evaluate all correspondences, we adopt the Princeton protocol, a method widely used in recent works [24, 25, 27, 29, 35]. Specifically, we calculate the pointwise geodesic error between the predicted maps and the ground truth map, normalizing it by the square root of the target shape's total surface area.
Results are presented in Table 1. As can be seen, our method outperforms axiomatic approaches and is comparable or superior to some training-based methods, particularly on the FAUST and SCAPE datasets. When it comes to the SHREC dataset, our method achieves state-of-the-art results across all methods. Most learning-based methods stumble over this challenging dataset, even though they have been trained on many training examples. This is because of the wide mix of poses and identities in the
Figure 4: Qualitative Results from the SHREC dataset. The correspondences are visualized by transferring a texture through the map. Our method results in visually superior outcomes.
Table 2: Quantitative results on the non-isometric benchmark. We highlight the best and the second best results. Our method achieves competitive results with trained methods.
METHOD	SMAL
Sup	FMNet [24]	47.1
GeoFMNet [25]	8.8
DiffGeoFMNet [64]	6.6
LIE [82]	20
Unup	WSupFMNet [29]	13.3
Deep Shells [35]	15.2
NeuroMorph [36]	23.1
Axiom	Smooth Shells [55]	16.3
SNK (Ours)	9.1
dataset. This highlights how our approach, which focuses on overfitting a single pair, is particularly effective in such challenging cases.
In Fig. 4, we showcase point-to-point maps generated by our method on the SHREC dataset. We compare these maps to those produced by the best-performing baseline method across each category - axiomatic, unsupervised, and supervised. Upon inspection, it is apparent that our method attains highly accurate correspondences. In contrast, the baseline methods either fail to achieve a precise matching or suffer from symmetry flip issues.
5.3 Non-Isometric Shape Matching
In the non-isometric, non-rigid context, we adhered to the same evaluation protocol and present our findings in Table 2. It can be seen from these results that our method outperforms both the axiomatic and unsupervised approaches, and competes strongly with, or even surpasses, some supervised techniques. This further demonstrates the efficacy and versatility of our method across diverse shape-matching scenarios.
Figure 5: Examples of non-isometric and non-homeomorphic matching on the SMAL and SHREC datasets, respectively.
In Fig. 5, we present qualitative examples of non-isometric and non-homeomorphic matching from the SMAL and SHREC datasets, respectively. The results further highlight the resilience of our approach to various deformations and topological changes.
5.4 Ablation Studies & Additional Details
In the supplementary material, we present an ablation study that underscores the efficacy of each component introduced in our methodology. Additionally, we provide comprehensive details regarding the implementation process, the computational specifications involved, as well as some qualitative results.
Importance of the neural prior In the supplementary we also evaluate the importance of using neural networks for training our method. In particular, we show that when unconstrained optimization replaces neural networks, the results degrade significantly. This illustrates the importance of the "neural prior" [83, 84] employed by our method and further highlights the difference between our approach and purely axiomatic methods.
In this work, we introduced SNK, a novel approach to zero-shot shape matching that operates independently of prior training. Our method hinges on the precise alignment of a pair of shapes, enabling the source shape to be deformed to match the target shape without any user intervention. By integrating both spatial and spectral losses, and architecting a new decoder that encourages smooth and visually pleasing deformations, our method has proven its effectiveness. Extensive experiments reveal that our approach competes with, and often surpasses, many supervised techniques, even achieving state-of-the-art results on the SHREC dataset.
Despite its success, SNK has certain limitations we aim to address in future research. Firstly, our method is currently tailored towards complete shape correspondence, necessitating adaptation for partial shape-matching scenarios. Secondly, while our method's convergence time of approximately 70 seconds per shape outpaces many competitors, exploring ways to further expedite this process would be beneficial. Finally, our method utilizes DiffusionNet as a backbone, which theoretically could accommodate both point clouds and triangular 3D meshes. Consequently, extending our approach to encompass other types of representations could prove useful.
Acknowledgements The authors would like to thank the anonymous reviewers for their valuable suggestions. Parts of this work were supported by the ERC Starting Grant No. 758800 (EXPROTEA) and the ANR AI Chair AIGRETTE.
[89] Michal Edelstein, Danielle Ezuz, and Mirela Ben-Chen. ENIGMA. ACM Transactions on Graphics, 39(4), August 2020. 19
[90] A. Gehre, M. Bronstein, L. Kobbelt, and J. Solomon. Interactive curve constrained functional maps. Computer Graphics Forum, 37(5):1-12, August 2018. 19
[91] D. Ezuz, B. Heeren, O. Azencot, M. Rumpf, and M. Ben-Chen. Elastic correspondence between triangle meshes. Computer Graphics Forum, 38(2):121-134, May 2019. 19
Supplementary Materials for: Shape Non-rigid Kinematics (SNK)
In this document, we have gathered all the results and discussions that, due to page limitations, were not included in the main manuscript.
More precisely, we initially detail the specifics of our implementation in Appendix A. Following this, we present a comprehensive analysis of the components introduced in our methodology in Appendix B. Further examination of our architecture's components can be found in Appendix C. Lastly, we include some qualitative results in Appendix D.
A.1 Network Architecture
In Section 4 of the main paper, we elaborated on our architecture, which is comprised of two primary modules: the functional map module and the shape reconstruction module.
For the functional map module, we utilized DiffusionNet [64] as a feature extractor, using the authors' original implementation  ${}^{1}$  . It consists of four blocks, each with a width of 128, and the feature dimension  ${F}_{i}$  is likewise 128. The functional map block possesses a dimension of  $k = {30}$  ,and we used  $\lambda  = {10}^{-3}$  in Equation 1,and  $\tau  = {0.07}$  in Equation 4 from the main paper.
The shape reconstruction module employs an encoder via DiffusionNet, comprising four blocks and possessing a width of 128, with an output dimension  $d_{1} = 512$ . We derive the latent code  $l$  by performing max-pooling over the vertex dimension.
In terms of PriMo modeling, we construct a volumetric layer by (virtually) extruding the input mesh along the vertex normals. This process generates rigid prisms for each mesh face, with each prism having a height,  $h$ , set to 0.02 (see Figure 2 of the main text).
Our novel decoder, the prism decoder, relies on DiffusionNet at its core. Given the source shape and the latent code  $l$ , we duplicate the latter, concatenate it with the XYZ coordinates of the source shape, and then process them with a DiffusionNet composed of four blocks with a width of 128. The output is a pointwise feature  $D \in \mathbb{R}^{n_2 \times d_2}$  with  $d_2 = 512$ . We transform these features into face-wise features  $G$ , as described in the main paper. These features are subsequently refined with a pointwise MLP, which consists of four layers of Linear layer and ReLU layer; however, the last layer does not contain a ReLU. The width of the MLP gradually narrows from 512 to 12 in the final layer. We obtain the matrix  $R$  in Equation 5 from the main paper by initially performing an SVD decomposition [75] of  $R_0$  such that  $R_0 = \mathbf{U}\mathbf{D}\mathbf{V}^\top$ . The solution is then obtained by  $R = \mathbf{USV}^\top$  with  $\mathbf{S} = diag(1, 1, \det(\mathbf{U})\det(\mathbf{V}))$ . Employing the predicted translation and rotations matrices  $t_i$  and  $R_i$  for each face (considered a prism in the PriMo design), we rigidly deform the source mesh faces. The deformed surface mesh is finally derived from the resulting prisms by updating the position of each vertex using the average transformation of its incident prisms.
For all our experiments, we utilized the Adam optimizer [85], with a learning rate preset to 0.001. With respect to the losses in Equation 6 from the main text, we used  $\lambda_{\mathrm{mse}} = \lambda_{\mathrm{fmap}} = \lambda_{\mathrm{cycle}} = \lambda_{\mathrm{primo}} = 1$
<https://github.com/nmwsharp/diffusion-net>
Table 3: Runtime comparison. A comparison of the runtime of various baseline methods with our method on the FAUST dataset. Time is presented in seconds.
METHOD	Runtime (s)
BCICP [19]	780
Smooth Shells [55]	430
SNK (Ours)	70
For every new pair, we trained our network over the course of 1000 iterations, terminating the training if no improvement in the loss was observed for over 100 consecutive iterations.
A.2 PriMo Loss Implementation
In the main paper, Equations 2 and 3 introduced the formulation for the PriMo energy. In this section, we further detail the practical computation of this energy.
Given two functions  $a(\mathbf{u})$  and  $b(\mathbf{u})$  defined by bi-linear interpolation of four values  $\{a_{00}, a_{10}, a_{01}, a_{11}\}$  and  $\{b_{00}, b_{10}, b_{01}, b_{11}\}$  respectively, we define the scalar product [86]:
$$
\begin{array}{l} \langle a, b \rangle_ {2} := \int_ {[ 0, 1 ] ^ {2}} a (\mathbf {u}) \cdot b (\mathbf {u}) d \mathbf {u} \\ = \frac {1}{9} \sum_ {i, j, k, l = 0} ^ {1} a _ {i j} \cdot b _ {k l} \cdot 2 ^ {(- | i - k | - | j - l |)} \tag {7} \\ \end{array}
$$
Next, referring to Figure 2 in the main paper, the energy  $E_{ij}$  is simply defined as:
$$
E _ {i j} = \left\langle \mathbf {f} ^ {i \rightarrow j} - \mathbf {f} ^ {j \rightarrow i}, \mathbf {f} ^ {i \rightarrow j} - \mathbf {f} ^ {j \rightarrow i} \right\rangle_ {2} \tag {8}
$$
Here,  $\mathbf{f}^{i\rightarrow j} = \{\mathbf{f}_{00}^{i\rightarrow j},\mathbf{f}_{10}^{i\rightarrow j},\mathbf{f}_{01}^{i\rightarrow j},\mathbf{f}_{11}^{i\rightarrow j}\}$  represents the four corners of the face of prism  $P_{i}$  that is adjacent to prism  $P_{j}$ .
A.3 Computational Specifications
We carry out all our experiments using Pytorch [87] on a 64-bit machine equipped with an Intel(R) Xeon(R) CPU E5-2630 v4 operating at  $2.20\mathrm{GHz}$  and an RTX 2080 Ti Graphics Card. For comparison, we use the original code provided by the authors of each competing method and adhere to the optimal parameters reported in their respective papers.
A.4 Runtime Comparison
Our method's runtime performance, compared to various baselines, is analyzed in Tab. 3. Specifically, we assess the overall runtime for all the baselines on the FAUST dataset, which consists of shapes averaging around 5000 vertices. The average time per pair, reported in seconds, demonstrates that our method outpaces the competition in terms of speed.
In Section 4 of the main text, we introduced the components of our methodology. In this section, we investigate the impact of each component independently. For illustrative purposes, we utilize the near-isometric matching experiment on the FAUST dataset (as referenced in Section 5.2 of the main text). Nevertheless, analogous conclusions can be drawn in other scenarios.
Initially, we examine the novel decoder architecture we proposed. In this case, we maintain all other aspects of our method and solely modify the decoder. We experiment with a decoder based on a
Table 4: Ablation Study. This table displays an ablation study of the various components of our method, conducted on the FAUST dataset. The study shows that every component is crucial to the overall performance. The best result is highlighted.
METHOD	FAUST
Encoder	■ MLP	35.9
◇ DiffusionNet	13.4
★ DiffusionNet + ARAP	14.8
Losses	● Latent Code	4.0
● Chamfer loss	80.7
⊕ Spectral only	8.7
▲ Ours (without Lprimo)	6.5
● Ours (without Lcycle)	2.7
SNK (Ours)	1.9
Multilayer Perceptron (MLP) network (■), and one based on the DiffusionNet network (◇), both possessing a comparable number of parameters to our decoder. Furthermore, given that several methods use the As-Smooth-As-Possible (ARAP) [65, 35, 36] energy for deformation regularization, we also compare our approach with a DiffusionNet decoder supplemented with ARAP loss (★).
Concerning the encoder, besides the shape encoder design we employed, we also explored the option of directly learning the latent code (marked with  $\clubsuit$  ) in a way similar to DeepSDF [88].
With respect to the loss functions, we compare our method against an approach that exclusively overfits the spectral losses [28], and then report the result of converting the predicted functional maps into point-to-point  $(\oplus)$ . Additionally, we also report the outcome of training our network with the unsupervised Chamfer loss as a substitute for the Mean Squared Error (MSE) loss  $\mathcal{L}_{\mathrm{mse}}$  that we used  $(\bullet)$ . Moreover, we experiment with our network trained without the PriMo loss  $\mathcal{L}_{\mathrm{primo}}(\triangleleft)$  and finally without the cycle loss  $\mathcal{L}_{\mathrm{cycle}}(\nabla)$ .
The results are summarized in Tab. 4, illustrating that all the components we incorporated are crucial for optimal performance. Specifically, the choice of the decoder is vital. Both the MLP and DiffusionNet decoders failed to match the results of our decoder, even when all our loss functions were used. The ARAP loss, in particular, hindered the performance of the DiffusionNet decoder, a contrast to its typical application in network training on a dataset.
Regarding the encoder, the shape encoder design outperforms the latent code design. This superior performance can be attributed to the shape encoder's ability to utilize the input shape as an added information source. This advantage is further enhanced by the intrinsic regularization provided by the network architecture, often referred to as the neural bias [83, 84].
In terms of the loss functions, training our architecture with the Chamfer loss was completely unsuccessful as it does not preserve the order of the points and is therefore unsuitable for matching. Using only the spectral loss yields decent results but falls short of state-of-the-art performance. This illustrates the importance of the interplay between spectral and spatial losses for optimal performance. The PriMo Energy plays a key role as, without it, the decoder fails to produce smooth, natural-looking deformations [67] that facilitate learning. Finally, the cycle loss proves critical in refining the point-to-point maps and achieving state-of-the-art results.
C In-Depth Exploration of SNK's Components
C.1 The Feature Extractor
The feature extractor plays a pivotal role in our architecture. The features it extracts are crucial for predicting both the fmap and the p2p maps, which in turn influence the loss. Consequently, consistency between the features of the two shapes is essential. To elucidate this, we have provided a
Features 2
Figure 6: Consistency between extracted features for source and target shapes.
visualization in Fig. 6 that highlights the congruence of features produced by the feature extractor for a shape pair. In this representation, we reduced the feature dimensions using PCA and then portrayed them as RGB values. The visualization readily demonstrates that the features from both shapes align closely, highlighting congruent areas.
C.2 SNK's Priors
In this section, we detail the priors employed by our method and highlight the unique neural prior that sets our approach apart from axiomatic methods.
- Elastic Energy as a Prior: PRIMO being an elastic energy, it penalizes non-linear stretching and bending. This directs our optimization towards smooth and realistic deformations.
- Cyclic Bijectivity via the cycle consistency loss.
- Near-isometry: Our method weakly enforces near-isometry in fmap predictions through Laplacian commutativity in a reduced  $(k = 30)$  functional basis. It should be noted that using elastic energy and weakly promoting isometries with low-frequency functional maps for non-isometric matching has precedents in works like [89, 90, 91, 55].
- Neural Networks: Finally, a subtle yet essential prior in our method is in our use of a neural network for feature extraction. Our approach follows studies like Deep Image Prior [84] and Neural Correspondence Prior [83] which show that neural networks can act as strong regularizers, generating features that enable plausible mappings even without pre-training
To demonstrate the neural prior empirically, we conducted two additional experiments:
- HKS-based Features: We replaced our feature extractor with the more traditional HKS features.
- Free Variables: We treat  $F_{1}$  and  $F_{2}$  as free variables, optimized via gradient descent without using a neural network.
Our findings (see Tab. 5) show that without the neural network regularization, the free variable model fails to converge. The HKS-based approach performs, but significantly worse than ours. This suggests neural networks offer extra regularization, surpassing traditional handcrafted features.
Overall, our method's priors are synergistic and versatile, making our approach applicable to a broad range of shapes.
In this section, we present a selection of visual results derived from our proposed method.
Firstly, in Fig. 7, we present the reconstructed shapes from the SCAPE dataset as produced by our method when overfitting a shape pair. This reconstruction is subsequently used to perform matching
Table 5: Ablation on the neural prior.
METHOD	FAUST
HKS features	20.3
Free Variables for Features	64.0
SNK (Ours)	1.9
Figure 7: Shape Reconstruction from the SCAPE dataset: The figure showcases the results of deforming the source shape to match the target shape using our architecture. It's noteworthy how well the shape has been reconstructed, a process that was completed in an entirely unsupervised manner.
using the nearest-neighbor approach. The high quality of the reconstruction is evident, even when dealing with challenging input pairs where the source and target pose exhibit significant variance.
Secondly, we inspect the quality of the shapes reconstructed by our novel decoder architecture during training. For this experiment, we deform a source shape to match a target shape using only the  $\mathcal{L}_{\mathrm{mse}}$  loss. Instead of utilizing the map generated by the functional map module, we employ the ground truth map, visualizing various shapes produced during the optimization process. We compare our decoder to a DiffusionNet-based decoder and a decoder utilizing DiffusionNet with ARAP loss applied, aligning with the approach in Appendix B. These results are illustrated in Fig. 8 using a shape pair from the FAUST dataset. Our method is the only one to produce natural-looking, smooth deformations, which in turn results in superior matching outcomes.
Lastly, we showcase some failure cases of our method on the partial setting. In fact, our method is geared toward complete shape matching, but in Fig. 9, we showcase our method's performance on three partial pairs from the Shrec 16' Cuts and Holes subsets [49]. Our method performs satisfactorily when dealing with moderate partiality.
Figure 8: Shape Interpolation using the FAUST dataset: Various decoders are employed to interpolate between a source and target shape. Compared to the baselines, our method yields smooth and natural-looking deformations.
Figure 9: Our method is tailored toward complete shape matching. Here we showcase the results of our method on some partial pairs from the Shrec 16 partial benchmark.