SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Souhaib Attaiki  
LIX, École Polytechnique, IP Paris  
attaiki@lix.polytechnique.fr
Maks Ovsjanikov  
LIX, École Polytechnique, IP Paris  
maks@lix.polytechnique.fr
We present Shape Non-rigid Kinematics (SNK), a novel zero-shot method for non-rigid shape matching that eliminates the need for extensive training or ground truth data. SNK operates on a single pair of shapes, and employs a reconstruction-based strategy using an encoder-decoder architecture, which deforms the source shape to closely match the target shape. During the process, an unsupervised functional map is predicted and converted into a point-to-point map, serving as a supervisory mechanism for the reconstruction. To aid in training, we have designed a new decoder architecture that generates smooth, realistic deformations. SNK demonstrates competitive results on traditional benchmarks, simplifying the shapematching process without compromising accuracy. Our code can be found online: https://github.com/pvnieo/SNK.
Shape matching, specifically non-rigid shape matching, is a fundamental problem in Computer Vision, Computational Geometry, and related fields. It plays a vital role in a wide range of applications, including statistical shape analysis [1, 2], deformation [3], registration [4], and texture transfer [5], among others. This problem involves finding a correspondence between two shapes that may differ due to non-rigid deformations, such as bending, stretching, or compression.
Traditional methods for addressing this problem have predominantly been axiomatic, offering several benefits, such as theoretical guarantees [6] and their application in different scenarios [7, 8, 9, 10]. Some of these methods include intrinsic isometries-based methods [11, 12], parametrization domain-based methods [13, 14], and functional map methods [15, 16, 17]. However, these methods often heavily rely on the availability of good initialization [18, 19, 20] or the use of handcrafted descriptors [21, 22, 23]. This dependence can be a limiting factor, constraining the performance of these methods and making them less effective in situations where these requirements cannot be adequately met.
Learning-based approaches have emerged as an alternative to these traditional methods, with notable examples including Deep Functional Maps methods [24, 25, 26, 27, 28, 29], dense semantic segmentation methods [30, 31, 32, 33], or template-based methods [34] which have shown significant promise in solving the shape-matching problem. The strength of these methods lies in their capacity to learn features and correlations directly from data, outperforming handcrafted descriptors. However, these techniques introduce a significant challenge. Even when the task is to match just a single pair of shapes, these methods necessitate the collection and training on a large data set, often in the tens of thousands [34]. The demand for these extensive data collections, coupled with the long training times that can extend into days [35, 36], can be particularly onerous.
Figure 1: Method overview. Given two shapes as input, we deform a source shape to resemble the target. Our network can be trained using just a single pair of shapes and uses a combination of spatial and spectral losses, as well as a regularization on the decoder.
Our proposed method, Shape Non-rigid Kinematics (SNK), positions itself uniquely in this landscape, combining the benefits of both axiomatic and learning-based approaches while addressing their limitations.
In particular, SNK works directly on a given pair of shapes, without requiring any prior training. This removes the need for large data sets or pre-existing examples. Unlike axiomatic methods, however, that rely on fixed input features such as HKS or SHOT, we employ a deep neural network for feature computation. As we demonstrate below, this use of a neural network introduces a significant prior for our method, setting it apart from purely geometric approaches. We thus refer to our method as "zero-shot" to highlight its ability to be trained on a single pair of shapes, while using a neural network for feature learning.
SNK is applied to each pair individually using a reconstruction-based strategy that employs an encoder-decoder architecture. By doing so, SNK effectively deforms the source shape to closely match the target shape. A point-to-point map is then extracted by a simple nearest neighbor between the target and the deformed source.
At the heart of our method lies the functional map paradigm. In our approach, a functional map is generated in an unsupervised manner and then converted into a point-to-point map. This map is subsequently used as supervision for the decoder, ensuring the accuracy of the shape reconstruction process. To confine the deformation to plausible transformations, we introduce a new decoder architecture that is grounded on the PriMo energy [37]. Lastly, our method is trained using losses in both the spatial (reconstruction) and spectral (functional map) domains, ensuring comprehensive performance improvement.
- We have devised a new decoder architecture rooted in the PriMo energy concept. This architecture facilitates the production of deformations that appear smooth and natural.
- We have shown that a loss function, which imposes penalties on both spatial and spectral quantities, is adequate for deriving matches on a single pair of shapes without any prerequisite training.
- We have developed a method for zero-shot shape matching that attains state-of-the-art results among methods that are not trained on shape collections. Furthermore, it competes with, and frequently outperforms, several supervised training-based approaches.
Non-rigid shape matching is a widely researched subject, and providing a comprehensive overview is beyond the scope of this paper. However, we will briefly discuss methods closely related to ours. For a more detailed exploration, we encourage interested readers to refer to recent surveys [38, 39, 40, 41].
Functional Map Our method is built upon the functional map paradigm, which since its introduction [15], has been widely used for nonrigid shape matching [27, 19, 18, 17, 16, 7, 9, 42, 29, 43, 44, 45, 46]. It offers a distinct advantage by transforming the complexity of pointwise maps optimization, originally quadratic relative to the vertex count, into a more manageable process of optimizing a functional map (which consists of small quadratic matrices), making the optimization process feasible, see [47] for an overview.
In the pursuit of identifying the functional map, initial studies depended on handcrafted feature functions defined on both source and target shapes, such as HKS [22], WKS [23], and SHOT [21] features. Subsequent research enhanced this pipeline by incorporating extra regularization [16, 8, 48], adjusting it to accommodate partial shapes [9, 49, 10, 7], and suggesting efficient refinement techniques [19, 50, 51]. Contrarily, while these techniques often rely on an initial map for refinement or handcrafted features that lack robustness [25], our approach distinctively learns features by precisely fitting the source and target shapes using a neural network.
Learning-based Methods for Shape Matching The field of shape matching has recently seen a surge in success with learning-based methods. These approaches take on various forms, including framing the issue as a segmentation problem, which involves training a network to segment each vertex on the source shape into a class corresponding to a vertex ID on a template shape [30, 31, 32, 33]. Other methods include template fitting, which deforms a template to align with source and target shapes [34], and the use of functional maps [24, 28, 52, 25].
The latter method involves a neural network that extracts features from source and target shapes to predict the functional map, which is then transformed into a point-to-point map. This innovative line of research was pioneered by FMNet [24] and subsequently expanded in numerous studies [27, 29, 25, 26]. These studies used ground-truth functional maps for supervision.
Simultaneously, a parallel line of research focuses on unsupervised learning, which proves to be beneficial when ground truth correspondences are unavailable. These methods ensure structural properties like bijectivity and orthonormality on functional maps in the reduced basis [28, 29], penalize the geodesic distortion of the predicted maps [53, 54], or harmonize intrinsic and extrinsic shape alignment [35, 55].
However, unlike axiomatic approaches, these methods are typically not competitive when applied on a single pair without prior training. In contrast, our method, though it is based on a neural network, can be directly applied to a single pair without any prerequisite training.
Shape Reconstruction Our method employs an encoder-decoder framework to deform the source shape to fit the target shape. One significant challenge in this domain is crafting a shape decoder architecture capable of generating a 3D shape from a latent code. Early efforts in this field, such as those by Litany et al., Ranjan et al., and Bouritsas et al. [56, 57, 58], built on graph architecture, defining convolution, pooling, and unpooling operations within this structure. However, they presumed uniform triangulation across all shapes, rendering them ineffective for shape matching.
Other methods, such as Zhang et al. [59], predict occupancy probabilities on a 3D voxel grid, but the resource-intensive nature of dense, volumetric representations restricted resolution. Different approaches decode shapes as either raw point clouds [60, 61, 62] or implicit representations [63, 63]. Nevertheless, these methods overlooked the point ordering, making them unsuitable for matching.
In our work, we propose a novel decoding architecture, which leverages the architecture of DiffusionNet [64] as a foundational element. Our approach computes a rotation and a translation matrix for every face on the source shape to align with the target shape. While previous methods relied on energies such as As-Rigid-As-Possible (ARAP) [65], Laplacian [66], or edge contraction [34] to restrict the decoder, our approach is adept at using the PriMo energy [37]. This energy encourages smooth, natural deformations that effectively preserve local features, particularly in areas of bending [67], providing a more constrained and intuitive approach to shape deformation.
3 Notation & Background
Our work adopts the functional map paradigm [68] in conjunction with the PriMo energy [37]. In the following section, we briefly introduce these concepts, where we will use consistent notation throughout the paper.
Given a 3D shape  $S_{i}$  composed of  $n_i$  vertices and represented as a triangular mesh, we perform a cotangent Laplace-Beltrami decomposition [69] on it. The first  $k$  eigenvalues of  $S_{i}$  are captured in the matrix  $\Phi_{i} \in \mathbb{R}^{n_{i} \times k}$ . We also create a diagonal matrix  $\Delta_{i} \in \mathbb{R}^{k \times k}$ , where the diagonal elements hold the first  $k$  eigenvalues of  $S_{i}$ .
Next, the diagonal matrix of area weights,  $M_{i} \in R^{n \times n}$ , is defined. It's worth mentioning that  $\Phi_{i}$  is orthogonal in relation to  $M_{i}$ , and that  $\Phi_{i}^{\top}M_{i}\Phi_{i}$  is equivalent to the  $\mathbb{R}^{k \times k}$  identity matrix, denoted as  $I_{k}$ . Lastly, we represent  $\Phi_{i}^{\dagger} = \Phi_{i}^{\top}M_{i}$  using the (left) Moore-Penrose pseudo-inverse symbol,  $\cdot^{\dagger}$ .
3.2 Functional Map Pipeline
Denoting source and target shapes as  $S_{1}$  and  $S_{2}$ , we define a pointwise map,  $T_{12}:S_1\to S_2$ , encapsulated in binary matrix  $\Pi_{12}\in \mathbb{R}^{n_1\times n_2}$ . To manage its quadratic growth with vertex count, we employ the functional map paradigm [15], yielding a  $(k\times k)$  size functional map  $C_{21} = \Phi_1^\dagger \Pi_{12}\Phi_2$  with  $k$  usually around 30, making the optimization process feasible.
Feature functions  $F_{1}$  and  $F_{2}$  are then derived from  $S_{1}$  and  $S_{2}$  respectively, with their spectral coefficients calculated as  $\mathbf{A}_i = \Phi_i^\dagger F_i$ . This enables the formulation of an optimization problem:
$$
\underset {\mathbf {C}} {\arg \min } \| \mathbf {C} \mathbf {A} _ {1} - \mathbf {A} _ {2} \| _ {F} ^ {2} + \lambda \| \mathbf {C} \Delta_ {1} - \Delta_ {2} \mathbf {C} \|, \tag {1}
$$
where  $\mathbf{C}$  is the desired functional map and the second term is a regularization to promote structural correctness [25]. Finally, the point map  $\Pi_{21} \in \{0, 1\}$  is derived from the optimal  $\mathbf{C}_{12}$ , based on the relationship  $\Phi_2 \mathbf{C}_{12} \approx \Pi_{21} \Phi_1$ , using methods such as nearest neighbor search or other post-processing techniques [18, 50].
3.3 PriMo Energy: Prism Based Modeling
The PriMo approach, introduced in [37], is a prominent method in shape deformation [67], employing a model that mimics the physically plausible behavior of thin shells and plates [70, 71]. In this model, a volumetric layer is formed by extruding the input mesh along vertex normals, thus generating rigid prisms for each mesh face (refer to Fig. 2). These prisms are interconnected by elastic joints, which stretch under deformation—the extent of this stretch effectively determines the deformation energy. Interestingly, this formulation permits a unique global minimum of the energy in the undeformed state, with any form of bending, shearing, twisting, or stretching causing an increase in this energy.
We elaborate on the elastic joint energy using the notation in Fig. 2. Two adjacent prisms, denoted  $P_{i}$  and  $P_{j}$ , are considered. In their undeformed state, they share a common face that might no longer match after deformation. The face of  $P_{i}$  adjacent to  $P_{j}$  is defined as a rectangular bi-linear patch  $\mathbf{f}^{i\rightarrow j}(u,v)$ , which interpolates its four corner vertices. The corresponding opposite face is denoted as  $\mathbf{f}^{j\rightarrow i}(u,v)\subset P_j$ . The energy between  $P_{i}$  and  $P_{j}$  is thus defined as:
$$
E _ {i j} := \int_ {[ 0, 1 ] ^ {2}} \| \mathbf {f} ^ {i \rightarrow j} (u, v) - \mathbf {f} ^ {j \rightarrow i} (u, v) \| ^ {2} d u d v \tag {2}
$$
$$
E := \sum_ {\{i, j \}} w _ {i j} \cdot E _ {i j}, \quad w _ {i j} := \frac {\left\| \mathbf {e} _ {i j} \right\| _ {2} ^ {2}}{\left| F _ {i} \right| + \left| F _ {j} \right|} \tag {3}
$$
Here,  $E$  is the deformation energy of the whole mesh, and the energy contribution from each pair  $P_{i}, P_{j}$  is influenced by the areas of their corresponding mesh faces  $F_{i}, F_{j}$ , and the squared length of their shared edge  $\mathbf{e}_{ij}$ . In the original PriMo paper, users are allowed to dictate the positions and orientations of some prisms, while an optimization technique is deployed to minimize the total
1. Initial state
2. Notation
4. Stretching
Figure 2: PriMo construction. The mesh vertices are represented as white circles. The elastic joints are colored yellow.
deformation energy. In contrast, in our approach, the deformation is the result of the decoder's output and the PriMo energy is utilized to regularize it.
In this section, we delve into our proposed method for zero-shot matching, which consists of two primary modules. The initial (functional map) module, is responsible for predicting functional maps between the source and target shapes and converting these into point-to-point (p2p) maps. The second module, an encoder-decoder, encodes the target shape into a latent code. Subsequently, the decoder utilizes this latent code to morph the source shape to align with the target shape.
Diverging from previous studies, we incorporate losses in both the spatial and spectral domains. Spatially, we utilize the p2p map, as predicted by the functional map module, to ensure the reconstruction closely resembles the target. Spectrally, we enforce structural properties such as bijectivity and orthogonality on the functional map. Furthermore, to facilitate the learning process, we minimize the PriMo energy on the decoder, which ensures it generates a natural-looking and smooth deformation.
In the subsequent sections, we elaborate on each stage of our proposed model, and we will use the same notation as Sec. 3.
4.1 Functional Map Module
The functional map module is designed with a two-fold objective - to predict both fmap and p2p maps reciprocally between the source and target shapes. This module is composed of two key elements: the feature extractor and the fmap solver, as illustrated in Fig. 1.
The feature extractor's purpose is to extract features from the raw geometry that are robust to shape deformation and sampling, which are later employed for predicting the fmap. To achieve this, we use the DiffusionNet architecture [64], following the recent approaches in the domain [27, 52, 26, 72]. The feature extractor processes both the target and the source shapes,  $S_{1}$  and  $S_{2}$  respectively, and extracts pointwise features in a Siamese manner, denoted as  $F_{1}$  and  $F_{2}$ .
Following the feature extraction, the fmap solver accepts the features  $F_{1}$  and  $F_{2}$ , and predicts an initial approximation of the fmaps  $\mathbf{C}_{12}^{0}$  and  $\mathbf{C}_{21}^{0}$  utilizing Eq. (1). These fmaps are matrix solutions to the least square optimization, and thus do not necessarily originate from p2p maps. To rectify this, we draw inspiration from recent methodologies [43, 18, 45, 73, 74] and predict a new set of fmaps that stem from p2p maps. Initially, we transform the preliminary fmaps into an early estimation of p2p maps by applying:
Figure 3: Prism Decoder. Our novel architecture, built upon DiffusionNet, initiates the process by extracting per-vertex features, which are subsequently consolidated into per-face features. These features are then processed by a Multilayer Perceptron (MLP) to generate per-face rotation and translation, which are used to rigidly transform the input faces. The transformed faces are then consolidated to produce the final reconstruction.
$$
\hat {\Pi} _ {2 1} ^ {0} (i, j) = \frac {\exp \left(\left\langle G _ {2} ^ {i} , G _ {1} ^ {j} \right\rangle / \tau\right)}{\sum_ {k = 1} ^ {n _ {1}} \exp \left(\left\langle G _ {2} ^ {i} , G _ {1} ^ {k} \right\rangle / \tau\right)}. \tag {4}
$$
Here,  $G_{1} = \Phi_{1}$ ,  $G_{2} = \Phi_{2}\mathbf{C}_{12}^{0}$ ,  $\hat{\Pi}_{21}^{0}$  is a soft p2p map predicted in a differentiable manner,  $\langle \cdot ,\cdot \rangle$  represents the scalar product, and  $\tau$  denotes a temperature parameter. We predict  $\hat{\Pi}_{12}^{0}$  in a comparable way, followed by predicting the final fmaps  $\mathbf{C}_{12}$  and  $\mathbf{C}_{21}$  that arise from p2p maps using the definition  $\mathbf{C}_{ij} = \Phi_j^\dagger \hat{\Pi}_{ji}^0\Phi_i$ . Ultimately, we estimate the final p2p map  $\hat{\Pi}_{ij}$  from  $\mathbf{C}_{ji}$  by applying Eq. (4) for a second time. This procedure aligns with two iterations of the ZoomOut algorithm [18].
4.2 Shape Reconstruction Module
The goal of the shape reconstruction module is to deform the source shape to match the target shape. It is composed of a encoder that extract a latent code that summarizes the target shape, and a decoder that takes the latent code as well as the source shape and deform it to match the target.
The encoder is a DiffusionNet network [64], that takes the 3D coordinates of the target shape, and produces a  $d_{1}$ -dimensional vertex-wise features. A max pooling is applied in the vertex dimension to obtain a latent code  $l \in \mathbb{R}^{d_1}$ .
The prism decoder We introduce a novel architecture for the decoder, utilizing DiffusionNet at its core. An overview of this architecture is presented in Fig. 3. The decoder receives two inputs: the 3D coordinates of the source shape and the latent code  $l$ . The latent code is duplicated and combined with the XYZ coordinates, which are subsequently inputted into the DiffusionNet architecture. This process generates vertex-wise features  $D \in \mathbb{R}^{n_2 \times d_2}$  of dimension  $d_2$ .
These vertex-wise features  $D$  are then transformed into face-wise features  $G \in \mathbb{R}^{f_2 \times d_2}$ , where  $f_2$  denotes the number of faces of the source shape. This transformation is achieved via mean pooling, where the three features corresponding to the vertices constituting each face are averaged:  $G_i = \frac{1}{3} \sum_{j \in \mathbf{F}_i} D_j$ , with  $\mathbf{F}_i$  representing the  $i^{th}$  face of the target shape.
These features are further refined by a compact network composed of pointwise linear layers and ReLU nonlinearities, producing outputs  $H \in \mathbb{R}^{f_2 \times 12}$ . The first three features are interpreted as a translation vector  $t_i$ , and the final nine features are reshaped into a  $3 \times 3$  matrix  $R_i^0$ . This matrix is subsequently transformed into a rotation matrix  $R_i$  by solving the Orthogonal Procrustes problem [75]:
$$
R = \underset {\Omega} {\arg \min } \| \Omega - R ^ {0} \| _ {F} \quad \text {s u b j e c t t o} \quad \Omega^ {\top} \Omega = I _ {3}. \tag {5}
$$
With  $t_i$  and  $R_i$  defined for each face, we transform the original source shape's faces  $\mathbf{F}$ , resulting in a new set of faces  $\mathbf{F}'$ . The vertices of the decoded shape are then constructed by averaging the new face coordinates' positions for all the faces they belong to.
4.3 Training Procedure
Our method operates in a zero-shot manner, meaning that it is trained on each new pair of shapes independently. This approach merges the benefits of axiomatic methods, which do not rely on large training sets and extended training periods, and the advantages of learning methods, which facilitate matching through learned features, transcending the limitations of handcrafted features used in axiomatic methods.
The process of our method unfolds as follows: the target and source shapes, designated  $S_{1}$  and  $S_{2}$ , are processed through the functional map module. The output of this stage comprises the fmap and p2p maps, represented as  $\mathbf{C}_{ij}$  and  $\hat{\Pi}_{ij}$ . Concurrently, a latent code,  $l$ , is extracted via the encoder and employed to deform the source shape to align with the target shape. The resulting deformed shape,  $S_{3}$ , maintains the same triangulation as the source shape. Utilizing all these outputs, a loss, denoted as  $\mathcal{L}$ , is computed, which subsequently guides the optimization of the parameters of the various networks through gradient descent. This process is iteratively executed until either the loss ceases to decrease or the maximum number of iterations is achieved. The output that yields the lowest loss is ultimately chosen for matching.
In this context, a p2p map between the source and target shape, known as  $T_{21}^{0}$ , is derived by performing a nearest neighbor search between the reconstructed shape  $S_{3}$  and the target shape  $S_{1}$ . As suggested in previous works [27, 52, 25, 29, 64, 26], we refine this map by applying the ZoomOut algorithm [18]. This method enhances the map by iterating between the spectral and spatial domains, gradually augmenting the number of spectral basis functions. The refined map,  $T_{21}$ , is then employed for evaluation.
Loss function In training our network, we employ multiple loss functions. Contrary to prior works [29, 28, 36, 35] that solely utilize either spatial or spectral losses, our method synergistically integrates both types for enhanced performance. The overall loss function is as follows:
$$
\mathcal {L} = \lambda_ {\mathrm {m s e}} \mathcal {L} _ {\mathrm {m s e}} + \lambda_ {\mathrm {f m a p}} \mathcal {L} _ {\mathrm {f m a p}} + \lambda_ {\mathrm {c y c l e}} \mathcal {L} _ {\mathrm {c y c l e}} + \lambda_ {\mathrm {p r i m o}} \mathcal {L} _ {\mathrm {p r i m o}} \tag {6}
$$
We denote  $\mathbf{S}_i$  as the XYZ coordinates of shape  $S_{i}$ . The Mean Square Error (MSE) loss,  $\mathcal{L}_{\mathrm{mse}}$ , is employed to ensure that the reconstructed shape closely resembles the target shape using the predicted p2p map, it's calculated as  $\mathcal{L}_{\mathrm{mse}} = \| \hat{\Pi}_{21}\mathbf{S}_1 - \mathbf{S}_3\| _F^2$ .