Shape Registration in the Time of Transformers

1 Introduction . Recent technological advancements of 3D acquisition pipelines have produced an abundance of available data . The direct consequence is the non-standardization of the acquisition process . Such technological democratization brings along a disparate amount of different representations , discretization , and arbitrary resolution . Given so , the request to align such data has become urgent . Furthermore , data-driven statistical approaches require aligned data to relate feature changes across the population , inferring underlying patterns . The Computer Vision community has devoted an extraordinary effort in the last decades to address 3D objects analysis . A common way to approach this problem is to align the geometry of one 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . source S target T output known shape to an incoming one . Such methodology is referred to as registration . Many different axiomatic pipelines have been proposed that address different kind of objects and domains . While many methods rely on the assumption that the shapes used in the registration task differ just by a rigid transformation , the non-rigid domain is far more complex and interesting . This category pertains organic objects ( e.g. , humans , animals , internal organs ) , which are particularly interesting as well . Non-rigid registration aims to align two geometries that may differ by bending and stretching of the geometry , which may also significantly modify its metric . This problem is even more complicated if the geometry representation is given just by a sparse point cloud . However , an emerging field merges data with classical algorithmic problems , exploiting such statistics as regularization . Among the different learning approaches , recently , the use of the Attention mechanism has become significantly popular in NLP domain , being later transferred to Computer Vision applications . Such architecture is called Transformers , and they represent one of the most significant groundbreaking methodological advancement since the introduction of CNNs . In this work , we aim to let non-rigid registration meet the transformers . Intuitively , we aim to use the transformer as a geometrical translator between two non-rigid point clouds . As the first element , we modified the attention mechanism , proposing to make it aware of the underlying density of the geometry . Hence , we apply such a mechanism in an autoencoder–like architecture , which takes a template point cloud as input and aims to modify its geometry to fit the target point cloud . The proposed method achieves better results than several state-of-the-art competitors in the shape matching task . We show results on humans case , but also inter-class objects . In this second case , our method is trained in an unsupervised manner , showing the power of our attention mechanism to infer the underlying geometry . Also , thanks to the attention mechanism , we are able to interpret what the network considers relevant for the registration . Finally , we can target texture transfer and shape interpolation showing applicability in real tasks as demonstrated in 1 . Our contributions could be summarized as follows : ( a ) We propose the first transformer for non-rigid registration task , showing the advantages of translation paradigm ; ( b ) We modified the attention mechanism to make it aware of the point cloud density and of the underlying geometry of a shape ; ( c ) We significantly improve the state-of-the-art performances on different datasets and challenging scenarios . All code and data is publicly available 1 . 1https : //github.com/GiovanniTRA/transmatching 2 Related work . Shape matching is a problem with a tradition of decades . For a complete overview , we refer to the surveys [ 54 , 51 ] ; below we cover the literature that more closely relates to our work . Surface matching Early attempts to match non-rigid objects work under the assumption of near isometry . Such is the case , for instance , of blended intrinsic maps [ 27 ] , which combine multiple conformal mappings with an additional penalty to preserve local areas . Similarly , several variants and applications of the functional maps framework [ 41 ] implicitly assume near isometries by requiring a special structure of the functional representation , or by means of dedicated regularizers [ 42 , 40 , 18 , 38 , 49 ] also designed for handling partial shapes [ 50 , 11 , 45 ] . A common drawback to these works is that they do not disambiguate the intrinsic shape symmetries ; further , the surface connectivity introduces a structural bias which may affect the performance , as recently shown in [ 36 ] . An attempt to overcome these issues was proposed in [ 37 ] , but with the extra assumption that the shapes to match are in the same pose . More recently , SmoothShells [ 15 ] proposes an iterative algorithm to recover dense correspondences by an alignment of intrinsic information . While these methods do not address 3D shape registration directly , they rely on the general idea that a correspondence can be recovered by aligning specialized , possibly high-dimensional embeddings of the shapes at hand . Template-free registration A popular approach to solve for a matching between 3D objects is to align their geometries extrinsically via ICP-like procedures [ 5 , 29 , 2 ] . In fact , registration and matching are intimately related problems with different goals . While the matching problem aims to find a combinatorial solution , which indicates for each point its image on the target shape , registration looks for a spatial transformation of the geometry . If the two shapes have a significantly different discretization , the latter problem is less ambiguous than the former . ICP-based approaches iteratively solve the two subproblems in an alternating fashion , by finding a point-to-point correspondence and the best transformation that adheres to such correspondence . These methods do not convergence to a good solution if the input shapes are significantly misaligned . Similarly , Coherent Point Drift [ 39 ] and variants [ 26 , 22 ] rephrase the registration problem as an alignment of probability densities . Template-based registration A different family of approaches make use of a given template , which is known a priori and is possibly parametric , toward which a given input shape is to be matched . Taking as an example the case of human bodies , it is common to model the surface deformation related to the subject identity using PCA [ 1 , 3 , 31 , 44 ] , while recent advancements in statistical data-science suggested that non-linear methods are more expressive to catch fine details of humans [ 48 , 59 , 9 ] . Similarly , the pose can be modeled by simple paradigms like Linear-Blend Skinning [ 31 , 44 ] , triangle deformations [ 3 , 23 ] , but also learning methods [ 59 ] . Efforts to register such templates to arbitrary target models have been carried out extensively by the community [ 23 , 61 , 32 , 33 ] . However , the requirement of a template is not always easy to satisfy . Learning methods With the rise of learning methods , several attempts have been made to introduce a statistical prior to the matching and registration process . For example , several extensions have been proposed to bring the functional maps formalism into a learning paradigm [ 30 , 13 , 53 ] . SmoothShells has also been extended to be data-driven [ 16 ] . The point cloud representation has received comparably less attention , mainly in a rigid alignment setting [ 24 , 43 , 52 ] . In the non-rigid domain , a seminal work is 3DCoded [ 19 ] that proposes a proper registration using an autoencoder architecture . However , having a fixed template forbids inter–class operations and limits the use of the point cloud structure . The constraint of having a fixed template has been relaxed in [ 21 ] but it still requires couple of isomorphic shapes in training . Recently , it has been proposed to learn a linearly-invariant embedding [ 34 ] , but the method requires training two separate networks and relies on simple PointNets [ 46 ] which are not able to catch the fine details of the objects . Finally , a recent trend in geometric deep learning suggests that implicit representations may also be used for shape matching [ 6 ] . Transformer-based architectures Transformers have been first introduced in the context of neural machine translation by the pioneering work [ 56 ] , and later procedeed to revolutionize the field of natural language processing [ 12 , 47 ] . The success obtained in NLP inspired further work to employ transformers in computer vision [ 14 ] , where they managed to outperform convolutional networks . Exploiting the input invariance property characterising the attention mechanism , many works have naturally extended transformers to handle point clouds [ 20 , 60 , 17 ] . These works show promising results , but work only in the context of object classification and segmentation . Instead , [ 57 ] proposes a network to find the rigid alignment between two point clouds imitating ICP , and using a transformer architecture to infer the residual term . Differently , we aim to solve for non-rigid registration , which is a more general case . 3 Notation and general objective . 3D shapes We model 3D shapes as compact 2-dimensional Riemannian manifoldsM , possibly with boundary ∂M . In the discrete setting , we represent the manifoldM as an unorganized point cloud of nM vertices embedded in R3 , and encoded in a coordinate matrix XM ∈ RnM×3 . source target registration Shape registration The main objective of this paper is to introduce a data-driven approach to perform shape registration . Given a source shape S and a target shape T , respectively represented by the sets of vertices XS and XT , our goal is to find a corresponding 3D position for each point in S on the surface of T . An example is shown in the inset figure , where the underlying surface is visualized for reference only . Our method does not assume the source S to be a fixed template , but can generalize to arbitrary shape pairs S and T ; this is in contrast , e.g. , with [ 19 ] , where the objective is to learn how to deform a fixed known template into another shape . Attention One of the most influential ideas in the recent years , attention originated in the realm of natural language processing but has since gained traction in other fields , such as computer vision and signal processing , due to the vast increase in performance and interpretability exhibited in several tasks . At its heart , the attention mechanism allows learning models to encode latent relations between inputs , assigning higher importance , or `` attention '' , to the parts deemed more relevant . In this sense , attention allows to efficiently capture context information as well as higher order dependencies . Formally , given two generic input sequences X1 ∈ Rn×d and X2 ∈ Rm×d ( for clarity of exposition we assume a constant embedding dimension d , but it is not a necessary assumption ) , the attention mechanism models the linearly encoded representation of the inputs as triplets of query , key , and value matrices , respectively Q ∈ Rn×d , K ∈ Rm×d , and V ∈ Rm×d . The attention score W is then defined as W = softmax ( QKT d−0.5 ) and used to compute the weighted mean of the value vectors , resulting in an output feature matrix A = WV . We refer to self–attention whenever X1 and X2 are the same object , and use the term cross–attention otherwise .

Summary: This paper introduces the use of transformers architecture for shape matching and the method is applied to 3D point clouds. The main innovation with respect to standard attention architecture on point clouds is the weight introduced in the attention mechanism which accounts for the local density of the shape. Experiments show a clear gain of the method compared to other state of the art method, both quantitatively and qualitatively.