Representation-Agnostic Shape Fields

3D shape analysis has been widely explored in the era of deep learning . Numerous models have been developed for various 3D data representation formats , e.g. , MeshCNN for meshes , PointNet for point clouds and VoxNet for voxels . In this study , we present Representation-Agnostic Shape Fields ( RASF ) , a generalizable and computation-efficient shape embedding layer for 3D deep learning . RASF is implemented with a learnable multi-channel 3D grid to store local geometry . Shape embeddings for various 3D shape representations ( point clouds , meshes and voxels ) are retrieved by coordinates indexing . While there are multiple ways to obtain the coefficients of RASF , we provide two effective schemes among all in this paper for RASF pre-training , that is shape reconstruction and normal estimation . Once trained , RASF becomes a plug-and-play performance booster with negligible cost . Extensive experiments on diverse data representation formats ( i.e. , meshes , point clouds and voxels ) , backbones networks and applications , validate the universal effectiveness of the proposed RASF . Code and pre-trained weights will be made publicly available upon acceptance . 1 INTRODUCTION 3D shape analysis is the foundation to understand the physical world . It has a wide range of applications in real life including robotics ( Liu et al. , 2020 ; Liang et al. , 2018 ; Lawin et al. , 2018 ; Deng et al. , 2018 ) , autopilot ( Shi et al. , 2020 ; Song et al. , 2019 ; Qi et al. , 2018 ; Zhou & Tuzel , 2018 ) , medical imaging ( Yang et al. , 2021 ; Li et al. , 2018a ; Isensee et al. , 2019 ) and movie animation ( Xu et al. , 2019 ; Harvey et al. , 2020 ; Aberman et al. , 2020 ; Hertz et al. , 2020 ) . In recent years , the research on this topic has prevailed and showed promising results in various tasks , such as object classification , part segmentation , scene segmentation , etc . Shape could be represented in different data formats , among which meshes , point clouds and voxels are most commonly used . In the deep learning era , most studies on these three representations use coordinates or coordinates- like feature as input to feed into the backbone network . For point clouds , the direct input to the network is the point coordinates . For meshes , the node feature of the graph is the vertices coordinates . For volumetric data , the 3D shape is denoted by whether a voxel in a particular position is occupied or not . Using coordinates to characterize a shape is simple and straightforward . However , the major problem with this is that coordinate lacks contextual geometric information . Hence the capacity of the backbone network could be restricted . Even though various operators and backbones are proposed to extract high-level feature from the combination of coordinates by aggregating the local geometry , the effect of the input feature is not clear yet . The practice in Natural Language Processing ( NLP ) and Data Mining ( DM ) might shed some light on this issue . Word embeddings ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) is proposed very early in NLP to map words into an embedding space using embedding layers , which work as lookup tables that are indexed by the one-hot encoding of words . The word embeddings retrieved from embedding layers have similar values for words with similar meanings . This technique is widely adopted across the NLP area and notably boosts the overall performance , regardless of how the text data is distributed and which language model is used ( Brown et al. , 2020 ; Devlin et al. , 2019 ; Radford et al. , 2019 ; Vaswani et al. , 2017 ) . Studies in the field of data mining ( DM ) also learn continuous embedding representation for graph nodes ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ) . Embedding learning in NLP and DM indicates that the effect of input feature and the capacity of the backbone network are somehow orthogonal to each other . It raises the question that whether there is a better way to denote a shape instead of vanilla coordinates , so as to facilitate the backbone network to execute downstream tasks , regardless of which backbone model is used . In this work , we introduce Representation-Agnostic Shape Fields ( RASF ) , a shape embedding layer that maps coordinates to shape embeddings with rich geometry information . RASF is implemented using a learnable multi-channel 3D grid . Similar to the lookup table in word embedding layer , coordinates within a local shape index from this 3D grid and retrieve shape embeddings . With simple operation on data , we make RASF compatible with major 3D representations , including point clouds , meshes and voxels . To obtain the weights of RASF , we investigate several pre-training schemes for RASF , among which we find that self-supervised training schemes , i.e. , reconstruction and normal estimation , yield the best performance and generalizability . Once trained , RASF could be seamlessly plugged into any 3D deep learning pipeline to improve performance across diverse downstream tasks and datasets , with little code modification and computation cost ( See Fig . 1 ) . We empirically show that RASF consistently brings significant improvement under diverse backbones and applications , including object classification , part segmentation and scene segmentation . Contributions In this work , we introduce a generalizable ( i.e. , it could be used in different 3D representations , backbones and downstream tasks ) and computation-efficient shape embedding layer for 3D deep learning , named Representation-Agnostic Shape Fields ( RASF ) . It applies a learnable multi-channel 3D grid to store local geometry . Shape embeddings for various 3D shape representations ( point clouds , meshes and voxels ) are retrieved by coordinates indexing . While there are multiple ways to obtain the coefficients of RASF , we provide two effective schemes among all in this paper for RASF pre-training , that is shape reconstruction and normal estimation . Abundant experiments across different representations , backbones and downstream tasks are conducted to validate the generalization and efficiency of our proposed RASF . Code and pre-trained weights will be made publicly available soon . 2 RELATED WORK . 2.1 3D SHAPE ANALYSIS . Point clouds Point cloud data could be directly obtained from 3D LiDAR sensors . Due to its compactness that represents only the surface of the objects and its aligned format ( an N × 3 matrix ) that suits the common deep learning frameworks , point clouds are the most extensively discussed representation in the field of 3D shape analysis . PointNet ( Qi et al. , 2017a ) is the pioneering deep learning network to process point clouds . It learns the feature of each point with a shared MLP and aggregates all points by global max-pooling . PointNet++ ( Qi et al. , 2017b ) supports hierarchical points aggregation to extract geometric information at different scales . DGCNN ( Wang et al. , 2019 ) builds a dynamic graph in each layer of the network to incorporate neighbor nodes by its proposed EdgeConv module . PointCNN ( Li et al. , 2018b ) , RSCNN ( Liu et al. , 2019b ) , ShellNet ( Zhang et al. , 2019 ) , KPConv ( Thomas et al. , 2019 ) and PCT ( Guo et al. , 2021 ) extend 3D convolution operation to the irregular point clouds data in different ways . Meshes Meshes are mostly used in computer graphics ( Kato et al. , 2018 ; Liu et al. , 2019a ; Pfaff et al. , 2020 ; Smirnov & Solomon , 2021 ) . GWCNN ( Ezuz et al. , 2017 ) maps unstructured geometric data to a regular domain for nonrigid shape analysis . MeshCNN ( Hanocka et al. , 2019 ) adopts specialized convolution and pooling operations on mesh edges . The convolution is conducted on the edge and the four adjacent edges around the incident triangles , while the pooling operation generates new geometry via adaptive edge collapse . Besides , MeshCNN designs several hand-crafted features that characterize the edge geometry , that is the dihedral angle , two inner angles and two edgelength ratios for each face . The 5-dimensional vector is fed into the MeshCNN network as input . Alternatively , MeshNet ( Feng et al. , 2019 ) regards faces as units . It introduces face-unit and featuresplitting to learn on meshes directly . A more recent work , HodgeNet ( Smirnov & Solomon , 2021 ) , operates on feature of vertices and edges simultaneously . There are also works in the field of single image reconstruction , which consider mesh data as graphs ( Wang et al. , 2018 ; Gkioxari et al. , 2019 ; Pan et al. , 2019 ; Wen et al. , 2019 ) . In this case , graph convolutions on nodes ( vertexes ) are adopted to aggregate local geometry . Voxels Volumetric data provides regular grids to represent 3D shapes . They could be processed by methods analogous to 2D grids . 3DShapeNets ( Wu et al. , 2015 ) proposed to represent 3D shapes with volumetric grids and introduced 3D Convolution Neural Networks ( 3DCNN ) for voxel classification . VoxNet ( Maturana & Scherer , 2015 ) utilized 3DCNN for robust 3D object recognition . The Voxception-ResNet ( VRN ) ( Brock et al. , 2016 ) introduced popular 2D network blocks into volumetric networks . The drawback of volumetric representations lies in that 3DCNN is computationally expensive . In recent years , LP-3DCNN ( Kumawat & Raman , 2019 ) is proposed to alleviate the computation issue . It applies 3D Short Term Fourier Transform ( STFT ) to replace the 3D convolution layers . 2.2 SHAPE DESCRIPTORS AND EMBEDDING LEARNING . Surface feature descriptors or shape descriptors for non-rigid objects ( generally in mesh format ) are another line of work in 3D shape analysis , often applied in shape correspondence , retrieval and segmentation . Shape descriptors aim at representing the local geometry around a point in a non-rigid shape . Classic surface descriptors describe a local shape patch based on diffusion and spectral geometry to achieve isometry-invariance , for example , heat kernel signature ( HKS ) ( Sun et al. , 2009 ) , wave kernel signature ( WKS ) ( Aubry et al. , 2011 ) and optimal spectral descriptors ( OSD ) ( Litman & Bronstein , 2013 ) . Masci et al ( Masci et al. , 2015 ) proposed to automatically learn various shape descriptors for Riemannian manifolds using a geodesic convolutional neural network . Studies in NLP embedding learning maps the one-hot encoding of words to a real-value vector via a learnable embedding layer . An embedding layer is a lookup table , using words as indices and retrieve word embeddings from it . The embedding layer could be trained in an unsupervised manner for general proposes , or in a supervised manner for a specific task , e.g. , document classification . 3 METHODOLOGY . 3.1 REPRESENTATION-AGNOSTIC SHAPE FIELDS ( RASF ) . We propose a shape embedding layer named Representation-Agnostic Shape Fields ( RASF ) , to facilitate 3D shape analysis across various representations . As shown in Fig . 2 ( a ) , RASF is implemented as a trainable 3D grid with the shape of R × R × R × C , where R is the resolution of the grids and C is the channel dimension . Common 3D shape representations are based on point coordinates . Suppose the shape is denoted as a point set X ∈ RN×3 . For each point P in the shape , we extract the K-Nearest-Neighbor points Pneigh from the whole point set . Regarding P as the central point , we normalize the coordinates of the local point set A = [ P , Pneigh ] to the range of the grid . In other words , P would be placed at the center of the grid , while the farthest neighbor to P would be placed at the border of the grid . Similar to the word embedding layer that is indexed by one-hot encoding of words , the normalized local point set serves as indexes to retrieve shape features from the grid . The difference is that one-hot encoding is discrete while coordinates are continuous . Therefore the indexing is accomplished via trilinear interpolation . This operation inspired by Spatial Transformer Networks ( STNs ) ( Jaderberg et al. , 2015 ) could be efficiently implemented by modern deep learning frameworks ( e.g. , grid sample in PyTorch ( Paszke et al. , 2017 ) ) . This function enables continuous indexing in batches to retrieve feature from discrete regular grid data . The trilinear interpolation returns a ( K + 1 ) × C matrix for P , which is then reduced to 1 × C by max-pooling on the first dimension . The C−dim vector , named shape embedding ( SE ) , encodes the shape in the local area A around the central point P . The whole process to obtain shape embedding SE ( P ) of point P could be formulated as follows : SE ( P ) = max ( GS ( N ( [ P , Pneigh ] ) ) ) , ( 1 ) where GS denotes the grid sample function and N denotes the normalization . Note that the central point always corresponds to the central feature of RASF due to the normalization . Since RASF is a cubic grid , we use L1-distance in KNN searching for parameters efficiency . After processing each point in this way , we obtain shape embeddings SE ∈ RN×C for all the points . The computation in RASF is negligible compared to the backbone networks , as analyzed in Sec . 5 .

Summary: This paper proposes a method for learning a latent spatial embedding of points in space to a feature space by pre-training on a pretext task (e.g., reconstruction). This embedding can the be used as part of any deep learning pipeline that inputs point coordinates as part of its architecture by first mapping the coordinates to their embedding. The authors demonstrate that utilizing these embeddings improves results of state-of-the-art learning methods on point clouds, meshes, and voxel grids.