Representation-Agnostic Shape Fields

3D shape analysis has been widely explored in the era of deep learning . Numerous models have been developed for various 3D data representation formats , e.g. , MeshCNN for meshes , PointNet for point clouds and VoxNet for voxels . In this study , we present Representation-Agnostic Shape Fields ( RASF ) , a generalizable and computation-efficient shape embedding layer for 3D deep learning . RASF is implemented with a learnable multi-channel 3D grid to store local geometry . Shape embeddings for various 3D shape representations ( point clouds , meshes and voxels ) are retrieved by coordinates indexing . While there are multiple ways to obtain the coefficients of RASF , we provide two effective schemes among all in this paper for RASF pre-training , that is shape reconstruction and normal estimation . Once trained , RASF becomes a plug-and-play performance booster with negligible cost . Extensive experiments on diverse data representation formats ( i.e. , meshes , point clouds and voxels ) , backbones networks and applications , validate the universal effectiveness of the proposed RASF . Code and pre-trained weights will be made publicly available upon acceptance . 1 INTRODUCTION 3D shape analysis is the foundation to understand the physical world . It has a wide range of applications in real life including robotics ( Liu et al. , 2020 ; Liang et al. , 2018 ; Lawin et al. , 2018 ; Deng et al. , 2018 ) , autopilot ( Shi et al. , 2020 ; Song et al. , 2019 ; Qi et al. , 2018 ; Zhou & Tuzel , 2018 ) , medical imaging ( Yang et al. , 2021 ; Li et al. , 2018a ; Isensee et al. , 2019 ) and movie animation ( Xu et al. , 2019 ; Harvey et al. , 2020 ; Aberman et al. , 2020 ; Hertz et al. , 2020 ) . In recent years , the research on this topic has prevailed and showed promising results in various tasks , such as object classification , part segmentation , scene segmentation , etc . Shape could be represented in different data formats , among which meshes , point clouds and voxels are most commonly used . In the deep learning era , most studies on these three representations use coordinates or coordinates- like feature as input to feed into the backbone network . For point clouds , the direct input to the network is the point coordinates . For meshes , the node feature of the graph is the vertices coordinates . For volumetric data , the 3D shape is denoted by whether a voxel in a particular position is occupied or not . Using coordinates to characterize a shape is simple and straightforward . However , the major problem with this is that coordinate lacks contextual geometric information . Hence the capacity of the backbone network could be restricted . Even though various operators and backbones are proposed to extract high-level feature from the combination of coordinates by aggregating the local geometry , the effect of the input feature is not clear yet . The practice in Natural Language Processing ( NLP ) and Data Mining ( DM ) might shed some light on this issue . Word embeddings ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) is proposed very early in NLP to map words into an embedding space using embedding layers , which work as lookup tables that are indexed by the one-hot encoding of words . The word embeddings retrieved from embedding layers have similar values for words with similar meanings . This technique is widely adopted across the NLP area and notably boosts the overall performance , regardless of how the text data is distributed and which language model is used ( Brown et al. , 2020 ; Devlin et al. , 2019 ; Radford et al. , 2019 ; Vaswani et al. , 2017 ) . Studies in the field of data mining ( DM ) also learn continuous embedding representation for graph nodes ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ) . Embedding learning in NLP and DM indicates that the effect of input feature and the capacity of the backbone network are somehow orthogonal to each other . It raises the question that whether there is a better way to denote a shape instead of vanilla coordinates , so as to facilitate the backbone network to execute downstream tasks , regardless of which backbone model is used . In this work , we introduce Representation-Agnostic Shape Fields ( RASF ) , a shape embedding layer that maps coordinates to shape embeddings with rich geometry information . RASF is implemented using a learnable multi-channel 3D grid . Similar to the lookup table in word embedding layer , coordinates within a local shape index from this 3D grid and retrieve shape embeddings . With simple operation on data , we make RASF compatible with major 3D representations , including point clouds , meshes and voxels . To obtain the weights of RASF , we investigate several pre-training schemes for RASF , among which we find that self-supervised training schemes , i.e. , reconstruction and normal estimation , yield the best performance and generalizability . Once trained , RASF could be seamlessly plugged into any 3D deep learning pipeline to improve performance across diverse downstream tasks and datasets , with little code modification and computation cost ( See Fig . 1 ) . We empirically show that RASF consistently brings significant improvement under diverse backbones and applications , including object classification , part segmentation and scene segmentation . Contributions In this work , we introduce a generalizable ( i.e. , it could be used in different 3D representations , backbones and downstream tasks ) and computation-efficient shape embedding layer for 3D deep learning , named Representation-Agnostic Shape Fields ( RASF ) . It applies a learnable multi-channel 3D grid to store local geometry . Shape embeddings for various 3D shape representations ( point clouds , meshes and voxels ) are retrieved by coordinates indexing . While there are multiple ways to obtain the coefficients of RASF , we provide two effective schemes among all in this paper for RASF pre-training , that is shape reconstruction and normal estimation . Abundant experiments across different representations , backbones and downstream tasks are conducted to validate the generalization and efficiency of our proposed RASF . Code and pre-trained weights will be made publicly available soon . 2 RELATED WORK . 2.1 3D SHAPE ANALYSIS . Point clouds Point cloud data could be directly obtained from 3D LiDAR sensors . Due to its compactness that represents only the surface of the objects and its aligned format ( an N × 3 matrix ) that suits the common deep learning frameworks , point clouds are the most extensively discussed representation in the field of 3D shape analysis . PointNet ( Qi et al. , 2017a ) is the pioneering deep learning network to process point clouds . It learns the feature of each point with a shared MLP and aggregates all points by global max-pooling . PointNet++ ( Qi et al. , 2017b ) supports hierarchical points aggregation to extract geometric information at different scales . DGCNN ( Wang et al. , 2019 ) builds a dynamic graph in each layer of the network to incorporate neighbor nodes by its proposed EdgeConv module . PointCNN ( Li et al. , 2018b ) , RSCNN ( Liu et al. , 2019b ) , ShellNet ( Zhang et al. , 2019 ) , KPConv ( Thomas et al. , 2019 ) and PCT ( Guo et al. , 2021 ) extend 3D convolution operation to the irregular point clouds data in different ways . Meshes Meshes are mostly used in computer graphics ( Kato et al. , 2018 ; Liu et al. , 2019a ; Pfaff et al. , 2020 ; Smirnov & Solomon , 2021 ) . GWCNN ( Ezuz et al. , 2017 ) maps unstructured geometric data to a regular domain for nonrigid shape analysis . MeshCNN ( Hanocka et al. , 2019 ) adopts specialized convolution and pooling operations on mesh edges . The convolution is conducted on the edge and the four adjacent edges around the incident triangles , while the pooling operation generates new geometry via adaptive edge collapse . Besides , MeshCNN designs several hand-crafted features that characterize the edge geometry , that is the dihedral angle , two inner angles and two edgelength ratios for each face . The 5-dimensional vector is fed into the MeshCNN network as input . Alternatively , MeshNet ( Feng et al. , 2019 ) regards faces as units . It introduces face-unit and featuresplitting to learn on meshes directly . A more recent work , HodgeNet ( Smirnov & Solomon , 2021 ) , operates on feature of vertices and edges simultaneously . There are also works in the field of single image reconstruction , which consider mesh data as graphs ( Wang et al. , 2018 ; Gkioxari et al. , 2019 ; Pan et al. , 2019 ; Wen et al. , 2019 ) . In this case , graph convolutions on nodes ( vertexes ) are adopted to aggregate local geometry . Voxels Volumetric data provides regular grids to represent 3D shapes . They could be processed by methods analogous to 2D grids . 3DShapeNets ( Wu et al. , 2015 ) proposed to represent 3D shapes with volumetric grids and introduced 3D Convolution Neural Networks ( 3DCNN ) for voxel classification . VoxNet ( Maturana & Scherer , 2015 ) utilized 3DCNN for robust 3D object recognition . The Voxception-ResNet ( VRN ) ( Brock et al. , 2016 ) introduced popular 2D network blocks into volumetric networks . The drawback of volumetric representations lies in that 3DCNN is computationally expensive . In recent years , LP-3DCNN ( Kumawat & Raman , 2019 ) is proposed to alleviate the computation issue . It applies 3D Short Term Fourier Transform ( STFT ) to replace the 3D convolution layers . 2.2 SHAPE DESCRIPTORS AND EMBEDDING LEARNING . Surface feature descriptors or shape descriptors for non-rigid objects ( generally in mesh format ) are another line of work in 3D shape analysis , often applied in shape correspondence , retrieval and segmentation . Shape descriptors aim at representing the local geometry around a point in a non-rigid shape . Classic surface descriptors describe a local shape patch based on diffusion and spectral geometry to achieve isometry-invariance , for example , heat kernel signature ( HKS ) ( Sun et al. , 2009 ) , wave kernel signature ( WKS ) ( Aubry et al. , 2011 ) and optimal spectral descriptors ( OSD ) ( Litman & Bronstein , 2013 ) . Masci et al ( Masci et al. , 2015 ) proposed to automatically learn various shape descriptors for Riemannian manifolds using a geodesic convolutional neural network . Studies in NLP embedding learning maps the one-hot encoding of words to a real-value vector via a learnable embedding layer . An embedding layer is a lookup table , using words as indices and retrieve word embeddings from it . The embedding layer could be trained in an unsupervised manner for general proposes , or in a supervised manner for a specific task , e.g. , document classification . 3 METHODOLOGY . 3.1 REPRESENTATION-AGNOSTIC SHAPE FIELDS ( RASF ) . We propose a shape embedding layer named Representation-Agnostic Shape Fields ( RASF ) , to facilitate 3D shape analysis across various representations . As shown in Fig . 2 ( a ) , RASF is implemented as a trainable 3D grid with the shape of R × R × R × C , where R is the resolution of the grids and C is the channel dimension . Common 3D shape representations are based on point coordinates . Suppose the shape is denoted as a point set X ∈ RN×3 . For each point P in the shape , we extract the K-Nearest-Neighbor points Pneigh from the whole point set . Regarding P as the central point , we normalize the coordinates of the local point set A = [ P , Pneigh ] to the range of the grid . In other words , P would be placed at the center of the grid , while the farthest neighbor to P would be placed at the border of the grid . Similar to the word embedding layer that is indexed by one-hot encoding of words , the normalized local point set serves as indexes to retrieve shape features from the grid . The difference is that one-hot encoding is discrete while coordinates are continuous . Therefore the indexing is accomplished via trilinear interpolation . This operation inspired by Spatial Transformer Networks ( STNs ) ( Jaderberg et al. , 2015 ) could be efficiently implemented by modern deep learning frameworks ( e.g. , grid sample in PyTorch ( Paszke et al. , 2017 ) ) . This function enables continuous indexing in batches to retrieve feature from discrete regular grid data . The trilinear interpolation returns a ( K + 1 ) × C matrix for P , which is then reduced to 1 × C by max-pooling on the first dimension . The C−dim vector , named shape embedding ( SE ) , encodes the shape in the local area A around the central point P . The whole process to obtain shape embedding SE ( P ) of point P could be formulated as follows : SE ( P ) = max ( GS ( N ( [ P , Pneigh ] ) ) ) , ( 1 ) where GS denotes the grid sample function and N denotes the normalization . Note that the central point always corresponds to the central feature of RASF due to the normalization . Since RASF is a cubic grid , we use L1-distance in KNN searching for parameters efficiency . After processing each point in this way , we obtain shape embeddings SE ∈ RN×C for all the points . The computation in RASF is negligible compared to the backbone networks , as analyzed in Sec . 5 .

Summary: This submissions introduces RASF (Representation Agnostic Shape Fields), an embedding layer that encodes local geometry and can be used for 3D deep learning with different input domains: point clouds, meshes or voxels. RASF is inspired by the idea of embedding layers in language models, where representations of tokens are indexed by one-hot vectors.   In this submissions the authors propose to implement RASF as volumetric latent embedding that is indexed via tri-linear sampling. For example, given an input pointcloud the process first extracts a local neighborhood around a point p and normalizes it, so that the point p becomes the origin. This pointcloud is then used to sample the volumetric latent embedding and afterwards a max-pooling operation is applied. The max-pooled feature becomes the representation of point p.  The experimental results show improvements when RASF is used on top of several baselines while incurring on a small additional computational complexity.