Representation-Agnostic Shape Fields
Authors: Xiaoyang Huang
Jiancheng Yang
Yanjun Wang
Ziyu Chen
Linguo Li
Teng Li
Bingbing Ni
Wenjun Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: shape embedding
3D deep learning
shape classification and segmentation

Abstract: 3D shape analysis has been widely explored in the era of deep learning. Numerous models have been developed for various 3D data representation formats, e.g., MeshCNN for meshes, PointNet for point clouds and VoxNet for voxels. In this study, we present Representation-Agnostic Shape Fields (RASF), a generalizable and computation-efficient shape embedding module for 3D deep learning. RASF is implemented with a learnable 3D grid with multiple channels to store local geometry. Based on RASF, shape embeddings for various 3D shape representations (point clouds, meshes and voxels) are retrieved by coordinate indexing. While there are multiple ways to optimize the learnable parameters of RASF, we provide two effective schemes among all in this paper for RASF pre-training: shape reconstruction and normal estimation. Once trained, RASF becomes a plug-and-play performance booster with negligible cost. Extensive experiments on diverse 3D representation formats, networks and applications, validate the universal effectiveness of the proposed RASF. Code and pre-trained models are publicly available\footnote{\url{https://github.com/seanywang0408/RASF}}.

Published as a conference paper at ICLR 2022
REPRESENTAATION-AGOSSIII SHAPE FIELDS
Xiaoyang Huang', Jiancheng Yang', Yanjun Wang', Ziyu Chen', Linguo Li', Teng Li,, Bingbing Ni'; Wenjun Zhang'
Scchool of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University 2 AAnhui University (huangxiaoyang, nibingbing 8s jtu. edu.edu
ABSTRACT
3D shape analysis has been widely explored in the era of deep learning. Numerous models have been developed for various 3D data representation formats, e.g., MeshCNN for meshes, PointNet for point clouds and VoxNet for voxels. In this study, we present Representaaion-Agnoosi generalizable and computationn-effccent shape embedding module for 3D deep learning. RASF is implemented with a learnable 3D grid with multiple channels store local geometry. Based RASF, shape embeddings for various 3D shape representations (point clouds, meshes and voxels) are retrieved by coordinate indexing. While there are multiple ways to optimize the learnable parameters of RASF, we provide two effective schemes among all in this paper for RASF pre-training: shape reconstruction and normal estimation. Once trained, RASF becomes plug-and-play performance booster with negligible cost. Extensive experiments on diverse 3D representation formats, networks and applications, validate the universal effectiveness of the proposed RASF. Code and pre-trained models are publicly available
Shape Fields (RASF),
INTRODUCTION
3D shape analysis is the foundation to understand the physical world. It has wide range of applications in real life including robotics (Liu al. 2020 Liang et al., 2018), autopilot (Shi al.
2020 [Sng et al. 2019 Qi et al. 2018) Zhou & Tuzel (2118), medical imaging (Yang et al. 202 lb [.i al.) 201 8a Yang et al. 2021a) and movie animation (Xu et al., 2019 Aberman et al. 2020
Hertz et al. 2020). In recent years, the research on this topic has prevailed and showed promising
results in various tasks, such as object classification, part segmentation, scene segmentation, etc.
Shape could be represented in different data formats, among which meshes, point clouds and voxels are most commonly used. In the
160
100
Sciteesst vr 3507, Heller eotets ei BNS?
O Naatecth
a100
deep learning era, most studies on these three
91
91
representations use coordinates or coordinates-
like feature as input to feed into the backbone network. For point clouds, the direct input
88
0
08
to the network is the point coordinates. For meshes, the node feature of the graph is the vertices coordinates.
06
For volumetric data, the 3D shape is denoted by whether a voxel in particular position is occupied or not. Using coordinates to characterize a shape is However, the
20
90
100
200
Labracy
Figure 1: Vertical Axis: Performance; Horizontal Axis: Latency. Diverse backbones with (solid symbols) and without RASF (hollow symbols) are evaluated. RASF could be seamlessly plugged into any 3D deep learning pipeline to improve performance across diverse downstream tasks and datasets, with little code modification and computation cost.
simple and straightforward. major problem with this is that coordinate lacks contextual geometric information. Hence the capacity of the backbone network could be restricted. Even though various operators and
"Corresponding Author https://gi thub. http://ttttttub..cosoee/eeeeaae/ httptttww...ooeeeeoaanaon///////

---

Published as a conference paper at ICLR 2022
backbones proposed to extract high-level feature from the combination of coordinates by aggregating the local geometry, the effect of the input feature is not clear yet.
The practice in Natural Language Processing (NLP) and Data Mining (DM) might shed some light on this issue. Word embeddings (Mikolov et al. 2013) Pennington et al. 2014) is proposed very early in NLP to map words into an embedding space using embedding layers, which work lookup tables that are indexed by the one-hot encoding of words. The word embeddings retrieved from embedding layers have similar values for words with similar meanings. This technique widely adopted across the NLP area and notably boosts the overall performance, regardless of how the text data distributed and which language model is used (Brown et al. 2020, Devlin et al., 2019] Radford et al. [219]] Vaswani et al.] (017). Studies in the field of data mining (DM) also learn continuous embedding representation for graph nodes (Grover & Leskovee, 2016 Perozzi et al.) 2014). Embedding learning in NLP and DM indicates that the effect of input feature and the capacity of the backbone network are somehow orthogonal each other. It raises the question that whether there is better way to denote shape instead of vanilla coordinates, so to facilitate the backbone network to execute downstream tasks, regardless of which backbone model is used.
In this work, we introduce Representation that maps coordinates to shape embeddings with rich geometry information. RASF is implemented using Icarnable multi-channel 3D grid. Similar to the lookup table in word embedding layer, coordinates within a local shape index from this 3D) grid and retrieve shape embeddings. With simple operation data, make RASF compatible with major 31) representations, including point clouds, meshes and voxels. To obtain the weights of RASF, we investigate several pre-training schemes for RASF, among which we find that self-supervised training schemes, i.e., reconstruction and normal estimation, yield the best performance and generalizability. Once trained, RASF could be seamlessly plugged into any 3D deep learning pipeline to improve performance across diverse downstream tasks and datasets, with little code modification and computation cost (See Fig. D. We empirically show that RASF consistently brings significant improvement under diverse backhones and applications, including object classification, part segmentation and scene segmentation.
Shape Fields (RASF), shape embedding layer
Contributions In this work, we introduce generalizable (i.e., it could be used in different 3D representations, backbones and downstream tasks) and computation-effiienn shape embedding layer for 3D deep learning, named Representation-AAnosii Shape Fields (RASF). It applies learnable multi-channel 3D grid to store local geometry, Shape embeddings for various 3D shape representations (point clouds, meshes and voxels) are retrieved by coordinates indexing. While there are multiple ways to obtain the coefficients of RASF, we provide two effective schemes among all in this paper for RASF pre-training, that is shape reconstruction and normal estimation. Abundant experiments across different representations, backbones and downstream tasks are conducted to validate the generalization and efficiency of our proposed RASE
2 RELATED WORK
2.1 3D SHAPE ANALYSIS
Point clouds Point cloud data could be directly obtained from 3D LiDAR sensors. Due to its compactness that represents only the surface of the objects and its aligned format (an N x 3 matrix) that suits the common deep learning frameworks, point clouds are the most extensively discussed representation in the field of 3D shape analysis. PointNet (Qi et al., 2017a) is the pioneering deep learning network to process point clouds. It learns the feature of each point with shared MLP and aggregates all points by global max- max-pooling. PointNet++ (Qi et al., 2017b) supports hierarchical points aggregation to extract geometric information at different scales. DGCNN (Wang et al.) 2019) builds a dynamic graph in each layer of the network to incorporate neighbor nodes by its proposed EdgeConv module. PointCNN (Li et al. 2018b), RSCNN (Liu et al. 2019c), DPAM (Liu et al., 2019a), ShellNet (Zhang et al. 2019) and KPConv (Thomas et al. 2019) extend 3D convolution operation the irregular point clouds data in different ways, while PAT (Yang al. 2019) and PCT (Guo et al. 202T) leverage transformer to process point clouds. Shape Sell-Correction (Chen et al.,
2021 proposed a self-supervised method to for point clouds analysis.
Meshes Meshes are mostly used in computer graphics (Kato et al. 2018 Liu et al. 2019b Pfaff
et al. 2020 Smirnov & Solomon 2021). GWCNN (Ezuz et al. 2017) maps unstructured geometric data to regular domain for nonrigid shape analysis. MeshCNN (Hanocka et al. 2019) adopts specialized convolution and pooling operations on mesh edges. The convolution is conducted on the
2

---

Published as a conference paper at ICLR 2022
edge and the four adjacent edges around the incident triangles, while the pooling operation generates new geometry via adaptive edge collapse. Besides, MeshCNN designs several hand-crafted features that characterize the edge geometry, that is the dihedral angle, two inner angles and two edge- length ratios for each face. The 5-dimensional vector is fed into the MeshCNN network as input. Afternatively, MeshNet (Feng al.) | (219)] regards faces as units. It introduces face-unit and feature- splitting to learn on meshes directly. A more recent work, HodgeNet (Smirnov & Solomon 2021), operates on feature of vertices and edges simultaneously. There are also works in the field single image reconstruction, which consider mesh data as graphs (Wang et al.) 2018 Gkioxari et al.. 2019 Pan et al., 2019 Wen et al. (2019). In this case, graph convolutions on nodes (vertexes) are adopted to aggregate local geometry.
Voxels Volumetric data provides regular grids to represent 3D shapes. They could be processed by methods analogous to 2D grids. 3DShapeNets (Wu et al. 2013) proposed to represent 3D shapes with volumetric grids and introduced 3D Convolution Neural Networks (3DCNN) for voxel classification. VoxNet (Maturana & Scherer 2015) utilized 3DCNN for robust 3D object recognition. The Voxception- ResNeet (VRN) (Brock et al., 2016) introduced popular 2D network blocks into volumetrie networks. The drawback of volumetric representations lies in that 3DCNN is computationally expensive. In recent years, LP-3DCNN (Kumawat & Raman 2019) is proposed to alleviate the computation issue. It applies 3D Short Term Fourier Transform (STFT) to replace the 3D convolution layers.
2.2 SHAPE DESCRIPTORS AND EMBEDDING LEARNING
Surface feature descriptors or shape descriptors for non-rigid objects (generally in mesh format) are another line of work in 3D shape analysis, often applied in shape correspondence, retrieval and segmentation. Shape descriptors aim at representing the local geometry around a point in a non-rigid shape. Classic surface descriptors describe local shape patch based on diffusion and spectral geometry to achieve
for example, heat kernel signature (HKS) (Sun jet al. 2009, wave kernel signature (WKS) (Aubry et al., 201T) and optimal spectral descriptors (OSD) (Litman & Bronstein 2013). Masci et al (Masci et al. (0155) proposed to automatically learn various shape descriptors for Riemannian manifolds using geodesic convolutional neural network.
Studies in NLP embedding learning maps the one-hot encoding of words to real-value vector via a learnable embedding layer. An embedding layer is a lookup table, using words as indices and retrieve word embeddings from it. The embedding layer could be trained in an unsupervised manner for general proposes, or in a supervised manner for a specific task, e.g., document classification.
3 METHODOLOGY
3.1 REPRESENTAAION-AGNOSTT SHAPE FIELDS (RASF)
We propose shape embedding layer named Representation-Agnonn facilitate 3D shape analysis across various representations. As shown in Fig. 2 (a), RASF is implemented trainable 3D grid with the shape of R x R x R x C, where R is the resolution of the grids and C is the channel dimension. Common 3D shape representations are based on point coordinates. Suppose the shape is denoted as point set X E R Nx3 For each point P in the shape, we extract the K-Nearest-Neighhor points Pneigh from the whole point set. Regarding P as the central point, we normalize the coordinates of the local point set A = [P, Paeiihh the range of the grid. In other words, P would be placed at the center of the grid, while the farthest neighbor to P would be placed at the border of the grid. Similar to the word embedding layer that indexed by one-hot encoding of words, the normalized local point set serves as indexes to retrieve shape features from the grid. The difference is that one-hot encoding is discrete while coordinates are continuous. Therefore the indexing is accomplished via trilinear interpolation. This operation inspired by Spatial Transformer Networks (STNs) (Jaderberg et al. 2015) could be efficiently implemented by modern deep learning frameworks (e.g., griplsample in PoTocc (Paszke et al. 2017)). This function enables continuous indexing in batches to retrieve feature from discrete regular grid data. The trilinear interpolation returns a K x C matrix for P (K includes the point P itself), which is then reduced to 1 x C by max max-poooing on the first dimension. The C' - dim vector, named shape embedding (SE), encodes the shape in the local area A around the central point P. The whole process to obtain shape
Shape Fields (RASF), to
3

---

Published as a conference paper at ICLR 2022
(a)
(b)
RASF Grid
loud
NX3
RASF Grid
N
Trilinear Interpolate
Normalize
Sample Surface
Mesh
K C
(V AM
RASF
(V RMXE
P.
Max-Pool
Voxel
1
Shape Embedding
NXNXNXI
RASF Grid NXNXNXC
Figure 2: (a) Inference of RASE. The process to obtain shape embedding of P via RASF grid. We extract the K-Nearest-Neighhen around P to the range of RASF grid. In other words, P would be placed at the center of the grid, while the farthest neighbor to P would be placed at the border of the grid. The normalized local point serves indexes fetch feature from the grid via trilinear interpolation. The K C interpolated feature is reduced to 1 X C' by max-pooling, named shape embedding. (b) RASF implementation for point clouds (top), meshes (middle) and voxels (bottom). C denotes the channel of RASF. Point Clouds: N denotes the number of points. Meshes: V, E denotes the vertexes and edges of meshes. We re-sample points on the faces of meshes and combine them with vertexes to fetch vertexes feature from RASE. Voxels: N denotes the size of voxels. We leverage the points with value of to fetch voxel feature from RASE.
P from the whole point set and normalize the local point set
embedding SE(P) of point P could be formulated as follows:
SE(P) max ( (GS (P,P,eeegg Preiih)))),
where GS denotes the sample function and denotes the normalization. Note that the central point always corresponds to the central feature of RASF due the normalization. Since RASF is cubic grid, we use Li-distance KNN searching for parameters efficiency. After processing each point in this way, obtain shape embeddings SE IR NxC for all the points. The computation in RASF is negligible compared to the backbone networks, as analyzed in Sec. 5
3.2 RASF FOR VARIOUS REPRESENTATIONS
RASF for Point Clouds. 3D shape using point clouds representation could be denoted by a N x 3 matrix, where N is the number of points. Therefore applying RASF on point clouds is natural. During the downstream tasks, the N x matrix fed into the backbone network together with coordinates auxiliary features.
RASF for Meshes. Suppose mesh (V,E, is denoted by three elements: vertexes V RNvx3 edges E and faces F. Since mesh-based backbone networks receive different elements as units, there are minor adjustments on how the shape embedding is obtained the RASF. Some backbone networks accept vertex features as input. Since the vertex positions in meshes are irregular (meaning that some vertexes are closer to others, while some are further), it hard to extract meaningful shapes given only the vertex positions. In this regard, we re-sample denser points P x3 on the faces of meshes and combine the re-sampled points along with the vertexes together, as shown in the middle of Fig. (b). Then we feed the combined point set [V, P] into RASE, to obtain shape embedding of the vertexes SE E R Ny xC Some backbone networks accept edge features input, e.g. MeshCNN (Hanocka al. 2019). For similar reasons, combine the midpoint of the edges with the re-sampled points to obtain the shape embedding the edges in similar way. For those backbone networks that receive face feature input (Feng et al.) 2019
4

---

Published as a conference paper at ICLR 2022
combine the barycenter of faces with the re-sampled points P to obtain the shape embedding of faces. Note that the shape embeddings obtained through the above approaches are entirely compatible with any mesh-based backbone networks, owing to the flexibility of RASF.
RASF for Voxels. An equivalent representation volumetric data is that points lie in positions where the voxel value is while no point exists when the voxel value is O. We leverage these "virtual" points to fetch voxel features from RASF during the downstream tasks, shown at the bottom of Fig. (b). Besides, we adjust the receptive field of RASF to a determinant distance instead of K-Nearest-Neighbors In this case, voxels that outside and far from the shape surface (no occupied voxel exists within the receptive field of RASF), would yield shape embedding of a zero vector. Voxels that are inside and far from the shape surface (all voxels around it are occupied), would yield a shape embedding of the same value. The N x N x N x volumetric data is transformed to an N N N tensor and fed into the backbone network.
3.3 LEARNING RASF IN PRETEXT TASKS
Reconstruction. We provide several schemes to pre-train RASE. The major scheme that we use in most of the experiments is reconstruction. Given the shape embeddings of each point SE E RR we randomly sample N, embeddings from N, and concatenate the embeddings with the coordinates. The coordinates are indispensable since the shape embeddings only encode the local geometry, without knowledge of the global geometry. The N, x (C + 3) tensor is fed into the reconstruction network, which consists of shared MLP, a max-pooling layer, and several linear layers. The last linear layer outputs (N 3) elements. We reshaped the output to N x 3, which is the predicted point set X prred- The loss function is the chamfer distance between Xpred and the ground-truth X.
Normal Estimation. This pretext task is to predict the normal of each point. We feed the shape embeddings and coordinates of each point SE E Nxx(C+3) into shared MLP, which outputs the predicted normals Npred E N x3 We use cosine similarity as the loss function.
The supervised task is shape classification, given only a small portion of points and their shape embeddings. The overall process is quite similar that of the reconstruction task, only that the output of the network the probability of each class, instead of coordinates.
Supervised.
Each of these pre- preitinining pushes RASF to encode the local geometry. We empirically find that self- supervised pre-training schemes (reconstruction and normal estimation) outperform the supervised one (classification). The self-supervised training enables RASF to be robust and transferable across different datasets and downstream tasks. For simplicity, we show the experimental results of reconstruction in Sec. 4 and then analyze different ways of pre-training in ablation study (Sec. [)) In practical implementations, we pre-train RASF using ShapeNetPart (Yi et al. (016) and fix its weights in downstream tasks. (All the downstream tasks use the same RASE) Empirical study shows that fine- tening the weights during the downstream tasks leads to unstable performance. The detailed pre-training settings and hyper-parameter analysis are presented in the appendix A.6
EXPERIMENTS
4.1 PERFORMANCE OF PRETEXT TASKS
We analyze the performance of pretext tasks by visualizing RASF weights in two ways (Fig. 3p and verify RASF using linear evaluation protocol.
The first visualization is to directly illustrate the channels in 3D and 2D. The 3D illustrations use visvis (Klein 2020) to show the three-dimensional grids, while the 2D illustrations show the max- pooling output on one axis of the grids. We figure that each channel focuses on a different area, representing a particular local geometry. The other visualization is to investigate RASF's response to geomerrically-varying shapes. Practically, we feed RASF with a set of semi-ellipsoids with different curvatures, as shown in Fig. 3 It is observed that some of the channels (channel 6, 11, 12, 20, 25, 29, 30) have strong correspondence to the ellipsoid curvature, i.e., the response of these channels changes gradually from large curvature to small curvature. On the other hand, some channels have the same response given different curvatures. We conjecture these channels could be related to other geometries. such as cones or cubes. (We randomly choose some channels for demonstrations. Visualizations of all channels and detailed settings could be found in appendix A.4
5

---

Published as a conference paper at ICLR 2022
Table Linear classification protocol. We test several kinds of linear classifier and report the ACC ModelNet40. shown that RASF feature is more discriminative than raw feature.
(3)
It is
Gamedad Elipodd
Classificr
Raw +RASF
Figure (a) RASF visualization in 3D and 2D. (b) We generate multiple Ellipsoids by deformation along Z-Axis and see their responses in RASF embeddings. Some channels have strong correspondence the ellipsoid curvature.
Max FC
25.57
32.17
PLY Point-Wise + Max FC 85,90
87.28
Flattened Feature & PC
9.52
39.10
Voxel
Flattened Feature FC
67.65
68.33
Moreover, we verify RASF by linear classification protocol on point clouds and voxels data. For point clouds, we experiment on three kinds of linear classifier: 1) max-pooling on the number of points followed by a fully-connected (FC) layer; 2) a point-wise (shared) FC layer followed by max- pooling and a FC layer; 3) flatten the N x C input to one dimension and use FC layer classifier. For voxels data, we simply flatten the N x N x N x C input to one dimension and use a FC layer classifier. The results are shown in Table [ It is observed that RASF feature outperforms raw feature in all settings, indicating RASF feature is more discriminative than raw feature. Especially for flattened points cloud, the out-of-order raw feature leads extreme performance degradation, while RASF feature still yields an accuracy of 39.10.
4.2 DATASETS IN DOWNSTREAM TASKS
We use five different datasets to evaluate the general effectiveness of RASF, which are various on tasks, characteristics, and shape representations. The summarized introduction is listed in Table All settings are identical for each backbone with and without RASF, including training hyper- parameters, train-test splits, and on.
Table 2: Datasets used in this study, with their train-test splits, data representation formats, descriptions and tasks, See Appendix A.5 for detailed demonstrations of each dataset.
Dalaset
Representation
Description
Task
ModelNer10 et al.
2015
3,991
908
Print Cloud & Voxel
3D Objects (Rigid)
Classification
ACC
ModeINer40
2015
9,843
2,468
Point Cloud Voxel
3D Objects (Rigid)
Classification
ACC
ShapeNeePart Tr
2016
12.137
2,874
Point Cloud
3D Objects (Rigid)
Part Sogmentation
mIOU
S3DIS Arment et
6-Fold
Cross
Point Cloud
Indoor
(Rigid)
Semantic Segmentation
SHRECTO Lian et al.
2011
300
300
Mesh
3D Objects (Rigid & Non Rigid)
Classification
ACC
SHRECI6
Lian et al:
2011
480
Mesh
Objects (Rigid Non-Rigid)
Classification
HUMAN
Maron al
2017
370
Mesh
Human Bodics (Non-Righ (ion-Rigid)
Part Sogmentation
mIOU
4.3 TRANSFER TO POINT CLOUDS
Settings. For point cloud representation, we conduct experiments on ModelNet40 for classification, ShapeNetPart for part segmentation and S3DIS for semantic scene segmentation. We compare the performance with and without the RASF input on various point cloud backbones. For classification, we experiment on PointNet, PointNet++, KPConv, DGCNN and PCT. For part segmentation, we experiment on PointNet, PointNet++, KPConv and DGCNN. For scene segmentation, we evaluate the performance of RASF DGCNN. All the results are obtained by running the public official code ourselves. For RASF, simply increase the input channel of all the backbone by C (the channel dimension of RASF). while other syper-parameters remain unchanged and default. Normals excluded for all the backbones. The number of sample points and the data augmentation techniques for each backbone are aligned with its original setting.
Results. As shown in Table 3 and Table RASF consistently improves the performance of all the backbone models on different tasks. For PointNet which lacks local operations, RASF notably improve 2.14% over baseline. For other backbones which contain various neighbor- based operators, RASF still brings consistent improvement, even though RASF is also based on local operations. It demonstrates that the effect of input feature and the capacity of the backbone network could

---

Published as a conference paper at ICLR 2022
Table 4: Results on point clouds data of 3D scenes (S3DIS). We compare performance on DGCNN with and without RASE
Table 3: Results on point clouds data of 3D objects. We compare performance on different point-clouds-based backbones and downstream tasks with and without RASF
Method
mIOU
Overall ACC
DGCNN
59.04
86.71
ModeINet40 ACC
ShapeNetPart mIOU
DGCNN +RASF
59.73
86.82
Method
Gain
+0.69
+0.11
PointNet
88.78
84.19
PointNet +RASF
90.92
84.85
Gain
+2.14
+0.66
Table 5: Results meshes data with MeshCNN.
PointNet++
91.55
84.86
SHREC10
SHREC16
HUMAN
PointNet++ +RASF
91.82
85.32
Method
ACC
ACC
mIOU
Gain
+0.27
+0.46
MeshCNN
96.00
99.17
91.35
KPConV
92.18
85.57
McshCNN +RASF
100.00
100.00
93.90
KPConv +RASF
92.80
86.24
Gain
+4.00
+0.83
+2,55
Gain
+0.62
+0.67
DGCNN
92.71
84.50
Table 6: Results voxels with VoxNet.
DGCNN +RASF
92.95
84.91
Method
ModelNet10
ModelNet40
Gain
+0.24
+0.41
PCT
92.93
VoxNet
91.48
82.78
PCT +RASF
93.18
VoxNet +RASF
92.46
83.07
Gain
+0.25
Gain
+0.98
+0.29
complement each other. In the scene segmentation task, where the point clouds contain richer and more complex local geometry, RASF notably improves the result over DGCNN. It demonstrates that RASF remains robust even when the downstream data distribution thoroughly distinct from the pre-training data. With the additional shape embedding input, networks have better understanding of the shape semantics than just using coordinates. RASF is able to generally boost the performance under various backbones and tasks with little memory and computation cost.
4.4 TRANSFER TO MESHES
Settings. For Mesh Representation, we use MeshCNN to evaluate the effectiveness of RASE. We conduct classification on SHREC dataset and segmentation on HUMAN dataset, following the settings MeshCNN. MeshCNN considers the edge as the unit of meshes. It manually calculates the edge feature based the two adjacent triangles. For classification,
train the model for 200 epochs using Adam (Kingma & Ba 2014) optimizer with an initial learning rate of 0.002 and linearly decrease the learning rate O from the 100/h epoch. For segmentation, the number of epochs is 600 while the initial learning rate is set to 0.002. For RASF, we extract shape embeddings of the edges following our proposed method for mesh concatenate it with its original feature, then increase the channel of the first layer of the model to feed them in the network.
Results. As shown in Table 5 RASF notably boosts the performance of MeshCNN over diverse datasets and tasks. Especially, MeshCNN with RASF reaches classification accuracy of 100% on SHRECI0 and SHREC16. Besides, we observed that the increment of the performance on meshes much more significant compared other representations. For one reason, the surface sampling of meshes yields a similar data distribution to point clouds, meaning little mismatch between pre- training and downstream training in terms of shape embedding. For another, sampling on the mesh surface actually introduces new geometry compared to using vertexes and edges only. Through RASF, the geometry is encoded in the shape embedding, resulting a higher performance boost. Moreover, RASF could be integrated mesh networks seamlessly without sacrificing efficiency in terms of training time and model size.
Noticed that MeshCNN accepts not only coordinates input but also the hand-crafted edge feature. In this case, RASF still able to increase the performance of the network. The comparison between the hand-crafted feature and RASF will be thoroughly discussed in Sec.

---

Published as a conference paper at ICLR 2022
4.5 TRANSFER TO VOXELS
Settings. We adopt VoxNet to evaluate RASF on voxels data, using ModelNet10 and ModelNet40 for classification task. The voxels data is in a shape of 32 x 32 32, We train the model for 100 epochs using Adam optimizer with an initial learning rate of 0.001 and multiply the learning rate by 0.5 for every 20 epochs. We obtain shape embedding of each voxel from RASF following the proposed RASF implementation for voxels. Then we feed the 32 x 32 X 32 x 32 shape embedding into VoxNet, by changing the input channel of the first layer. The best accuracy on test-set among all epochs is reported.
Results. RASF boosts the performance of VoxNet on both datasets (Table 6). Note that RASF is pre-trained on point clouds, in which the scale is different from voxels. What's more, point clouds only exist on the surface of the objects, while voxels are solid, which indicates they have entirely different distributions. Even in this case, RASF could still increase the performance by aggregating local geometry. It enriches the shape semantics of voxel representation by transferring the learned geometry to distinct data distributions, demonstrating the robustness RÁSF.
5 DISCUSSION
5.1 ABLATION STUDY
Module Design The adoption of learnable grid in the shape embedding layer is motivated from embedding layer in NLP. In this part we investigate how a learnable grid is superior to other module architecture choices, including a PointNet model (consisting of point-wise MLP, a Max-pooling layer and global MIP) and a simplified PointNet-like module (consisting of a single fully-connected layer and Max-pooling layer). Besides, we include EdgeConv module in DGCNN, which has been proved to be an effective neighborhood- based point clouds operator. We pre-train these three modules using the same reconstruction pre-text task described in Sec. 3 As the results shown in Table RASF has better performance in each downstream task. EdgeConv yields comparable performance on point clouds, yet it rapidly deteriorates on mesh data. We argue that RASF is a better choice with respect to generalizability.
Comparison of Pre-Training Schemes We compare the performance of the three pre-training schemes, including reconstruction, normal estimation and classification. We experiment on ModelNet40 for point clouds, SHREC10 for meshes and 1odelNet for voxels. We use PointNet, MeshCNN and VoxNet as the backbones for point clouds, meshes and voxels respectively. As shown in the middle bar of Table 8 all the pre-training schemes improve the performance over baseline. RASF obtained from self-supervised pre-training consistently outperforms the supervised one. We argue that self-superviss that are more related to the local geometry while less related to high-level semantics.
pre-training enables RASF to learn more general and robust embeddings
Fixed weights vs. Fine-Tuned vs. Random- Initialized. We evaluate the effect of the pre-trained RASF weights by comparing fixed, fine-tuned, and random-initialized RASF in the downstream tasks. Fixed RASF is the setting we use in all the experiments. Fine-tuned RASF refers to optimizing the pre-trained RASF together with the backbone network in the training of the downstream tasks, while random-initializen refers to a non-pre-trained RASF that is optimized within the downstream tasks. The detailed settings are the same as described in the previous paragraph. As shown in the bottom bar of Table iinitiiiitiillzzed RASFs are consistently worse than fixed RASFs by a large margin. It proves that RASF considered Besides, fine-tuned RASFS yield unstable results compared to fixed RASFs. This phenomenon demonstrates the pre-trained RASF generalizes well in downstream tasks.
feature layer would not work without pre-training.
5.2 COMPLEXITY ANALYSIS
We analyze the complexity of RASF, The most computationally-intess step in RASF occurs in K- X-Nearest-Neighhors algorithm, in which the time complexity is O(V2), while the trilinear step yields O88NK). The number of parameters in RASF is R x R x R x C. As RASF could be fixed in the downstream tasks, it introduces evaluate the actual running time of RASF and multiple backbones with a batch size of 1 (Table 9. Note that our current implementation of K-Nearest-Neighhors is naive version with PyTorch. The
additional memory cost apart from its parameters. We
8

---

Published as a conference paper at ICLR 2022
Table 7: Analysis of module design as shape
Table 9: Time complexity analysis of RASE. The
embedding layers.
actual running time of RASF is almost negligible compared to the backbones.
Point Cloud
Mesh SHREC10
Mesh HUMAN
Methods
ModeINet40
Task
Latency of Backbone
Backbone+RASF
Baseline
88.78
96.00
91.35
PointNet
2.47ms
2.75ms (+11.3%)
KPConw
2.54ms
2.71ms (+6.6%)
PointNet
87.84
Cls.
DGCNN
2.96ms
3.43ms (+15.8%)
FC Max
90.59
98.00
93.68
MeshCNN
86. 14ms
86.98ms (+0.9%)
EdgeConv
90.90
85.33
80.53
RASF
90.92
100.00
93.90
PointNet
2.54ms
2.80ms
+10.2%)
KPConv
12.8ms
13.5ms (+5.44%)
Sog.
DGCNN
10.9ms
11.8ms (4-8.2%)
Table 8: Analysis of RASF pre-training schemes (top) and the effeets of fixed / fine-tuned random- initialized weights (bottom).
MeshCNN
214ms
217ms (+1.44%)
Table 10: Comparison between RASF and hand-
Point Cloud
Mesh SHREC10
Voxel ModelNet10
crafted (HC) features. The baselines are PointNet for ModelNet40 (ACC). PointNet for ShapeNetPart (mIOU) and MeshCNN for SHREC10 (ACC).
Method
ModelNet40
Baseline
88.78
96.00
91.48
Reconstruction
90.92
100.00
92.46
Point Cloud
Point Cloud ShapeNetPart
Mesh
Normal Estim.
91.97
99.67
92.32
Method
ModelNet40
SHRECIO
Supervised
89.34
98.00
92.21
Baseline
88.78
84.19
64.67
Fixed
90.28
98.17
92.46
+HC
89.85
84.66
96.00
Fine-Tuned
89.64
99.33
91.36
+RASF
90.92
84.85
97.33
Random 88.47
95.67
92.09
+HC +RASF
91.33
85.00
100.00
actual time cost is expected to be lower by integrating more sophisticated KNN implementation, e.g., Faiss (Johnson et al. 2017). All the results are measured on an RTX 2080 Ti.
5.3 COMPARISON WITH HAND-CRAFTED FEATURES
In point clouds analysis, the point normals are widely used as auxiliary features to improve performance. In meshes analysis, MeshCNN proposed to use the geometric statistics around the edges as auxiliary features, that is the dihedral angle, two inner angles and two edge-length ratios for each face. In this work, learn local geometry by optimizing the RASF grid in pre-text tasks. We compare shape embeddings from RASF with these two hand-crafted features. For the point cloud baseline, we only feed the point coordinates mesh baseline, we remove the hand-crafted features on edges and replace them with the mid-points of the edges, representing the position of the edges. We experiment on SHRECI0 for meshes and ModelNet40 (classification), ShapeNetPart (segmentation) for point clouds.
input, using PointNet backbone. For the
Backbones with RASF consistently outperform baselines by large margins (Table 1Op. Hand-crafted features also improve the performance, but not as good as RASF, indicating that RASF provides richer geometric information than hand-crafted features. We reckon that RASF is pre-trained through reconstruction task which requires RASF to provide comprehensive local geometric information. We also noticed that combining RASF and the hand-crafted feature could further increase the performance. It demonstrates that the effect of RASF is somehow orthogonal to these hand-crafted features. RASF brings additional notable improvements over the existing methods.
CONCLUSION
We propose Representation-Annosiee Shape Fields (RASF), a generalizable and computation- efficient shape embedding layer for 3D deep learning. Shape embeddings for various 3D shape representations (point clouds, meshes and voxels) are retrieved by coordinates indexing. We provide two effective schemes for RASF pre-training, that is shape reconstruction and normal estimation, to enable RASF to learn robust and general shape embeddings. Once trained. RASF could be plugged into any 3D neural network with negligible cost. RASF widely boosts the performance for various 3D representations, neural backbones and applications.

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGEMENT
This work was supported by National Science Foundation of China (U20B2072, 61976137).
REFERENCES
Kfir Aberman, Peizh Uo Li, Dani Lischinski, Olga Baoquan Chen. Skeleton-aware networks for deep motion retargeting, ACM Transactions on Graphics (TOG), 39(4):62-1, 1, 2020.
Daniel Cohen-Or, and
Adobe. Adobe fuse 3d characters, 2016. https: /wwp::///wwwww// www.wiixamo... com
com
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In ACM SIGGRAPH 2005 Papers, pp- 408-416. 2005.
Iro Armeni, Ozan Sener, Amir Zamir, Helen Jiang, Ioamnis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1534-1543, 2016.
Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. The wave kernel signature: A quantum mechanical approach to shape analysis. In 2011 IEEE international conference computer vision workshops (ICCV workshops), pp- 1626-1636 IEEE, 2011.
Federica Bogo, Javier Romero, Matthew Loper, and Michael J Black. Faust: Dataset and evaluation for 3d mesh registration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7709-3801, 2014.
Andrew Brock, Theodore Lim, James Millar Ritchie, and Nicholas J Weston. Generative and discriminative voxel modeling with convolutional neural networks. Neural Inofrmation Processing Conference: 3D Deep Learning, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. In H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL [ttps::/pproeeedingg neeriis cc/paper/202C/
Ye Chen, Jinxian Liu, Bingbing Ni, Hang Wang, Jiancheng Yang. Ning Liu, Teng Li, and Qi Tian. Shape self-correction for unsupervised point cloud understanding. In IEEE International Conference on Computer Vision (ICCV), pp. 8382-8391, 2021.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171-4186, 2019.
Danielle Ezuz, Justin Solomon, Vladimir G Kim, and Mirela Ben-Chen. Gwenn: A metric alignment layer for deep shape analysis. In Computer Graphics Forum, volume 36, pp. 49-57. Wiley Online Library, 2017.
Yutong Feng, Yifan Feng, Haoxuan You, Xibin Zhao, and Yue Gao. Meshnet: Mesh neural network for 3d shape representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8279-8286, 2019.
Daniela Giorgi. Silvia Biasotti, and Laura Paraboschi. Shape retrieval contest 2007: Watertight models track. SHREC competition, 8(7), 2007.
10

---

Published as a conference paper at ICLR 2022
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cmn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9785- 9795, 2019.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855-864, 2016.
Meng-Haue Hao Guo, Jun-Xion Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7(2):187-199, 2021.
Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshenn: a network with an edge. ACM Transactions on Graphics (TOG), 38(4):1-12, 2019.
Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or. Deep geometric texture synthesis.
ACM Transactions on Graphics (TOG), 39(4): 2020.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. arXiv preprint 1rXiv:1500.02025, 2015.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint 1XXiv:1702.08::44 2017.
Evangelos Kalogerakis, Aaron Hertzmann, and Karan Singh. Learning 3d mesh segmentation and labeling. In ACM SIGGRAPH 2010 papers, pp. 1-12. 2010.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3907-3916, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2014.
Almar Klein. visvis. https: http:///ss/tttaooo..oo////a//aaa
com
2020.
Sudhakar Kumawat and Shanmuganathan Raman. Lp-3denn: Unveiling local phase in 3d convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4903-4912, 2019.
Xiaomeng Li, Hao Chen, Xiaojuan Qi, Qi Dou, Chi- Fu, and Pheng-Ann Heng. H-denseunet: hybrid densely connected unet for liver and tumor segmentation from et volumes. IEEE Iransactions on medical imaging, 37(12)2263326674 2018a
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcon: Convolution on X-transformed points. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 828-838, 2018b.
Lian, Godil, B Bustos, M Daoudi, Hermans, Kawamura, Kurita, G Lavoua, and P Dp Suetens. Shape retrieval on non-rigid 3d watertight meshes. In Eurographics workshop on 3d object retrieval (3DOR). Citeseer, 2011.
Ming Liang, Bin Yang. Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi- sensor 3d object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 641-656. 2018.
Roee Litman and Alexander M Bronstein. Learning spectral descriptors for deformable shape correspondence. IEEE transactions on pattern analysis and machine intelligence, 36(1):171-180 2013.
Jinxian Liu, Bingbing Ni, Caiyuan Li, Jiancheng Yang, and Qi Tian. Dynamic points agglomeration for hierarchical point sets learning- In IEEE International Conference on Computer Vision (ICCV), pp. 7546-7555, 2019a.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7708-7717, 2019b.
11

---

Published as a conference paper at ICLR 2022
Xingyu Liu, Rico Jonschkowski, Anelia Angelova, and Kurt Konolige. Keypose:
Multi-view 3d labeling and keypoint estimation for transparent objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11602-11610, 2020.
Yongcheng Liu, Bin Fan, Shiming Xiang. and Chunhong Pan. Relation-shapp convolutional neural network for point cloud analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 8895-8904, 2019c.
Haggai Maron, Meirav Galun, Noam Aigerman, Miri Trope Nadav Dym, Ersin Yumer, Vladimir G Kim, and Yaron Lipman. ACM Trans. Graph., 36(4):71-1, 2017.
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. Geodesic convolutional neural networks riemannian manifolds. In Proceedings of the IEEE international conference on computer vision workshops, pp. 37-45 2015.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922-922 -28. IEEE, 2015.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings the 2013 conference of the north american chapter of the association for computational linguisties: Human language technologies, pp. 746-751, 2013.
Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9964-9973, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Veio,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 701-710, 2014.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint urXiv:2010.03400 2020.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017 7a.
Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++ deep hierarchical feature learning on point sets metric space. Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 5105-5114, 2017b.
Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 918-929 927, 2018
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Hanyu Shi. Guosheng Lin, Hao Wang, Tzu- Yi Hung, and Zhenhua Wang. Spsequencenet: Semantic segmentation network on 4d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4574 4583, 2020.
Dmitriy Smirnov and Justin Solomon. HodgeNet: Learning spectral geometry on triangle meshes. SIGGRAPH, 2021.
12

---

Published as a conference paper at ICLR 2022
Xibin Song. Peng Wang, Dingfu Zhou, Rui Zhu, Chenye Guan, Yuchao Dai, Hao Su, Hongdong Li, and Ruigang Yang. Apollocar3d: A large 3d car instance understanding benchmark for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5452-54622 2019.
Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. A concise and provably informative multi-scale signature based on heat diffusion. In Computer graphics volume 28, pp. 1383-1392, Wiley Online Library, 2009.
Hugues Thomas, Charles R Qi, Baan-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6411-642) 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000-6010, 2017.
Daniel Vlasic, Ilya Baran, Wojeiech Matusik, and Jovan Popović, Articulated mesh animation from multi-view silhouettes. In ACM SIGGRAPH 2008 papers, pp. 1-9. 2008.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 52-67, 2018.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics (tog). 38(5): 2019.
Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu. Pixel2mesh++: Multi-view 3d mesh generation via deformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1042-1051, 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Zhan Xu, Yang Zhou, Evangelos Kalogerakis, and Karan Singh. Predicting animation skeletons for 3d articulated models via volumetric nets. In 2019 International Conference on 3D Vision (3DV), pp. 298-307. IEEE, 2019.
Jiancheng Yang, Qiang Zhang, Binghing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3323-3332, 2019.
Jiancheng Yang, Shixuan Gu, Donglai Wei, Hanspeter Pfister, and Bingbing Ni. Ribseg dataset and strong point cloud baselines for rib segmentation from et scans. In International Conference on Medical Image Computing and Computer-Asiisted Intervention (MICCAI), pp. 611-621. Springer, Cham, 2021a.
Jiancheng Yang, Xiaoyang Huang, Yi He, Jingwei Xu, Canqian Yang, Guozheng Xu, and Bingbing Ni. Reinventing 2d convolutions for 3d images. IEEE Journal of Biomedical and Health Informatics, 2021b.
Li Yi, Vladimir Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions on Graphics (ToG), 35(6):1-12, 2016.
Zhiyuan Zhang, Binh-Son Hua, and Sai-Kit Yeung. Shellnet: Efficient point cloud convolutional neural networks using concentric shells statistics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1607-1616, 2019.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4490- 4499, 2018.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 DETAILS OF PRE-TRAINING RASF
To demonstrate the general effectiveness of RASF, we use the same RASF weights in all the experiments below. We pre-train RASF based on large point clouds dataset, ShapeNetPart (Yi et al. 2016). It includes training samples of 12, 137 and testing samples of 2, 847. We randomly sample 048 points for each object the input shape to RASF. We set the resolution R of RASF to 16 and the channel dimension C to 32. During the pretraining in ShapeNetPart, we set the number of neighbors to for 2048 points per sample. K changes adaptively according to the total number of points in one sample, so to preserve relatively stable scale in RASF across different shapes. In the reconstruction pre-text task, we sample N, = 24 rows in the 2048 32 shape embeddings to feed into the reconstruction network. The choosing of the RASF hyper-parameters analyzed in Sec. A.6
The training lasts for 150 epochs, with an initial learning rate of 0.001 using Adam. We decay the learning rate by 0.2 for every 50 epochs. The chamfer distance on the test-set converges to 0.003 at the end. It is hard to tell the difference between the reconstructed shape and the ground-truth with human eyes.
A.2 DETAILS OF IMPLEMENTING RASF
When RASF is used in down-stream tasks, it is used only before the first layer in the backbones networks, making very simple to implement. Suppose RASF embeddings have C channels and the original backbone receives input of C, channels. We simply enlarge the channels of the first layer to C + C. to implement backbones with RASF, All other settings are identical between bakcbones with and without RASE.
A.3 VISUALIZATION OF RECONSTRUCTED SHAPE
We illustrate the reconstructed shapes during the reconstruction task for RASF pre-training in Fig. 4 The predicted shapes are obtained on test-set after convergence. As shown in the figure, the reconstructed shapes and the ground truths are hardly distinguishable with human eyes, demonstrating that RASF has learned the local geometry for favourable reconstruction performance.
C.K.K.))
52
91
00:00
90.56
50.70 9002
Ground Truth
90
D022
BLSB
BB22
89
88
87
Predicted
02 35,
02
10 12,
(1 12, 64 24.
(8. 3,3,1% 16 o0)
(16, 16, an
33, 32,24 24
33 32, m
16
64.33
32
64,24 A,
32,32.2
24)
24)
Figure 4: The predicted shapes and ground-truths during the reconstruction task.
Figure 5: Sensitivity analysis of RASF parpeepas
A.4 VISUALIZATION OF RASF
We design approaches visualize the pre-trained RASE. One is directly illustrating the weights. The other is feeding RASF with a set of semi-ellipsoids with different curvatures to explore how RASF would respond geometricalll-varyingg shapes.
Illustrations of RASF Weights. We visualize each channel of the pre- trained RASF. First, visualize the farce-dimensiona, grid using visvis, a python library (Klein 2020). Besides, we reduce the three-dimensional grid of each channel two-dimensional by conducting max-pooling the
14

---

Published as a conference paper at ICLR 2022
x-axis (the other two axes have similar phenomenon). The max- max-looling here corresponds to the max-pooling in RASF to obtain shape embeddings. As illustrated in Fig. 6 we find that each channel focus a different area, representing a particular local shape. Besides, some of the channels show a characteristic of symmetry, which might represent a symmetrical local shape.
Visualization with Deformed Ellipsoids. We explore how RASF would respond to geometrically-varyyngg shapes by designing proof-of-coneepl experiment. We generate a set of semi-ellipsoids in diverse curvatures by varying the radius lengths on the three axes. A portion of the generated semi-ellipsoids are illustrated in Fig. (left). A semi-ellipsoid could be parameterized by spherical coordinates. Suppose the semi-ellipsoid axes coincide with coordinate axes, we have:
asin())cos({))
y bsin())sin(())
(3)
2 = ccos(()
where a, b, e denote the radius lengths on x-axis, y-axis and z-axis, 0 < 5 T/2,0 To vary the radius length along the x-axis, we set c as and increases a from 0.1 of 0.1, so as y-axis and z-axis. In this way, we generate three groups of semi-ellipsoids, cach of which contains 20 semi-ellipsoids in different shapes for one axis. We feed these shapes into the pre-trained RASF and obtain their shape embeddings. We consider the peak of the semi-ellipsoids as the central point of RASF, which are marked red in Fig. 7 (left), while the others are considered as the local shape of the peak. At last, we obtain 20 shape embedding vectors for 20 shapes in each group. We illustrate the shape embeddings by arranging them in order in rows, as shown in Fig. (right).
2m.
step
is observed that some channels (channel 6, 11. 12, 20, 25, 29, 30) have strong correspondence to the ellipsoid curvature. The response of these channels changes gradually from large curvature shapes to small curvature shapes. On the other hand, some channels have the same response given different curvature shapes. We suppose these channels could be related to other geometries, such as cones or cubes.
A.5 DATASETS IN DOWNSTREAM TASKS
ModelNet. The ModelNet datasets are introduced by Wu et al. (Wu et al., 2015) for object classification. ModelNet40 includes 40 categories of 3D rigid objects. ModelNetTO is a subset of ModelNet40. Point clouds (Qi et al. 2017a) and volumetric representations (Wu et al., 2015) are available for this dataset.
ShapeNetPart. ShapeNetPart (Yi et al. 2016) is point cloud dataset for benchmarking 3D shape segmentation. It contains 16, 881 shapes from 16 categories. Each point in the point cloud is annotated with one of 50 labels. Most categories are labeled with two to five parts. The size of the train-set and test-set are 12, 137 and 2, 847 respectively.
S3DIS. The Stanford Large-Scale 3D Indoor Spaces (S3DIS) (Armeni et al. 2016) is for a semantic indoor scene segmentation task, containing 6 indoor areas with 271 rooms. Each point annotated with one of the 13 categories, e.g., board, chair, ceiling, etc. plus clutter. We conduct cross-validation on the 6 areas, the same protocol as prior works (Armeni et al. 2016 Qi et al., 2017a Wang et al.) 2019).
SHREC. SHREC (Lian et al. 2011) is 30-classes dataset for mesh classification, with 20 examples for each class. The categories include rigid objects such as lamps, and also non-rigid objects such as aliens. We follow the protocol (Ezuz al. (017), which generates two kinds of split for training and testing. The first is to randomly sample 10 examples from 20 per class for training to form SHRECIO, yielding a 1: train-test-split. The other is to randomly sample 16 from 20 for training, named SHREC16. Since (Ezuz et al. 2017) 7 did not release their train test split, we conduct the random split ourselves.
HUMAN. The HUMAN dataset is proposed by Maron al. (Maron et al. 2017) for mesh segmentation. The train-set includes 370 models from SCAPE (Anguelov et al. 2005 FAUST (Bogo et al., 2014), MIT (Vlasic et al., (2088) and Adobe Fuse (Adobe 2016). while the test-set includes 18 models from SHREC07 (Giorgi et al., 2007) (humans) dataset. The 388 models in total
manually annotated with 8 labels based (Kalogerakis et al.
2010).
15

---

Published as a conference paper at ICLR 2022
A.6 HYPER-PARAMETEE PARAMETERS.
We analyze the sensitivity of the hyper-parameters in RASE. There are four hyper-paramnetrss in RASF, that is the resolution R. the channel C, the number of neighbor points K obtain shape embedding, and the number of shape embeddings N, to feed into the reconstruction network. We change one parameter a time, using the set of hyper-parameters to pre-train the RASF and adopt in the downstream task. At last, we decrease the R, C' and the neighbor points, while increasing N, the same time, given that the reconstruction network needs more input shape embeddings when RASF models smaller shapes. We adopt PointNet (Qi et al. 2017a) on ModelNet40 (Wu et al.) 2013) for demonstration. The results are shown in Fig. The results show that RASF grid needs proper size to achieve the best performance. However, increasing the size (resolution and channel) brings more harm than decreasing it. We argue that RASF needs a proper resolution and number of channels to achieve optimal performance.
16

---

Published as a conference paper at ICLR 2022
Channel
Channel
Channel
Channal 3
Channel 4
Channel 5
Channel 6
Channel 7
Channel 8
Channel 9
Channel 10
Channel 11
Channel 12
Channel 13
Channel 14
Channel 15
Channel 16
Channel 17
Channel 18
Channel 19
Channel 20
Channel 21
Channel 22
Channel 23
Charnel 24
Channel 25
Chernel 26
Charnel 27
Charnel
Channel 29
Channal 30
Charnel
Figure 6: Visualization of RASF weights. For each channel, show the 3D visualization and the 2D figure after max- max-pooling mux-pooling on the x-axis. It is observed that each channel focus on a different area, representing partieular local shape.
17

---

Published as a conference paper at ICLR 2022
(a)
(b)
2
Charnele Stope Erahedting
(c)
Charnele Erabrddinge
Figure left: The input semi-ellipsoids with different curvatures. right: The visualization of shape embeddings given three groups of semi-ellipsoids input. (a)(b)(c) are the three groups of semi-ellipsoids (deformed along three axes) with their corresponding shape embeddings. The shape embeddings are arranged in order row by row. Each row represents one shape embedding. It is observed that the response of several channels changes gradually from large curvature shapes to small curvature shapes.
18


Reviews:
[{"review_id": "waWqYodR9_1", "reviewer": "Reviewer_Y3fs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper takes inspiration from word embeddings in NLP and applies the idea in a novel way to 3D deep learning. It is a simple idea, and it is exciting that it appears to be effective across various tasks and modalities. I think moving towards \"universal\" shape representations is an interesting direction for 3D deep learning, and this paper makes a nice step in that vein.\n\nIt is shown in the appendix that increasing the resolution and number channels in the embedding grid degrades quality, which is surprising. The authors claim that this is due to overfitting, but is this quantitatively validated? It would be nice to see metrics on the training set if this is indeed the case.\n\nThe authors should explicitly confirm that the learning set-ups are identical for each backbone with and without RASF---i.e., that the test/train splits are the same and architectures only differ in the embedding layer. It also might be helpful to include some error bars across different random seeds.\n\nHave you tried using your approach on some more recent state-of-the-art 3D learning methods? The ones compared against are fairly representative but a bit outdated, so I wonder if latest advances might make this method obsolete.\n\nWhile the paper is generally clear, there are typos and minor issues in exposition, some of which are listed below. In particular, I would encourage the authors to make some adjustments to some of the figures so that they are easier to parse just from the illustrations and caption contents.\n\n\nFigure 1 is difficult to parse. Some description of the actual contents of the plot in the caption would make it more clear.\n\nThe introduction has very many citations that are not particularly central to the paper and are mostly cited again in the related works section. It might be worth removing some of them from the introduction, because currently it makes the text appear somewhat cluttered and difficult to read.\n\nPage 1: \"In recent computer vision and deep learning community...\" sentence needs some restructuring\n\nPage 1: \"coordinate lacks geometric information\" not entirely clear what this means at this point in the paper\n\nPage 2: \"indices by\" -> \"indexed by\"\n\nDGCNN is mentioned several times but with the wrong citation---the actual paper is [Wang et al. 2018].\n\n\"HodgeNet: Learning Spectral Geometry on Triangle Meshes\" [Smirnov and Solomon 2021] should be mentioned in the related work on learning on meshes.\n\nThe description of the point extraction and normalization in 3.1 is a bit hard to follow.\n\nPage 5: \"by visualize\" -> \"by visualizing\"\n\nFigure 3 is very small, and it is not immediately clear what it's trying to show. It would help to add some more labeling.\n\nFigure 3 caption: \"geometrically-various\" -> \"geometrically-varying\"", "summary_of_the_paper": "This paper proposes a method for learning a latent spatial embedding of points in space to a feature space by pre-training on a pretext task (e.g., reconstruction). This embedding can the be used as part of any deep learning pipeline that inputs point coordinates as part of its architecture by first mapping the coordinates to their embedding. The authors demonstrate that utilizing these embeddings improves results of state-of-the-art learning methods on point clouds, meshes, and voxel grids.", "summary_of_the_review": "I think this paper proposes a simple yet effective framework for pertaining embedding for 3D deep learning tasks. Despite some minor questions and issues in exposition, I think it would be a nice addition to ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "hFcLCvET5Eh", "reviewer": "Reviewer_ESBS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I found the idea behind RASF compelling. The concept is simple and yet it does boost accuracy while keeping the added computational cost to a reasonable degree. \n\nStrengths:\n- Clear idea and exposition. I like that the authors didn't try to over-complicate the exposition of the idea.\n\n- Good evaluation on multiple geometry representations for different problems (reconstruction, normal estimation, segmentation, classification). It seems that adding RASF improves the accuracy for all the methods.\n\n\nWeaknesses:\n- Implementation details are unclear sometimes. Are RASF embedding layers used for all layers in pointnet for classification? I think the reader would benefit from an appendix where implementation details are clear.\n\n- The baselines used to add RASF on top might be outdated. It would be good if the authors could provide at least a couple of comparisons where baselines are state-of-the-art. Specially, those architectures using transformers (eg. https://arxiv.org/abs/2012.09688), which can learn both local and global geometry. I would like to see these comparisons before recommending this paper to be accepted.\n\n\nFinally, I have a couple of curiosities that I would like the authors to provide some feedback if possible:\n\n- Have you though about using RASF at different scales? One could easily do this by having multiple RASF embeddings which are queried by neigborhoods at different scales?\n\n- What would be the expected performance of a transfer learning task? Where you train RASF on point clouds and then test it on meshes? This could be an interesting direction for future work,", "summary_of_the_paper": "This submissions introduces RASF (Representation Agnostic Shape Fields), an embedding layer that encodes local geometry and can be used for 3D deep learning with different input domains: point clouds, meshes or voxels. RASF is inspired by the idea of embedding layers in language models, where representations of tokens are indexed by one-hot vectors. \n\nIn this submissions the authors propose to implement RASF as volumetric latent embedding that is indexed via tri-linear sampling. For example, given an input pointcloud the process first extracts a local neighborhood around a point p and normalizes it, so that the point p becomes the origin. This pointcloud is then used to sample the volumetric latent embedding and afterwards a max-pooling operation is applied. The max-pooled feature becomes the representation of point p.\n\nThe experimental results show improvements when RASF is used on top of several baselines while incurring on a small additional computational complexity.", "summary_of_the_review": "This paper presents a conceptually simple approach that seems to boost performance for most baseline models for 3D deep learning. I like the exposition of the idea and the extensive experimental results. However, I'm concerned that the baselines chosen for comparison might be outdated. If the authors can provide results with updated baselines (specially those that use transformers as backbones) I would be happy to upgrade my current score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QHoVl8D6LpS", "reviewer": "Reviewer_uSYH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(1) This work has a good motivation of trying to unify various 3D representations: meshes, point clouds, and voxels.\n\n(2) The design is simple and brings certain improvements to most (basic) networks employed in the experiments, and the method can be used a plugin for various method\n\n(3) The paper is well written and quite clear\n\n\nWeaknesses:\n\n(1) The first concern is on the motivation of the design. Though the paper claims \"generalizable\", the method simply re-samples points for mesh and voxel inputs, so that we can obtain points for any kind of inputs and then use these points for local shape extraction and embedding.  So, it seems to me that it is too strong to claim that the method is generalizable.\n\n(2) From the results shown in Section 4, the method seems to be effective mainly for earlier networks such as PointNet (2018) and MeshCNN (2019), which explore very local information in the input.\n\n(3) The idea of finding K nearest neighbors for feature extraction sounds very similar to EdgeConv in DGCNN, which improves over PointNet with features from local neighborhood of K nearest points. From Table 3, the improvement of the proposed method over DGCNN is quite marginal.\n\n(4) I am also concerned about the effectiveness of the design.  As shown in Table 8, the randomly-initialized RASF achieves comparable performance with the pre-trained one. If the backbone is changed from PointNet to DGCNN, it may be hard to see such improvement.", "summary_of_the_paper": "This paper presents a method for 3D shape analysis by means of local shape embedding.  The key idea is to use a learnable multi-channel 3D grid to embed local shapes in the input 3D object, similar to word embedding in NLP; hence, the method can be pre-trained and applied for various downstream tasks.\n\nThe procedure is:\n(1) If the input is mesh-based or voxel-based, first generate sample points on mesh or in voxels; if input is point-based, no need to have this pre-processing;\n(2) For each point p, find a set of K nearest neighbors points, i.e., P_neigh; and\n(3) normalize { p, P_neigh } and take the result as an index to retrieve shape features from the 3D grid, etc.\n\nThe paper claims that the proposed method is generalizable (i.e., could be used in different 3D representations, backbones and downstream tasks) and computation-efficient shape embedding layer for 3D deep learning.", "summary_of_the_review": "Overall, I am lukewarm for this paper and slightly more on the negative side.\n\n\n--------------------------------------\n\n\nOther issues:\n\nI am not sure how sensitive the method is to K and the density of the point samples in the inputs.\n\n\nP.3:\n\ndescriptors( -> descriptors (\n\n(Sun et al., 2009) , -> (Sun et al., 2009),\n\n\nP.6:\n\na accuracy -> an accuracy\n\nShapenetPart -> ShapeNetPart\n\nrepresentation, We -> representation, we \n\nclassfication -> classification\n\n\nP.7:\n\nAdam(Kingma -> Adam (Kingma", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6pzEIxdvk6", "reviewer": "Reviewer_quxB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength \n- The proposed shape embedding is general and can be plug-and-play in various 3D representations. \n- The proposed method has been extensively evaluated with different representations in a range of downstream applications, including classification, part segmentation, and semantic segmentation. All the experiments have witnessed a boost in performance when incorporating the proposed shape embedding, which indicates the effectiveness of the proposed approach.\n- The paper is well written and easy to follow.\n\n### Weakness\n- Though the performance can be boosted using the proposed method, the increase of the quantitative measurement seems to be a bit marginal. ", "summary_of_the_paper": "This paper presents a generalizable shape embedding layer for 3D deep learning. The proposed shape embedding can be used for a number of 3D shape representations including point cloud, mesh, and voxel. The shape embeddings can be obtained via self-supervised learning, where the paper has experimented with shape reconstruction and normal estimation as the pre-training tasks. The pre-training would enable the proposed method to learn general shape embeddings. At deploy time, the 3D priors are retrieved using coordinate indexing. The experimental results have shown that the proposed method can provide a boost on various representations in different applications.", "summary_of_the_review": "The idea of introducing an embedding layer to the 3D shape analysis just as the NLP community is interesting and novel. \nThe proposed framework is general to many mainstream 3D representations and does not require many additional changes to existing 3D learning frameworks in order to be deployed. The experimental results are positive and have shown the effectiveness of the proposed method in a number of applications. Though the performance boost has been verified, the magnitude of the increase seems to be a bit marginal. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an "embedding layer" in which points on a model are mapped into a feature space, trained using a reconstruction-based pretext task.  Then, the resulting embedding layer can be applied to shape data before using different learning architectures for modalities like meshes and point clouds.  The work is particularly interesting in its attempt to derive a learned shape representation that is agnostic to modality.  Some questions remained about experiments (e.g. baselines), but these are relatively minor and partially addressed in the rebuttal phase; also, sometimes the improvement seems to be marginal in practice.  

Two reviewers championed this work during the discussion phase.  The AC tends to agree this work is an interesting direction for future work and contains insight that the vision/learning communities might be able to use in other settings.
Decision: Accept (Poster)