Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for learning a latent spatial embedding of points in space to a feature space by pre-training on a pretext task (e.g., reconstruction). This embedding can the be used as part of any deep learning pipeline that inputs point coordinates as part of its architecture by first mapping the coordinates to their embedding. The authors demonstrate that utilizing these embeddings improves results of state-of-the-art learning methods on point clouds, meshes, and voxel grids.
main_review: This paper takes inspiration from word embeddings in NLP and applies the idea in a novel way to 3D deep learning. It is a simple idea, and it is exciting that it appears to be effective across various tasks and modalities. I think moving towards "universal" shape representations is an interesting direction for 3D deep learning, and this paper makes a nice step in that vein.

It is shown in the appendix that increasing the resolution and number channels in the embedding grid degrades quality, which is surprising. The authors claim that this is due to overfitting, but is this quantitatively validated? It would be nice to see metrics on the training set if this is indeed the case.

The authors should explicitly confirm that the learning set-ups are identical for each backbone with and without RASF---i.e., that the test/train splits are the same and architectures only differ in the embedding layer. It also might be helpful to include some error bars across different random seeds.

Have you tried using your approach on some more recent state-of-the-art 3D learning methods? The ones compared against are fairly representative but a bit outdated, so I wonder if latest advances might make this method obsolete.

While the paper is generally clear, there are typos and minor issues in exposition, some of which are listed below. In particular, I would encourage the authors to make some adjustments to some of the figures so that they are easier to parse just from the illustrations and caption contents.


Figure 1 is difficult to parse. Some description of the actual contents of the plot in the caption would make it more clear.

The introduction has very many citations that are not particularly central to the paper and are mostly cited again in the related works section. It might be worth removing some of them from the introduction, because currently it makes the text appear somewhat cluttered and difficult to read.

Page 1: "In recent computer vision and deep learning community..." sentence needs some restructuring

Page 1: "coordinate lacks geometric information" not entirely clear what this means at this point in the paper

Page 2: "indices by" -> "indexed by"

DGCNN is mentioned several times but with the wrong citation---the actual paper is [Wang et al. 2018].

"HodgeNet: Learning Spectral Geometry on Triangle Meshes" [Smirnov and Solomon 2021] should be mentioned in the related work on learning on meshes.

The description of the point extraction and normalization in 3.1 is a bit hard to follow.

Page 5: "by visualize" -> "by visualizing"

Figure 3 is very small, and it is not immediately clear what it's trying to show. It would help to add some more labeling.

Figure 3 caption: "geometrically-various" -> "geometrically-varying"
summary_of_the_review: I think this paper proposes a simple yet effective framework for pertaining embedding for 3D deep learning tasks. Despite some minor questions and issues in exposition, I think it would be a nice addition to ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This submissions introduces RASF (Representation Agnostic Shape Fields), an embedding layer that encodes local geometry and can be used for 3D deep learning with different input domains: point clouds, meshes or voxels. RASF is inspired by the idea of embedding layers in language models, where representations of tokens are indexed by one-hot vectors. 

In this submissions the authors propose to implement RASF as volumetric latent embedding that is indexed via tri-linear sampling. For example, given an input pointcloud the process first extracts a local neighborhood around a point p and normalizes it, so that the point p becomes the origin. This pointcloud is then used to sample the volumetric latent embedding and afterwards a max-pooling operation is applied. The max-pooled feature becomes the representation of point p.

The experimental results show improvements when RASF is used on top of several baselines while incurring on a small additional computational complexity.
main_review: Overall I found the idea behind RASF compelling. The concept is simple and yet it does boost accuracy while keeping the added computational cost to a reasonable degree. 

Strengths:
- Clear idea and exposition. I like that the authors didn't try to over-complicate the exposition of the idea.

- Good evaluation on multiple geometry representations for different problems (reconstruction, normal estimation, segmentation, classification). It seems that adding RASF improves the accuracy for all the methods.


Weaknesses:
- Implementation details are unclear sometimes. Are RASF embedding layers used for all layers in pointnet for classification? I think the reader would benefit from an appendix where implementation details are clear.

- The baselines used to add RASF on top might be outdated. It would be good if the authors could provide at least a couple of comparisons where baselines are state-of-the-art. Specially, those architectures using transformers (eg. https://arxiv.org/abs/2012.09688), which can learn both local and global geometry. I would like to see these comparisons before recommending this paper to be accepted.


Finally, I have a couple of curiosities that I would like the authors to provide some feedback if possible:

- Have you though about using RASF at different scales? One could easily do this by having multiple RASF embeddings which are queried by neigborhoods at different scales?

- What would be the expected performance of a transfer learning task? Where you train RASF on point clouds and then test it on meshes? This could be an interesting direction for future work,
summary_of_the_review: This paper presents a conceptually simple approach that seems to boost performance for most baseline models for 3D deep learning. I like the exposition of the idea and the extensive experimental results. However, I'm concerned that the baselines chosen for comparison might be outdated. If the authors can provide results with updated baselines (specially those that use transformers as backbones) I would be happy to upgrade my current score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a generalizable shape embedding layer for 3D deep learning. The proposed shape embedding can be used for a number of 3D shape representations including point cloud, mesh, and voxel. The shape embeddings can be obtained via self-supervised learning, where the paper has experimented with shape reconstruction and normal estimation as the pre-training tasks. The pre-training would enable the proposed method to learn general shape embeddings. At deploy time, the 3D priors are retrieved using coordinate indexing. The experimental results have shown that the proposed method can provide a boost on various representations in different applications.
main_review: ### Strength 
- The proposed shape embedding is general and can be plug-and-play in various 3D representations. 
- The proposed method has been extensively evaluated with different representations in a range of downstream applications, including classification, part segmentation, and semantic segmentation. All the experiments have witnessed a boost in performance when incorporating the proposed shape embedding, which indicates the effectiveness of the proposed approach.
- The paper is well written and easy to follow.

### Weakness
- Though the performance can be boosted using the proposed method, the increase of the quantitative measurement seems to be a bit marginal. 
summary_of_the_review: The idea of introducing an embedding layer to the 3D shape analysis just as the NLP community is interesting and novel. 
The proposed framework is general to many mainstream 3D representations and does not require many additional changes to existing 3D learning frameworks in order to be deployed. The experimental results are positive and have shown the effectiveness of the proposed method in a number of applications. Though the performance boost has been verified, the magnitude of the increase seems to be a bit marginal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method for 3D shape analysis by means of local shape embedding.  The key idea is to use a learnable multi-channel 3D grid to embed local shapes in the input 3D object, similar to word embedding in NLP; hence, the method can be pre-trained and applied for various downstream tasks.

The procedure is:
(1) If the input is mesh-based or voxel-based, first generate sample points on mesh or in voxels; if input is point-based, no need to have this pre-processing;
(2) For each point p, find a set of K nearest neighbors points, i.e., P_neigh; and
(3) normalize { p, P_neigh } and take the result as an index to retrieve shape features from the 3D grid, etc.

The paper claims that the proposed method is generalizable (i.e., could be used in different 3D representations, backbones and downstream tasks) and computation-efficient shape embedding layer for 3D deep learning.
main_review: Strengths:

(1) This work has a good motivation of trying to unify various 3D representations: meshes, point clouds, and voxels.

(2) The design is simple and brings certain improvements to most (basic) networks employed in the experiments, and the method can be used a plugin for various method

(3) The paper is well written and quite clear


Weaknesses:

(1) The first concern is on the motivation of the design. Though the paper claims "generalizable", the method simply re-samples points for mesh and voxel inputs, so that we can obtain points for any kind of inputs and then use these points for local shape extraction and embedding.  So, it seems to me that it is too strong to claim that the method is generalizable.

(2) From the results shown in Section 4, the method seems to be effective mainly for earlier networks such as PointNet (2018) and MeshCNN (2019), which explore very local information in the input.

(3) The idea of finding K nearest neighbors for feature extraction sounds very similar to EdgeConv in DGCNN, which improves over PointNet with features from local neighborhood of K nearest points. From Table 3, the improvement of the proposed method over DGCNN is quite marginal.

(4) I am also concerned about the effectiveness of the design.  As shown in Table 8, the randomly-initialized RASF achieves comparable performance with the pre-trained one. If the backbone is changed from PointNet to DGCNN, it may be hard to see such improvement.
summary_of_the_review: Overall, I am lukewarm for this paper and slightly more on the negative side.


--------------------------------------


Other issues:

I am not sure how sensitive the method is to K and the density of the point samples in the inputs.


P.3:

descriptors( -> descriptors (

(Sun et al., 2009) , -> (Sun et al., 2009),


P.6:

a accuracy -> an accuracy

ShapenetPart -> ShapeNetPart

representation, We -> representation, we 

classfication -> classification


P.7:

Adam(Kingma -> Adam (Kingma
review_title: null

Meta-review: This paper proposes an "embedding layer" in which points on a model are mapped into a feature space, trained using a reconstruction-based pretext task.  Then, the resulting embedding layer can be applied to shape data before using different learning architectures for modalities like meshes and point clouds.  The work is particularly interesting in its attempt to derive a learned shape representation that is agnostic to modality.  Some questions remained about experiments (e.g. baselines), but these are relatively minor and partially addressed in the rebuttal phase; also, sometimes the improvement seems to be marginal in practice.  

Two reviewers championed this work during the discussion phase.  The AC tends to agree this work is an interesting direction for future work and contains insight that the vision/learning communities might be able to use in other settings.
Decision: Accept (Poster)