Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes SE(3)-equivariant coordinate-based model for shape reconstruction. The model consists of a self-attention model to encoder the points in the point cloud and a cross-attention model to output the occupancy of any query point. Both models are made SE(3) equivariant. The local attention and the SE(3) equivariance allows compositional generalization. 

strength_and_weaknesses: Strengths: 

(1) Both the self-attention and cross attention models are well designed and prove to be effective. 

(2) The design of SE(3) equivariant attention is based on rigorous group representation consideration. 

(3) The experimental results on compositional generalization are impressive. 

Weakness: 

The difference and novelty relative to existing equivariant attention networks should be explained more carefully. 
clarity,_quality,_novelty_and_reproducibility: Both the theoretical formulation and experimental results are of high quality. 

The paper appears to be novel, although the novelty relative to existing equivariant attention networks is not entirely clear. 

The part on matrix groups and Peter-Weyl is a bit dense in terms of math notation. Some intuitive explanations can be helpful. 
summary_of_the_review: This paper makes a solid contribution to the problem of shape reconstruction from point cloud. The proposed self-attention and cross-attention modules are interesting and useful.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This manuscript describes a novel method to reconstruct continuous surfaces of scenes or objects from sparse point clouds. The method applies recent advances in SE(3)-equivariant transformer architectures to this problem. The transformer uses an encoder-decoder architecture. The encoder is an SO(3)-equivariant transformer which computes a per-point feature vector based on a local neighborhood of points (the encoder is translation invariant because only relative positions of neighbors are used). Different parts of the feature vector can be interpreted as being scalar-valued (as in a traditional network), vector-valued (as in Vector Neurons), or tensor-valued for arbitrarily high-order tensors. Because of the geometric interpretability of these representations, an SE(3) transformation of the input points induces a corresponding transformation of the representations output by the encoder. The decoder then takes as input the point cloud, the features output by the encoder, and a query point anywhere in 3D, and outputs an occupancy value. This network then implicitly defines a continuous occupancy field. The decoder is also SE(3)-equivariant, such that passing a transformed point cloud to the network results in an occupancy field transformed by the same SE(3) transformation. The manuscript demonstrates how this can be used to infer shapes in novel poses that were not seen during training, which has implications for both training efficiency and composability of representations.
strength_and_weaknesses: Ultimately, this manuscript presents a straightforward application of an existing technique (SE(3) Transformers) to an existing problem (occupancy modeling). The novelty is somewhat limited, but the work is timely and does contribute novel and valuable experiments to the literature. Occupancy modeling is a great application for SE(3)-equivariant networks, which remove the need for rotational data augmentation to handle arbitrary rotations. Even more compelling, when combined with local feature extraction, the proposed method can be applied to infer scenes comprising novel combinations of familiar shapes, which would otherwise require data augmentation that generates an immense number of combinations of shapes and poses. The experiments clearly demonstrate the value of the approach compared to a variety of baselines.
clarity,_quality,_novelty_and_reproducibility: The manuscript is very dense and somewhat challenging to read, but given the deeply technical mathematical backing of the proposed method this is to be expected. I personally found it necessary to reference prior work (e.g. SE(3)-Transformers and Tensor Field Networks) to fully understand the approach. Again, this is not necessarily a bad thing, but the manuscript could perhaps suggest doing so. The appendices were useful but also quite extensive. Some more concrete example might be useful, e.g. showing the values of the Wigner-D matrices for a particular order J and / or showing the block sparsity structure of the weight matrices for a particular feature vector with a particular multiplicity of each order. 
summary_of_the_review: This paper is a good example of a step change in performance; the proposed networks process input point clouds in a fundamentally different way (i.e. they are SE(3)-equivariant) from prior works, and the result is a large gap in performance in scenarios where this is important (e.g. inference of shapes in a different pose than they appeared in training. The novelty may be limited, but the empirical contribution is nevertheless compelling.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents an SE(3)-equivariant transformer architecture for occupancy prediction from point clouds. The network uses two types of attention layers, one attention layer on edges in the point graph and one cross attention layer to aggregate information from local neighborhoods in the output layers.
Further, the network utilizes Wigner-D harmonic bands of degree 2, allowing the network to model certain symmetries in the data.
The network is evaluated on I/I, I/SO(3) and SO(3)/SO(3) setups on ShapeNet, a synthetic scene dataset based on ShapeNet and Matterport3D, a real scene dataset.
strength_and_weaknesses: Strengths:
- Utilizing harmonic bands of degree > 1 makes sense to allow the network to model symmetries within the input. To my knowledge, it is first applied to 3D reconstruction in this work. Similar ideas have been seen in 2D before (harmonic networks).
- I like that the SE(3) equivariant formulations are well grounded in group theoretic considerations of irreducable representations.
- The paper is technically sound
- The paper is well written

Weaknesses:
- The novelty of the work is limited. There have been SE(3)-equivariant networks in a very similar setting before [1]. The notable addition to this work are the additional harmonic bands.
- The paper leaves out some of those work in its comparisons.
- In general, it is not evaluated if the additional harmonic bands (type-2) improve the results. The claim by the authors is not supported in experiments or theory. An ablation study, showing the benefits, is needed. Also, the a qualitative evaluation of these feature maps would be interesting. It might be that one can actually see different behaviour on symmetric objects.
 
clarity,_quality,_novelty_and_reproducibility: The paper is well written, sound and of high quality. It is probably not easily reproduced without code due to the more complex Wigner-D basis (but the authors provide code). There is some originality within this work by bringing the harmonic basis in a 3D point cloud architecture.
summary_of_the_review: I lean towards rejecting this work, based on the above mentioned problems. I am not fully convinced by the experiments. There are many methods doing 3D reconstruction with neural networks now and I have seen much more detailed reconstructions [1, 2, 3] against which this method is not compared. The authors claim that their SE(3)-equivariant formulation is very beneficial for scene reconstruction but at the same time, they only show very few examples of 3D reconstructions on real datasets and do not provide a quantitative evaluation or ablation studies.

[1] Chen et al.: 3D Equivariant Graph Implicit Functions, ECCV 2022

[2] Jiang et al.: Local Implicit Grid Representations for 3D Scenes, CVPR 2020

[3] Chabra et al.: Deep Local Shapes, ECCV 2020

Minor notation problems:
- sometimes, x_i is used to describe the relative position (Fig2), sometimes to describe the absolute position (3.1)
- neighboring point pairs are sometimes denoted as x_i, x_j, and sometimes as x,y
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel architecture to obtain local shape modelling and SE(3) equivariance for the task of 3D shape reconstruction. These properties are obtained by generalizing the attention mechanism and imposing specific equivariance constraints on the layer weights. The method can be trained on single aligned objects, and at inference, time can naturally work both with single roto-translated objects and also with complex scenes.
strength_and_weaknesses: STRENGTH
===========
S1) GENERALIZATION: The method can generalize to complex scenes without seeing them at training time. I am not aware of other works with the same capability. This is a particularly compelling property.

S2) SIGNIFICANCE: The design of equivariant networks is an active research field which collects significant interest from Computer Vision, Graphics, and ML communities. The new technique proposed in the paper can open novel research directions with a consequent tangible impact.

WEAKNESSES
============
W1) CLARITY: I am not completely familiar with all the involved math, and not all passages are clear to me. Often, the maths heavily breaks the text (e.g., after Eq. 5), notation is pretty heavy (e.g., Eq. 6), and some background concepts are introduced during the method explanation interrupting the narration (e.g., after Eq. 6). In this sense, I highly recommend to revise the text, grouping symbols and concepts definition before method presentation, and well separating mathematical derivation from the textual narration.

W2) APPLICATIONS AND LIMITATIONS: The proposed method is tested only on the shape reconstruction task. Other reported networks have shown results on different applications and challenges. There is no mention of possible limitations, and I do not see methodological limits to applying the formulated attention also on other tasks (e.g., segmentation, classification, matching) with minor modification. Testing on different tasks (and also domains) is standard for novel feature extractors (e.g., Vector Neurons).


MINORS
=======
Q1) Is it possible to manipulate the Query Tokens to obtain a modification of the output shape?
Q2) How the model performs in scenes where target objects are not separated in space (i.e., they are close, as objects in clutter)?
clarity,_quality,_novelty_and_reproducibility: As mentioned above, the work presents clarity problems, which directly harm the potential reproducibility. However, the novelty looks significant, even if I cannot fully understand all the passages and check their correctness.
summary_of_the_review: Given that I am not high-confident about the math and parts of the explanation are confusing, I cannot assess the correctness of the method. I think the method requires substantial clarification before publication, and there are unjustified lacks in the experiments and no mention of limitations or future works. For this reason, I think the paper requires some work and cannot vote for acceptance yet.
Looking forward to the rebuttal; I will be happy to raise my score if it addresses my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces a principally designed SE3 equivariant network for predicting implicit shape representations from point cloud observations. The proposed network, building on earlier SE3-Transformers work, allows impressive performance on recovering dense reconstruction from a single point cloud, while also enabling zero-shot generalization to multi-object scenes. The reviewers unanimously recommend acceptance, and the AC concurs.
justification_for_why_not_higher_score: While the results are impressive and the architecture well designed, the proposed approach dies depend on prior SE3 transformers work for the formulation and there is a limited technical innovation in that context.
justification_for_why_not_lower_score: The proposed methods shows clear improvements over prior work and presents a well-designed architecture for a common task. All the reviewers are unanimous in recommending acceptance.
Decision: Accept: poster