Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method for language grounding in 3D scenes (i.e. detecting a specific 3D object instance based on an input natural text describing the object in the context of the scene).  The method is based on a transformer architecture, with the key novelty lying in a spatial self-attention mechanism and a teacher-student distillation training setup.

The method is evaluated on the ScanRefer and ReferIt3D datasets, with comparisons against ablations to demonstrate the value of method components, and also against baselines taken from prior work to demonstrate the relative improvement in 3D object grounding.  The experiments report grounding accuracy and show measurable improvements over prior work.
strengths_and_weaknesses: Strengths
+ The paper is clearly motivated in the need to better model 3D spatial relations in the context of the language grounding problem. The described spatial self-attention mechanism is to my knowledge a novel contribution in this area
+ The exposition is overall quite clear, and the experiments present a thorough spectrum of ablations that do a good job of analyzing the impact of different method components (in particular, the attention to contrasts between view-dependent vs view-independent text inputs is appreciated)

Weaknesses
- Given the big focus on spatial relations, I would have liked to see a more systematic analysis of the results dependent on the need for attending to spatial relations to disambiguate the specific object instance that the input text describes (as illustrated well in Fig 1).  For example, a comparison of overall performance on sentences like those seen in Fig 1 vs sentences that do not use spatial relations to disambiguate would offer more detailed evidence that the proposed architecture is indeed better able to ground objects in such cases.  In addition, a breakdown by spatial relation types (involving distance, involving orientation, and involving both) would be quite informative.
questions: I would like the authors to address the identified weakness above, at least in terms of the broad strokes of how the approach performs for input text exhibiting different degrees of spatial relation disambiguation.
limitations: The conclusion mentions the two-stage nature of the approach as a limitation (i.e. that object detections are assumed to be given, and may be imperfect).  The conclusion also briefly states that the presented work only has minimal potential negative societal impact.  I would agree with this statement given the domain (3D interiors), but I would still point out that inherent dataset biases may misrepresent the rich diversity of environments in which people around the world may live.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on improving spatial relationship reasoning in the task of 3D object referring expression comprehension. The key ideas are a dedicated spatial attention mechanism integrated into the basic transformer architecture, and a teacher-student training mechanism. The paper significantly outperforms existing baselines on ReferIt and ScanRefer datasets.
strengths_and_weaknesses: The paper is clearly motivated and well written. The model is based on a straightforward transformer architecture, and integrates spatial attention models. Although I would argue that the color feature and spatial relationship feature used in the paper is a little ad-hoc but it is simple and effective. I have only very few questions and suggestions about the weakness of the paper.

Perhaps the biggest weakness of the paper is that, based on my understanding, the base architecture (e.g., the R1 model in Table 1) is not an existing baseline model. Thus, the authors are both changing the basic architecture (c.f., InstanceRefer and Multi-View) and add-ons (teacher-student, spatial attention). To better understand the effectiveness of the proposed modules, the authors should consider integrating the presented techniques with other baselines too.


questions: L159: Can you be more specific about the horizontal and vertical angles?

L165: Why do you use sigsoftmax, in contrast to, for example, softmax？

L235: Is there any reference on using only the first 3 layers of BERT to encode the sentence? Why don't use all layers? Also, it seems that some baselines (InstanceRefer) use (perhaps) weaker textual encoders (GloVe+GRU), can you provide an ablation study on the textual encoder module?

L239/L205: If the pointnet is pretrained and fixed, why are you adding $L_{obj}^u$?

L240: The author should explain the details of the rotation augmentation.

limitations: What are the limitations of this paper?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new method for 3D object grounding with a transformer that combines input of a BERT text encoder and a Point++ object encoder to match the detected object proposals with the input sentence description.

The key insight here is that explicitly encoding spatial relations between detected objects significantly help with spatial reasoning. Based on this observation, the method incorporates a spatial self-attention layer that computes spatial relevance from language inputs and the explicitly defined pairwise object relations (of relative distance and angle). This spatial relevance is then used to weight the self attention between objects. 

Combining this novel spatial reasoning layer with teacher-student knowledge that distills knowledge from a teacher network with access to ground truth labels and color of objects, the proposed method is able to achieve significantly better performance than prior works on multiple benchmarks, where reasonable control of variables are attempted. Thorough ablation are also performed to justify the importance of each of the novel design choices.

strengths_and_weaknesses: First and foremost: I have not worked on the problem studied in this paper and was not familiar with the related works until reviewing this paper. So this is more of a common-sense review judging from the technical soundness and quality of evaluation, as well as quick skim through of some of the relevant works discussed. I'll defer the judgment about the magnitude of contribution to reviews who are more familiar with this problem.

That being said, I like this work in general: it has a very clear motivation (improving spatial reasoning for 3D object grounding) and proposes an effective solution that clearly outperforms relevant prior works. My concern about this paper is largely around the magnitude of the contribution and if that have significant impact on future works. Again, I don't think I am really qualified to judge about this, but based on other points below, I lean towards accepting this paper.

### Strengths:
+ Simple yet important key observation about the importance of explicitly encoding pairwise spatial relations, solid design of a novel module based on this observation.
+ A generalizable teacher-student training method that can be extended to other methods for the problem studied.
+ Exceptional results: significantly outperforms relevant prior works on multiple benchmarks and even outperforms works with more assumptions on input.
+ Extensive ablations validating each of the design choices.
+ Good writing quality, easy to follow.

### Weaknesses:
- Magnitude of contribution might be slightly limited since it follows a rather standard pipeline. The novel part of this paper can, to some extent, be seen training tricks / inductive biases to help the model better picking up certain information (spatial relation) that is important to the task.
- Following the previous point: it might be argued that the techniques introduced in this network might be limiting eventually if solutions emerge to better capture the spatial relations from the object attributes alone.
- Ablations only justify that each of the component help with the final result, without discussion of alternative method that might achieve the same goal. I think this understandable though - should not expect a work that explores an idea to exhaustively study the design space.
questions: ### Design Choices
- What is the motivation of computing Euclidean distance from the centroids, instead of between the bounding boxes? It seem to me that distance between bounding boxes might describe proximity between objects more unambiguously as opposed to centroid-to-centroid distances.
- Similarly, it seems that the vertical angles between objects might be very sensitive to height of the objects. Would it be better to decouple the impact of dimension e.g. by computing this angle based on the center of the bottom face?
- I wonder what's the benefit of hardcoding spatial relations as opposed to, say, using a MLP to automatically compute spatial relations from the bounding boxes of the objects, probably in their local coordinate frame?
- I imagine many text descriptions are probably related with the room architecture itself. Would there be benefit in also trying to encode object-room relations explicitly?
- I know its mentioned in the supplementary that determining the orientations from noisy inputs is challenging. Still, it appears to be that most natural language descriptions of directions (left, right, etc) is highly depending on the orientation of the objects, so the model will need to reason about orientation in some way. Have you tried to more explicitly handle orientations when defining and encoding the spatial relations?  If, does inclusion of them degrades the performance because of the ambiguity? If not, what are some possible ways to potential handle such cases?

### Evaluation
- As mentioned, some discussion of alternative ways to design the spatial relation layer might be helpful, but definitely not necessary.

limitations: Only one limitation that is common to all method of this class is mentioned towards the end. More discussion of limitation specific to this method would help.
Qualitative failure cases are provided in the supplementary, with analysis. One example (Supp Figure 2, right) shows a weakness specific to this method regarding object orientation.

I agree that there is no apparent negative social impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a language-conditioned transformer model for localizing objects in 3D scenes and reasoning about relative spatial distances based on natural language description. The model is based on the transformer architecture and includes spatial attention layers and multi-head design to learn and reason about relative distances and orientations among multiple objects. The spatial attention output is fused with standard self-attention by the sigmoid softmax function proposed by the authors. The model is trained using a teacher-student approach accompanied by rotation augmentation to facilitate learning under scarce data and to produce reasonable inference with point cloud inputs. The final model is tested on object grounding tasks Nr3D, Sr3D, and ScanRefer3D, and outperformed state-of-the-art models.
strengths_and_weaknesses: Strengths: 
1. This paper addresses a challenging and interesting task of 3D object reasoning based on natural language description, with a focus on distinguishing similar objects referred by the text using different spatial relationships. 
2. Using multi-head attention, the proposed language-conditioned transformer model effectively learns to capture and reason about different spatial relationships among objects in the self-attention layers. 
3. The model is systematically evaluated on multiple datasets and achieves strong performance, and the learned attention is clearly demonstrated in Figure 4. 
4. The ablation study is carefully designed and shows the value of different components the authors introduce in the paper.

Weaknesses: 
1. In equation 3, it is not very intuitive what g (language conditioned weight) means. What is the role of the classification token s_cls being an input in the conditioning? 
2. Related to the previous point, in Figure 2 (left), the Text BERT serves as input to both the spatial self-attention and cross-attention. It is not clear if the self-attention takes the text features (s_1, …, s_M) as input or only takes the s_cls token.
3. In line 189, the authors state that the teacher model’s input object representation is the sum of object class label embedding and the averaged color embedding. Does the sum introduce any difficulty during training, especially when it requires the model to use object appearance (such as color) rather than spatial relationships to locate objects?
4. It is nice to see the visualization of the teacher model in Figure 4. It will be compelling if the authors show that the student model attention layers behave similarly.
5. The loss function in equation 8 is very complicated. There is little detail about the auxiliary losses and the rationale for including them in the objective.
6. The final model only marginally outperforms 3D-SPS, according to Table 5. Although the authors state that the comparison is not fair due to detection optimization or single-stage pipeline, it would be helpful to reflect on how to improve the current detected object proposal.
questions: As mentioned in the weaknesses above, clarifying the language conditioned weight, BERT text feature input, and auxiliary tasks would be helpful. In addition, it would be helpful to include more details about the cross-attention computation in the model.
limitations: The object proposal in the two-stage pipeline is critical because it constrains the hypothesis space for cross-modal matching. Thus the prospect of improvement in the object proposal stage would be beneficial.
review_title: null

Meta-review: metareview: Reviewers where in agreement that the method and manuscript are strong and provide a valuable connection between 3D perception and language.

The evaluation suffers somewhat from the fact that there are no good datasets targeted at this problem. Authors mitigate this by performing a thorough evaluation with numerous ablations/alternatives that demonstrate their method works as intended.

Beyond just this application, the general approach taken of incorporating priors about a problem domain into a self-attention layer for a Transformer to combine multimodal input is timely and will be of wide interest.

I encourage the authors to update the manuscript and include the additional experiments and results they produced for reviewers.

confidence: Certain
Decision: Accept