Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: For a given 3D point cloud of a scene, 3D bounding boxes, and language instructions, the method aims to identify the target object from a set of potential candidates. To achieve this, four modules (classification, spatial embedding, language embedding, and spatial-language model) have been used in the system. The method was evaluated on the ReferIt3D dataset (containing Sr3D and Nr3D datasets), and the results show that the suggested system outperforms the baseline models. Further, the experiments present that the method performs better for the expressions with spatial references (Sr3D) than natural references(Nr3D), which include the color features or appearance of the objects.
strengths_and_weaknesses: Strengths:
- The paper addresses a critical topic and different important subtopics. It focuses on understanding language instructions in 3D. It further discusses the importance of viewpoint orientations and the impacts of perception noise in spatial reasoning tasks. These issues have been less focused in computer vision studies, and they are critical in robotics applications.
- The system is evaluated in detail.  It is compared with five previous approaches for Nr3D and four existing approaches for Sr3D. Further ablations, additional results with ground-truth classes, and more evaluations with view-dependent and independent expressions are also presented in the experiments section.

Weaknesses:
- The suggested system is complex, composed of many different modules, and used four different losses. In this case, failure in one module can mislead the main prediction and might cause identifying the wrong object as a target. Further, in ablation studies, the paper shows that the mask loss does not affect the accuracy much, and the text loss decreased the accuracy. Therefore, it is not clear why these components are suggested as parts of the proposed system.
- The method focuses on understanding spatial references, and it mainly considers the positional embeddings. Accordingly, it obtains lower accuracies in the Nr3D dataset than the Sr3D dataset since the Nr3D dataset contains more features such as color and shape. However, in many robotics applications, only spatial relations might not be enough to disambiguate the target objects, and users might use varying features while describing the target objects. Such a system might fail when an object is described only with its appearance.
summary_of_recommendation: Although the suggested system is complex and mostly focuses on spatial references, it includes critical analyses about viewpoint orientations and perception noise, and these are crucial aspects for robots to build the bridge between the natural language and vision in real-world environments (see strengths and weaknesses for details). 
issues: - The paper does not emphasize enough what has been achieved in the 3D visual grounding and what are the contributions of the suggested system over existing approaches. The previous approaches on comprehending referring expressions in 3D aimed to identify the target object among candidate objects, as in this paper. It would be nice to focus more on the novel aspects of the suggested system.
- What is the number of object classes that can be identified by the semantic classifier? Can it classify all the objects categories in the ReferIt3D dataset?
- It is not clear how the spatial embedding is obtained in the system. The paper states that positional encoding PE(·) is employed to transform the bounding box vector, but it is not straightforward to understand how PE(·) converts bounding box vectors to spatial embedding.
- The target classification loss is defined as L_tar, but it is used as L_clf for the rest of the paper.
- Did you fine-tune the baseline methods (i.e., the ones used for comparison in Table 1) on the ReferIt3D dataset?
- What are the easy and hard categories reported in the tables? It has been forgotten to explain in the paper.
- The authors state that the model trained with corrected orientations achieved comparable accuracy to the best model even without corrections at inference (+0.4%, −0.4%). Which columns of Table 4 were compared to obtain these values?
- Figure 2 is not easy to follow. Spatial embedding representation might be positioned closer to DistilBert to reduce the length of arrows.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a model for grounding referring expressions to a 3D point cloud scene. Authors take the approach of encoding the language component and the the 3D scene and decode the referring object using a combination of losses. The 3D scene encoder uses a standard model for predicting the 3D bounding boxes and a classification label. The proposed approach treats the classification outputs as a sentence and determines a language-encoding for the same. The positional encoding is then encoded as a spatial embedding which is later combined with the language embedding prior to decoding the object labels. The model uses a combination of loss functions to arrive at the target object. Authors present thorough experiments on a number of benchmark data sets. 
strengths_and_weaknesses: Strengths 
- The authors address an important problem in human-robot interaction. 
- The model uses a realistic 3D point cloud data as input. 
- The approach also shows robustness to the orientation of the person providing the instruction. 



summary_of_recommendation: The paper contributes an approach for grounding language understanding in 3D scenes. The results are comprehensive. 
issues: - The approach directly processes the predicted classification labels and bounding boxes without explicitly modeling uncertainty in the classification labels or the geometry of the bounding boxes. 
- How extensible is the approach to richer inter-object relations. For example, "bring me the cup in the microwave" where an object can contain another object referred to by the instruction.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work develops a novel model for 3D visual grounding based on a 3D point cloud representation of a scene and a language utterance. This model simplifies earlier method by using pretrained language embeddings and a decoupled perception system, the method also also outperforms earlier methods on 2 datasets. Ablation studies motivate the design choices for this model also analyze the view dependence of the two Nr3D and Sr3D. 
strengths_and_weaknesses: Strengths:
* The transformer based, decoupled LanguageRefer model is well described and it is clear why all the components are needed. 
* The ablations with perception noise help understand how much of the performance is purely from perception which may motivate further improvements
* Thorough description of experimental results with several baselines means the results are convincing.

Weakness
* The improvements on Nr3D seem convincing but are incremental, 41.7% vs 43.9%
* Some error analysis would help understand in which ways is LanguageRefer better than previous models? Are there any particular cases that it gets that InstanceRefer does not? 
* The performance on Nr3D is still quite low, even with ground truth perception, why is this the case? What are some examples of errors that might motivate further improvements.
summary_of_recommendation: This work proposes an incremental improvement for 3D visual grounding with a simpler perception-decoupled transformer based architecture. 
issues: * Some error analysis of baselines and LanguageRefer on the Nr3D dataset to show examples why LanguageRefer is qualitatively better, also some analysis of the remaining errors.
* Figure 2, there is a token "###acing" which is confusing, is this a typo?
* Numbers in tables would be more convincing if they had errors reported as well
* Table 4, mention the dataset in the caption for consistency.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a model to address the 3D visual grounding problem. Given a set of candidate 3D bounding boxes, the goal is to identify the object being referred to by a natural language expression. The proposed model combines spatial embeddings of the bounding boxes with a linguistic embedding from DistilBert. The resulting model is shown to outperform several baselines. In addition, the authors show that viewpoint-specific information is critical to grounding in 3D environments where the spatial information assumes a viewpoint for the speaker.

strengths_and_weaknesses: One strength of this paper is the extensive set of comparisons and ablations, to which the proposed model achieves strong performance. The paper evaluates several design choices such as how important using ground truth class labels are and the relative importance of different components of the loss function. One particularly valuable insight is the importance of using a viewpoint that is consistent with viewpoint-dependent language. This is common for situated environments and is a valuable insight to the community.

The main weaknesses of this paper result from missing details needed to fully appreciate the take-aways of the proposed model. For example, I was not able to find descriptions of the baselines used in Table 1. This makes it difficult to state which component of the proposed model most contributed to the performance gain. It would be helpful to see these descriptions that highlight the key differences with the proposed model in Section 4.2 (or in the appendix if space does not allow). Relatedly, although the paper presents many interesting ablations, it would be additionally informative to compare the relative contributions of both the spatial encodings and the pretrained language model. 

Additionally, the following points addressing the clarity of the paper would further strengthen the contribution:
- Visual information only factors into the prediction through class labels. This decoupling is shown to help the approach transfer to different datasets (Table 2). It would strengthen the comparison to show how a coupled perception-language model transfers between these datasets but perhaps this is outside the scope of this work. It is still an interesting result that the model performs well without further information and it would be useful to discuss any limitations that arise from this assumption.
- Regarding the loss function ablations, when the text loss and classification losses are left out, does the inference procedure described on line 171 change? Which combination of loss functions are used for the Table 1 evaluations?
- What is the value of “k” in the experiments? (Line 172)
- It would be helpful to describe the differences between the “Easy” and “Hard” splits in the result tables.

summary_of_recommendation: I recommend this paper be accepted to CoRL. The importance of viewpoint is an interesting contribution and I believe the identified issues can be addressed by the authors.
issues: The main issues with this work are missing details mentioned above. Adding these would greatly help the reader recognize the key contributions of this work.
review_title: null

Meta-review: All reviewers agree that the visual grounding is an important and interesting topics and that the proposed approach, taking advantage of 3D visual data and language instructions, is promising.

However, the paper could make an even better case for the proposed method by improving clarity and by discussing the robustness, scalability and the limitations of the approach in further detail.

## Post-Rebuttal Update
The authors addressed and clarified the concerns raised by the reviewers, improving the paper in the process.
Decision: Accept (Poster)