LanguageRefer: Spatial-Language Model for 3D Visual Grounding
Authors: Junha Roh
Karthik Desingh
Ali Farhadi
Dieter Fox
Venue: CoRL2021 Poster (2021)
Keywords: Referring task
Language model
3D visual grounding
3D Navigation

Abstract: For robots to understand human instructions and perform meaningful tasks in the near future, it is important to develop learned models that comprehend referential language to identify common objects in real-world 3D scenes. In this paper, we introduce a spatial-language model for a 3D visual grounding problem. Specifically, given a reconstructed 3D scene in the form of point clouds with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model successfully identifies the target object from a set of potential candidates. Specifically, LanguageRefer uses a transformer-based architecture that combines spatial embedding from bounding boxes with fine-tuned language embeddings from DistilBert to predict the target object. We show that it performs competitively on visio-linguistic datasets proposed by ReferIt3D. Further, we analyze its spatial reasoning task performance decoupled from perception noise, the accuracy of view-dependent utterances, and viewpoint annotations for potential robotics applications.

LanguageRefer: Spatial-Language Model for 3D Visual Grounding
Junha Roh, Karthik Desingh, Ali Farhadi, Dieter Fox Paul G. Allen School, University Washington, United States (rohjunha, kdesingh, ali, fox] washington edu
Abstract: For robots to understand human instructions and perform meaningful tasks in the near future, it is important to develop learned models that compre- hend referential language to identify common objects real-world 3D scenes. In this paper, we introduce spatial-language model for a 3D visual grounding problem. Specifically, given a reconstructed 3D scene the form of point clouds with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model successfully identifies the tar- get object from set of potential candidates. Specifically, LanguageRefer uses architecture that combines spatial embedding from bounding
a transformer-basee boxes with fine-tuned language embeddings from DistilBert [1] to predict the tar- get object. We show that it performs competitively on Disio-linguistic datasets proposed by ReferIt3D [2]. Further, analyze its spatial reasoning task per- formance decoupled from perception noise, the accuracy of view-dependent ut- terances, and viewpoint annotations for potential robotics applications. Project website: https://sites.googgtt www.gggooo..ooooo/////ggaggeeeeo
Keywords: Referring task, Language model, 3D visual grounding. 3D Navigation
Language description
"Pacing the foot of the bed, the bed on the right."
Classification
Predicted target object
3D Pointclouds in bounding boxes
Spatial Embedding
Language
Embedding
Bounding boxes
Spatial-Lang.
Model
Figure I: Simplified overview of LanguageRefer. The LanguageRefer model takes input grounding language description a single object the scene, a 3D point cloud a scene, and bounding boxes objects the scene and predicts the target object. Its four modules include: a classifier, spatial embedder, a language embedder, and spatial- language model.
Introduction
For robots to communicate scamlessly with humans to perform meaningful tasks in indoor environ- ments, they must understand natural language utterances and ground them to real-world elements. Several recent advances have combined language and visual elements, producing methods for tasks such as visual question and answering (VQA) involving spatio-temporal reasoning tasks [3, 4. 5], embodied QA [6], and pre-training for visual recognition tasks with language descriptions [7]. Fur- ther, embodied agents can follow visually grounded language instructions to perform embodied tasks [8, 9]. However, for real robots to intelligently perform these tasks, need 3D representa- tions from raw sensor data; ReferIt3D [2] proposes benchmarking dataset of language utterances referring objects in 3D scenes from the ScanNet [10] dataset.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Our long-term goal is to enable robots to visually navigate indoor environments based on referential instructions. In this paper we take a step towards this goal by leveraging the ReferIt3D dataset to build model that can identify the 3D object referred to in a language utterance.
Referential language to identify an object in real-world 3D scenes poses a challenging problem. Consider the sample utterance Facing the foot of the bed, the bed on the right" along with 3D scene as shown in Figure 1. Humans can easily follow the language clues, infer the point of view of the speaker, locate all the referenced elements, and spatially reason to locate the bed in the scene despite two instances of beds. However, viewpoint prediction, object identification, and spatial reasoning remain open- open-ended research problems in robotics and vision. The 3D reference task proposed by ReferIt3D is difficult because: (1) reconstructed 3D scenes of the real world in the ScanNet dataset are noisy and lack fine details compared to 2D images or rendered 3D scenes, (2) fine-grained class labels and expressions in natural language utterances are diverse and not exactly matched, (3) view-dependent utterances often require guessing the original viewpoints, which deters the model from properly learning spatial concepts, and (4) the combined complexity of multiple challenges complicates efforts to analyze what the model learns understands.
Inspired by the success of the methods in [11, 12] on CLEVR and CLEVRER domains for the spatial reasoning task, we hypothesized that for the 3D reference task, decoupling the spatial-reasoning from the perceptual task of identifying the objects in the 3D scene would improve performance and clearly track the role of perception noise in performance. More specifically, instead of developing an integrated multi-modal perception system, we assumed a pre-trained instance classification model or ground-truth classes for the objects that informed the spatial reasoning task. We focused on how the language model with spatial information could handle the reference task.
The ReferIt3D dataset includes two sub-datasets containing natural and synthetic language utter- ances, namely Nr3D and Sr3D, respectively. our experiments, our model achieved comparable scores on both with predicted instance class labels. We observed high accuracy with ground-truth la- bels in Sr3D, which indicates that our model better understood template-based language data. Since our pipeline is modular and features multiple models (perceptual, spatial embedding, pre-trained language embedding, and vironments and object entities.
our approach flexible and adaptable to different en-
Another aspect of the 3D reference task is viewpoint prediction. The Referlt3D dataset contains utterances that can be grouped into view-independent (VI) and view-dependent (VD) categories. An example of VI utterance is "The lamp closer the white armchair" and a VD utterance is "The lamp on the right in-between the beds". The VD utterance requires viewpoint prediction. This distinction is crucial for robotics applications where the agent must infer the viewpoint to which the speaker is referring. In the ReferIt3D dataset, some VD utterances lack information to guess the valid orientation of the agent (who utters the language description), which prevents a model from understanding the significance of spatial relationships such as 'left of.' This is due to the annotation process of ReferIt3D datasets, where both a speaker and listener can freely rotate the scene to infer an ill-defined orientation in the utterance. Human annotators who are aware of spatial concepts tend to validate otherwise arbitrary orientations. However, data-driven model will suffer from degraded learning when it cannot verify orientations as well as human annotators. Niewpoint-free annotations may suit most robotics applications; nonetheless, predicting or verifying whether a given statement and the viewpoint match remains important. To better train and investigate the agent in view dependencies, we provide an extra collection orientation annotations for VD utterances and compare the models with and without viewpoint correction from them.
To summarize, the paper contributes: (1) a novel transformer-basse spatial- model in a modular pipeline that better understands spatial relationships in a 3D visual grounding task. We show that our model achieves comparable performance with state-of-the methods on the Nr3D) and Sr33 datasets. (2) analysis of the ReferIt3D dataset with viewpoint orientation annotations to remove potential artifacts from implicit orientations.
(3) ablation and additional experiments with ground-trute classes that decouple the impact of perception noise in the spatial reasoning task.
2

---

2 Related Work
2.1 Vision-and-Language Navigation and Robot Navigation
-aiio--ad-language navigation (VLN) has been extensively studied and made remarkable progress over the last few years ([13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]). Given a language instruction in the simulation environment, the goal is for an agent to reach the desired node the pre-defined traversal graph using images as input. The literature offers several extensions. ALFRED [8] pro- poses an extended VLN task that grounds sequence of sub-tasks to achieve a higher level task in the A12Thor environment [9]. ALFRED navigation sequences have (implicit) goals that are often close to objects of interest in the subsequent sub-tasks, e.g.. when an agent is asked to move in front of the sink because is going to clean a cup there. With high-level semantic tasks, object-centric spatial understanding is of even greater importance.
In another direction, recent approaches have relaxed the constraint of discrete traversal in VLN into continuous space ([24, 25, 26, 27, 28J). Here, agent encounters more complex tasks involving time and space. Thus, expanding the space representation to 3D can be an effective solution. In the context of VLN, consider the 3D visual grounding task as a proxy for 3D indoor navigation that includes the full observability assumption and goal-oriented language descriptions. In particular, approach focuses understanding spatial relationships among objects, which plays key role in VLN and robot navigation.
2.2 2D and 3D Visual Grounding
The 2D visual grounding task localizes an object or region in an image given language description about the object or region ([29, 30, 31J). Most methods use two-stage approaches: they first gen- erate proposals and then compare the proposals to the language description choose the grounded proposal ([32, 33, 34, 35, 36]).
The 3D visual grounding task localizes 3D bounding box from the point cloud of scene given language description. Recently, Referlt3D [2] and ScanRefer [37] were proposed as datasets for 3D visual grounding tasks, with language annotation
the ScanNet [10] dataset. Most 3D grounding approaches (12, 37, 38, 39, 40)) follow a two-stage schemes similar to many 2D visual grounding tasks. First, multiple bounding boxes are proposed or the ground-truth bounding boxes are used, and then features from the proposals are combined or compared with features from the language description. InstanceRefer [38] extracts attribute features both from point clouds and utterances and compares them to select the object that best matches. FFL-3DOG [39] matches features from language and point clouds with guidance from language and a visual scene graph. These methods rely on specific designs, c.g., sard-eye-view mappings with different types of operations or intensive language pre-processing for graph generation. In contrast, our approach leverages the language embedding space from the pre-trained language model. Following the pipeline and general ar- chitecture of the language model therefore requires minimal manual design compared to previous works. SAT [40], like our approach, relies on transformer [41] models; it learns a fused embed- ding of multi-modal inputs and uses auxiliary 2D images. In contrast, our approach uses semantic classifier to predict object class labels and takes these labels input. It has a marginal cost of learning fused embeddings compared to training multiple BERT models [42] from scratch in SAT [40]. Even given only semantic information from point clouds, our model still achieved comparable performance with state-of-the-ar art methods on Nr3D and Sr3D datasets. In addition, the decoupled perception module makes approach modular and thus transferable different data.
3 Problem Statement and Methodology
Given 3D scene S with a list of objects (O1, Oxs) ) and language utterance U, the problem is
to predict the target object Or, T E Im = (1.... .M} referred to in the language. A single object
O consists of a bounding box B, E B = R and corresponding point cloud P. E P = B N. x6 (xyz positions and RGB values) in the bounding box with N; number of points.
We propose approach based on language models, called RefguageRefer, to solve a 3D visual grounding task. Our model focuses on understanding spatial relationships between objects from language descriptions and 3D bounding box information. We chose this approach due to (1) the high
3

---

Longivgge Embberdiigg
"Facling the foot of the beel the bad on the right."
Spotiar Longtaa-Lonugggeeee Model
Language description
[cLa]
Prodicted class label
"P
"Ihe'
Mask
"hei"
[M4AS]]
CIt Lons
Movee'
Spatial Emebedeing
Macunddig
Pointelou
(127)
Rei Low
'iss'"
Podffors-enccded
Figure 2: Detailed overview of LanguageRefer. A semantic classifier predicts class lahels from 3D point cloud in each bounding hox (using color and xyz positions). The language description utterance (e.g.. "Facing the foot of the bed, bed on the right") is transformed into a sequence of tokens. The input token embedding in DistilBert [I] converts the tokens into embedded feature vectors (green squares). Bounding box position and size information are positional-encoded to form encoded vectors using techniques from [41] (orange squares); they added the corresponding embedded feature vectors (green squares). After the addition, our reference model processes the modified features and feeds them to multiple tasks. The main task a reference task, i.c., it chooses the referred object from the object features. The instance classification task is a binary classification, i.e., determines whether the given object feature belongs to the target class. Finally, the masking task, commonly used in language modeling, recovers the original token from randomly replaced token in the utterance.
dependency on spatial relationship descriptions in language, and (2) the holistic nature of spatial relationship information, which differs from unary attribute information such color and shape. As shown in Figure. 2, we use a two-stage approach. First, we determine class labels of objects lpatial-language embedding identify the referenced object. The
in the scene. Second, following subsections describe these steps in detail.
Semantic Classification Model and Tokenization. For semantic classification of the point cloud bounding box, we employed PointNet++ [43], which achieved 69% accuracy on average in the test dataset. In training and inference, we use sampled point cloud P/ E R 1024400 and PointNet++ predicts the semantic label i E L, where L is set of class labels in text. Each scene has pairs of predicted class labels and bounding box values ((4, B.) I, E L, B; B) from objects. Predicted labels are concatenated to the utterance U with a separator [SEP] and then split by a tokenizer into a list of indices of tokens: U becomes (uI, ",,,)) and each predicted class label i becomes (of..... ,o,,), ), where each token index is in In and D the size of the dictionary.
Language Model and Token Embedding Generation. Our model uses a pre-trained language model. DistilBert, for the reference task. Transformers consider relationships among all pairs of elements through attention, and they can be effectively leveraged to explain spatial relationships between objects discrete entities. In our formulation, predicted class labels are considered to be sentences, so they are concatenated to the utterance with separation by [SEP]. Therefore, the final sequence of token indices would V ((CLS), u1. ut [SEP], of. ... o,.,, [SEP]. ... [SEP], ... o [SEP]). Though the token index sequence complies with the specification of Distil- Bert, it violates the number of sentences. Then, we transform V into the token embedding sequence W = (W), ... WT), wi E IR 768 using DistilBert's word embeddings. For concise notation, we de- fine the indices of utterance tokens in V or W as mask My = (2, ++1 1) and the indices of first tokens from objects (o).... MM) as a mask Mo. We also define a mask operator [-] to manipulate specific elements in the sequence; for instance, W[Mu W.
empties all utterance embeddings in
Spatial-I Language Model and Spatial Embeddings. To combine spatial information from raw bounding box values into the token embedding W, we employ sinusoidal positional encoding PE(-) from [41] to transform the bounding box vector (center position and size) B, E B c R to b. =
Spatial-Language

---

(E( E R which is then added to [ The token embedding W is then combined with spatial information and finally transformed into the output embedding X (II,-.. .IT , I; E 7 by the reference model, which is fine-tuned from the pre-trained DistilBert. The final reference task performed by the reference classifier from X[Mo], as explained in the following subsection.
Loss Functions. We use three tasks for training and corresponding loss values. First, we use the reference loss, Cref. following the original proposal in [2]. We ask the model to choose one object as the target instance from M candidates. We collect scalar values from X [Moo] by a linear layer and take the argmax on those values to choose the target instance.
Second, we add a binary target classification loss Las on X [MMo] to determine whether a given object belongs to the target class.
Last, we employ mask loss from language model pre-training, Lmask- We randomly replace the tokens from nouns in the utterance with a probability of 15 %. The noun token is replaced by [MASK] with an 80 chance, by a random token with 10 chance, or it remains the same with probability of 10 %. Then, the model is asked to recover the original token index. We expect the model to fill in the replaced tokens in the utterance by understanding the relationship between objects. We use cross entropy loss for all tasks and compute the final loss as
L = L + 0.5C.l + 0.5.mask-
(1)
At inference, we followed the approach of InstanceRefer [38] filter objects that do not belong to the predicted target class. We used an extra DistilBert-baase based target classification model of 94 % accuracy that takes the language utterance as input predict the target class. To reduce the chance of removing the true target instance in the filtering process, the top-k class predictions (from the semantic classifier) for each object are compared to the predicted larget class. We use k throughout the experiments. For masking loss computation, we extracted nouns in the utterance. We used flair [44] for part-of-speech (POS) tagging.
4 Experiments
4.1 Datasets
We evaluated our model on the reference task from ReferIt3D [2] with two datasets, Nr3D (Natural reference in 3D) and Sr3D (Spatial reference in 3D, which contains spatial descriptions only). Both datasets augment ScanNet [10], a reconstructed 3D indoor scene dataset with language descriptions. Nr3D has 41,503 natural language utterances, and Sr3D contains 83,572 template-based utterances, on 707 scenes following the official splits of ScanNet [10]. The datasets have 76 target classes and designed to have multiple same-class distractors in the scene.
4.2 Experiment Settings
ReferIt3D [2] provides the ground-truth bounding boxes of objects in the scene, point clouds of the scene, utterances and corresponding target objects. We measured the accuracy of the model by comparing the object selected from M candidates the ground-truth target object. When the number of same-class distractors exceeded two, we classified the instance as "hard" according to [2]. The other cases were classified as "easy."
We trained models with a learning rate of 0.0001 using AdamW optimization, warm-up and linear scheduling. Our model was initialized with pre-trained model of the cased Distilbert base [1] from the Hugging Face implementation [45].
We compared the performance of our model with state-of-the-art methods on Refer1t3D ([2, 40, 37, 38, 391) based on the reported numbers on the challenge website [46] and corresponding papers. Since SAT [40] uses an extra 2D image dataset in their training, we separated it from non-SAT, their baseline model that is not trained with the extra dataset.
Each value in b, is extended to 128-dimensions by PE and then concatenated to form 768- -o-eeimesiionl vector.
5

---

Dataset
Method
Overall
Easy
Hard
View-dep.
View-indep.
ReferIt3D
35.6 %
43.6 %
27.9 %
32.5 %
37.1 %
ScanRefer [37]
34.2 %
41.0 %
23.5 %
29.9 %
35.4 %
InstanceRefer [38]
38.8 %
46.0 %
31.8 %
34.5 %
41.9 %
Nr3D
FFL-3DOG [39]
41.7 %
48.2 %
35.0
37.1 %
44.7 %
non-SAT [40]
37.7 %
44.5 %
31,2 %
34.1 %
39.5 %
Ours
43.9 %
51.0 %
36.6
41.7 %
45.0 %
Nr3D w/ 2D images
SAT [40]
49.2 %
56.3 %
42.4
46,9 %
50.4 %
ReferIt3D [2]
40.8 %
44.7 %
31.5 %
39.2 %
40.8 %
InstanceRefer [38]
48.0 %
51.1 % N/A
40.5 %
45.4 %
48.1 1 %
Sr3D
non-SAT [40]
47.4 %
N/A
N/A
N/A
Ours
56.0 %
58.9 %
49.3
49.2 %
56.3 %
Sr3D w/ 2D images
SAT [40]
57.9 %
61.2 %
50.0
49.2 %
58.3 %
Table I: Accuracy on ReferIt3D [2]. Our model outperformed state-of the models on both Nr3D and
Sr3D except for models with additional training data (SAT with 2D images). The average performance gap between ours and other models Sr3D (10.6%) is larger than that on Nr3D (6.3%) since our model uses only spatial reasoning for the reference task.
4.3 Evaluation on ReferIt3D [2]
Table shows the accuracy on Nr3D and Sr3D. Our model outperformed other models (without extra training datasets) on both Nr3D and Sr3D. It achieved 43.9% on Nr3D with an +8.3% improvement over the baseline method of Referlt3D [2] and a 2.2% increase over the best accuracy from other models in Nr3D. For Sr3D, model achieved 56.0%, which is 15.2% and 8.0% increase the baseline and the best accuracy in the Sr3D section, respectively. It is also comparable to the accuracy of SAT (with 1.9% difference), which was trained with additional 2D image training data. These results prove that our model can accurately reason about spatial relationships from spatial-language embeddings and outperforms other models both datasets despite the loss of information in appearance. In addition, less diverse language expressions and utterances only about spatial reasoning can explain our model's strong performance on Sr3D.
4.4 Evaluation with Ground-Truthe Class Labels
We further investigated the model's spatial reasoning ability by removing noise in class labels. We replaced predicted class labels with ground-truth class labels, which was possible due to the explicit usage of class labels in our model.
Table 2 shows the result with and without ground-truth class labels. The first and second columns demonstrate the type of dataset in training and evaluation, respectively. For instance, the second row shows the evaluation result of the model trained with the Nr3D dataset with ground-truth class labels and evaluated on the Nr3D dataset with predicted class labels. When we trained and evaluated our model with ground-trute truth labels on Nr3D and Sr3D (row 4 and 8), we achieved 54.3% and 91.1% accuracy, respectively. Compared to the accuracy of models trained and evaluated with noisy labels, we realized an improvement is 10.4% and 35.1% 1%, respectively, for each dataset. When we evaluated with ground-truth labels the models trained with noisy labels, their performance increase was 9.7% and 24.2%. respectively. Multiple reasons account for the difference in performance on Nr3D and Sr3D: diversity in the natural language dataset, higher portion of view-dependen utterances or view-dependent utterances with no mention of orientation, and descriptions other than spatial relationships, such as color appearance.
In addition, we examined the accuracy given switched datasets, namely, when we train model on Nr3D and evaluate it using Sr3D, and vice versa. The last two rows in Table 2 show two switched evaluation results: 40.0% and 37.6 67 on Sr3D and Nr3D, respectively. The 37.6% accuracy on Nr3D from the model trained on Sr3D is comparable to the accuracy of non-SAT [40] and InstanceRefer [38]. It shows our model's s generalizability of spatial reasoning and potential to transfer different perception modules and datasets.

---

Dataset
Overall
Easy
Hard
View-dep.
View-indep.
Training
Evaluation
Nr3DD-pred
Nr3D--pred
43.9 %
51.0 %
36.6 %
41.7 %
45.0 %
Nr3D-gt
Nr3D-pred
41.4 %
48.0
34.6
38.1
43.0 %
Nr3D-pred
Nr3D-gt
53.6 %
64.4 %
42.5 %
50.7 %
55.0 %
Nr3D-gt
Nr3D-gt
54.3 %
65.5
42.8
49.1
56.8 %
Sr3D-pred
Sr3D-pred
56.0 %
58.9
49.3
49.2
56.3 %
Sr3D-gt
Sr3D-pred
52.1 %
53.8
48.2 %
43.9
52.5 %
Sr3D-pred
Sr3D-gt
80.2 %
83.2 %
73.1 %
62.5 %
81.0 %
Sr3D-gt
Sr3D-gt
91.1 %
93.1
86.2 %
67.0
92.1 %
Nr3D-pred
Sr3D-pred
40.0 %
43.6
31.5
41.2
39.9 %
Sr3D-pred Nr3D-ppred
37.6 %
45.3 %
29.6 %
34.5 %
39.0 %
Table 2: Ablation with ground-truth class labels. First and second columns show types of data used in of the model both trained and evaluated
training and evaluation. The high overall accuracy (91.1% at with ground-trun truth class labels on Sr3D shows its spatial reasoning ability besides perception noise. Ac- curacy gaps between ground-truth and predicted class labels on Nr3D and Sr3D (9.7%. 24.2% respectively) indirectly tell us about language complexity and information loss due to classification. Transferring the model trained with Sr3D Nr3D evaluation shows overall number (37.6% other methods. Our model can easily accommodate different classification models datascts.
10) comparable to those from
4.5 Ablation on Loss Terms
We trained models with different combinations of loss terms, and Table 3 shows the results. In addition to three tasks, we examined the effect of text classification task that predicts the target class label 1 L from the tokens from utterance X [MM.].
We found that only the binary classification loss shows its effect clearly (+3.5% from Ref. Ref.- Clf., +3.9% from Ref.-Mask to Mask-Mas +100% from Ref.-Text to Ref.- Ref.-Text-Clf.). The mask loss was not effective, and the text classification loss degraded accuracy. One hypothesis here is that the text classification loss does not help the model to transform the initial language embedding to spatial-language embedding. The model may not require strong language constraints because (1) already has good initial language embedding. and (2) the text classification loss is independent of the scene's spatial configuration. Due to the performance drop, we chose our model without the text classification loss.
Ref.
Clf.
Mask
Text
Overall
Easy
Hard
View-dep.
View-indep.
40.6 %
48.2 %
32.8 %
38.5 %
41.6 %
44.1 %
51.3 %
36.7 %
41.0 %
45.6 %
V
40.0 %
47.2 %
32.5 %
36.8 %
41.5 %
/
40.0 %
48.7 %
30.8 %
37.8 %
40.9 %
43.9
51.0 %
36.6 %
41.7 %
45.0 %
41.0 %
49.7 %
31.9 %
39.3 %
41.8 %
40.0
48.2 %
31.5 %
38.9 %
40.6 %
Table 3: Ablation of loss terms Nr3D. The classification loss effective, the mask loss did not significantly affect accuracy, and the text loss degraded accuracy. We chose the model without text losses (fifth row, in blue).
4.6 Viewpoint Annotation
View-dependent utterances without information about original viewpoint make the reference task in ReferIt3D [2] more challenging. For instance, utterances such as "The door is wood with the handle on the left side." assume specific orientations of the agent, and it is impossible to recover the true orientation without knowing the referred object; this differs from view-dependent utterances with explicit viewpoint information, such as "Facing the foot of the bed." However, the original dataset
7

---

Correction
Overall
Easy
Hard
View-dep.
View-indep.
Training Evaluation
43.5 %
50.7 %
36.0 %
37.0 %
46.6 %
49.0
56.0
41.8
54.4
46.4 %
43.1 %
50.3 %
35.6 %
40.4 %
44.4 %
43.9
51.0 %
36.6 %
41.7 %
45.0 %
Table 4: Comparison of accuracy with and without corrected orientations on Nr3D.
(a) An example of stan- dard orientation I.
(b) An example of stan- dard orientation
(c) An example of stan- (d) An example of stan- dard orientation
dard orientation
Figure 3: Examples of standard orientations for viewpoint annotation Nr3D (a-d). Wc assume that the robot is always inside the room except for cases specified by utterances.
of Referlt3D [2] does not distinguish the utterances without orientation information from those with
it. Therefore, we split the view-dependent (VD) utterance category two subcategories, VD- explicit and VD-implicii, where VD-explicci has explicit viewpoint information in the utterance. We then collected orientations from human annotators that validated the utterances. We set four standard orientations assuming the agent is in the room (around the center of the scene) and asked annotators to select all orientations that could be considered valid from the utterance. Figure shows examples of the four orientations. We found that four orientations were sufficient to recover the original viewpoints of the speakers. In total, 12,680 view-dependent utterances the Nr3D dataset were annotated; from these, 5,942 utterances were classified VD-explicit. For train and test split, 10,206 and 2,474 utterances were annotated, respectively.
From the orientation for siew-dependent utterances, we revised the dataset with the corrected ori- entation; we rotated the scene with respect to the annotation so all scenes remained valid at the canonical orientation. For view-independent utterances, were valid in any direction. Table shows the accuracy values for models trained with and without corrected orientations. At inference, we evaluated each model with and without corrected orienta- tions, as well. The second row shows the accuracy the model that was trained and evaluated with corrected orientations. Its overall accuracy was improved by +5.1% from the final model without correction that was used in Table (the last row Table 4). Note that the improvement on view- new-independent utterances was significant (+ (+172.%). The first row shows the accuracy of a model trained with corrected orien- tations and evaluated on the test data without correction. This model achieved accuracy comparable to the final model (-0.4%) This implies the correction helped the model to accurately interpret the view-dependent scene when the orientation was consistently aligned, introducing unwanted bias.
randomly rotated the scenes since they
marginal (+1.4%), but the improvement
view-dependent ones
5 Conclusion
We proposed LanguageRefer, a LanguageRefer combines language embeddings from utterances and class labels with positional- encoded spatial information for efficient learning of the spatial-language embedding space without different modules for individual modality. Experimental results show that LanguageRefer outper- formed state-of-the, are models on ReferIt3D with no additional training data. Analysis and ab- lations we performed demonstrate the effects of 1) noisy class labels, 2) arbitrary viewpoints in view-dependent utterances, 3) and the transfer of our model to different datasets for future robotics applications.
model for 3D visual grounding in a reference task.

---

Acknowledgments
We would like to thank to Xiangyun Meng and Mohit Shridhar for discussion and feedback. This work is in part supported by NSF IIS 1652052, ISS 17303 DARPA 16-2001-1---4031, DARPA W91 19F1N-5--1-054
Honda Research Institute part of the Curious Minded Machine initiative, and gifts from Allen Institute for Artificial Intelligence.
References
[1] V. Sanh, I.. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[2] P. Achlioptas, A. Abdelreheem, F. Xia, M. Elhoseiny, and L. Guibas. Referit3d: Neural listen- ers for fine-grained 3d object identification in real-world scenes. 16th European Conference on Computer Vision (ECCV), 2020.
[3] J. Johnson, B. Hariharan, L. van der Maaten, L.. Fei-Fei, C. L. Zitnick, and R. Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[4] K. Yi, Gan, Y. Li, P. Kohli, Wu, A. Torralba, and Tenenbaum. CLEVRER: collision events for video representation and reasoning. In ICLR, 2020.
[5] R. Girdhar and D. Ramanan. CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning. In ICLR, 2020.
[6] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra. Embodied question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1-10, 2018.
[7] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. arXiv preprint arXiv:2103.00022 2021.
[8] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10737- 10746. 2020.
[9] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
[10] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5828-5839, 2017.
[11] D. Ding, F. Hill, A. Santoro, and M. Botvinick. Object-based attention for spatio-temporal rea- soning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08500 2020.
[12] A. Kamath, M. Singh, Y. LeCun, I. Misra, G. Synnaeve, and N. Carion. Adetr-modulated detection for end-to-end multi-modal understanding. arXiv preprint arXiv:2104. 12763, 2021.
[13] Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. V. Hengel. Vision-and-languuaee navigation: Interpreting uisuall--grundder navigation in- structions in real environments. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3674-3683, 2018
[14) A. Ku, P. Anderson, R. Patel, E. Ie, and Baldridge. Room-across-room: Multilingual vision- and-language navigation with dense spatiotemporal grounding. In EMNLP. 2020.

---

[15] Y. Qi, Q. Wu, P. Anderson, X. Wang, W. Wang, C. Shen, and A. V. Hengel. Reverie: Remote embodied visual referring expression real indoor environments. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9979-99999 2020.
[16] K. Chen, J. K. Chen, J. Chuang, M. V'azquez, and S. Savarese. Topological planning with transformers for vision-and-language navigation. ArXiv,
2020.
[17] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A platform for embodied ai research. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9338-93444 2019.
[18) C.-Y. Ma, J. Lu, Z. Wu, G. Al-Regib, Z. Kira, R. Socher, and C. Xiong. Self-monitoring navigation agent via auxiliary progress estimation. ArXiv, abs/1901.03035, 2019.
[19] L. Ke, X. Li, Y. Bisk, A. Holtzman, Z. Gan, J. Liu, J. Gao, Y. Choi, and S. Srinivasa. Tactical rewind: Self-correction via backtracking in vision-and -ad--anlllanguaee navigation. 2019 1EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6734 6742,66424211 2019.
[20] P. Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. Zamir. On evaluation of embodied navigation agents.
ArXiv, abs//807..n 000118770067777 .07757677772 2018.
[21] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg- Kirkpatrick, K. Saenko, D. Klein, and T. Darrell. Speaker-followed models for vision-and lan-andd--n navi- gation. In NeurlPS, 2018.
[22] Y. Hong, Q. Wu, Qi, C. Rodriguez-Opaza and S. Gould. recurrent vision-ande -ad---nn-llnnnng and-language bert for navigation. ArXiv, abs/2011.13922 2020.
[23] A. Shrivastava, K. Gopalakrishnan, Y. Liu, R. Piramuthu, G. Tur, D. Parikh, and D. Hakkani- Tur. Visitron: Visual semantics-aligned interactively trained object-navigator. ArXiv, abs/2105.1 1589, 2021.
[24] P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee. Sim- to-real transfer for rision-and-langeee navigation. arXiv preprint 20XXx:201I.03300 2020.
[25] Krantz, Wijmans, A. Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: Vision- and-language navigation in continuous environments. In European Conference on Computer Vision, pages 104-120. Springer, 2020.
[26] V. Blukis, R. A. Knepper, and Y. Artzi. Few-shot object grounding and mapping for natural language robot instruction following. ArXiv, abs/201 01/22011.0730 1.07384, 2020.
[27] J. Roh, C. Paxton, A. Pronobis, A. Farhadi, and D. Fox. Conditional driving from natural language instructions. ArXiv, abs/1910.0761. 2019.
[28] M. Z. Irshad, C.-Y. Ma, and Z. Kira. Hicrarchical cross-modat agent for robotics vision-and- language navigation. ArXiv, has/2104.10674, 2021.
[29] B. A. Plummer, L. Wang, C. M. Cervantes, J. Caicedo, Hockenmaier, and S. Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. International Journal of Computer Vision, 123:74-93, 2015.
[30] S. Kazemzadeh, V. Ordonez, M. Matten, and T. L.. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP. 2014.
[31] J. Mao, J. Huang, A. Toshev, O.-M. Camburu, A. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 11-20, 2016.
[32] L. Yu, Z. L. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg. Mattnet: Modular attention network for referring expression comprehension. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1307-1315, 2018
10

---

[33] S. Yang, G. Li, and Y. Yu. Dynamic graph attention for referring expression comprehen- sion. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4643- 4652, 2019.
[34] L. Yu, P. Poirson, S. Yang. A. Berg, and T. L. Berg. Modeling context in referring expressions. ArXiv, hbb//600.00272, 2016.
[35] Y. Liu, B. Wan, X.-D. Zhu, and X. He. Learning cross-modal context graph for visual ground- ing. ArXiv, (bss/1911t 9111000422 2020.
[36] L. Ye, M. Rochan, Z. Liu, and Y. Wang. Cross-modal self-attention network for referring image segmentation, 2019.
[37] D. Z. Chen, A. X. Chang, and M. Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. 16th European Conference on Computer Vision (ECCV), 2020.
[38] Z. Yuan, X. Yan, Liao, R. Zhang, Z. Li, and S. Cui. Instancerefer: Cooperative holistic understanding for visual grounding point clouds through instance multi-level contextual referring, 2021.
[39] M. Feng. Z. Li, Q. L. Zhang, X. Zhang, G. Zhu, H. Zhang, Y. Wang, and A. Mian. Free- form description guided 3d visual graph network for object grounding point cloud, 2021.
[40] Z. Yang, S. Zhang, L. Wang, and Luo. SAT: 2d semantics assisted training for 3d visual grounding. CoRR, abs/2105. 11450, 2021. URL https: //apx///////xv arggab///////a 11450,
[41] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, I.. Kaiser, and I. Polo- sukhin. Attention is you need. arXiv preprint arXiv: vwXivv:776.03: 00700000000222. 2017.
[42] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018.
[43] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. In NIPS, 2017.
[44] A. Akbik, T. Bergmann, D. Blythe, K. Rasul, S. Schweter, and R. Vollgraf. Flair: An easy-to- use framework for state-of-the-art nlp. In NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 54-59 2019.
[45] Wolf, L. Debut, Sanh, J. Chaumond, Delangue, Moi, Cistac, Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jemite, J. Plu, C. Xu, T. L. Scao, Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Huggingface's transformers: State- of-the-art natural language processing, 2020.
[46] Achlioptas.
Referit3d benchmarks.
URL https://referittd. gi github.c.. 1o/
benchmarks html.


Reviews:
[{"review_id": "ygHQlS9Q7P_", "reviewer": "Reviewer_PcsE", "summary": "This paper proposes a model to address the 3D visual grounding problem. Given a set of candidate 3D bounding boxes, the goal is to identify the object being referred to by a natural language expression. The proposed model combines spatial embeddings of the bounding boxes with a linguistic embedding from DistilBert. The resulting model is shown to outperform several baselines. In addition, the authors show that viewpoint-specific information is critical to grounding in 3D environments where the spatial information assumes a viewpoint for the speaker.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "One strength of this paper is the extensive set of comparisons and ablations, to which the proposed model achieves strong performance. The paper evaluates several design choices such as how important using ground truth class labels are and the relative importance of different components of the loss function. One particularly valuable insight is the importance of using a viewpoint that is consistent with viewpoint-dependent language. This is common for situated environments and is a valuable insight to the community.\n\nThe main weaknesses of this paper result from missing details needed to fully appreciate the take-aways of the proposed model. For example, I was not able to find descriptions of the baselines used in Table 1. This makes it difficult to state which component of the proposed model most contributed to the performance gain. It would be helpful to see these descriptions that highlight the key differences with the proposed model in Section 4.2 (or in the appendix if space does not allow). Relatedly, although the paper presents many interesting ablations, it would be additionally informative to compare the relative contributions of both the spatial encodings and the pretrained language model. \n\nAdditionally, the following points addressing the clarity of the paper would further strengthen the contribution:\n- Visual information only factors into the prediction through class labels. This decoupling is shown to help the approach transfer to different datasets (Table 2). It would strengthen the comparison to show how a coupled perception-language model transfers between these datasets but perhaps this is outside the scope of this work. It is still an interesting result that the model performs well without further information and it would be useful to discuss any limitations that arise from this assumption.\n- Regarding the loss function ablations, when the text loss and classification losses are left out, does the inference procedure described on line 171 change? Which combination of loss functions are used for the Table 1 evaluations?\n- What is the value of “k” in the experiments? (Line 172)\n- It would be helpful to describe the differences between the “Easy” and “Hard” splits in the result tables.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend this paper be accepted to CoRL. The importance of viewpoint is an interesting contribution and I believe the identified issues can be addressed by the authors.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The main issues with this work are missing details mentioned above. Adding these would greatly help the reader recognize the key contributions of this work.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "pQAUedEMqOz", "reviewer": "Reviewer_3bBQ", "summary": "For a given 3D point cloud of a scene, 3D bounding boxes, and language instructions, the method aims to identify the target object from a set of potential candidates. To achieve this, four modules (classification, spatial embedding, language embedding, and spatial-language model) have been used in the system. The method was evaluated on the ReferIt3D dataset (containing Sr3D and Nr3D datasets), and the results show that the suggested system outperforms the baseline models. Further, the experiments present that the method performs better for the expressions with spatial references (Sr3D) than natural references(Nr3D), which include the color features or appearance of the objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper addresses a critical topic and different important subtopics. It focuses on understanding language instructions in 3D. It further discusses the importance of viewpoint orientations and the impacts of perception noise in spatial reasoning tasks. These issues have been less focused in computer vision studies, and they are critical in robotics applications.\n- The system is evaluated in detail.  It is compared with five previous approaches for Nr3D and four existing approaches for Sr3D. Further ablations, additional results with ground-truth classes, and more evaluations with view-dependent and independent expressions are also presented in the experiments section.\n\nWeaknesses:\n- The suggested system is complex, composed of many different modules, and used four different losses. In this case, failure in one module can mislead the main prediction and might cause identifying the wrong object as a target. Further, in ablation studies, the paper shows that the mask loss does not affect the accuracy much, and the text loss decreased the accuracy. Therefore, it is not clear why these components are suggested as parts of the proposed system.\n- The method focuses on understanding spatial references, and it mainly considers the positional embeddings. Accordingly, it obtains lower accuracies in the Nr3D dataset than the Sr3D dataset since the Nr3D dataset contains more features such as color and shape. However, in many robotics applications, only spatial relations might not be enough to disambiguate the target objects, and users might use varying features while describing the target objects. Such a system might fail when an object is described only with its appearance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the suggested system is complex and mostly focuses on spatial references, it includes critical analyses about viewpoint orientations and perception noise, and these are crucial aspects for robots to build the bridge between the natural language and vision in real-world environments (see strengths and weaknesses for details). ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- The paper does not emphasize enough what has been achieved in the 3D visual grounding and what are the contributions of the suggested system over existing approaches. The previous approaches on comprehending referring expressions in 3D aimed to identify the target object among candidate objects, as in this paper. It would be nice to focus more on the novel aspects of the suggested system.\n- What is the number of object classes that can be identified by the semantic classifier? Can it classify all the objects categories in the ReferIt3D dataset?\n- It is not clear how the spatial embedding is obtained in the system. The paper states that positional encoding PE(·) is employed to transform the bounding box vector, but it is not straightforward to understand how PE(·) converts bounding box vectors to spatial embedding.\n- The target classification loss is defined as L_tar, but it is used as L_clf for the rest of the paper.\n- Did you fine-tune the baseline methods (i.e., the ones used for comparison in Table 1) on the ReferIt3D dataset?\n- What are the easy and hard categories reported in the tables? It has been forgotten to explain in the paper.\n- The authors state that the model trained with corrected orientations achieved comparable accuracy to the best model even without corrections at inference (+0.4%, −0.4%). Which columns of Table 4 were compared to obtain these values?\n- Figure 2 is not easy to follow. Spatial embedding representation might be positioned closer to DistilBert to reduce the length of arrows."}}, {"review_id": "bltO-c720_", "reviewer": "Reviewer_rNyP", "summary": "This paper presents a model for grounding referring expressions to a 3D point cloud scene. Authors take the approach of encoding the language component and the the 3D scene and decode the referring object using a combination of losses. The 3D scene encoder uses a standard model for predicting the 3D bounding boxes and a classification label. The proposed approach treats the classification outputs as a sentence and determines a language-encoding for the same. The positional encoding is then encoded as a spatial embedding which is later combined with the language embedding prior to decoding the object labels. The model uses a combination of loss functions to arrive at the target object. Authors present thorough experiments on a number of benchmark data sets. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- The authors address an important problem in human-robot interaction. \n- The model uses a realistic 3D point cloud data as input. \n- The approach also shows robustness to the orientation of the person providing the instruction. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper contributes an approach for grounding language understanding in 3D scenes. The results are comprehensive. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- The approach directly processes the predicted classification labels and bounding boxes without explicitly modeling uncertainty in the classification labels or the geometry of the bounding boxes. \n- How extensible is the approach to richer inter-object relations. For example, \"bring me the cup in the microwave\" where an object can contain another object referred to by the instruction. "}}, {"review_id": "Uum8FVXZCdF", "reviewer": "Reviewer_tKLL", "summary": "This work develops a novel model for 3D visual grounding based on a 3D point cloud representation of a scene and a language utterance. This model simplifies earlier method by using pretrained language embeddings and a decoupled perception system, the method also also outperforms earlier methods on 2 datasets. Ablation studies motivate the design choices for this model also analyze the view dependence of the two Nr3D and Sr3D. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The transformer based, decoupled LanguageRefer model is well described and it is clear why all the components are needed. \n* The ablations with perception noise help understand how much of the performance is purely from perception which may motivate further improvements\n* Thorough description of experimental results with several baselines means the results are convincing.\n\nWeakness\n* The improvements on Nr3D seem convincing but are incremental, 41.7% vs 43.9%\n* Some error analysis would help understand in which ways is LanguageRefer better than previous models? Are there any particular cases that it gets that InstanceRefer does not? \n* The performance on Nr3D is still quite low, even with ground truth perception, why is this the case? What are some examples of errors that might motivate further improvements.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work proposes an incremental improvement for 3D visual grounding with a simpler perception-decoupled transformer based architecture. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* Some error analysis of baselines and LanguageRefer on the Nr3D dataset to show examples why LanguageRefer is qualitatively better, also some analysis of the remaining errors.\n* Figure 2, there is a token \"###acing\" which is confusing, is this a typo?\n* Numbers in tables would be more convincing if they had errors reported as well\n* Table 4, mention the dataset in the caption for consistency."}}]

Author rebuttal:

Meta-review: All reviewers agree that the visual grounding is an important and interesting topics and that the proposed approach, taking advantage of 3D visual data and language instructions, is promising.

However, the paper could make an even better case for the proposed method by improving clarity and by discussing the robustness, scalability and the limitations of the approach in further detail.

## Post-Rebuttal Update
The authors addressed and clarified the concerns raised by the reviewers, improving the paper in the process.
Decision: Accept (Poster)