Language Grounding with 3D Objects
Authors: Jesse Thomason
Mohit Shridhar
Yonatan Bisk
Chris Paxton
Luke Zettlemoyer
Venue: CoRL2021 Poster (2021)
Keywords: Benchmark
Language Grounding
Vision
3D

Abstract: Seemingly simple natural language requests to a robot are generally underspecified, for example "Can you bring me the wireless mouse?" Flat images of candidate mice may not provide the discriminative information needed for "wireless." The world, and objects in it, are not flat images but complex 3D shapes. If a human requests an object based on any of its basic properties, such as color, shape, or texture, robots should perform the necessary exploration to accomplish the task. In particular, while substantial effort and progress has been made on understanding explicitly visual attributes like color and category, comparatively little progress has been made on understanding language about shapes and contours. In this work, we introduce a novel reasoning task that targets both visual and non-visual language about 3D objects. Our new benchmark ShapeNet Annotated with Referring Expressions (SNARE) requires a model to choose which of two objects is being referenced by a natural language description. We introduce several CLIP-based models for distinguishing objects and demonstrate that while recent advances in jointly modeling vision and language are useful for robotic language understanding, it is still the case that these image-based models are weaker at understanding the 3D nature of objects -- properties which play a key role in manipulation. We find that adding view estimation to language grounding models improves accuracy on both SNARE and when identifying objects referred to in language on a robot platform, but note that a large gap remains between these models and human performance.

Language Grounding with 3D Objects
Jesse Thomason" University of Southern California
Mohit Shridhar* University of Washington
Yonatan Bisk
Chris Paxton NVIDIA
Luke Zettlemoyer University of Washington
Carnegie Mellon University
Abstract:
Seemingly simple natural language requests to a robot are generally underspecified, for example Can you bring me the wireless mouse? ? Flat images candidate mice may not provide the discriminative information needed for wireless. The world, and objects in are not flat images but complex 3D shapes. a human requests an object based on any of its basic properties, such color, shape, or texture, robots should perform the necessary exploration to accomplish the task. particular, while substantial effort and progress has been made on understanding explicitly visual attributes like color and category, comparatively little progress has been made understanding language about shapes and contours. In this work, we introduce a novel reasoning task that targets both visual and non-visual language about 3D objects. Our new benchmark ShapeNet Annotated with Referring Expressions (SNARE) requires a model to choose which of two objects is being referenced by natural language description." We introduce several CLIP-based [1] models for distinguishing objects and demonstrate that while recent advances in jointly modeling vision and language useful for robotic language understanding, it is still the case that these image-basee models are weaker at understanding the 3D nature of objects properties which play key role in manipulation. We find that adding view estimation to language grounding models improves accuracy on both SNARE and when identifying objects referred to in language robot platform, but note that large gap remains between these models and human performance.
Keywords: Benchmark, Language Grounding, Vision, 3D
Introduction
Joint language and vision models often trained image captions which have a bias towards canonically "visual" attributes of objects, such color, rather than functional ones like shape. Image captions omit properties that require understanding objects as 3D, rather than flat, concepts. this work, we show how the effect of this disconnect is that while robotics research has benefited greatly from advances computer vision, aadion-and-language models cannot always be directly applied to robotics. People use geometric and physical properties of objects when describing them. For example, robot carrying out request to Bring me the mug with the wide handle needs to be able to spot the wide handle, even the mug rests on countertop with the handle oriented out of view, but a robot can rotate such an object to investigate in further detail. Generally, identifying objects based on natural language is more challenging from non-canonical viewpoints, which are common for robots in home, office, and industrial environments.
Choose the mug with
handle
Rotate (B)
no logo
Rotate (A)
Botata (B)
How?
Potate thnii
Figure 1: Deciding between 3D objects described in natu- ral language involves rotating objects to see multiple faces.
'Equal contributions. Ptttps://gt httpggttthhuu.tee hww.hhubb.ommeeeeeeeeeeeeeeeeeee http://///nneeeettttteeeeeeeee
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

The codification and commodification of ResNet [2] architectures pretrained on ImageNet [3] has yielded immense progress in off-the-shelf computer vision techniques. Such representations provide strong foundation for visual reasoning. With the introduction of pretrained Transformer [4] archi- tectures like BERT [5], natural language processing has similar off-the-shelf tools for representing language. Modern visual language grounding is done through massive, internet-scale pretraining to combine these strengths: aligning images with their captions and content descriptions in natural language using Transformer models [6, 1, 7, 8].
However, internet images suffer from type of reporting bias. These images are captured by sighted humans, from human-centric vantage points [9]. There is a domain shift between such canonical depictions of objects in images and objects as seen through a robot camera [10, 11], as studied in computer vision between such images and those captured by the blind [12]. For example, images of mugs online almost exclusively feature a full or wild can easily be oriented such that the handle is occluded from the camera by the mug body.
view of the handle, while a mug the
person requesting mug with wide handle has a mental model of the referent object that includes its full 3D spatial information; humans do not imagine objects only in 2 dimensions. Further, someone trying retrieve said mug will differentiate it from other mugs on the countertop by viewing the mugs from different angles to inspect the handles.
Our goal is to similarly imbue robots with three dimensional notions of language grounding, by encouraging them to rotate an object when selecting the referent of a language description. We introduce the ShapeNet Annotated with Referring Expressions (SNARE) dataset, which provides discriminative natural language descriptions of 3D ShapeNet [13] object models. SNARE includes both visual descriptions that focus on colors and object categories and blindfolded descriptions that focus on shapes and parts, We finetune large off-the-shelf models to select the correct object given its description, and find that estimating the input views as an auxiliary prediction improves accuracy.
We introduce Language Grounding through Object Rotation (LAGOR) which performs view esti- mation as auxiliary loss to encourage 3D object understanding (Section 4) during the SNARE task. We find that using multiple views to select the correct object improves over single-view object selection, using the large-scale vision-and and-language CLIP [1] model as backbone to score how well an image and language MATCH. We show that models taking in multiple views of objects objects, while performing auxiliary view estimation, can result in accuracy approaching that of models consuming panoramic views of objects, while also being more realistic for a robot.
Our key contributions are:
SNARE, benchmark dataset for identifying 3D object models given natural language referring expressions including visual and blindfolded descriptions;
Baseline models for SNARE demonstrating both zero-shot and fine-tuned performance of state- of-the-art vision- and languagg models; and LAGOR, an initial SNARE model that looks at two random views of an object, estimates the angle of those views, then makes a referent prediction, which we demonstrate using robot.
2 Related Work
Using natural language work with robot partners is a long-standing goal in robotics [14]. We argue that internet-scale, pretrained
models offer powerful starting point for human-robot collaboration. Unlike 2D internet images, physical objects can be picked up and moved by robot agents. Agents can perform information-seeking behaviors on objects to improve their world [15, 16, 17] and language understanding [18, 19, 20J. Such world interaction is inextricable from language grounding [21], motivating language annotations for higher-fidelity referents than static images. We introduce SNARE, comprised of language referring expressions for 3D objects, and the LAGOR model to combine language and multiple object views to achieve better 3D understanding. SNARE extends lines of work tying language to static images, 3D object models, and even physical objects. We demonstrate that LAGOR generalizes to physical objects manipulated by tabletop robot $5.2. LAGOR is inspired by models that perform information-seeken actions informed by learned world models better achieve goals.
Image-based Object Identification Object classification [3, 2] is the first step towards image cap- tioning and visual question answering [22]. Particular object instances can be found with region seg-

---

Glot is similar in spirit, it contains train- ing data only for chair objects, and tests on four additional categories. By contrast, mentation models such as MaskRCNN [23], enabling object referent tasks such as Guess What!? [24]. Combining visual recognition with Transformer-basee language understanding to jointly attend to language and visual tokens leads to improved downstream performance on many Vision-and-lanuuaa tasks [6, 25, 8]. Joint embedding approaches that learn a shared subspace for representing language and vision tokens [26] also achieve state-of-the-ar performance when trained at scale [1]. We show that these large-scale, pretrained models fall short of human performance on the SNARE task, and that synthesizing 2D views from multiple vantage points improves object identification performance.
Language Grounding in 3D Prior works have associated single word attributes with 3D object mod- els based on latent representations of 3D meshes [27]. To learn spatial language, vision-and-lange lan-lagg navigation (VLN) [28] models infer navigation actions from instructions and visual observations in 3D simulated worlds. Such tasks can be extended to include simulated world interactions such as picking and placing objects [29] and using appliances and tools [30]. Models for these tasks extend Transformer representations to include action taking [3] and invoke object classification methods to create semantic understanding of the world [33]. SNARE poses complementary challenge, providing data for selecting referent objects in the presence of distractors by taking into account the multiple views possible of objects in 3D space.
The work most similar to ours is Shape- Glot [34]. a collection of referring expres- sions for ShapeNet objects to discriminate between two distractors. While Shape-
Data
Fold
# Cats
# Objs
# Ref Exps
ShapeGGot
Chairs
1
4,511
78,789
Other
200
400
Total
5
4,711
79,189
Train
207
6,153
39,104
SNARE
Val
7
371
2,304
SNARE spans 262 distinct object cate- gories (Table 1). ShapeGlot focuses a
Test
48
1,357
8,751
model's abilities to learn particular parts of objects, such as chair arms and legs, while SNARE aims to ground language more generally to 3D features of objects span- ning color, shape, category, and parts. Further, ShapeGlot models take in pointcloud representation of objects along with visual features, where LAGOR model does not assume access the 31) model of an object and instead operates solely on different 2D views achieved through object rotation.
Total
262
7,881
50,159
Table Fold summaries in SNARE, with the Shape- Glot benchmark size for contrast.
Physical Object Identification Modeling the connections between language to physical objects enables robots to identify objects for manipulation by color, shape, and category [35, 36). The physical forces and sounds objects make during manipulation actions can also be associated with words such as ratiling and heavy for multimodal understanding beyond vision [19, 37J. Prior work has gathered language annotations for the YCB Benchmark object set [38] to explore how language descriptions provide priors on object affordances [39]. In our tabletop robot experiments, we use camera views of novel objects evaluate zero shot transfer of LAGOR to robot camera views of real objects, which are processed by an off-the-shelf segmentation algorithm.
Information-Seekknng Actions Embodiment, whether in simulation or the physical world, affords agents the opportunity to seek out new information to help with a given task. By predicting the new information that can be gotten from different actions, agents can prospect over potential futures for planning [40, 41, 42]. the aforementioned VLN task, predicting what will be seen along different potential routes facilitates more efficient navigation [43]. Our LAGOR model estimates the angles of object input views, encouraging 3D understanding in the learned object representations.
3 ShapeNet Annotated with Referring Expressions (SNARE)
We introduce SNARE, as a new benchmark for grounding natural language referring expressions to distinguish 3D objects. The annotations are collected to complement the ACRONYM' [44] grasping dataset and include language that targets both visual and tactile attributes of objects. Our goal is to enable future research to ground both from multi-view vision, as in this work, and directly from robotic grasp contact point data (Section 6).
Yess that dataset title is "ACRONYM" [44].
3

---

Visual
Tea
Cylindrical cup
Semi-spherical cup
White mug
Round coffee mug
Cylindrical coffee mug
Blindfolded
Pac Man shaped handle
Thick C shaped handle
Rounded bottom
Flat bottom
Handle is semi-circular
Has rectangular handle
Figure Example object pair and object referring expressions in SNARE. Given a referring expression and one (or more) views of two contrasting objects, model must decide which object being referenced by the language. Language annotations are collected in two forms: Visual and Blindfolded. The latter expressions tasked annotators with describing objects tactilely.
To construct SNARE, we select subset of 7,897 ACRONYM [44] object models from ShapeNet- Sem [45, 13). We obtain over 50,000 natural language referring expressions in English for these object models via Amazon Mechanical Turk (AMT). Figure gives an example of referring expressions collected for two mug objects used distinguish between the two.
To elicit referring expressions with high specificity, we frame the annotation as discriminative task. AMT workers were presented with two object models side-by-side from the same ShapeNet category, for example two different OutdoorTabe abble object meshes, and asked to complete sentences like
The way to tell Object A from Object B is that Object A looks like a(n) Blindfolded, the way to tell Object from Object B is that Object B is a(n)
with referring expressions. We displayed the object models as GIFs rendered to give a 360 rotating view of each object. The resulting SNARE task asks models take in one such referring expression and decide whether it applies to Object A or Object B.
Priming in the annotation prompts invokes visual features looks like...) and non-visual features (Blindfolded, ...) in the referring expressions. By pairing objects from the same ShapeNet category, referring expressions must go beyond categorical information like brown dresser differentiate one object from the other with higher specificity, similar to ShapeGlot's s choice to use distractor objects from single training category [34]. We collected six referring expressions per object-three primed to be visual and three primed to blindfolded. Priming for visual expressions mentioned the name of the object, shapes, and colors, while for blindfolded mentioned shapes and parts. Each referring expression was vetted through a secondary task on AMT where, given the language expression, workers had to correctly select the referent object. Every referring expression in SNARE was correctly associated to its referent object by a majority of such annotators.
SNARE referring expressions average 4.27 words, with blindfolded expressions longer (4.95) than visual expressions (3.63). We estimated that blindfolded expressions use more shape words (14% vs 5% of all words) while visual more color words (22% vs 1% of all words), by traversing the WordNet [46] hierarchy for each word and noting whether it is a hyponym of color or shape. This distribution difference is consistent with the priming used to elicit visual and blindfolded expressions.
Each SNARE instance is tuple of (referring expression, Object A, Object B), and models must select which of Object A B the referent of the natural language expression. We split these data instances into train, validation, and test folds by ShapeNet category. We ensure that closely related and AccentTabl
categories such as 2She ves and 3She ves or the same fold. For example, all shelf-related and bed-related categories are sorted into the train and test folds, respectively, that inter-category information does not leak across folds. More details can be found Section 7.3. Table summarizes the overall data statistics.
within
Methods
There has been recent proliferation of off-the-shelf, increasingly large-scale pretrained vision and language alignment models applicable for language grounding in robotics [1, 6. 7, 8, 47]. We set out to answer two questions about such models, using SNARE as a testbed:
ASection 7.1 contains additional details about the AMT study.

---

90% O
180 6
Visual red gray and white display case MATCH score after rotation + 1388% %
Blind cube botton with knabs
MATCH score after rotation +246%
135
45
Visual red white and blue device MATCH score after rotation +131%
Blind -nghhstand with smooth flat top and four legs MATCH score after rotation +205%
increases after performing object rotation. Rotations involve exposing colors (top left, bottom left), canonical faces (top right), and parts (bottom right). robot encountering objects in the wild will find initial object orientations that do not line up with natural language referring expressions for those objects, as reflected in SNARE.
Figure 3: Example MATCH
Can existing language and vision models ground language in the SNARE benchmark? Do SNARE models generalize to a robot object selection task better than off-the-shelf models?
To answer the first question, first train MATCH module that learns predictive head top of a CLIP [1] backbone, and evaluate this module on single and multiple views of 3D objects. To answer the second question, introduce the LAGOR model, which uses view estimation as an auxiliary loss while predicting language expression referent objects for SNARE, and evaluate on both SNARE $5.1 and physical robot platform $5.2. LAGOR examines an initial 3D object view and an additional, post-rotation view, striking a balance between single-view models and those that attempt to capture each object's entire 3D structure.
4.1 Language-Vin View Match (MAтCH) Module
The MATCH module takes in language expression L and single view of an object, V, and produces match score s(L, V,). To decide whether Object A or Object B is the referent a language expression on SNARE, interpret (A.B) Vi,o) the model's prediction.
CLIP [1] serves as backhone which we add additional, learnable layers for the SNARE task. We use CLIP's transformer-based sentence encoder to extract language features : IR 1=52 IR. x512 and image Vi,+. VLB RI5512 These modality-specifie feature vectors are concatenated: [ViA:l] : RR 1x24 and [i Rxx1024 and independently run through learnable, multi-layer perceptron that gradually reduces the dimensionality from 1024 to 512, then 256, and finally to a single-dimensional score This final score comparison training mirrors the multiple-choice formulation used in existing unimodal [5] and multimodal transformers [7]. We keep the CLIP encoders frozen during training.
We use CLIP VIT-B/32 to extract visual features for image ViA, ViA
ShapeNet objects are 3D meshes, incompatible with the 2D input expected by off-the-shell vision and language models. We sample eight rendered viewpoints around each object 45 degree increments, and compare off-the-shelf CLIP and MATCH module accuracies when considering single or multiple of these views. To aggregate multiple views, we maxpool over view embedding vectors ViA YJ.A-
MATCH is trained with cross-entropy- L,, to predict a binary label of whether the referring expression matches the object represented in the view image. MATCH is trained for 50 epochs SNARE, with object views chosen at random during each step.
Baselines. We compare fine-tuned MATCH against zero-shot CLIP classifier. Instead of fine- tuning CLIP, we use the cosine distance between visual and language features to pick the referred object. That is, we select argmax vLo- We also evaluate against trained ViLBERT baseline that consumes multiple object views at once, but find that it does not perform well as the MATCH
5

---

view = 2
Initial
CLIP VIT-B/32
A
CLIP VIT-B/32
view
"the chair with skinny legs"
CLIP Lang.
view 5
Initial
CLIP ViT-B/32
0 X
B
CLIP VIT-B/32
view
Figure LAGOR model relies the pretrained CLIP architecture backbone encoder across multiple views compared to the encoded referring expression. The model sees improvements from auxiliary losses that predict the orientation of the initial and final view.
module; more details about the ViLBERT baseline can be found in Section 7.4. The MATCH model can be thought of as a fine-tuned CLIP model; rather than back-propagating through the CLIP representation itself we learn additional tuning layers top.
4.2 Language Grounding through Object Rotation (LAGOR)
robot tasked with retrieving an object given a natural language expression should not have to gather 360 degree view of each candidate referent object to make decision. We have the intuition that estimating the input viewing angles can enable a model to develop a global reference frame for language grounding against 3D objects (Figure 3). Thus, we propose LAGOR, which takes in two views of each candidate object, performs view estimation on those views as an auxiliary prediction (Figure 4), then predicts the language referent.
In addition to predicting the object referent using a pre-trained MATCH module that considers two views, LAGOR predicts the input views of each object using cross-entropy loss, L2, against 1-hot vector representing a discrete set of 8 views at 45 degree offsets. LAGOR learns a multi-layer perceptron that takes in visual embedding V; and reduces dimensionality from 512 256 128 to 64 to vector of 8 logits representing the 8 discrete object views. The MATCH loss, L, is combined with the view estimation loss for final loss function L +.2
4.3 Robot Demonstration
We evaluate LAGOR trained on SNARE on a robot platform, taking two random views of objects and evaluating the robot's ability to select which is the correct referent of a language expression (Section 5.2). We compare LAGOR to the off-the-shelf CLIP model that considers single view of the object only, before performing any view-gathering rotations.
We set two objects on the workspace of a Franka Emika Panda with a D414.5 We capture initial view of the scene, then segment the objects from the workspace using the Unseen Object Clustering [48] algorithm. We feed the segmented objects as the initial views to LAGOR and CLIP. We move the to second vantage point above the workspace and capture another image of the objects and repeat segmentation to obtain the second views for LAGOR.
Intel Realsense
"httss//wwwfrankaat. and http://wwwwiineellleee...ooomeee respectivee

---

Validation Blind
Test
Model
Views
Visualc
All
Visualc
BlindC
All
ViLBERT
All
89,5
76.6
83.1
80,2
73.0
76,6
CLIP
All
83.7 #0.0
65.2 10.0
74.5 10.0
80.0 =0.0
61.4 =0.0
70.9 #0.0
MATCH
All
89.2 +0.9
75.2 10.7
82.2 10.4
83.9 10.5
68.7 =0.9
76.5 #0.5
CLIP
Single
79.0 +0.0
63.0 +0.0
71.1 +0.0
74.0 +0.0
59.7 +0.0
67.0 +0.0
MATCH
Single
88.4 +0.4
73.3 +0.6
80.9 +0.4
83.2 +0.3
68.0 +0.5
75.8 +0.3
CLIP
Two
81.0 #0.0
64.1 10.0
72.6 10.0
76.0 10.0
60.8 =0.0
68.6 10.0
MATCH
Two
89.2 +0.6
70.7
81.8 +0.4
83.7 +0.4
68.7 +0.5
76.4 +0.4
LAGOR
Two
89.8 +0.4
75.3 +0.7
82.6 +0.4
84.3 +0.44
69.4 +0.5
77.0 +0.5
Human (U)
All
94.0
90.6
92.3
93.4
88.9
91.2
Human (M)
All
100.0
100.0
100.0
100.0
100.0
100.0
Table 2: Accuracy on the SNARE benchmark. Mean accuracy standard deviation over 10 seeds. CLIP zero-shot accuracy is well above random chance, and adding more object views improves all models' performance. LAGOR, which needs only two views, making reasonable to deploy on robot platform, outperforms two-view MATCH statistically significantly on both the validation and training set, by adding level of 3D object understanding in the form of view estimation auxiliary losses. Human accuracy is conservatively calculated as the number of SNARE instances for which the correct referent object was identified by every voting annotator (Unanimous); for all SNARE instances majority of voting annotators correctly selected the referent (Majority).
5 Results
In this section we show that CLIP [1] and ViLBERT [6] select correct 3D object referents in SNARE substantially less often than the MATCH and LAGOR models. Then, we deploy CLIP and LAGOR on a robot tasked with selecting objects conditioned on natural language referring expressions, and find that view estimation and gathering additional object viewpoints improves referent identification.
5.1 Performance on SNARE
We train all models for 50 epochs on SNARE, with object views chosen random during each step, and report the best validation performance across those epochs, as well as the test performance at that best validation epoch checkpoint. We train each model 10 times with different random seed to estimate performance variance across training runs.
Table 2 gives LAGOR accuracy compared to other view alternatives, well as models taking in all views or single views. We compare to existing models trained (ViLBERT) and zero-shot (CLIP), as recommended in their respective papers. We do not re-run ViLBERT training because it is extremely time-consuming; the same reason we build on CLIP for our model development. To compare models, perform an unpaired, Welch's 's two-tailed r-test overall accuracy considering both visual and blindfolded subsets. We perform five such tests in all: for validation and test, LAGOR against two- view MATCH, LAGOR against all-view MATCH, and one additional pooling test (Section 7.5). We perform a conservative Bonferroni multiple-comparisse correction applied to threshold of < 0.05.
LAGOR statistically significantly outperforms wiew MATCH on both the validation (p 0.0008) and test (p = 0.0055) sets. The modelling difference between LAGOR and MATCH with two views only the view estimation auxiliary losses. LAGOR averages higher accuracy than the MATCH that uses all views on every metrie; however, these differences are significant for neither the validation (p 0.0844) test (p- 0.0315) set.
While no models achieve human level accuracy, the performance difference is particularly striking on the blindfolded referring expression subset of data, which lags 5-15% behind visual referring expressions across models. That gap supports our intuition that blindfolded referring expressions capture complementary and challenging linguistic space currently understudied for vision-language models but key to everyday manipulation. Thereby, SNARE opens interesting avenues for future work exploring the shape and grasp-points of objects.
'Note that with the Bonferroni correction, significance is reached only with < 0.01 given the desired threshold of 0.05 with tests run.
7

---

mag with the thin metal fradeee
LaGOR
Figure 5: With physical robot platform, LAGOR outperforms zero-shot CLIP for selecting tabletop referent objects whose views are captured via multiple camera angles and off-the-shelf segmentation.
Notably, zero-shot CLIP performance on SNARE, even when considering only one random view of each object, is well above random chance. While MATCH fine-tuning and LAGOR auxiliary losses improve accuracy over zero-shot CLIP, the cero-shot performance good indicator that massively pre-trained language and vision alignment like CLIP could be useful for robotics tasks for which there is little or no in-domain data.
5.2 Robot Results
We performed a set of experiments on a real robot (Figure 5). We run single view, zero-shot CLIP against LAGOR on 11 referring expressions and pairs of objects. Two objects time were placed on the table, and we selected two arbitrary viewing angles for each object. We used unknown object instance segmentation [48] to segment out each object. Zeroshot CLIP correctly selects the referent object 7 out of 11 times, while LAGOR selects the correct referent 9 out of 11 times. LAGOR corrects CLIP's mistakes on expressions like Mug with the thin metal handle where multiple views expose potentially occluded parts like handle, as well as when information gathering makes decisions more confident, such as Orange box when both objects have orange components, but one object is orange from all views. These experiments introduce a number of new sources of noise not present in the simulated setting, for example changes camera angle, shifting lighting during rotation, and automatic segmentation to account for cluttered scenes. For example, segmentation leads to the creation of voids inside of objects not present in SNARE. View extraction details, referring expressions, and extracted views are shown in Section 7.6.
6 Conclusions
We introduce ShapeNet Annotated with Referring Expressions (SNARE), a challenge task to ground language 3D object models. We show that fine-tuned, massively pretrained vision and language models fall short human performance at identifying object referents of natural language expressions by wide margin, while still achieving well above random chance performance (Table 2). Models improve as more 2D views of 3D objects are available, but a robot can only capture one view at time. We introduce Language Grounding through Object Rotation (LAGOR), model that considers two views of candidate referent objects trained with current view estimation as an auxiliary loss, and builds on a pre-trained CLIP model. We transfer the trained LAGOR model to a physical robot to study its generalization to noisy images of physical objects.
In the future, rotation policy could be learned to examine up to N views across M candidate objects subject to a referring expression, rather than performing random rotation to a novel view. By assigning rotation actions expected discriminative values using trained rotation module (though that could introduce a cyclic dependency with the MATCH module), and penalties based on the time it takes a particular hardware achieve object rotation, one could learn POMDP policy aggregating object views seen so far to decide whether to next obtain new view or make guess about the referent object. Obtaining different object views could be done either by picking up and rotating the object or by moving the robot camera to see the object from another angle. Further, since 3D meshes are available for each object, it may be possible to extract shape-level information using PointNet [49], similar to experiments attempted by ShapeGlot [34]. Because we target physical robot applications, we may be able to tie language to sets of graspable points encoding gripper orientation and position [44] for each object.
8

---

Acknowledgements
We would like to thank the anonymous CoRL 2021 reviewers for their feedback and suggestions. This paper was made stronger by the review and revise model adopted by CoRL this year via OpenReview. This work was funded in part by ONR under award 140209-405780.
References
[1] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. arXiv, 2021.
[2] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Computer Vision and Pattern Recognition (CVPR), 2016.
[3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Computer Vision and Pattern Recognition (CVPR), 2009.
[4] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
I. Polosukhin. Attention is all you need. In Neural Information Processing Systems (NeurIPS), 2017.
[5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
[6] Lu, Batra, D. Parikh, and S. Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-anddlanguagge tasks. In Neural Information Processing Systems (NeurIPS), pages 13-23, 2019.
[7] Y.C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and Liu. UNITER: Universal image-text representation learning. In European Conference Computer Vision (ECCV), 2020.
[8] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsich, and K.-W. Chang. VisualBERT: A simple and performant baseline for vision and language. arXiv, 2019.
[9] A. Torralba and A. A. Efros. Unbiased look at dataset bias. In Computer Vision and Pattern Recognition (CVPR), 2011.
[10] T. Kanade and M. Hebert. First-person vision. Proceedings of the IEEE, 100(3))444,-2453 2012.
[11] Anderson, A. Shrivastava, Truong, Majumdar, D. Parikh, D. Batra, and S. Lee. Sim- to-real transfer for vision-and lan---uag navigation. In Conference on Robot Learning (CoRL), 2020.
[12] D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, Vizwiz grand challenge: Answering visual questions from blind people. In Computer Vision and Pattern Recognition (CVPR), June 2018.
[13] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang. Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, I.. Yi, and F. Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv: www.:11120000010 [es.GR] Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
[14] S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek. Robots that use language. The Annual Review of Control, Robotics, and Autonomous Systems, 15, 2020.
[15] K. Takahashi, T. Ogata, J. Nakanishi, G. Cheng, and S. Sugano. Dynamic motion learning for dulti-dof flexible-joint robots using active-passive motor babbling through deep learning. Advanced Robotics. 100-0110022-105, 2017.

---

[16] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In International Conference on Computer Vision (ICCV), pages 2901-2910, 2019.
[17] T. Nagarajan and K. Grauman. Learning affordance landscapes for interaction exploration in 3D environments. In Neural Information Processing Systems (NeurIPS), 2020.
[18] F. Hill, S. Mokra, N. Wong, and T. Harley. Human fnstluction-folloiinng with deep reinforcement learning via transfer-learingg from text. arXiv, 2020.
[19] Thomason, A. Padmakumar, J. Sinapov, N. Walker, Y. Jiang, H. Yedidsion, Hart, Stone, and R. J. Mooney. Jointly improving parsing and perception for natural language commands through human-robot dialog. The Journal of Artificial Intelligence Research (JAIR), 67, 2020.
[20] C. Lynch and Sermanet. Grounding language in play. In arXiv, 2020.
[21] Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May, A. Nisnevich, N. Pinto, and J. Turian. Experience grounds language. In Empirical Methods in Natural Language Processing (EMNLP), 2020.
[22] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Computer Vision and Pattern Recognition (CVPR), 2018.
[23] K. He, G. Gkioxari. P. Dollár, and R. B. Girshick. Mask R-CNN. International Conference on Computer Vision (ICCV), 2017.
[24] H. De Vries, F. Strub, S. Chandar, O. Pietquin, H. Larochelle, and A. Courville. Guess- what?! visual object discovery through multi-modal dialogue. In Computer Vision and Pattern Recognition (CVPR), 2017.
[25] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee. 12-in-1: Multi-task vision and language representation learning. In Computer Vision and Pattern Recognition (CVPR), June 2020.
[26] C. W. Leong and R. Mihalcea. Going beyond text: A hybrid image-text approach for measuring word relatedness. In International Joint Conference on Natural Language Processing (IJCNLP). 2011.
[27] V. Cohen, B. Burchfiel, T. Nguyen, N. Gopalan, S. Tellex, and G. Konidaris. Grounding language attributes to objects using bayesian eigenobjects. In Intelligent Robots and Systems (RROS),015)
[28] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-andd-anguage navigation: Interpreting visually-grounded navigation instructions in real environments. In Computer Vision and Puttern Recognition ( (CVPR), 2018.
[29] J. Abramson, A. Ahuja, A. Brussee, F. Carnevale, M. Cassin, S. Clark, A. Dudzik, P. Georgiev, A. Guy, T. Harley, Hill, A. Hung. Z. Kenton, J. Landon, T. Lillicrap, K. Mathewson, A. Muldal, A. Santoro, N. Savinov, V. Varma, G. Wayne, N. Wong, C. Yan, and R. Zhu. Imitating interactive intelligence. arXiv, 2020.
[30] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In Computer Vision and Pattern Recognition (CVPR), 2020.
[31] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra. Improving vision- and-language navigation with image-text pairs from the web. In European Conference Computer Vision (ECCV), 2020.
[32] Y. Hong, Q. Wu, Y. Qi, C. Rodriguez-Opazo, and S. Gould. A recurrent vision-and -anon-aad-- BERT for navigation. arXiv, 2020.
[33] K. Singh, S. Bhambri, B. Kim, R. Mottaghi, and Choi. MOCA: A modular object-centric approach for interactive instruction following. arXiv, 2020.
10

---

[34] P. Achlioptas. J. Fan. R. X. Hawkins, N. D. Goodman, and L. J. Guibas. ShapeGlot: Learning language for shape differentiation. In International Conference on Computer Vision (ICCV), 2019.
[35] M. Shridhar, D. Mittal, and D. Hsu. INGRESS: Interactive visual grounding of referring expressions. The International Journal of Robotics Research (LJRR), 39(2-3)2217-222 2020.
[36] T. Kollar, S. Tellex, M. R. Walter, A. S. Huang, A. Bachrach, S. Hemachandra, E. Brunskill, A. Banerjee, D. Roy, S. Teller, and N. Roy. Generalized grounding graphs: A probabilistic framework for understanding grounded commands. arXiv, 2017.
[37] J. Thomason, J. Sinapov, M. Svetlik, P. Stone, and R. J. Mooney. Learning multi-modal grounded linguistic semanties by playing "I spy". In International Joint Conference on Artificial Intelligence (IJCAI), 2016.
(38) B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar. fale-cmu-berkeley dataset for robotic manipulation research. The International Journal of Robotics Research (IJRR), 36(3):261-265 2017.
[39] R. Scalise, J. Thomason, Y. Bisk, and S. Srinivasa. Improving robot success detection using static object data. In Intelligent Robots and Systems (IROS), 2019.
[40] X. Zhang, J. Sinapov, and S. Zhang. Planning multimodal exploratory actions for online robot attribute learning. In Robotics: Science and Systems (RSS), 2021.
[41] S. Nair, S. Savarese, and C. Finn. Goal-aware prediction: Learning to model what matters. In International Conference on Machine Learning 2020.
[42] C. Paxton, Y. Bisk, J. Thomason, A. Byravan, and D. Fox. Prospection: Interpretable plans from language by predicting the future. In International Conference on Robotics and Automation (ICRA). IEEE, 2019.
[43] J. Y. Koh, H. Lee, Y. Yang. J. Baldridge, and P. Anderson. Pathdreamer: A world model for indoor navigation. arXiv, 2021.
[44] C. Eppner, A. Mousavian, and D. Fox. ACRONYM: A large-scale grasp dataset based on simulation. In 202/ IEEE Inf. Conf. on Robotics and Automation, ICRA, 2020.
[45] A. X. Chang. M. Savva, and P. Hanrahan. Semantically 3d models for common-senss knowledge. In FPIC Workshop, Computer Vision and Pattern Recognition (CVPR), 2015.
[46] G. A. Miller. Wordnet: A lexical database for english. Communications of the ACM, 38(11):
39-41, 1995.
[47] P. Zhang, X. Li, X. Hu, J. Yang. L. Zhang, L. Wang, Y. Choi, and J. Gao. VinVL: Making visual representations matter in vision-language models. Computer Vision and Pattern Recognition (CVPR), 2021.
[48] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning rgb-d feature embeddings for unseen object instance segmentation. In Conference on Robot Learning (CoRL), 2020.
[49] C. R. Qi, L. Yi. H. Su, and L. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Neural Information Processing Systems (NeurIPS), 2017.
[50] J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), 2014.
[51] Sharma, N. Ding, S. Goodman, and R. Soricut, Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Association for Computational Linguistics (ACL), 2018
[52] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.


Reviews:
[{"review_id": "o9C5l07IkG", "reviewer": "Reviewer_GQzH", "summary": "This paper addresses the problem of grounding referring expressions for 3D objects. The authors claim that a single 2D image is not enough to identify an object in the 3D world. To formally study the problem, a new dataset, SNARE, based on ShapeNet objects is introduced to test models’ understanding of both visual and non-visual features. In addition, two CLIP-based models, MATCH and LAGOR, are introduced to align images with text descriptions of objects and to leverage an implicit 3D representation of objects to further promote understanding of non-visual cues. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem this paper studies is very relevant to the robotics community. As the authors mentioned, it’s difficult for robots to collect complete 360 views of real-world objects. Grounding 3D objects in a small number of 2D images from different views has practical value. \n2. The data collection process is carefully designed and produces high-quality data. It’s interesting that the authors deliberately elicit specific descriptions of objects by pairing objects. The wordnet analysis of the dataset also provides insights into the semantic content of the language descriptions. \n3. The dataset uses the standard ShapeNet objects and the referring expressions also include non-visual features. Therefore the dataset could be useful for future research that incorporates other sensor modalities as well. \n\nWeakness:\n1. It’s unclear whether the choice of the pooling method (i.e., max pooling) has affected the performance of models that use images from all viewpoints. \n2. The authors should more formally discuss the physical meaning behind the 8 viewpoints. Is there a canonical pose for every category of objects? If not, how can the model learn to predict the correct viewpoint of a novel object, especially since the data are split at the object class level. What is the accuracy of view estimation? Does it correlate to the performance of language grounding? \n3. More insights about the design of the LAGOR model is needed to support the authors' claim. It’s unclear why the LAGOR model only predicts the orientation of the first view? Is there a fixed rotational offset between the first and the second views? If not, why shouldn’t the model have access to the pose information for the second view? Will that further help improve the performance of the model? \n4. The novelty of the MATCH model is limited. I suspect that the performance difference between the base CLIP and the MATCH model mainly comes from fine tuning, which is expected when using pretrained models. \n5. Compared to the around 10% performance increase after fine tuning the CLIP model, the 2% difference between MATCH (two) and LAGOR (two) is relatively small. Due to the small improvement, it’s important that the authors further analyze the cause of it. It would help if the authors could test on different data splits and consistently see this improvement. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes an interesting solution to a relevant task. However, due to the lack of explanation for the model design and further analysis of the relatively small performance gain, it’s hard to judge the significance of the paper.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Besides questions and suggestions from above, I strongly suggest the authors restructure section 4. At this moment, the section contains information about baselines, hypotheses, proposed methods, experimental designs, and experiment results.\n"}}, {"review_id": "abR1YYNJrQp", "reviewer": "Reviewer_9Pxm", "summary": "The authors collected a dataset (SNARE) with 3D object models and natural language referring expressions. Moreover, they propose the MATCH method to predict the described object among two candidate objects (one target and one distractor). Using this method, they show that using multiple views to select the correct object improves over single-view object selection. They further improve the MATCH model and suggest the LAGOR method estimating the current view of objects to improve the performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The suggested dataset includes both visual descriptions that focus on colors and object categories and blindfolded descriptions that focus on shapes and parts. Computer vision studies commonly suggest visual descriptions, but blindfolded descriptions do not often address existing studies. The blindfolded descriptions suggested in the dataset can be useful when the robot is verbally instructed to grasp a specific object. Moreover, this dataset includes many object categories (262 in total) compared to the previous works.\n- The paper presents interesting findings. The results show that the performance of the MATCH method increases when two views of an object are used instead of a single view, but the accuracies do not increase when the 360-degree view is provided. Further, the authors show that estimating the current view helps identifying referred objects.\n\nWeaknesses:\n- The suggested system chooses which of two objects is being referenced by a natural language description.  Only having two candidates oversimplifies the problem, given that there might be many distractors in real-world environments. Because of this simplification, one of the critical features of referring expressions (i.e., spatial relations between objects) is excluded in the paper.\n- The paper suggests rotating the objects before identifying the described one because some parts of the object might not be visible from the robot's viewpoint, and these invisible features might be used in the user's description. This is a very valid point, but the suggested system rotates the objects even if it is not necessary. In other words, if the visible part is enough to predict the referred object, the system still performs the rotation to make a prediction.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper simplifies the problem and the method might perform unnecessary rotations, it suggests a valuable dataset and presents very interesting findings. These are the main reasons for my recommendation (see the strengths and weaknesses for details).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- The CLIP baseline method is not fine-tuned in the suggested dataset. Are there any specific reasons to only report the zero-shot performances of this baseline?\n- In the instructions of MTUR workers (provided in the appendix), the workers were instructed that visual descriptions can involve the name of the objects, shapes, and colors; by contrast, blindfolded descriptions can not involve colors but can involve shape and parts. However, in the paper manuscript, this is provided as an estimation (\"We estimated that blindfolded expressions use more shape words 177 (14% vs 5% of all words) while visual use more color words (22% vs 1% of all words)\"). It would be better to acknowledge readers that these were the instructions that should be followed by the MTURK workers.\n- It would be nice to include further information about the training of MATCH and LAGOR methods. For instance, what were the negative samples during training? Were they the distractor objects? Did you train these methods for a single view and include the max-pooling during testing for multi-views?\n- In Table 2, the results are reported for single, two, and 360. Did you use the 3D model of the objects as an input for the results of 360? Or was it many 2D views of the objects? If it is the second case, the paper's title and abstract might be misleading because the suggested systems are not trained/evaluated on 3D object models but multiple 2D views of the objects. \n- In the robotics demonstration results (Section 5.2), did you use a single view both for CLIP and LAGOR? Or was it a single view for CLIP and two views for LAGOR? If the second one is the case, why not provide two views to both of the methods? Further, the accuracy of LAGOR is reported as 50%; isn't it the random chance level (selecting one of two candidates)?\n- It would be nice to include a reference to the statement in line 51 (\"However, internet images suffer from a type of reporting bias\").\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nPost-Rebuttal Comments:\n\nI would like to thank the authors for replying to my comments in detail. The revisions are helpful to highlight the contributions and interpret the results. However, I still have the following concerns after the rebuttal:\n- I think simplifying the problem by selecting the target object among two candidates might be the reason for the high accuracies of all the methods shown in Table 2. As also pointed out by Reviewer ch3k, the performances of the zero-shot CLIP models are high on the SNARE benchmark, and this might be because of the high chance level (i.e., 50%). I think the system needs to be tested with more than two candidate objects to present the advantages of estimating the current view of objects and be usable in real-world applications.\n- In the robotics demonstration, the authors present the results for a single view for the zero-shot CLIP model and two views for LAGOR. For a fair comparison, it would be better to show these results for the zero-shot CLIP, MATCH, and LAGOR methods with a single view and two views (if resolving the technical issues is possible)."}}, {"review_id": "TdM0Tj_9ch3", "reviewer": "Reviewer_XGP9", "summary": "Paper presents a novel benchmark, SNARE, which requires the agent to ground a referring expression in natural language to a 3D object. The agent might need to actively rotate the objects to help determine the selection. A CLIP-based baseline model (LAGOR) is proposed. Results demonstrate that LAGOR attains superior performance over other vision-language models (ViLBERT and raw CLIP).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n[+] The main problem (language grounding in 3D) that this new benchmark concerns is fairly interesting. My personal take for the difference between canonical LG and LG in 3D can be primarily about embodiment. The authors did a good job incorporating it in their benchmark and design an interesting task. Although the results are less surprising, it still seems to be a nice contribution to the community.\n\n[+] The paper is overall clear and well-written. The authors make their motivations clear and convincing. The dataset itself and the proposed learner seem ground and I cannot find any significant technical issue.\n\n[Weaknesses]\n\n[-] The author could have made their presentations more clear by including enough details on the embodiment in their main text. After reading this paper, I'm still not quite sure how the four compared learners (ViLBERT, CLIP, MATCH, LAGOR) select the action. The authors are expected to clarify this in a rebuttal.\n\n[-] If my understanding is correct, the only action considered in this benchmark is rotation (along a predefined axis). I won't say it is overclaiming but indeed I find the action space can be rather limited. This can significantly weaken the point of bringing embodiment to language grounding. A hotfix I can think of is to provide the intermediate action sequence for some hard examples (maybe also some failure mode). Otherwise, it can be hard to tell whether the learner masters some embodiment for 3D interaction or just trivialize it with a (hand-crafted?)short-cut strategy (e.g. continuously rotate the object if the uncertainty is high).\n\n[-] The overall performance gap between the baselines and the proposed learner seems to be marginal and the authors do not provide a confidence interval. I strongly encourage the authors to include it in a rebuttal.\n\n[Suggestions & Questions]\n\nThe authors are expected to address the issues I mentioned in [Weaknesses].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Given the technical quality at this point, I tend to recommend rejection. I may change my mind per other reviewers' opinion and the rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See [Suggestions & Questions].", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "CnairqkJsyV", "reviewer": "Reviewer_ch3k", "summary": "* Introduces SNARE, a new benchmark for grounding natural language referring expressions to distinguish 3D objects.\n* Introduces the concept of 'blindfolded descriptions', a complementary and challenging linguistic space. Introduces a language annotation process that explicitly gathers these (less collected) tactile natural language descriptions of objects.\n* Introduce several CLIP-based models for distinguishing objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n* Good motivation: imbue robots with three dimensional notions of language grounding as opposed to purely visual ones.\n* Good description of the CLIP-based models for distinguishing objects and the auxiliary losses. I feel like this carries enough information to begin implementing.\n* Informative related work section highlighting how this paper sits at the intersection of multiple active fields.\n* Raises an important question: does modern visual language grounding have important limitations for language-conditioned robotic learning, i.e. domain shift between canonical depictions of objects in images and objects as seen through a robot camera?\n\n[Weaknesses]\n* LAGOR has quite strict requirements for training (8 simultaneous labeled views of each object at 45 degree offsets), and seems to offer minor benefits in performance over single-view trained methods or pretrained methods. In table 1, we see 79.4% (LAGOR) vs 78.5% (MATCH) averaged over visual and blind descriptions. \n* Claims in the conclusion like L303: \"We find that LAGOR is able to identify referent objects using only an initial and final view more accurately than models utilizing 360 views\" seem quite strong, given the experiments.\n* Claims like L297 \"We show that fine-tuned, massively pretrained vision and language models fall short of human performance at identifying object referents of natural language expressions by a wide margin (Table 2).\" seem odd. You are taking models trained on one domain (massive internet collections) and applying them in zero shot to new domain (simulated object) data. If anything, I'd say the gap between the zero shot performance and the performance of models finetuned on in-domain SNARE data is much smaller than I would have expected. \nThe more important takeaway from this for me is that pretrained models actually do very well on the considered benchmark. This seems like it could be emphasized more instead of focusing on the relatively minor gains LAGOR offers.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper has some issues with exposition, but presents interesting results showing how large pretrained language models (and their finetuned variants) handle referent-picking challenges when the language used is more tactile than visual. As pointed out by this work, this different mode of object description may be more likely in the real world than in the large datasets CLIP-style models are trained on. Despite this supposed domain shift, the results show that pretrained models fare surprisingly well in zero shot. This seems like an interesting insight for the broader community. \n\nUPDATE: \n\n* I highly appreciate the additional experiments and improved statistical rigor, as well as the removal of overly strong interpretations.\n* In regards to my prior comment \"LAGOR has quite strict requirements for training (8 simultaneous labeled views of each object at 45 degree offsets)\". I appreciate the clarification that not all need to be present at training time, but computational cost was not what I was concerned with. I think the criticism remains valid that requiring 8 labeled views (even if only two are sampled for each training instance) represents a very significant additional assumption to the overall training process, for pretty minor gains over baselines. In general, I think this limits its applicability to many new scenarios. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "This paper would benefit from:\n* Cleaned up exposition.\n* Statistically valid experiments or toned down interpretation of the existing experiments.\n* Highlighting the seemingly impressive zero shot performance of CLIP-style models on tactile descriptions."}}]

Author rebuttal:

Meta-review: This paper studies an interesting problem, and puts together an interesting dataset that I expect will be impactful in future works. With only a few exceptions, the paper is also well-written and well-motivated. The only exceptions are that the paper includes some claims that are not fully supported (see reviewers XGP9, ch3k, GQzH), and a few details mentioned by reviewers XGP9 and 9Pxm. The paper also proposes several CLIP-based models, and the results of these models are not entirely convincing, especially given the lack of error bars.

Overall, the paper is borderline. I think that the paper currently leans towards accept, but this may change during the author response and reviewer discussion.

---------------------

After the author response period, some of the reviewer's concerns have been addressed, especially the concerns about the robustness of the results. Beyond the comments above, the paper also includes experiments on a real robot platform, which I expect to be appreciated by the CoRL community. I recommend accept.
Decision: Accept (Poster)