Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Paper presents a novel benchmark, SNARE, which requires the agent to ground a referring expression in natural language to a 3D object. The agent might need to actively rotate the objects to help determine the selection. A CLIP-based baseline model (LAGOR) is proposed. Results demonstrate that LAGOR attains superior performance over other vision-language models (ViLBERT and raw CLIP).
strengths_and_weaknesses: [Strengths]

[+] The main problem (language grounding in 3D) that this new benchmark concerns is fairly interesting. My personal take for the difference between canonical LG and LG in 3D can be primarily about embodiment. The authors did a good job incorporating it in their benchmark and design an interesting task. Although the results are less surprising, it still seems to be a nice contribution to the community.

[+] The paper is overall clear and well-written. The authors make their motivations clear and convincing. The dataset itself and the proposed learner seem ground and I cannot find any significant technical issue.

[Weaknesses]

[-] The author could have made their presentations more clear by including enough details on the embodiment in their main text. After reading this paper, I'm still not quite sure how the four compared learners (ViLBERT, CLIP, MATCH, LAGOR) select the action. The authors are expected to clarify this in a rebuttal.

[-] If my understanding is correct, the only action considered in this benchmark is rotation (along a predefined axis). I won't say it is overclaiming but indeed I find the action space can be rather limited. This can significantly weaken the point of bringing embodiment to language grounding. A hotfix I can think of is to provide the intermediate action sequence for some hard examples (maybe also some failure mode). Otherwise, it can be hard to tell whether the learner masters some embodiment for 3D interaction or just trivialize it with a (hand-crafted?)short-cut strategy (e.g. continuously rotate the object if the uncertainty is high).

[-] The overall performance gap between the baselines and the proposed learner seems to be marginal and the authors do not provide a confidence interval. I strongly encourage the authors to include it in a rebuttal.

[Suggestions & Questions]

The authors are expected to address the issues I mentioned in [Weaknesses].
summary_of_recommendation: Given the technical quality at this point, I tend to recommend rejection. I may change my mind per other reviewers' opinion and the rebuttal.
issues: See [Suggestions & Questions].
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of grounding referring expressions for 3D objects. The authors claim that a single 2D image is not enough to identify an object in the 3D world. To formally study the problem, a new dataset, SNARE, based on ShapeNet objects is introduced to test models’ understanding of both visual and non-visual features. In addition, two CLIP-based models, MATCH and LAGOR, are introduced to align images with text descriptions of objects and to leverage an implicit 3D representation of objects to further promote understanding of non-visual cues. 

strengths_and_weaknesses: Strengths:
1. The problem this paper studies is very relevant to the robotics community. As the authors mentioned, it’s difficult for robots to collect complete 360 views of real-world objects. Grounding 3D objects in a small number of 2D images from different views has practical value. 
2. The data collection process is carefully designed and produces high-quality data. It’s interesting that the authors deliberately elicit specific descriptions of objects by pairing objects. The wordnet analysis of the dataset also provides insights into the semantic content of the language descriptions. 
3. The dataset uses the standard ShapeNet objects and the referring expressions also include non-visual features. Therefore the dataset could be useful for future research that incorporates other sensor modalities as well. 

Weakness:
1. It’s unclear whether the choice of the pooling method (i.e., max pooling) has affected the performance of models that use images from all viewpoints. 
2. The authors should more formally discuss the physical meaning behind the 8 viewpoints. Is there a canonical pose for every category of objects? If not, how can the model learn to predict the correct viewpoint of a novel object, especially since the data are split at the object class level. What is the accuracy of view estimation? Does it correlate to the performance of language grounding? 
3. More insights about the design of the LAGOR model is needed to support the authors' claim. It’s unclear why the LAGOR model only predicts the orientation of the first view? Is there a fixed rotational offset between the first and the second views? If not, why shouldn’t the model have access to the pose information for the second view? Will that further help improve the performance of the model? 
4. The novelty of the MATCH model is limited. I suspect that the performance difference between the base CLIP and the MATCH model mainly comes from fine tuning, which is expected when using pretrained models. 
5. Compared to the around 10% performance increase after fine tuning the CLIP model, the 2% difference between MATCH (two) and LAGOR (two) is relatively small. Due to the small improvement, it’s important that the authors further analyze the cause of it. It would help if the authors could test on different data splits and consistently see this improvement. 

summary_of_recommendation: This paper proposes an interesting solution to a relevant task. However, due to the lack of explanation for the model design and further analysis of the relatively small performance gain, it’s hard to judge the significance of the paper.

issues: Besides questions and suggestions from above, I strongly suggest the authors restructure section 4. At this moment, the section contains information about baselines, hypotheses, proposed methods, experimental designs, and experiment results.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors collected a dataset (SNARE) with 3D object models and natural language referring expressions. Moreover, they propose the MATCH method to predict the described object among two candidate objects (one target and one distractor). Using this method, they show that using multiple views to select the correct object improves over single-view object selection. They further improve the MATCH model and suggest the LAGOR method estimating the current view of objects to improve the performance.
strengths_and_weaknesses: Strengths:
- The suggested dataset includes both visual descriptions that focus on colors and object categories and blindfolded descriptions that focus on shapes and parts. Computer vision studies commonly suggest visual descriptions, but blindfolded descriptions do not often address existing studies. The blindfolded descriptions suggested in the dataset can be useful when the robot is verbally instructed to grasp a specific object. Moreover, this dataset includes many object categories (262 in total) compared to the previous works.
- The paper presents interesting findings. The results show that the performance of the MATCH method increases when two views of an object are used instead of a single view, but the accuracies do not increase when the 360-degree view is provided. Further, the authors show that estimating the current view helps identifying referred objects.

Weaknesses:
- The suggested system chooses which of two objects is being referenced by a natural language description.  Only having two candidates oversimplifies the problem, given that there might be many distractors in real-world environments. Because of this simplification, one of the critical features of referring expressions (i.e., spatial relations between objects) is excluded in the paper.
- The paper suggests rotating the objects before identifying the described one because some parts of the object might not be visible from the robot's viewpoint, and these invisible features might be used in the user's description. This is a very valid point, but the suggested system rotates the objects even if it is not necessary. In other words, if the visible part is enough to predict the referred object, the system still performs the rotation to make a prediction.
summary_of_recommendation: Although the paper simplifies the problem and the method might perform unnecessary rotations, it suggests a valuable dataset and presents very interesting findings. These are the main reasons for my recommendation (see the strengths and weaknesses for details).
issues: - The CLIP baseline method is not fine-tuned in the suggested dataset. Are there any specific reasons to only report the zero-shot performances of this baseline?
- In the instructions of MTUR workers (provided in the appendix), the workers were instructed that visual descriptions can involve the name of the objects, shapes, and colors; by contrast, blindfolded descriptions can not involve colors but can involve shape and parts. However, in the paper manuscript, this is provided as an estimation ("We estimated that blindfolded expressions use more shape words 177 (14% vs 5% of all words) while visual use more color words (22% vs 1% of all words)"). It would be better to acknowledge readers that these were the instructions that should be followed by the MTURK workers.
- It would be nice to include further information about the training of MATCH and LAGOR methods. For instance, what were the negative samples during training? Were they the distractor objects? Did you train these methods for a single view and include the max-pooling during testing for multi-views?
- In Table 2, the results are reported for single, two, and 360. Did you use the 3D model of the objects as an input for the results of 360? Or was it many 2D views of the objects? If it is the second case, the paper's title and abstract might be misleading because the suggested systems are not trained/evaluated on 3D object models but multiple 2D views of the objects. 
- In the robotics demonstration results (Section 5.2), did you use a single view both for CLIP and LAGOR? Or was it a single view for CLIP and two views for LAGOR? If the second one is the case, why not provide two views to both of the methods? Further, the accuracy of LAGOR is reported as 50%; isn't it the random chance level (selecting one of two candidates)?
- It would be nice to include a reference to the statement in line 51 ("However, internet images suffer from a type of reporting bias").

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Post-Rebuttal Comments:

I would like to thank the authors for replying to my comments in detail. The revisions are helpful to highlight the contributions and interpret the results. However, I still have the following concerns after the rebuttal:
- I think simplifying the problem by selecting the target object among two candidates might be the reason for the high accuracies of all the methods shown in Table 2. As also pointed out by Reviewer ch3k, the performances of the zero-shot CLIP models are high on the SNARE benchmark, and this might be because of the high chance level (i.e., 50%). I think the system needs to be tested with more than two candidate objects to present the advantages of estimating the current view of objects and be usable in real-world applications.
- In the robotics demonstration, the authors present the results for a single view for the zero-shot CLIP model and two views for LAGOR. For a fair comparison, it would be better to show these results for the zero-shot CLIP, MATCH, and LAGOR methods with a single view and two views (if resolving the technical issues is possible).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: * Introduces SNARE, a new benchmark for grounding natural language referring expressions to distinguish 3D objects.
* Introduces the concept of 'blindfolded descriptions', a complementary and challenging linguistic space. Introduces a language annotation process that explicitly gathers these (less collected) tactile natural language descriptions of objects.
* Introduce several CLIP-based models for distinguishing objects.
strengths_and_weaknesses: [Strengths]
* Good motivation: imbue robots with three dimensional notions of language grounding as opposed to purely visual ones.
* Good description of the CLIP-based models for distinguishing objects and the auxiliary losses. I feel like this carries enough information to begin implementing.
* Informative related work section highlighting how this paper sits at the intersection of multiple active fields.
* Raises an important question: does modern visual language grounding have important limitations for language-conditioned robotic learning, i.e. domain shift between canonical depictions of objects in images and objects as seen through a robot camera?

[Weaknesses]
* LAGOR has quite strict requirements for training (8 simultaneous labeled views of each object at 45 degree offsets), and seems to offer minor benefits in performance over single-view trained methods or pretrained methods. In table 1, we see 79.4% (LAGOR) vs 78.5% (MATCH) averaged over visual and blind descriptions. 
* Claims in the conclusion like L303: "We find that LAGOR is able to identify referent objects using only an initial and final view more accurately than models utilizing 360 views" seem quite strong, given the experiments.
* Claims like L297 "We show that fine-tuned, massively pretrained vision and language models fall short of human performance at identifying object referents of natural language expressions by a wide margin (Table 2)." seem odd. You are taking models trained on one domain (massive internet collections) and applying them in zero shot to new domain (simulated object) data. If anything, I'd say the gap between the zero shot performance and the performance of models finetuned on in-domain SNARE data is much smaller than I would have expected. 
The more important takeaway from this for me is that pretrained models actually do very well on the considered benchmark. This seems like it could be emphasized more instead of focusing on the relatively minor gains LAGOR offers.

summary_of_recommendation: This paper has some issues with exposition, but presents interesting results showing how large pretrained language models (and their finetuned variants) handle referent-picking challenges when the language used is more tactile than visual. As pointed out by this work, this different mode of object description may be more likely in the real world than in the large datasets CLIP-style models are trained on. Despite this supposed domain shift, the results show that pretrained models fare surprisingly well in zero shot. This seems like an interesting insight for the broader community. 

UPDATE: 

* I highly appreciate the additional experiments and improved statistical rigor, as well as the removal of overly strong interpretations.
* In regards to my prior comment "LAGOR has quite strict requirements for training (8 simultaneous labeled views of each object at 45 degree offsets)". I appreciate the clarification that not all need to be present at training time, but computational cost was not what I was concerned with. I think the criticism remains valid that requiring 8 labeled views (even if only two are sampled for each training instance) represents a very significant additional assumption to the overall training process, for pretty minor gains over baselines. In general, I think this limits its applicability to many new scenarios. 
issues: This paper would benefit from:
* Cleaned up exposition.
* Statistically valid experiments or toned down interpretation of the existing experiments.
* Highlighting the seemingly impressive zero shot performance of CLIP-style models on tactile descriptions.
review_title: null

Meta-review: This paper studies an interesting problem, and puts together an interesting dataset that I expect will be impactful in future works. With only a few exceptions, the paper is also well-written and well-motivated. The only exceptions are that the paper includes some claims that are not fully supported (see reviewers XGP9, ch3k, GQzH), and a few details mentioned by reviewers XGP9 and 9Pxm. The paper also proposes several CLIP-based models, and the results of these models are not entirely convincing, especially given the lack of error bars.

Overall, the paper is borderline. I think that the paper currently leans towards accept, but this may change during the author response and reviewer discussion.

---------------------

After the author response period, some of the reviewer's concerns have been addressed, especially the concerns about the robustness of the results. Beyond the comments above, the paper also includes experiments on a real robot platform, which I expect to be appreciated by the CoRL community. I recommend accept.
Decision: Accept (Poster)