Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: By combining the robustness in open-vocabulary scene understanding from a vision-language model in RGB images and a semantic-agnostic spatial understanding from a 3D model, it achieves open-world 3D scene understanding with a small set of 3D datasets.
It proposes two tasks completing partially observed objects and localizing hidden objects to show out-of-distribution generalization and spatial understanding ability on the partially observed 3D dataset. It provides various ablation models to help understand the effect of each part of the model.
strengths_and_weaknesses: Strengths
* Provides a simple but powerful decomposition and abstraction of perception and localization for out-of-distribution detection in robotics applications.
* Can leverage various VLMs as its module.
* Uses multi-scale proposals from VLMs to handle small objects with batched high-speed inference

Weaknesses
* Ambiguous evaluation on newly proposed datasets and tasks; not clear why they are proposed other than the existing tasks such as detection or visual grounding.
* Does not compare with existing methods in detection or visual grounding tasks; hard to understand the empirical contribution to real-world applications.
* Proposed datasets on AI2-THOR which is less diverse in appearance than other datasets such as ScanNet or SUN-RGBD.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: It proposes a modular approach for 3D understanding with out-of-distribution generalization by combining vision-language model (VLM) with a semantic-abstract 3D model.
It is one of the challenging tasks in robotics due to the limited size of 3D datasets but is crucial to real-world applications.
It shows that the proposed method provides better generalization both in detecting small objects by introducing multi-scale feature extraction from VLM and identifying novel and fine-grained classes in 3D.
While its application is crucial for advancing the 3D understanding, its evaluation is not based on the existing tasks such as 3D detection or visual grounding, nor on existing approaches; therefore, it is not trivial to clearly understand their contribution compared to the existing literature.
Therefore, I recommend a weak acceptance.
issues: In general, evaluation is a little bit confusing; the first task, open vocabulary semantic scene completion (OVSSC), seems to be similar in open vocabulary detection/segmentation and the second task, visually obscured object localization (VOOL), can be considered as a specialized case of visual grounding. Besides the ablations, it lacks direct comparison to existing models or tasks. For instance, applying the proposed approach to Refer-it-in-RGBD (https://arxiv.org/abs/2103.07894) might be interesting to see how different approaches to spatial understanding work. How the model works in the existing detection tasks is another task of interest.

* Sec 3.3; how the 3D semantic-abstracted module is robustly working with ambiguous saliency or relevancy from the previous module.
* Hard to understand the effect of the Z_target in the VOOL task; 
* How are query points Q chosen?
* Why 32x32x32? 3D detection methods can use refined volume up to 1 cm resolution.
* Why AI2-THOR? AI2-THOR is less diverse in terms of appearance, object classes and instances in each class, and types of rooms. While arguing its novelty in out-of-distribution generalization in vocabulary and appearance, testing only on AI2-THOR makes this approach less compelling. 
* Visual grounding methods in either 
a referring expression comprehension task or a referring expression segmentation task should be compared as a baseline such as `LAVT: Language-Aware Vision Transformer for Referring Image Segmentation', `CRIS: CLIP-Driven Referring Image Segmentation', `OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework' (https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco-4, https://paperswithcode.com/sota/referring-expression-comprehension-on-refcoco) in order to show the evidence of the statement argued in line 237. Some of the spatial relationships can be interpreted in 2D (such as ``left of'') and it would be interesting to compare two different approaches that have different levels of task decompositions. At least, some 2D visual grounding models can be used in ClipSpatial.
* Instead of taking a single expression per instance, is choosing K objects at once in the OVSSC task affect the result? I wonder if exclusion happens or is encouraged during the process.
* Known-class accuracy (in comparison to existing detection methods) is missing.
* While being trained on six spatial prepositions, no information on spatial prepositions in the custom dataset is not provided; providing a success rate on each preposition may help better understand the problem.

* Typo in line 221: FOr > For
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper formulates and studies the task of open-world 3D scene understanding. The key insight of the authors is that the task of spatial/geometric reasoning can be decoupled from the task of learning semantic embeddings of images and text. The paper presents a method that decouples the semantic component (using pre-trained a CLIP transformer) from the spatial component (3D point cloud completion), and show results in a simulator as well as some real-world 3D scans, along with generalization results to new environments and textual classes.
strengths_and_weaknesses: Strengths
---
- The VLM relevancy module seems like a neat little idea by combining the GradCam-style visualization technique with vision transformers. To my best knowledge, this is the first work that demonstrates an effective way to do this, and I can think of several research areas that would benefit for this.
- Despite being restrictive, the way the authors handle the spatial prepositions seems very simple, yet unique. I have seen a lot of CLIP-based papers struggle with this and it's very exciting to see this work well enough for a downstream application.
- The evaluation benchmark, designed on the AI2-THOR simulator, seems very thorough and provides good insights on which components of the method contribute the most. This could be a valuable contribution to the 3D scene understanding community if made public along with reproducible baselines etc.
- The supplemental video does a very good job of conveying the results and qualitatively comparing to baselines — I really like it!

*** 
Weaknesses
---
- [Implementation Details & Reproducibility] My biggest concern with the paper in its current shape is the lack of implementation details in multiple sections throughout the paper. This, coupled with the lack of any code release makes the paper very hard to parse, reproduce, and draw any generalizable insights from (the authors mention “code, models will be publicly available” but my review focuses on the provided material). This further makes it difficult to judge what components of the method are novel and what components are used directly from prior (unreferenced) work. Below are some notable examples of crucial implementation/training details missing:
	- Section 3.3.1: There is no information here regarding the models or training pipeline. What is the learning objective, and how is it achieved? Is this a novel component, or do you borrow this from prior work?
	- Section 3.3.2: This is particularly hard to parse, since the “method” is barely described in text (how is the query point allocation/matching done? etc.), and there are absolutely no training details provided.
	- Section 3.2: There are some implementation details here but the important stuff is only mentioned in the supplemental material, making the text seem incomplete. Further, this is not referenced in the main text.
- [Robotics Focus] The paper reads a lot like a typical computer vision paper that creates and evaluates on a task-specific benchmark without considering effects on a potential downstream robotic task (in simulation or real). Barring the conclusion (Line 292), the paper makes no effort to justify why the method may be well-suited for an embodied navigation task and why the proposed benchmark and metrics matter — why is the voxel IoU a good metric for a semantic navigation task where the robot just needs to reach a general location and may not care about the exact 3D voxels? Especially to the CoRL audience, the paper would benefit from a better discussion of the motivation and positioning wrt applications in robotics, if not by adding a relevant downstream application.
- [Organization] In addition to missing implementation details, the paper is also missing any qualitative visualization of the comparative experiments and there is no qualitative or quantitative table/figure corresponding to the sim2real claim (Line 270) — this only makes an appearance in the supplemental material with only a placeholder in the main text. Maybe this is a late addition after the paper deadline, but in its current form, the paper just feels incomplete by itself and needs to be re-organized with more essential information in the main text.
- [Experiments/Baselines] While the authors do a good job of evaluating the method on the proposed suite of tasks, I would argue that the evaluation is not against any “baselines”, and are closer to a set of ablations to the proposed method (except the one with Chefer et al.). This is not immediately a bad thing, but the authors should consider an analysis with respect to existing algorithms that work with a stricter set of assumptions to get a sense for how well the method performs. For instance, one such comparison can be against a SoTA closed-vocabulary scene completion algorithm, to understand how well SemAbs (proposed) done in comparison and where it lacks — does the CLIP-based model perform just as well as a more carefully engineered system or is it a long way from the best 3D scene understanding algorithms that the community has been developing.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: In my review, I raised a number of concerns regarding the current submission: primarily on the lack of implementation details and related reproducibility concerns and the inability to assess significance of claims for downstream robotics applications. While I really like some ideas proposed in the paper, in its current form, I cannot recommend acceptance of the paper, but I am very willing to engage with the authors in the discussion phase and reconsider my recommendation.

---

**Update**: The authors' responses have addressed my concerns regarding lack of implementation details and presentation, and the last round of revisions with qualified claims of "open-vocabulary" mostly addresses my concern as well. However, I maintain that the relevance of the findings (and the choice of metrics to determine superior performance) to eventual downstream applications is not very clear, especially for applications in robotics (per the CoRL guidelines). Contingent on the promised revisions to qualify the open-vocabulary claims and the added discussion of relevance to robotics, I do not have any strong issues with the paper. Erring on the side of being conservative, I am still keeping my original rating as I cannot (enthusiastically or weakly) recommend acceptance but would not argue against it.
issues: - [Points Raised Above] I raised some concerns in the above section and some clarification in response to those points would really help me re-evaluate the submission score.
	- Reproducibility concerns
	- Positioning wrt robotics
	- Experiments
- [Limited Spatial Understanding] The authors should consider toning down their “open-vocabulary” claims in light of the fact that (i) they expect a very specific type of phrase (Line 182) to describe the object in the scene, and (ii) they further reduce the set of spatial prepositions to just *six* (Line 195), which is much more restrictive. Generally speaking, the repeated claim of “small/finite words for describing spatial relations” [citing Landau and Jackendoff] is not entirely true and there are several other ways spatial relations may be described ([see commentary by Bennett on the same paper](https://ase.tufts.edu/cogstud/jackendoff/papers/what_where.pdf)). While this may not be a big limitation in practice, the current claims need to be qualified (or clarified) better to present a more accurate discussion of prior work and motivation.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes SemanticAbstraction; a framework that incorporates 3D spatial capabilities to 2D vision-language models while allowing said models to keep their zero-shot learning abilities. The framework is comprised of 2 modules:
- Semantic Aware Wrapper which abstracts the input image given a semantic label using ViT-CLIP into a relevancy map projected to 3D
- Semantic Abstracted 3D Module which converts the projected 3D relevancy map into 3D occupancy prediction using a 3D encoder-decoder based model operating on 3D voxel grids

The framework is trained on synthetic only data and evaluated on 2 main tasks:
- Open Vocabulary Semantic Scene Completion (OVSSC): predicting 3D occupancy of objects given a single RGB-D image with object labels
- Visually Obscured Object Localization (VOOL): predicting possible occupancy locations of referred objects given an input image
The proposed method is compared against baselines and performs favorably.
strengths_and_weaknesses: Strengths:
- The paper is very well written, easy to follow
- The work addresses an important problem for robotics; the ability to link language and visual models while being robust to long-tail distribution descriptions
- The experimental results are quite promising
- Releasing code, dataset and models is beneficial to the advancement of this area of research

Weaknesses & Questions:
- It is not very clear to the reviewer why the semantic abstracted 3D module is needed? Why not just use the 3D projected relevancy map as an output with some post-processing?
- Using 3D voxel grids as a basis for the occupancy prediction can be potentially problematic for accurately localizing objects of small scale. Furthermore, maintaining a reasonable voxel size with large environments can lead to large memory consumption. It would be nice to see some information regarding the memory required or ablations on larger environments with small objects perhaps
- Is the OVSSC task a realistic one? Normally one would either want the location of a single object (as in the VOOL task) for which a prerequisite is segmenting the environment (where one would not expect giving categories as input). My question here is what does this task help us understand that we could not before?
- The framework was only evaluated on self-constructed baselines. While I understand that this area is rather novel with not enough related work that one can directly compare against. On the other hand, there are similar datasets like ScanRefer that offer a benchmark for a similar problem (albeit bounding box regression instead of segmentation). Can one perhaps adapt one of the methods from there or compare the current approach on said dataset?

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper covers a key topic in robotics and presents very nice results showing generalization capabilities from synthetic dataset to real-world. 

I would like to thank the authors for their responses. Based on the discussion, I will maintain my score.
issues: - What does BCE in line 230 stand for?
- What's the motivation behind using a 3D CNN for encoding? Did the authors try something in the direction of SRT: https://srt-paper.github.io/ which would help side-stepping a number of problems relating to using voxel-based representations.
review_title: null

Meta-review: metareview: All reviewers agree that the paper addresses important issues in robot learning.
1. Although the method is simple, the experimental results are promising.
2. The authors constructed a new dataset for this task.
3. The paper and supplemental appendix are well organized. The supplemental video does a good job of conveying the qualitative results.

However, the reviewers raised a number of concerns.
1. The experimental comparison was conducted using only self-constructed baselines. It would have been helpful if the paper also presented evaluation on similar datasets.
2. Implementation details are not fully presented in the paper.
3. The expression "open vocabulary" should be reconsidered because spatial prepositions are limited to six.

### Post-rebuttal comment 
The reviewers initially raised a few concerns regarding the experimental comparison and the implementation details. They also questioned some of the claims made in the initial submission. The authors provided a detailed response to each of these points and made changes to the paper to resolve most of the concerns. Therefore, I recommend acceptance of the paper.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)