Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models
Authors: Huy Ha
Shuran Song
Venue: CoRL 2022 Poster (2022)
Keywords: 3D scene understanding
out-of-domain generalization
language

Abstract:  We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs -- a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extracted from CLIP and learn 3D spatial and geometric reasoning skills on top of those abstractions in a semantic-agnostic manner. We demonstrate the usefulness of SemAbs on two open-world 3D scene understanding tasks: 1) completing partially observed objects and 2) localizing hidden objects from language descriptions. Experiments show that SemAbs can generalize to novel vocabulary, materials/lighting, classes, and domains (i.e., real-world scans) from training on limited 3D synthetic data. 

Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models
Huy Ha
Shuran Song
Columbia University
cemantic-abstraatioo.... wwwp://ww..cooooccccco..cc
We study open-world 3D scene understanding, a family of tasks that require agents reason about their 3D environment with open-set vocabulary and out-of-domain visual inputs a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs). a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extracted from CLIP, and learn 3D spatial and geometric reasoning skills on top of those abstractions in semantic- aggnssic manner. We demonstrate the usefulness of SemAbs on two open-world 3D scene understanding tasks: 1) completing partially observed objects and 2) localizing hidden objects from language descriptions. SemAbs can generalize to novel vocabulary for object attributes and nouns, materials/lighting classes, and domains (i.e., sal-world scans) from training on limited 3D synthetic data.
Abstract:
Keywords: 3D scene understanding, out-of-domain generalization, language
Posed RGB-D Input
Open Vocabulary Scene Completion
Visually Obscured Object Localization
Used N95s garbage
"COVID-19 rapid behind Harry book'
HARKY
deak bookshelf floor office chair trash basket lego technic liebherr excavator wall with colorful cartoon murals rubiks cube harry potter and sorcerer's stone
Figure 1: Open- 3D Scene Understanding. Our approach. Semantic Abstraction. unlocks 2D VLM's capabilities to 3D scene understanding, Trained with small simulated data, our model generalizes unseen classes a novel domain (i.e., real world scans), from small objects like "rubiks cube", to long-tail tail concepts like "harry potter". to hidden objects like the "used N95s in the garbage hin".
1 Introduction
To assist people in their daily life, robots must recognize a large, context-dependent,, and dynamic set of semantic categories from their visual sensors. However, if their 3D training environment contains only common categories like desks, office chairs, and bookshelves, how could they represent and localize long-tail objects, such as the "used N95s in the garbage bin" (Fig 1)?
This is an example of an open-world 3D scene understanding task (Fig. 2), a family of 3D vision-language tasks which encompasses open-set classification (evaluation novel classes) with extra generalization requirements to novel vocabulary (i.e., object attributes, synonyms of object nouns), visual properties (e.g. lighting, textures), and domains (e.g. sim v.s. real). The core challenge in such tasks is the limited data: all existing 3D datasets [1-6] are limited in diversity and scale compared to their internet-ccale 2D counterparts [7-9], so training them does not prepare robots for the open 3D world.
On the other hand, large-scale 2D vision language models (VLM) [7-11] have demonstrated impressive performance open-set image classification. Through exposure to many maaee-caption pairs from the internet, they have acquired an unprecedented level of robustness to visual distributional shifts and a large repertoire of visual-semantic concepts and vocabulary. However, fine-tuning these pretrained models reduces their open-set classification robustness [7, 10, 12, 13). Compared to internet-scale image-caption data, the finetuning datasets, including the existing 3D datasets [1-6], show significant reduction in scale
6th Conference on Robot Leaming (022),002)) Auckland, New Zealand.

---

and diversity, which causes finetuned models to become task task--speiaaized and lose their generality. Hence, we investigate the following question:
How can we equip 2D VLMs with new 3D capabilities, while maintaining their zero-shot robustness?
We propose Semantic Abstraction (SemAbs), a framework for tackling visual-semantic reasoning in open-world 3D scene understanding tasks using 2D VLMs. We hypothesize that, while open-world visual-semantic reasoning requires exposure to intemet-scale datasets, 3D spatial and geometric reasoning tractable even with limited synthetic dataset, and could generalize better if learned in semantic-agnostie manner. For instance, instead of learning the concept of "behind the Harry Potter book", the 3D localization model only needs to learn the concept of "behind that object".
To achieve this abstraction, we leverage the relevancy maps extracted from 2D VLMS and used them as the "abstracted object" representation that is agnostic to their semantic labels. The SemAbs module factorizes into two submodules: 1) A semantic-aware wrapper that takes input RGB-D image and object category label and outputs the relevancy map of a pre-trained 2D VLM model (i.e., CLIP [7]) for that label, and 2) a cemaatic-abstracted 3D module that uses the relevancy map to predict 3D occupancy. This 3D occupancy can either represent the 3D geometry of a partially observed object or the possible 3D locations of a hidden object (e.g. mask in the trash) conditioned on the language input.
While we only train the 3D network on a limited synthetic 3D dataset, it generalizes to any novel semantic labels, vocabulary, visual properties, and domains that the 2D VLM can generalize to. As result, the SemAbs module inherits the VLM's visual robustness and open-set classification abilities while adding the spatial reasoning abilities it lacks [14-16]. In summary, our contributions are three-fold
Semantic Abstraction, framework for augmenting 2D VLMs with 3D reasoning capabilities for open-world 3D scene understanding tasks. By abstracting semantics away using relevancy maps, our SemAbs module generalizes to novel semantic labels, vocabulary, visual properties, and domains (i.e., sim2real) despite being trained on only a limited synthetic 3D dataset.
Efficient Multiscale Relevancy Extraction. To support Semantic Abstraction, we propose a
multi-scale relevancy extractor for vision transformers [17] (ViT'), which robustly detects small, long-tail objects and achieves over x60 speed up from prior work [18].
Two novel Open-world 3D Scene Understanding tasks copen-vocabulary semantic scene completion and visually obscured object localization), a data generation pipeline with A12-THOR [1] simulator for the tasks, and systematic open-world evaluation procedure. We believe these two tasks are important primitives for bridging existing robotics tasks to the open-world.
2 Related Works
2D Visual Language Models. The recent advancements in scaling up contrasting learning have enabled the training of large 2D vision language models (VLM). Through their exposure millions of internet image-caption pairs, these VLMS [7, 9-1 I] acquire a remarkable level of zero-shot robustness - they can recognize a diverse set of long-tail semantic concepts robustly under distributional shifts. However, the leamed image-level representations are not directly applicable to spatial-reasoning tasks which require pixel-level information. To address this issue, approaches were proposed to extract dense features [19-21] or relevancy maps [18, 22]. These advancements complement our contribution, which leverages such techniques for interfacing 2D VLMs to open-world 3D scene understanding tasks.
Transferring 2D VLMs 2D Applications. Encouraged by their impressive cabilities, many downstream tasks build around these pretrained VLMs [14, 23-28), often finetuning them (either end-to-end or train learablee weights on top of the pretrained encoder) on a small task-specific dataset. However, many recent studies show that finetuning these VLMs significantly weakens their zero-shot robustness [7, 10, 12, 13, 19), which motivates new paradigm for using pretrained models. By combining a fixed pretrained VLM with prompting [29] or attention extraction [15, 19-21], the new generation of zero-shot transfer techniques can inherit the VLM's robustness in their downstream tasks without overfitting to any visual domain. However, all existing zero-shot transfer techniques have been confined to the realms of 2D vision.
Closed-Workde 3D Scene Understanding. Understanding the semantics and 3D structure of an unstructured environment is fundamental capability for robots. However, prior works in 3D object detection [30-32], semantic scene completion [33-35], and language-informed object localization [6, 36] are typically concerned with limited visual diversity and object categories, limitations imposed by their 3D training data. Despite great efforts from the community [1-6], existing 3D datasets' scale, diversity,
All code, data, and models is publicly available at https://gihub..oo//omummt hiw.:/uumbbbiiiooooooooo/eeeeeen
2

---

Training distribution
Novel synonyms
Novel semantic classes
Novel domain (Real)
Pouse plant
potted alocasia
feddy bear
Wooden pouf
Iapsop
computer
hydrangea
iphone
ighh
00D
Nouns
(+) Geometry
(+) ODD Visual Properties
Openworidness
Figure 2: Open-world generalization requirements build top of each other, forming tiers of open-worldness (distinct property of each tier outlined yellow).
and coverage pale in comparison to intemet-scale image-text pairs [7-9]. However, our key insight is that reasoning be learned separately from 3D reasoning. That is, with the right abstraction, the complex visual-semantic reasoning can be offloaded to 2D VLMs, while the 3D model specializes in semantic-agnostie spatial and geometry reasoning.
3 Method: Semantic Abstraction
Evidence from linguistics and cognitive science suggests that semantic concepts are much more diverse than spatial and geometric concepts. While the average adult knows roughly 10,000 nouns for object categories. there are only 90 words in English for describing spatial relations [37]. fact, there are so few spatial prepositions that they are typically considered closed-class vocabulary [38]. Similarly for 3D geometry, the idea that our 3D world can be decomposed into sets of primitives [39] has been applied in many works, such generalized cylinders [40] and block worlds [41]. In contrast, the space of nouns for semantic concepts large and often cannot be further decomposed into common primitives. This suggests that while we may need large and diverse dataset for semantic and visual concept leaming, relatively small 3D dataset could cover spatial and geometric reasoning. This observation motivates our approach, Semantic Abstraction.
3.1 Abstraction via Relevancy
The Semantic Abstraction (SemAbs) module (Fig 3c) takes as input an RGB-D image I "x and an object class text label T (e.g. "biege armehair") and outputs the 3D occupancy O for objects of class T. factorizes into two submodules:
The semantic-aware wrapper (Fig 3c, green background) abstracts I and T into a relevancy map R HWW where each pixel's value denotes that pixel's contribution to the VLM's classification score for T. Introduced
model explainability [18, 22], relevancy maps can treated as coarse localization of the text label. Using the depth image and camera matrices, R is projected into a 3D point cloud, H where R (a 3D location with scalar relevancy value). Only Reroj but neither
Re = [x the text nor the image I, passed to the second submodule.
The semantic-bsstracted 3D module (Fig 3c, grey background) treats the relevancy point cloud the localization of a partially observed object and completes it into that object's s 3D occupancy. To do this, we first scatter Rrooj into 3D voxel grid Rox E IR Dx 128x 128x 128 Then, we encode Rox as a 3D feature volume: feneode (R Z ER ERDx Dx x 128x 128, where femende is 3D UNet [42] with levels. Z can be sampled using trilinear interpolation to produce local features o ERD for any 3D query point q E 3R3 Finally, decoding o6 with a learned MLP fáecode gives us an occupancy probability for each point g, Jeicode (<b²) o=) E [0.1]. In this submodule, only Semende and Sdecnde (Fig. 3c, yellow boxes) are trained with the 3D datasct.
Although the semantic-abstracke 3D module only observes relevancy abstractions of the semantic label T but not itself, the semantic-abstree -bbttaccdabttttceee 3D module's output can interpretted as the 3D occupancy for T. This means it generalizes to any semantic label that can be recognized by the 2D VLMs' relevancy maps even it was trained on limited 3D dataset. In our implementation, we use CLIP [7] as our VLM. However, our framework is ILM-agnostic, as long as relevancy maps can be generated. It is interesting future direction to investigate how different VLMs [7-11], their training procedure [43], and different relevancy approaches [18, 22, 44, 45] affect the performance of different downstream 3D scene understanding tasks. In the next sections, we explain how ensure the visual-semantie concepts are reliably recognized by the relevancy maps (Sec. 3.2), and how to apply SemAbs to 3D scene understanding tasks (Sec. 3.3, Sec. 3.4).

---

Semantic Labels [seen/unseen]
a) Open Vocabulary Semantic Scene Completion
b) Visually Obscured Object Localization
tarpel
frselace candle
large Indoor plant a pot blege armchair)
sample
CoRL ticket
foreat
cheetahh-pateemed pillow
(xX.2)
on top of fireplace
RGB-D Image
floor
3D festures
sample (Kx22)
f
Appatiat
index
c) Semantic Abstraction (SemAbs)
Semantic-Aware Wrapper
biege armchair
Semantic-Abstracie 3D Module
Relevancy Extractor
RGB
project
scatter
Sencode
Seccode
Depth Camera
(xyy2)
Occupancy Semantic Label
Matrices
Relevancy Point Cloud
Query Points
3D Occupancy
Figure 3: Semantic Abstraction Overview. Our framework can be applicd to open-world 3D scene understanding tasks (a-b) using the SemAhs module (c). It consists of semantic-aware wrapper (green background) that abstracts the input image and semantic label into relevancy map, and a semantic-absstraated 3D module (grey background) completes projected relevancy map into 3D occupancy. This abstraction allows our approach to generalize long-tail semantic labels unseen (bolded) during 3D training, such as the "CoRL ticket on top of the fireplace".
3.2 A Multi-Scale Relevancy Extractor
We choose to use ViT-based CLIP [7] due to their supe- rior performance over the ResNet variants. However, existing VIT relevancy techniques [18] often produce noisy, low-resolution maps that highlight irrelevant regions or miss small objects [27, 46).
Chefer et
(+) Augmentations (+) scale (Ours)
USSA
To reduce noise from irrelevant regions, use hor- izontal flips and RGB augmentations [27] of and obtain a relevancy map for each augmentation. To reliably detect small objects, we propose a multi-scale relevancy extractor that densely computes relevancies at different scales and locations in a sliding window fashion (an analogous relevancy-baset approach Li et al.'s local attention-pooling) The final relevancy map is averaged across all augmentations and scales.
Minterad swicch
Figure 4: Our relevancy extractor robustly detects even small, long-tail objects, like the "nintendo switch".
Qualitatively, while augmentations help in reducing irrelevant regions (e.g. Fig. 4, from localizing the entire fireplace in the 1st column to only the vase in the 2nd column), they still often miss small objects. However, with our multi-scale relevancy extractor, our relevancy maps can detect small objects such as the tiny "nintendo switch" (Fig. 4, bottom right). To support efficient multi-scale relevancy extraction, we implemented a batch- parallelized ViT relevancy extractor. At 0.4 seconds per text label on average", we achieve over 60 speed up compared to the non-parallelized implementation. With the rise in popularity of ViTs, we believe this efficient ViT relevancy extractor implementation will be a useful primitive for the research community and have released it Github and Hugging Face Spaces.
3.3 Application to Open Vocabulary Semantic Scene Completion (OVSSC)
Task. Given input a single RGB-D image I and a set of K object class labels, represented open- vocabulary text K the task is to complete the 3D geometry of all the partially observed objects referred by { k-1- In contrast to its closed-world variant [33], where the object list is fixed for all scenes, OVSSC formulation accounts for context-dependency. which allows the class labels to change from scene to scene and from training to testing. From this 3D semantic scene completion, robots can use the detailed 3D geometry for low level planning such grasping and motion planning.
measured over 10 repetitions with 100 text labels on a GTX 1080 Ti machine with light load, We provide extra details in the appendix.

---

Applying SemAbs to OVSSC is matter of applying the SemAbs module K times, once per object label. First, the semantic-aware wrapper abstracts I and {Tx ) into K relevancy point clouds { ] Then, the semantic-abstracted 3D module extracts K feature volumes fe=1 used to decode K 3D occupancies [Os))== The final OVSSC output is point-wise argmax of occupancy probability over the K classes. Query points with the maximum occupancy probability less than a threshold e are labeled as empty. We provide data generation and training details in the appendix.
3.4 Application to Visually Obscured Object Localization (VOOL)
Motivation. Even after completing partially visible objects in the scene, OVSSC's output may not be complete- - the scene may contain visually obscured or hidden objects (e.g., the ticket on the fireplace in Fig. 3b). In such cases, the human user can provide additional hints to the robot using natural language to help localize the object, which motivates our VOOL task.
Task. Given as input a single RGB-D image I of a 3D scene and a description D of an object's location in the scene, the task is to predict the possible occupancies of the referred object. Similar to prior works [47], we assume that the description Specifically, D SS where the open-vocabulary target object label TTunger (e-g- "rapid test") is the visually obscured object to be located, the open- vecabvocabulary reference object label Tae (e.g. "harry potter book") is presumably visible object, and the glosed-vocabull spatial preprosition S describes Turger's 's location with respect to Ter (e.g. "behind"). Unlike the referred expression 3D object localization task [6] which assumes full visibility (i.e., fused pointcloud), VOOL takes as input only the current RGB-D image, and thus is more suited for dynamic, partially observable environment. Further, in contrast to bounding boxes, VOOL's occupancy can be disjoint regions in space, which accounts the multi-moda uncertainty when objects are hidden. Downstream robotic applications, such as object searching, benefit more from the uncertainty information encoded in VOOL's output than bounding boxes.
in the standard linguistic representation of an object's place [37].
Applying SemAbs to VOOL. We propose to learn an embedding (S) E R22 for each spatial relation (Fig. 3b, blue box), since the set of spatial prepositions is small and finite [37], and use them as follows. As in OVSSC ($3.3), we can get occupancy feature volumes Zurgax and Zee for the target and reference objects respectively. Given set of fixed query points Q {q: J
we extract their
local feature point clouds &Q 7runs Zni .QQ E: ERNND, then concatenate them to get a new feature pointcloud
7nd E N22D Finally, to get the 3D localization occupancy O we perform a point-wise cosine- Zuugall ll QQ
similarity between d Zin and (Spatial(S). For this task, we only need to learn the spatial embeddings fepatial conditioned on the occupancy information encoded in Zurget and Zea since semantic reasoning of Turger and Tre has been offloaded to the SemAbs (and, therefore, to CLIP). We provide data generation and training details in the appendix.
Experiments
We design experiments to systemically investigate Semantic Abstraction's open-world generalization abilities to novel' concepts in 3D scene understanding. Our benchmark include the following categories: 1) Novel Rooms: we follow A12-THOR [1]'s split, which holds out 20 rooms for as novel rooms, while training on 100 rooms. 2) Novel Visual: we use A12 12-THOR's 's randomize material and lighting functionality to generate more varied 2D visual conditions of novel rooms. 3) Novel Synonyms: we manually picked out 17 classes which had natural synonyms (e.g. "pillow" to "cushion"", complete list included in appendix) and replace their occurrences in text inputs (i.e., object label in OVSSC, description in VOOL) to these synonyms. 4) Novel Class: we hold out 6 classes out of the 202 classes in A12-THOR. For OVSSC, any view which contains one of these novel classes is held out for testing. For VOOL, any scene with description that contains a novel class (as either the target or reference object) is held out. 5) Novel Domain: We also provide qualitative evaluations (Fig. 5) using real world scans from HM3D [4].
Metrics & Baselines. For both tasks, we measure the learner's 's performance with voxel IoU of dimension 32 x x2,, where the ground truth of each task is generated as described in $3.3 for OVSSC and $3.4 for VOOL.. We compare with following categories of baselines:
Semantic-Aware (SemAware). To evaluate the effectiveness of semantic abstraction, we design baselines that perform visual-semantic reasoning themselves by taking RGB point clouds as input
Here, we define "novel" as to whether concept observed during the 3D training process.
5

---

Partial RGB pointcloud
Scene completion
Localization prediction
"Hogwarts right of leather sofa'
leather sofa roomba outlet
in the roomba'
Real Warld Matterport Scans
koys behind small plant'
"pink and blue hydrangea on window sill"
red sofa
wooden pouf blue recliner plant
"Vaccination the blue folder"
'Vaccination in the red folder'
black aluminum globe book shelf house plant
"eychainn on side table"
"banana inside side table"
sofa plano side table
Simulation
keychain 0.00
keychainn loU:
floor
wall
pillow
window
carpet
light
ceiling
chair
table
curtain
Figure 5: Semantic Abstraction inherits CLIP's visual-semantic reasoning skils From distinguishing colors (e.g. "red folder" v.s. "blue folder") to recognizing cultural (e.g. "hogwarts box") and long-tail -ail semantic concepts (e.g. "roomba", "hydrangea"), our approach offloads such visual-semantic reasoning challenges CLIP. Indoing so, its learned 3D spatial and geometric reasoning skills transfers sim2real zero-shot manner.
(instead relevancy point clouds). For OVSSC, the baseline's feature point cloud supervised with BCE to be aligned (i.e., using cosine similarity) with the correct object class text embedding. Similarly, for VOOL, feature point cloud's cosine similarity with an embedding
entire localization description (e.g. "N95s inside the trash") is supervised using BCE to the target object occupancy.
Semantic & Spatial Abstraction (ClipSpatial). What we off-load both 3D and
reasoning to 2D VLMs (instead of just the latter, in approach)? To answer this, designed CLIPSpatiil, a VOOL baseline that uses relevancy maps for the entire description (e.g. "N95s inside the trash") as input. We expect this approach perform poorly since it has been demonstrated that current 2D VLMs [7, 8] struggles with spatial relations [14-16, 48).
Naive Relevancy Extraction. (SemAbs + [18] To investigate the effects of relevancy quality, we replace our multi-scale relevancy extractor with Chefer et al. [18]'s approach.
The first two baselines be seen as two extremes
spectrum of how much rely pretrained 2D VLMs. While SemAware learns both semantics and 3D spatial reasoning. CLIPSpatial delegates both to pretrained 2D VLMs. Our approach is in between these extremes, designed such that it builds on 2D VLMs' visual robustness and open-classification strengths while addressing its spatial reasoning weaknesses.
4.1 Open-world Evaluation Results
Tab. and 2 summarize the quantitative results, and Fig. 5 shows real world qualitative results tested Matterport scenes [4]. More results and comparisons can be found on the project website.
Semantic Abstraction simplifies open-world generalization. lines especially struggle
SemAware
base-
generalize to novel semantic classes
(Tab. 1, Tab. 2).

---

Input RGB-D
Completion
Relevancy Maps
puppy toy
cark hoard book
bed
certificate
curtain
lamp
night stand
bed
certificate
puppy key
curk hoard
alarm clock
ipod
towel
hand sculpture
redietor
alarm clock
hand sculpture
ipod
pipe
container
shoe
shoe
ratiator
towel
dresser
floor
well
Figure 6: Qualitative result on NYUCAD [49, 50J dataset. Our model can distinguish between "puppy loy", "ipod", "certificate", and "hand sculpture", while all prior works would've ve output "obj".
We hypothesize that observing text embeddings during training causes these baselines to specialize to this dis- tribution of embeddings, such that some synonyms and most novel class labels are diitt--distiiuutin inputs. Our approach significantly outperforms the SemAware baseline across the board, suggesting that Semantic Ab- straction not only simplifies open-world generalization is also a strong inductive bias for learning task.
Novel
Approach
Room
Visual Synonyms Class
SemAware SemAbs+[18]
32.2
31.9
20.2
0.0
26.6
24.3
17.8
12.2
Ours
40.1
36.4
33.4
37.9
Table I:
Open-Vocab Vocab Semantic Scene Completion
Semantic & Spatial Abstraction generalizes poorly. The CLIPSpatial baseline achieves 8.0-9.0 IoU worse than our approach in all generalization scenarios (Tab.
2). This indicates that our design of learning spatial reasoning instead of relying on CL.IP was crucial to semantic abstraction's performance.
Semantic Abstraction requires high quality relevancy maps. Intuitively, the model can only output high quality completions its relevancy maps are also high quality (i.e.. highlights the target object, especially when the object is small). Our average IoU is x1.5 (novel room, visual, synonyms, Table 1), over x2.0 (novel classes, Table 1, novel room, visual, synonyms, Table 2) or x5.0 (novel class, Table 2) compared to SemAbss [18], which demonstrates the importance our relevancy extractor in successful application of Semantic Abstraction.
Novel
Approach
Room
Visual Synonyms Class
SemAware SemAbs+[18)
12.1
11.9
11.4
3.0
9.1
8.8
10.7
4.0
CLIPSpatial
11.7
10.1
14.3
11.2
Ours
20.9
19.2
23.4
19.7
Table 2: Visually Obscured Object Localization
Inheriting VLM zero-shot robustness enables Sim2Real transfer. Despite training in simulation, our approach can perform completion and localization from real-world matterport scans [4] (Fig. and 5). From novel vocabulary for describing materials (e.g. "leather sofa", "wooden pouf"", colors (e.g. "red folder", "blue folder"), conjunctions of them (e.g. "black aluminum file cabinet"") novel semantic classes (e.g. "roomba", "Hogwarts box"), our approach unlocks CLIP's visual robustness and large repertoire of -esualisemattic concepts 3D scene understanding. When OVSSC's completion misses objects (e.g. "hydrangea", "keychain"), VOOL can still localize them (e.g. +0.13 IoU for keychain).
4.2 Zero-shot Evaluation Results
The NYUv2 CAD dataset [49, 50] is a popular dloeed-vocabulary SSC benchmark. Although it contains 894 object classes, prior works [33, 51, 52] collapse it down to classes to avoid the challenge in learning the "long-tail" categories (e.g. "puupy toy", "ippo"", "hand sculture"). We evaluate our model zero-shot on NYUv2 directly on all 894 classes as well as the 11 classes for comparison (Fig. 6, Tab. 3, Ours). We also train a SemAbss model from scratch in NYU to investigate the generalization gap between our dataset and NYUv2 (Tab. 3, Ours NYU). We use the same training/testing split [33]. Similar to prior works [33], we report volumetric IoU at voxel dimension of 60. As in our THOR dataset, we only train Seneade. and Steewie in the semantic abstracted 3D module, not the VL.M. To our knowledge, our approach is the first to output the full list of 894 object categories in this benchmark.
7

---

Approach
ceil.
floor
wall
win.
chair
bed
sofa
table
fur..
objs.
avg.
SSCNet [33]
32.5
92.6
40.2
8.9
33.9
57
59.5
28.3
8.1
44,8
25.1
36.6
SISNet [51]
63,4
94.4
67.2
52.4
59.2
77.9
71.1
51.8
46.2
65.8
48.8
63.5
Ours Supervised
22.6
46.1
33,9
35.9
23.9
55.9
37.9
19.7
30.8
39.8
27.7
34.0
Ours Zeroshot
13.7
17.3
13.5
25.2
15.2
33.3
31.5
12.0
23.7
25.6
19.9
21.0
Table 3: NYUv2 CAD Semantic Scene Completion.
Open-world versus Closed-world formulation. Our work investigates open-world 3D scene understand- ing, which means specializing to one data distribution is not our focus. However, we hope these results on standard closed-worke benchmark along with our discussion below are helpful to the research community in understanding practical trade-off's in algorithm and evaluation design between the open-world versus closed-world formulation. Below summarizes our key findings:
Flexibility to object labels. Our framework directly outputs the full 894 object categories, of which 108 do not appear in the training split, and its predictions can map back to the collapsed 11 classes. This has never been attempted by prior works. This flexibility is achieved by building on top of 2D VLMs.
Robust detection of small, long-tail object categories. "Due to the heavy-tailed distribution of object instances in natural images, large number of objects will occur too infrequently to build usable recognition models" [41]. However, our approach can robustly detect not just long-tail tail object classes, but also small ones challenging scenarios for perception systems. For instance, from Fig. 6, our multi-scale relevancy extractor can output highly localized relevancy maps around "puppy ",""""" "ipood", and "hand sculpture".
Inability to finetune large pretrained models. As shown in prior works [7, 10, 12, 13, 19], directly finetuning 2D VLMs on small datasets hurts their robustness. We cannot finetune the visual encoders (and thus, its relevancy activations) to specialize on the visual and class distribution on a small dataset (e.g.. NYUv2) without sacrificing it robustness and generality. This is open and important research direction and recent work [53] has demonstrated promising results in image classification.
Bias from image caption data. The typical training paradigm of current 2D VLMs [7, 10, 11] is contrastive learning over image-caption pairs on the internt.. While their dataset covers a large number of semantic concepts, they still have their limits. For instance, we observed that for "wall" and "ceiling" two classes which closed-works approaches typically achieve high performance for our approach achieved an much lower IoU of only 33.9 and 22.6. Since these classes are often not the subject of internet captions, our approach inherit this bias from 2D VLMs. However, our framework is VLM- agnostic, which means it can continue to benefit from future and more powerful versions of 2D VLMs.
4.3 Limitations and assumptions
Our key assumption is that geometry and spatial reasoning can be done in semantic agnostic manner. This assumption does not hold when we care about completing detailed object geometry, which is often semantic-dependent However, we believe that coarse 3D reasoning with paired semantic lahels may already useful for open-world robotic applications such as object retrieval. Second, while we have assumed simple description syntax and a small set of spatial prepositions for object localization, incorporating natural language understanding (i.e., to support open-set phrases for richer spatial descriptions) into wpendworld 3D scene understanding is an important extension of our work. Lastly, we assumed that relevancy maps highlight only relevant regions. Limited by current VLMs, our relevancy extractor can sometimes fail to distinguish highly related concepts (e.g. book v.s. bookshelf). However, since Semantic Abstraction is VI.M-agnostic, this limitation could be addressed by using other VLMS.
5 Conclusion
We proposed Semantic Abstraction, framework that equips 2D VLMs with 3D spatial capabilities for open-worke 3D scene understanding tasks. By abstracting RGB and semantic label inputs as relevancy maps, our learned 3D spatial and geometric reasoning skills generalizes to novel vocabulary for object attributes and nouns, visual properties, semantic classes, and domains. Since scene completion and 3D object localization are fundamental primitives for many robotic tasks, we believe our approach can assist in bridging currently fose-world robotics tasks, such as object retrieval, into the open- world formulation. More importantly, we hope our framework encourages roboticists embrace building systems around large pretrained models, as path towards open-world robotics.
8

---

Acknowledgments: We would like to thank Samir Yitzhak Gadre, Cheng Chi and Zhenjia Xu for their helpful feedback and fruitful discussions. This work was supported in part by NSF Award #2143601, #2132519 JP Morgan Faculty Research Award, and Google Research Award. The views and conclusions contained herein are those the authors and should not be interpreted as necessarily representing the official policies, either expressed implied, of the sponsors.
References
[1] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. AI2-THOR: An Interactive 3D Environment for Visual AL. arXiv, 2017.
[2] S. Song, S. P. Lichtenberg, and J. Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567-576, 2015.
[3] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1/22v25//..03012. 2015.
[4] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Tumer, E. Under- sander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra. Habitat-matterpon 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AL. In Thirty-fifih Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL http://ppeeeeeeeenne/oommmdddddd v4Qg.s.
[5] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistants to
rearrange their habitat. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
[6] D. Z. Chen, A. X. Chang, and M. Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In European Conference on Computer Vision, pages 202-221. Springer, 2020.
[7] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 9748-8763. PMLR, 2021.
[8] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi. Aligm before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34, 2021.
[9] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Bar, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[10] H. Pham, Z. Dai, G. Ghiasi, H. Liu, A. W. Yu, M.-T. Luong, M. Tan, and Q Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2/11.10055 2021.
[11] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
[12] M. Wortsman, G. Ilharco, M. J. W. Kim, H. Hajishirzi, A. Farhadi, H. Namkoong, and L. Schmidt. Robust fine-tuning of zero-shot models. arXiv preprint arXiv:2109.01900 2021.
[13] A. Andreassen, Y. Bahri, B. Neyshabur, and R. Roclofs. The evolution oitttdiiittttnn robustness throughout fune-tuning, arXiv preprint arXiv:2106. 1ae.::210..1111... 2021.
[14] Subramanian, W. Merrill, T. Darrell, M. Gardner, Singh, and A. Rohrbach. Reclip: strong zero-shot bascline for referring expression comprehension. arXiv preprint 2rXiv:2204.05991, 2022.

---

[15] S. Y. Gadre, M. Wortsman, G. Ilbarco, L. Schmidt, and S. Song, Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprint arXiv:2203.10421, 2022.
[16] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXhv:2204.01125, 2022.
[17] Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[18] H. Chefer, S. Gur, and L. Wolf. Generic attention-mode model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397-406 2021.
[19] C. Zhou, C. C. Loy, and B. Dai. Denseclip: Extract free dense labels from clip. arXiv preprint 1rX1:2212..1001, 2021.
[20] Y. Rao, W. Zhao, G. Chen, Y. Tang, Z. Zhu, G. Huang. Zhou, and J. Lu. Denseelip: Language- guided dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages 18082-18091, 2022.
[21] J. Li, G. Shakhnarovich, and R. A. Ych. Adapting clip for phrase localization without further training. arXiv preprint arXiv:2204.08 036477, 2022.
[22] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[23] R. Mokady, A. Hertz, and A. H. Bermano. Clipcap: Clip prefix for image captioning, arXiv preprint arXi:2211.009334, 2021.
[24] H. Song, L. Dong, W.-N. Zhang, T. Liu, and F. Wei. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. arXiv preprint arXfv:2203.07190, 2022.
[25] Z. Gao, J. Liu, S. Chen, D. Chang, H. Zhang, and Yuan. Clip2tv: An empirical study on transformer-bssel methods for video-text retrieval. arXiv preprint arXiv:21 2021.
[26] M. Wang, J. Xing, and Y. Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arxiv:2 08420 2021.
[27] N. Zabari and Y. Hoshen. Semantic segmentation in-the-wild without seeing any segmentation examples. CoRR, abs/2112.03185; 2021. URL http:://arrivooo/////1111111111
[28] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation.
In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
[29] A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani,
J. Lee, V. Vanboucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXhv:2204.0053 2022.
[30] Shuran Song and Xiao. Sliding Shapes for 3D object detection in depth images. 2014.
[31] Shuran Song and J. Xiao. Deep sliding shapes for amodal 3D object detection in rgb-d images. 2016.
[32] J. Hou, A. Dai, and M. Nießner. 3d-sis: 3d semantic instance segmentation of rgh-d scans. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2019.
[33] S. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, and T. Funkhouser. Semantic scene completion from single depth image. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 1746-1554 2017.
[34] A. Dai, D. Ritchie M. Bokeloh, S. Reed, J. Sturm, and M. Nießner. Scancomplete: Large-scale scene completion and semantic segmentation for 3d scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4578-4581 2018
10

---

[35] A. Avetisyan, M. Dahnert, A. Dai. M. Savva, A. X. Chang, and M. Nießnner Scan2cad: Learning cad model alignment in rgb-d scans. In Proceedings the IEEE/CVF Conference on computer vision and pattern recognition, pages 2614-2622 2019.
the
[36] J. Roh, K. Desingh, A. Farhadi, and D. Fox. Languagerefer: grounding. In Conference on Robot Learning, pages 1046-1046 PMLR, 2022.
model for 3d visual
[37] L. Barbara and R. Jackendoff. "what" and "where" in spatial language and spatial cognition. Behavioral and brain sciences, 16:217-238, 1993.
[38] W. G. Hayward and M. J. Tarr. Spatial language and spatial representation. Cognition, 55(1):39-84 84, 1995.
[39] S. Tulsiani, H. Su, L. Guibas, A. Efros, and Malik. Learning shape abstractions by assembling volumetric primitives, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2635-2643, 2017.
[40] I. Binford. Visual perception by computer. In IEEE Conference of Systems and Control, 1971.
[41] Gupta, A. A. Efros, and M. Hebert. Blocks world revisited: Image understanding using qualitative geometry and mechanics. In European Conference on Computer Vision, pages 482-496. Springer, 2010.
[42] Ö. Çişek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger. 3d u-net: Learning dense volumetric segmentation from sparse annotation. CoRR, abbs/1606.06650, 2016. URL http: /arxxv.org/abs/ www..orggaaa//106666665555
[43] H. Chefer, I. Schwartz, and L.. Wolf. Optimizing relevance maps of vision transformers improves robustness. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022. URL
[44] H. Chefer, Gur, and L. Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recoguition, pages 782-791, 2021.
[45] Y. Liu, H. Li, Y. Guo, C. Kong, J. Li, and S. Wang. Rethinking attention modell explainability through faithfulness violation test. arXiv preprint arXiv:2201.121 2022.
[46] S. Shen, L. H. Li, H. Tan, M. Bansal, A. Rohrbach, K.W. Chang, Z. Yao, and K. Keutzer. How much can clip benefit vision-and-languugge tasks? arXiv preprint arXiv:2 107.06383. 2021.
[47] N. Liu, Li, Y. Du, J. Tenenbaum, and A. Torralba. Leamingg to compose visual relations. Advances in Neural Information Processing Systems, 34, 2021.
[48] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. When and why vision-language models behave like bags-of-words, and what to do about it?, 2022.
[49] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. Indoor segmentation and support inference from rghd images. In European conference computer vision, pages 746-760. Springer, 2012.
[50] R. Guo, C. Zou, and D. Hoiem. Predicting complete 3d models of indoor scenes. arXiv preprint arXiv: see.l00444224 2015.
[51] Y. Cai, X. Chen, C. Zhang, K.-Y. Lin, X. Wang, and H. Li. Semantic scene completion via integrating instances and scene in loop. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 324-333- 2021.
[52] P. Zhang, W. Liu, Y. Lei, Lu, and X. Yang. Cascaded context pyramid for full-resolution 3d semantic scene completion. In Proceedigss of the IEEF/CVF International Conference on Computer Vision, pages 7801-7810, 2019.
[53] M. Wortsman, G. Ilharco, Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. URL https//arxiv.orr/ abs/2203.05482.
11


Reviews:
[{"review_id": "uI8B-fQxrel", "reviewer": "Reviewer_jSHt", "summary": "By combining the robustness in open-vocabulary scene understanding from a vision-language model in RGB images and a semantic-agnostic spatial understanding from a 3D model, it achieves open-world 3D scene understanding with a small set of 3D datasets.\nIt proposes two tasks completing partially observed objects and localizing hidden objects to show out-of-distribution generalization and spatial understanding ability on the partially observed 3D dataset. It provides various ablation models to help understand the effect of each part of the model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Provides a simple but powerful decomposition and abstraction of perception and localization for out-of-distribution detection in robotics applications.\n* Can leverage various VLMs as its module.\n* Uses multi-scale proposals from VLMs to handle small objects with batched high-speed inference\n\nWeaknesses\n* Ambiguous evaluation on newly proposed datasets and tasks; not clear why they are proposed other than the existing tasks such as detection or visual grounding.\n* Does not compare with existing methods in detection or visual grounding tasks; hard to understand the empirical contribution to real-world applications.\n* Proposed datasets on AI2-THOR which is less diverse in appearance than other datasets such as ScanNet or SUN-RGBD.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It proposes a modular approach for 3D understanding with out-of-distribution generalization by combining vision-language model (VLM) with a semantic-abstract 3D model.\nIt is one of the challenging tasks in robotics due to the limited size of 3D datasets but is crucial to real-world applications.\nIt shows that the proposed method provides better generalization both in detecting small objects by introducing multi-scale feature extraction from VLM and identifying novel and fine-grained classes in 3D.\nWhile its application is crucial for advancing the 3D understanding, its evaluation is not based on the existing tasks such as 3D detection or visual grounding, nor on existing approaches; therefore, it is not trivial to clearly understand their contribution compared to the existing literature.\nTherefore, I recommend a weak acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "In general, evaluation is a little bit confusing; the first task, open vocabulary semantic scene completion (OVSSC), seems to be similar in open vocabulary detection/segmentation and the second task, visually obscured object localization (VOOL), can be considered as a specialized case of visual grounding. Besides the ablations, it lacks direct comparison to existing models or tasks. For instance, applying the proposed approach to Refer-it-in-RGBD (https://arxiv.org/abs/2103.07894) might be interesting to see how different approaches to spatial understanding work. How the model works in the existing detection tasks is another task of interest.\n\n* Sec 3.3; how the 3D semantic-abstracted module is robustly working with ambiguous saliency or relevancy from the previous module.\n* Hard to understand the effect of the Z_target in the VOOL task; \n* How are query points Q chosen?\n* Why 32x32x32? 3D detection methods can use refined volume up to 1 cm resolution.\n* Why AI2-THOR? AI2-THOR is less diverse in terms of appearance, object classes and instances in each class, and types of rooms. While arguing its novelty in out-of-distribution generalization in vocabulary and appearance, testing only on AI2-THOR makes this approach less compelling. \n* Visual grounding methods in either \na referring expression comprehension task or a referring expression segmentation task should be compared as a baseline such as `LAVT: Language-Aware Vision Transformer for Referring Image Segmentation', `CRIS: CLIP-Driven Referring Image Segmentation', `OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework' (https://paperswithcode.com/sota/referring-expression-segmentation-on-refcoco-4, https://paperswithcode.com/sota/referring-expression-comprehension-on-refcoco) in order to show the evidence of the statement argued in line 237. Some of the spatial relationships can be interpreted in 2D (such as ``left of'') and it would be interesting to compare two different approaches that have different levels of task decompositions. At least, some 2D visual grounding models can be used in ClipSpatial.\n* Instead of taking a single expression per instance, is choosing K objects at once in the OVSSC task affect the result? I wonder if exclusion happens or is encouraged during the process.\n* Known-class accuracy (in comparison to existing detection methods) is missing.\n* While being trained on six spatial prepositions, no information on spatial prepositions in the custom dataset is not provided; providing a success rate on each preposition may help better understand the problem.\n\n* Typo in line 221: FOr > For"}}, {"review_id": "T5omFjG6ne1", "reviewer": "Reviewer_ZWJc", "summary": "The paper proposes SemanticAbstraction; a framework that incorporates 3D spatial capabilities to 2D vision-language models while allowing said models to keep their zero-shot learning abilities. The framework is comprised of 2 modules:\n- Semantic Aware Wrapper which abstracts the input image given a semantic label using ViT-CLIP into a relevancy map projected to 3D\n- Semantic Abstracted 3D Module which converts the projected 3D relevancy map into 3D occupancy prediction using a 3D encoder-decoder based model operating on 3D voxel grids\n\nThe framework is trained on synthetic only data and evaluated on 2 main tasks:\n- Open Vocabulary Semantic Scene Completion (OVSSC): predicting 3D occupancy of objects given a single RGB-D image with object labels\n- Visually Obscured Object Localization (VOOL): predicting possible occupancy locations of referred objects given an input image\nThe proposed method is compared against baselines and performs favorably.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written, easy to follow\n- The work addresses an important problem for robotics; the ability to link language and visual models while being robust to long-tail distribution descriptions\n- The experimental results are quite promising\n- Releasing code, dataset and models is beneficial to the advancement of this area of research\n\nWeaknesses & Questions:\n- It is not very clear to the reviewer why the semantic abstracted 3D module is needed? Why not just use the 3D projected relevancy map as an output with some post-processing?\n- Using 3D voxel grids as a basis for the occupancy prediction can be potentially problematic for accurately localizing objects of small scale. Furthermore, maintaining a reasonable voxel size with large environments can lead to large memory consumption. It would be nice to see some information regarding the memory required or ablations on larger environments with small objects perhaps\n- Is the OVSSC task a realistic one? Normally one would either want the location of a single object (as in the VOOL task) for which a prerequisite is segmenting the environment (where one would not expect giving categories as input). My question here is what does this task help us understand that we could not before?\n- The framework was only evaluated on self-constructed baselines. While I understand that this area is rather novel with not enough related work that one can directly compare against. On the other hand, there are similar datasets like ScanRefer that offer a benchmark for a similar problem (albeit bounding box regression instead of segmentation). Can one perhaps adapt one of the methods from there or compare the current approach on said dataset?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper covers a key topic in robotics and presents very nice results showing generalization capabilities from synthetic dataset to real-world. \n\nI would like to thank the authors for their responses. Based on the discussion, I will maintain my score.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- What does BCE in line 230 stand for?\n- What's the motivation behind using a 3D CNN for encoding? Did the authors try something in the direction of SRT: https://srt-paper.github.io/ which would help side-stepping a number of problems relating to using voxel-based representations."}}, {"review_id": "L9sh8HwKyia", "reviewer": "Reviewer_Vpqh", "summary": "This paper formulates and studies the task of open-world 3D scene understanding. The key insight of the authors is that the task of spatial/geometric reasoning can be decoupled from the task of learning semantic embeddings of images and text. The paper presents a method that decouples the semantic component (using pre-trained a CLIP transformer) from the spatial component (3D point cloud completion), and show results in a simulator as well as some real-world 3D scans, along with generalization results to new environments and textual classes.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\n- The VLM relevancy module seems like a neat little idea by combining the GradCam-style visualization technique with vision transformers. To my best knowledge, this is the first work that demonstrates an effective way to do this, and I can think of several research areas that would benefit for this.\n- Despite being restrictive, the way the authors handle the spatial prepositions seems very simple, yet unique. I have seen a lot of CLIP-based papers struggle with this and it's very exciting to see this work well enough for a downstream application.\n- The evaluation benchmark, designed on the AI2-THOR simulator, seems very thorough and provides good insights on which components of the method contribute the most. This could be a valuable contribution to the 3D scene understanding community if made public along with reproducible baselines etc.\n- The supplemental video does a very good job of conveying the results and qualitatively comparing to baselines — I really like it!\n\n*** \nWeaknesses\n---\n- [Implementation Details & Reproducibility] My biggest concern with the paper in its current shape is the lack of implementation details in multiple sections throughout the paper. This, coupled with the lack of any code release makes the paper very hard to parse, reproduce, and draw any generalizable insights from (the authors mention “code, models will be publicly available” but my review focuses on the provided material). This further makes it difficult to judge what components of the method are novel and what components are used directly from prior (unreferenced) work. Below are some notable examples of crucial implementation/training details missing:\n\t- Section 3.3.1: There is no information here regarding the models or training pipeline. What is the learning objective, and how is it achieved? Is this a novel component, or do you borrow this from prior work?\n\t- Section 3.3.2: This is particularly hard to parse, since the “method” is barely described in text (how is the query point allocation/matching done? etc.), and there are absolutely no training details provided.\n\t- Section 3.2: There are some implementation details here but the important stuff is only mentioned in the supplemental material, making the text seem incomplete. Further, this is not referenced in the main text.\n- [Robotics Focus] The paper reads a lot like a typical computer vision paper that creates and evaluates on a task-specific benchmark without considering effects on a potential downstream robotic task (in simulation or real). Barring the conclusion (Line 292), the paper makes no effort to justify why the method may be well-suited for an embodied navigation task and why the proposed benchmark and metrics matter — why is the voxel IoU a good metric for a semantic navigation task where the robot just needs to reach a general location and may not care about the exact 3D voxels? Especially to the CoRL audience, the paper would benefit from a better discussion of the motivation and positioning wrt applications in robotics, if not by adding a relevant downstream application.\n- [Organization] In addition to missing implementation details, the paper is also missing any qualitative visualization of the comparative experiments and there is no qualitative or quantitative table/figure corresponding to the sim2real claim (Line 270) — this only makes an appearance in the supplemental material with only a placeholder in the main text. Maybe this is a late addition after the paper deadline, but in its current form, the paper just feels incomplete by itself and needs to be re-organized with more essential information in the main text.\n- [Experiments/Baselines] While the authors do a good job of evaluating the method on the proposed suite of tasks, I would argue that the evaluation is not against any “baselines”, and are closer to a set of ablations to the proposed method (except the one with Chefer et al.). This is not immediately a bad thing, but the authors should consider an analysis with respect to existing algorithms that work with a stricter set of assumptions to get a sense for how well the method performs. For instance, one such comparison can be against a SoTA closed-vocabulary scene completion algorithm, to understand how well SemAbs (proposed) done in comparison and where it lacks — does the CLIP-based model perform just as well as a more carefully engineered system or is it a long way from the best 3D scene understanding algorithms that the community has been developing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In my review, I raised a number of concerns regarding the current submission: primarily on the lack of implementation details and related reproducibility concerns and the inability to assess significance of claims for downstream robotics applications. While I really like some ideas proposed in the paper, in its current form, I cannot recommend acceptance of the paper, but I am very willing to engage with the authors in the discussion phase and reconsider my recommendation.\n\n---\n\n**Update**: The authors' responses have addressed my concerns regarding lack of implementation details and presentation, and the last round of revisions with qualified claims of \"open-vocabulary\" mostly addresses my concern as well. However, I maintain that the relevance of the findings (and the choice of metrics to determine superior performance) to eventual downstream applications is not very clear, especially for applications in robotics (per the CoRL guidelines). Contingent on the promised revisions to qualify the open-vocabulary claims and the added discussion of relevance to robotics, I do not have any strong issues with the paper. Erring on the side of being conservative, I am still keeping my original rating as I cannot (enthusiastically or weakly) recommend acceptance but would not argue against it.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- [Points Raised Above] I raised some concerns in the above section and some clarification in response to those points would really help me re-evaluate the submission score.\n\t- Reproducibility concerns\n\t- Positioning wrt robotics\n\t- Experiments\n- [Limited Spatial Understanding] The authors should consider toning down their “open-vocabulary” claims in light of the fact that (i) they expect a very specific type of phrase (Line 182) to describe the object in the scene, and (ii) they further reduce the set of spatial prepositions to just *six* (Line 195), which is much more restrictive. Generally speaking, the repeated claim of “small/finite words for describing spatial relations” [citing Landau and Jackendoff] is not entirely true and there are several other ways spatial relations may be described ([see commentary by Bennett on the same paper](https://ase.tufts.edu/cogstud/jackendoff/papers/what_where.pdf)). While this may not be a big limitation in practice, the current claims need to be qualified (or clarified) better to present a more accurate discussion of prior work and motivation."}}]

Author rebuttal:

Decision: Accept (poster)