OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data
Authors: Shiyang Lu
Haonan Chang
Eric Pu Jing
Abdeslam Boularias
Kostas Bekris
Venue: CoRL 2023 Poster (2023)
Keywords: Open-Vocabulary
3D Instance Retrieval

Abstract: This work presents OVIR-3D, a straightforward yet effective method for open-vocabulary 3D object instance retrieval without using any 3D data for training. Given a language query, the proposed method is able to return a ranked set of 3D object instance segments based on the feature similarity of the instance and the text query. This is achieved by a multi-view fusion of text-aligned 2D region proposals into 3D space, where the 2D region proposal network could leverage 2D datasets, which are more accessible and typically larger than 3D datasets. The proposed fusion process is efficient as it can be performed in real-time for most indoor 3D scenes and does not require additional training in 3D space. Experiments on public datasets and a real robot show the effectiveness of the method and its potential for applications in robot navigation and manipulation.

OVIR-3D: Open- 3D Instance Retrieval Without Training on 3D Data
Shiyang Lu Haonan Chang Eric Pu Jing Abdeslam Boularias Kostas Bekris
Rutgers University
https://github: wwttluu..o//sshiihooeee hwtp//hhoooggg//oV-----
Abstract: This work presents OVIR-3D, straightforward yet effective method for open-vocabulary 3D object instance retrieval without using any 3D data for training. Given language query, the proposed method able to return ranked set of 3D object instance segments based the feature similarity of the instance and the text query. This is achieved by multi-view fusion of text-aligned 2D re- gion proposals into 3D space, where the 2D region proposal network could lever- age 2D datasets, which are more accessible and typically larger than 3D datasets. The proposed fusion process is efficient as can be performed in real-time for most indoor scenes and does not require additional training in 3D space. Ex- periments on public datasets and real robot show the effectiveness of the method and its potential for applications in robot navigation and manipulation.
Keywords: Open Vocabulary, 3D Instance Retrieval
Introduction
There has been recent progress in open- open-vocabulary 2D detection and segmentation methods [1, 2, that rely pre-trained vision-language models [4, 5, 6]. However, their counterparts in the 3D) domain have not been extensively explored. One reason is the lack of large 3D datasets with suf- ficient object diversity for training open-vocabulary models. Early approaches for dense semantic mapping [7, 8, 9, 10] project multi-view 2D detections to 3D using closed-set detectors but can- not handle arbitrary language queries. More recently, OpenScene [11] and Clip-fields [12] achieve open-vocabulary 3D semantic segmentation by projecting text-aligned pixel features to 3D points and distilling 3D features from aggregated 2D features. Given a text query during inference, OpenScene [11] generates a heatmap of the point cloud based the similarity between point fea- tures and the query feature. Nevertheless, manual thresholding is required for object search vert a heatmap to a binary mask and it lacks the ability to separate instances from the same category. This limits its use in robotic applications, such as autonomous robotic manipulation and naviga- tion. Clip-fields [12], on the other hand, requires training on additional ground truth annotation for instance identification, which makes it less open-vocabulary at the instance level.
This work focuses open-vocabulary 3D instance retrieval, aiming to return ranked set of 3D instance segments given 3D point cloud reconstructed from RGB-D video and language query. Some examples are shown in Figure 1. While 2D segmentation from a single viewpoint is often insufficient for robot grasping and navigation, 3D instance retrieval methods generate a more complete and accurate segmentation of objects 3D space by multi-view fusion and smoothing. In particular, this work considers a scenario, where a mobile robot navigates in an indoor scene and automatically reconstructs the 3D environment using its RGB-D sensor and an off-the-shelf SLAM module. Instead of fusing pixel-level information and then either grouping them into instances by thresholding inference time [11] or training object identification model with additional ground truth data [12], the proposed method directly fuses instance-level information into the 3D scene without additional training, so that given a text query such as "lamp" or "bed", robot can immediately locate the top-related object instances and perform required tasks.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

b) Top 2 instances retrieved given the query: "bed"
c) Top 3 instances retrieved given the query: "lamp"
a) Original scan
d) Ground-truth annotations from ScanNet200
e) All text-queryable 3D Instances detected the scene.
Figure 1: Examples of open-vocabulary 3D instance retrieval using the proposed system, (a-c) Given 3D scan reconstructed from an RGB-D video (e.g., scene0645 from ScanNet [13]) and a text query (e.g., bed, lamp), the proposed method retrieves a set 3D instances ranked based on their semantic similarity to the text query. (d-e) Instances that are not even in the ground-truth annotations can also be detected and queried by the proposed method, such as the cushions on the sofa.
The proposed method addresses this problem by first generating 2D object region proposals and their corresponding text-aligned features by querying a 2D open-vocabulary detector with an extensive vocabulary. It then performs data association and periodic filtering and merging of 3D instances to improve instance masks and remove noisy detections. Finally, a post-processing step handles isolated objects and filters small segments that are likely to be noise. Extensive experiments on real scans from both room-scale dataset ScanNer200 [14] and tabletop-scale dataset J'BB-idee [15] demonstrate the effectiveness of the proposed method, which offers an efficient 2DD-to33 instance fusion module (~~ 30 fps for a scene in ScanNet [13] on an NVIDIA RTX 3090) and an open- vocabulary 3D instance retrieval method with near-instant inference time for text query.
The main contributions of this work are: (i) an efficient 2D-to-3D instance fusion module given text-aligned region proposals, which results in (ii) an open- vocabulayy 3D instance retrieval method that ranks 3D instances based on semantic similarity given a text query.
2 Related Work
2.1 2D Open- Voceauacyyylary Detection and Segmentation
With the advent of large vision-language pre-trained models, such as CLIP [4], ALIGN [5] and LIT [16), number of 2D open-vocabulary object detection and segmentation methods have been proposed [17, 18, 19, 1, 2, 20, 21J. For 2D semantic segmentation, LSeg [17] encodes 2D images and aligns pixel features with segment label embeddings. OpenSeg [18] uses image-level supervi- sion, such as caption text, which allows scaling up training data. GiT [19] performs bottom- up hierarchical spatial grouping of semantically-related visual regions for semantic segmentation. For 2D object detection, ViLD [1] achieves open-vocabulary detection by aligning the features of class-agnostic region proposals with text label features. Detic [2] attempts to address the long-tail detection problem by utilizing data with bounding box annotations and image-level annotations.

---

OWL-VIT [20] proposes pipeline for transferring image-text models to open-vocabulary object detection, Our proposed method adopts Detic [2] as a backbone detector to locate objects in 2D im- ages since can provide pixel-level instance segmentation and text-aligned features. Furthermore, it can be queried with a large vocabulary without sacrificing much speed.
2.2 3D Reconstruction and Glosed-Vocabulary, Semantic Mapping
Early works have addressed the 3D reconstruction problem either through online SLAM meth- ods [22, 23, 24, 25, 26] or offline methods like drructurefrrommmotion [27, 28] using a variety of 3D representations, such as TSDF [29], Surfel [30], and more recently NeRF [31, 32, 33, 34]. With the advancement of learning-hasee 2D object detection and segmentation methods, recent efforts have focused on point-wise dense semantic mapping of 3D scenes [7, 8, 10, 9, 35J. Despite being effective, these methods have not yet been designed to open-vocabulary detectors. They either assume mutually exclusive instances [7. 10] or utilize category labels for data association [8. 9, 35]. The proposed method in this paper adopts off ffffthh--helf 3D reconstruction method and focuses integrating 2D information with point-cloud information to achieve open-vocabulary 3D instance segmentation. A key contribution in this context is method that associates open-vocabulary 2D instance detections and fuses them into a 3D point cloud while keeping them open-vocabulary.
2.3 3D Open- Vece-Vocayulary Scene Understanding
More recently, research efforts aim for apen-vocabulary 3D scene understanding [12, 11, 36, 37, 38, 39, 40). Given that existing 3D datasets tend to be significantly smaller than 2D image datasets, this is mainly accomplished by fusing pretrained 2D image features into 3D reconstructions. Open- Scene [11] projects pixel-wise features from 2D open-vocabulary segmentation models [17, 18] to a 3D reconstruction and distills 3D features for better semantic segmentation. ConceptFusion [38] fuses multi-modal features, such as sound, from off-the-shelf foundation models that can only pro- duce image-level embeddings. LeRF [39] fuses multi-scale CLIP features to neural radiance field for open- ppen-vocabulary query. These methods can generate heatmap of a scene that corresponds to a query, but they do not provide instance-level segmentation, which limits their use in tasks that require robot interact with specific object instances. PLA [40] constructs hierarchical 3D-text pairs for 3D open-world learning and aims to perform not only 3D semantic segmentation but instance seg- mentation as well. Nevertheless, the method so far has been demonstrated only on certain furniture- scale objects, and performance in other categories is unclear. On the other hand, our method focuses on instance-level. open- vocabulayyy 3D segmentation without manual 3D annotation.
Problem Formulation
A 3D scan X represented by N points reconstructed from an RGB-D video V (I,,I,..... Ir} given known camera intrinsics C' and camera poses P., where I, is the video frame time The objective in open-vocabulary 3D instance retrieval is to return a set of K ranked instances represented as binary 3D masks M {mali [1, K]} over the 3D scan N given a text query Q and the desired number of instances K to be retrieved. The ranking of instance masks is based on the semantic similarity between the 3D instance and the text query, where the most similar instance should be ranked first.
4 Method
The overall pipeline of the proposed method is illustrated in figure 2. To summarize, given video frame, the method first generates 2D region proposals R2D = [ri,.., with text-aligned fea- tures FED = { ,.., {2D] using an off-the-shelf 2D open-vocabulary method trained on large 2D datasets. The 2D region proposals R2D of each frame I, are then projected the reconstructed 3D point cloud given the camera intrinsics C' and poses Pt. The projected 3D regions R3D are either matched to existing 3D object instances O {o1,.., o.) with 3D features påD = {fpD ..., f?? stored in the memory bank B, or added as a new instance if not matched with anything. The 2D region to 3D instance matching is based on feature similarity siy = cos f)D) and region over- lapping ?r?? . o,) in the 3D space. Matched regions integrated into the 3D instance. To remove unreliable detections and improve segmentation quality, periodic filtering and merging of
3

---

21k categories text queries
Pre trairranndd 2D Open- Vocabulary Instance Segmentation Method, Detic (Frozen)
2D region proposals with text-aligned features
Inference
Text-aligned 2D Region Proposal Generation
Query: "Sofa"
0
T
Region Proposals
Text Feature
Ranking
3D Projection Instance Fusion
Memory Bank
Memory Bank of Queryable 3D Instances
Post-processing
b) ZD-to-3D Instance Fusion
Periodic filtering and ieerinng of instances in the memory bank
Top 1 is shown in blue.
Figure 2: Pipeline of the proposed method.
3D instances in the memory bank B is performed every T frames. A final post-processing step moves 3D instances that are too small and separates object instances that are isolated in 3D space but incorrectly merged. During inference time, the text query will be used to match with set of rep- resentative features of each 3D instance, and the instances O will be ranked based on the similarity and returned. Details of the proposed method are presented below.
4.1 Text-aligned Region Proposal
Learning-based region proposal networks have served as critical module for many instance seg- mentation methods, such MaskRCNN [41]. However, directly generating 3D region proposals for open-voca voccbulaayyl instance retrieval is hard due to the lack of annotated 3D data with enough category varieties. This work views the 3D region proposal problem as a fusion problem from 2D region proposals. In particular, leverages the power of an off-the-shelf open-vocabulary 2D detec- tor Detic [2], which is trained with multiple large image datasets, to generate 2D region proposals (masks) R2D by querying it with an extensive number of categories, i.c. 21k categories from ImageNet21k [42] dataset. In addition 2D masks R2D associated text-aligned features F2D are extracted before the final classification layer of the 2D model. The output category labels are dropped without being used as they are rather noisy given the input vocabulary size. Both region proposals RD and their text-aligned features F2D are used for data association the fusion step.
Though the proposed method does not critically depend on a specific 2D detector, not all open- vocabulary 21) detectors can as region proposal backbone. There
requirements that 2D detector must exhibit: 1) It must be able to generate pixel-wise masks for wide variety of objects in a timely manner; and 2) It should provide a text-aligned feature for each region so that it can be queried with language. During the development of this work, it's found that Detic [2] naturally meets the requirement without additional modifications and thus it is adopted for this work. Some other options, such as SAM [43] and Grounding-DDie -IIN [44] were not adopted because they are either too slow unable to directly output text-aligned features for proposed regions, which turn out be critical for data association in the experiments. Detic [2] also has fast inference speed even when queried with all the categories from ImageNet21k [42] (~ 10fps NVIDIA RTX3090), which makes it favorable for this task.
4.2 2D-to-3D Instance Fusion
2D region proposals R2D {r ...,, 2D) and their corresponding features F2D {f?D S2D) for each frame I, are first projected the 3D scan using camera intrinsics C and pose Pt. The projected 3D regions R3D either matched to existing 3D object instances 0 (01, ... 06 with
(r.....,,7)
3D features F3D {J3D_.. _..)) where b is the number of 3D instances already stored in the

---

memory bank B, or added as a new instance if it is not matched with anything. The 3D feature f instance i for matching is simply the average of 21D features 12D from all the associated 2D regions, ie. f = fjD, where j is the index of 2D regions that are associated to 3D instance i, and n is the number of 2D regions that have already been merged. The memory bank is emply at the beginning.
The matching of 2D region r; to 3D instance Oy is based on cosine similarity Si = cos (PDD) and 3D intersection over union (r ,g)) between the projected region ra2 and visible part of the 3D instance oj in the current frame. If Sij is greater than a predefined threshold 8, (default 0. 0.75) and the overlapping (r .6)) is also greater than predefined threshold Ozn (default match. Matched regions will be aggregated the 3D
0o 0.25), then they are considered instance, i.e, 0j == Oj U r and f3D x+1 f3D 1 !.DD. The matching is not restricted to one-to-one multiple 2D region proposals may correspond to the same instance.
x2
4.3 Periodic 3D Instance Filtering and Merging
The fusion process is fast but it will generate redundant 3D instances when 2D region proposal fails to match properly, potentially leading to low-quality segmentation and inaccurate data association. To address this, periodic filtering and merging of 3D instances stored in memory bank B occur every T (default T 300) frames. Point filtering is based the detection rate ydle of point p. where pdtt = G/ /cp² Intuitively, this is the frequency of point being considered as part of the instance over the frequency of it being visible (cwie). Points with ydde < Odet (default fdet = 0.2) are removed from instance o;, Meanwhile, the number of points in each projected 3D segment from a single view that corresponds to o, is recorded. If after point filtering, instance o, contains fewer than the median number of points in its corresponding segments, then it is filtered entirely. This dynamic threshold that automatically adapts instance sizes is critical in the filtering process.
Merging of two instances Op, Og is determined by feature similarity Spg = cos(s2p JJDD) and 3D intersection over union IoU (ap. 0p) between two instance segments, using the same thresholds 0, and foom as in Section 4.2. Additionally, instances Op and o, are merged if recall(o, o2) = lop ogl/|og| I 2 frecall (default frecall = 0.25) and Spg 2 8,, indicating that Og is mostly contained in op and both instances have similar features.
are justified through ablation studies in Section 6, and these values remain fixed for experiments in Section 5 across different datasets.
4.4 Post-processing
A simple post-processing step is executed to separate object instances that are isolated in 3D space and filter small segments that are likely to be noise. This is achieved by using DBSCAN [45] to find 3D point clusters in each instance, where the distance parameter eps is set to 10cm. If an instance o has segments not connected in 3D space, DBSCAN will return more than one point cluster and og will be separated in multiple instances.
4.5 Inference
During inference time, text query is converted to feature vector f, (g)) using CLIP [4]. Instead of representing each 3D instance with the average feature of associated 2D regions, the K clustering centers by K-Means of associated features, which can be viewed as representative features from a set of viewpoints, are used. The 3D instances are then ranked by the largest cosine similarity between the text query q and K representative features of an instance. An ablation study on different strategies of feature ensemble presented in section 6.3.
Experiments
5.1 Datasets
The first dataset used for the experiment is ScanNer200 [14], which contains validation set of 312 indoor scans with 200 categories of objects. Uncountable categories "floor", "wall", and "ceiling" and their subcategories
not evaluated. The second dataset is YCB- Video [15], which contains a validation set of 12 videos. It's tabletop dataset that was originally designed for object 6DoF
5

---

pose estimation for robot manipulation. The 3D scans of the tabletop scene are reconstructed by KinectFusion [22]. The ground truth instance segmentation labels are automatically generated given the object mesh models and annotated 6DoF poses.
5.2 Metrics
Standard mean average precision (m.P) metric for instance retrieval at different IoU thresholds is adopted for the evaluation purpose. In partieular, mAP22 and m.PPsu at the IoU threshold 0.25 and 0 = 0.5 respectively, as well as the overall mAP,, i.e Σ m. Po, where 0 [0.5:0.05:0.95 reported. Only annotated object categories in 3D scene are used as text queries for evaluation. The mAP results were computed for each 3D scene and then averaged for the whole dataset.
5.3 Baselines
OpenScene [11], which is the most relevant work to date, is used as the first comparison point. Given an object query, it returns a heatmap of the input point cloud. A set of thresholds = [0.5:0.03:0.9] are tested for each category to convert the heatmap into a binary mask and then foreground points are clustered into 3D instances using DBSCAN [45], similar to the post-processing step section 4.4. The one with the best overall performance is reported. Furthermore, series of prior research has focused semantic mapping using closed-vocabulary detectors. Two representative works, Fu- sion++ [9] and PanopticFusion [10], are used comparison points with two revisions: 1) Instead of using their whole SLAM system, this work assumes the 3D reconstruction and ground truth camera poses are given, and only tested their data association and instance mapping algorithms. 2) Their backbone detector MaskRCNN [41] is replaced with Detic [2] for open- vocabulary detection, and the mean feature of associated 2D detections for each instance used to match text queries.
Ner2000 [14]
YCB- VideVidoo [15]
Method
mAP2s
mAP30
MAP
m.AP2s
mAPSo
mAP
OpenScene [11]
01,268
0.190
0.089
0.421
0.333
0.116
*Fusion++ [9]
0.414
0.253
0.094
0.817
0.464
0.120
*PanopticFusion [10]
0.539
0.370
0.150
0.851
0.803
0.393
Ours
0.564
0.443
0.211
0.863
0.848
0.465
Table I: Results on ScanNet200 [14] and YCB-Video [15] dataset
mAPPo
MAP
Method
head
common
tail
head
common
tail
OpenScene [11]
0.308
0.178
0.067
0.150
0.076
0.033
*Fusion++ [9]
0.235
0.243
0.288
0.094
0.090
0.098
*PanopticFusion [10]
0.335
0.360
0.424
0.145
0.146
0.162
Ours
0.417
0.433
0.469
0.224
0.214
0.193
Table 2: Results on three sets of categories with different frequencies in ScanNet200 [14]
5.4 Results
Quantitative results of instance retrieval on ScanNet200 [14] and YCB-video [15] datasets are shown in Table 1. Furthermore, results on different sets of categories with different frequencies in Scan- Net200 are shown in Table 2. The proposed method outperforms all other baselines by a large margin terms of instance retrieval mAP. It seems that OpenScene[11] does not perform well on this task even with an automatically tuned threshold for each category because fused point features not distinguishable enough. As result, grouping points into segments with accurate boundaries by thresholding is rather difficult. The proposed method, on the other hand, directly fuses instance- level information and improves segment quality by periodic merging and filtering. The proposed method outperforms the other two baselines primarily because of the use of instance feature simi- larity as an additional metric for data association while the baselines only consider 3D overlapping, which can easily fail when the 2D detections are noisy, especially in the open- open-vocabulary setup.
5.5 Running Time
The inference time nearly instant (~ 20ms) for a text query. The rumning time for the fusion process depends on the number of detections (NV frame and the number of fused 3D instances (M), i.e. OMMN). As mentioned in Section 4.2, it requires two dot products compute the region overlapping and feature similarity, which in practice is a fast process that operates at 30F with an NVIDIA RTX 3090 GPU for most 3D scans in the ScanNer200 [14] dataset.

---

Ablation Studies
6.1 Input queries to the 2D region proposal method
The proposed method utilizes an open-vocabulary 2D detector as a region proposal method by querying it with large vocabulary. One concern is whether the input query of the 2D method would affect the performance of the 3D instance retrieval. This ablation study tests queries from multiple datasets as input to the region proposal method and displays their impact on the overall performance. In addition to ScanNet200 [14] and ImageNet21K IK [42], COCO [46] (80 categories), LVIS [47] (1203 categories), and more aggressively, queries with ImageNer21K categories but with- out ScanNer200 categories are tested. Results of 3D instance retrieval on the ScanNet200 dataset are shown in Table 4. It turns out that an extensive vocabulary is helpful for generating regions of interest for arbitrary objects. Furthermore, the results show that the region proposal network has cer- tain generalizability, such that even when ScanNer200 categories ImageNet21k Ik categories, it can still find most regions based on similar categories in the vocabulary, and the final performance of retrieving objects in ScanNer200 only slightly dropped.
completely removed from the
COCO
ScanNer200
LVIS
ImageNe(21k
ImageNe12Ik ScanNer200
mo
0.228
0,419
0.429
0.443
0,410
Table 3: Results on ScanNet200 [14] with different input queries to the 2D region proposal network
6.2 Instance features and 2D masks
In this ablation study, alternative approaches are explored to replace instance features and 2D masks derived from Detic in order to demonstrate the adaptability of the proposed method across different backbones. Rather than utilizing the mask feature directly extracted from Detic, detected bounding boxes are cropped and fed to the CLIP model to extract features from these cropped regions. For the generation of 2D masks, SAM [43] is adopted to create segmentations for the regions proposed by Detic bounding boxes. In this experimental setup, instance fusion performed every three frames, primarily due the relatively slow performance of SAM. Substituting the Detic feature with the CLIP feature from cropped images yields slightly inferior results, whereas replacing the Detic mask with the SAM mask leads to an improvement in performance. This outcome is expected since SAM generally produces higher-quality masks, albeit with tradeoff in terms of speed. It is anticipated that as open- opecbvocabylary 2D detection techniques advance, more potent and efficient methods may emerge as viable alternatives to the current backbone.
Proposed
w/ CLIP feature w/ SAM segmentation
m.l A.P Poo
0,414
0.406
0.440
Table 4: Results on ScanNet200 [14] with different instance features and 2D masks
6.3 Feature ensemble strategies
Three different feature ensemble strategies are tested represent 3D instance based on associated 2D features. The first strategy to compute the average of all 2D features. The second strategy involves clustering the 2D features from different viewpoints using the K-Means algorithm, and the clustering centers are used to represent each instance. During instance retrieval, the feature similarity is determined as the maximum similarity between the query feature and the clustering centers. The third strategy to use the feature from the largest associated 2D region. Results of 3D instance retrieval on the ScanNet200 dataset are presented in Table 5. The approach of using multiple features through clustering outperforms simple averaging, while using the feature from the largest associated 2D region yields the poorest results.
average
clustered (K=16)
clustered (K=64) feature from largest 2D detection
m.APso 50
0.428
0.429
0.443
0.380
Table 5: Results on ScanNer200 [14] with different feature ensemble strategies
6.4
Time intervals and visibility threshold for periodic instance filtering and merging
This ablation study tested different time intervals T and visibility threshold ouse for filtering men- tioned in section 4.3. Results of 3D instance retrieval on the ScanNet200 dataset are shown in
7

---

Table 6 and Table respectively. The frame interval T = 300 and visibility threshold Puis = 0.2
yields the best results.
T=
T 100
T
300
T= 500
T 1000
mo 50
0,340
0.417
0.443
0,410
0,412
Table 6: Results on Scan Ner200 [14] with different time intervals periodic filtering and merging
Oeis
Bus. 0.1
Dois
0.15
Bois
0.2
Ovis 0.25
B.s 0.3
mo
0.256
0,386
0.407
0.443
0.418
0.408
Table m3P3 on ScanNet200 [14] dataset
7
Robotics Experiments
"handle" "
"bottle cap"
Tabletop Scene
3D Scan
c) Text Query
d Grasping
Figure 3: Robot Experiments.
Contrasting to conventional closed-set semantic methods, the superiority of open- open-vocabulary detec- tors for manipulation lies in their ability to pinpoint the grasp region with a language specification. A part-based grasping experiment was devised given this inspiration. In particular, the robot is asked to grasp the "bottle cap" and "handle" respectively in sets of experiments with five distinct table setups in each set. An RGB-D camera is mounted on the robot's wrist that captures videos of the objects on the table. For each scene, there is a short scanning phase shown in Figure 3(b) that the robot arm went through a predefined trajectory to get a more complete view of objects on the table. Reconstruction is performed using KinectFusion. OVIR-3D is compared against OpenScene to segment parts given the text query and located segments are used guide the robot's grasp. For OVIR-3D, the 5/5 graspable bottle caps and 4/5 "handle" of the pitcher were detected and the robot grasping success rate was 90%. The detection rate for OpenScene was low at 0/5 and 3/5 respectively, and the overall grasping success rate was 30%. An object is considered detected if reasonable visual segment is found.
Conclusion and Limitations
This paper presents OVIR-3D, a rather straightforward but effective method for open-vocabulary 3D instance retrieval. By utilizing an off-the-shelf open-vocabulary 2D instance segmentation method for region proposal and fusing its output 2D regions and text-aligned features 3D space, the proposed method can achieve much better performance than other baselines without using any 3D instance annotation, additional training. or manual heatmap thresholding during inference. This method can also be used for 3D instance pseudo-label generation for self-supervised learning.
A limitation of the proposed method is that it is not always able to merge segments of very large instances, such as long dining tables. It can also miss tiny objects as they are likely to be treated as noise and removed during the fusion process. Furthermore, while the proposed method can improve segmentation quality due multi-view noisy filtering, it still relies on the 2D region proposal model not consistently miss an object or generate bad segmentation, since does use any 3D data for fine-tuning tuning. A promising direction is to integrate this method with 3D learning- rearning-based method to utilize the scareer but cleaner 3D annotations.

---

Acknowledgments
We thank Dr. Yu Wu for providing useful suggestions during the development of this work. This work is supported by NSF awards 2309866, 1846043, and 2132972.
References
[1] x. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. In International Conference on Learning Representations.
[2] X. Zhou, R. Girdhar, A. Joulin, Krähenbühl, and Misra. Detecting wwenty-thousand classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX, pages 350-368. Springer, 2022.
[3] R. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Mar- culescu. Open-vocabulayy semantic segmentation with mask adasted clip. arXiv preprint arXiv:2210.04155 2022.
[4] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 8748 8763. PMLR, 2021.
[5] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and rision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 9904-4916 PMLR, 2021.
[6] L. Yuan, D. Chen, Chen, N. Codella, X. Dai, Gao, H. Hu, X. Huang, B. Li, C. Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.1 1142, 2021.
[7] McCormac, A. Handa, A. Davison, and S. Leutenegger. Semanticfusion: Dense 3d seman- tic mapping with convolutional neural networks. In 2017 IEEE International Conference on Robotics and automation (ICRA), pages 4628-4635. IEEE, 2017.
[8] M. Runz, M. Buffier, and L. Agapito. Maskfusion: Real-time recognition, tracking and recon- struction of multiple moving objects. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). pages 10-20. IEEE, 2018
[9] J. McCormac, R. Clark, M. Bloesch, A. Davison, and S. Leutenegger. Fusion++: Volumetric object-level slam. In 2018 international conference on 3D vision (3DV), pages 32-41 41. IEEE, 2018.
[10] G. Narita, T. Seno, T. Ishikawa, and Y. Kaji. Panopticfusion: Online volumetric semantic map- ping at the level of stuff and things. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4205-4212. IEEE, 2019.
[11] Peng, K. Genova, C. M. Jiang, A. Tagliasacchi, M. Pollefeys, and T. Funkhouser. Open- scene: 3d scene understanding with open vocabularies. CVPR, 2023.
[12] A. Submission. Clip-fields: Weakly supervised semantic fields for robotic memory. RSS 2023,
2023.
[13] A. Dai, A. X. Chang, M. Savva, M. Halber, Funkhouser, and M. Nießner. Scannet: Richly- annotated reconstructions of indoor scenes. Proc. Computer Vision and Pattern Recog- nition (CVPR), JEEE, 2017.
[14] D. Rozenberszki, O. Litany, and A. Dai. Language grouunded indoor 3d semantic segmentation in the wild. In Proceedings of the European Conference Computer Vision (ECCV), 2022.

---

[15] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. 2018.
[16] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero- shot transfer with locked- locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022.
[17] B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03546, 2022.
[18] G. Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin. Scaling open-vocabulary image segmentation with image-level labels. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XXXVI, pages 540-557. Springer, 2022.
[19] Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. arXiv preprint ar arXiv:2202.11094 2022.
[20] A. S. D. W. A. D. M. A. A. M. D. S. X. W. X. Z. T. K. N. H. Matthias Minderer, Alexey Gritsenko. Simple open-vocabulary object detection with vision transformers. ECCV, 2022.
[21] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang. L. Yuan, L. Zhang, J.- N. Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[22] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shot- ton, S. Hodges, and A. Fitzgibbon. Kinectfusion: Real-time dense surface mapping and track- ing. In 2011 10th IEEE international symposium on mixed and augmented reality, pages 127-136. leee, 2011.
[23] T. Whelan, S. Leutenegger, R. Salas-Moreno, B. Glocker, and A. Davison. Elasticfusion: Dense slam without pose graph. Roboties: Science and Systems, 2015.
[24] A. Dai, M. Nieβner, M. Zollhöfer, S. Izadi, and C. Theobalt. Bundlefusion: Real-time glob- ally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG), 36(4):1, 2017.
[25] E. Sucar, S. Liu, Ortiz, and A. Davison. imap: Implicit mapping and positioning in real- time. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6229 1229-6238, 2021.
[26] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys. Nice- slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[27] L.. Schönberger and J.-M. Frahm. Structure-from-mmatee revisited. Conference on Com- puter Vision and Paterr Recognition (CVPR), 2016.
[28] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference Computer Vision (ECCV), 2016.
[29] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303-312, 1996.
[30] H. Pfister, M. Zwicker, J. Van Baar, and M. Gross. Surfels: Surface elements rendering primitives. In Proceedings of the 27th annual conference Computer graphics and interactive techniques, pages 335-342, 2000.
10

---

[31] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[32] I. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Trans. Graph., 41(4):102:11022215 July 2022. doi: 10.1 10.114553528223333331222 URL https: //t/i/ ott.llrrgg/1.//
3530022..
[33] Sara Fridovich-Keil and Alex Yu, M. Tancik, Q Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[34] C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5459-5469, 2022.
[35] X. Kong, S. Liu, M. Taher, and A. J. Davison. vmap: Vectorised object mapping for neural field slam. arXiv preprint arXiv:2302.01883. 2023.
[36] J. Zhang, R. Dong, and K. Ma. Clip-fo3d: Learning free open-world 3d scene representations from 2d dense clip. arXiv preprint arXiv:2303.04748, 2023.
[37] D. Hegde, M. J. Valanarasu, and V. M. Patel. Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition. arXiv preprint arXiv:, 2022.
[38] K. Jatavallabhula, A. Kuwajerwala, Q Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi,
N. Keetha, Tewari, J. Tenenbaum, C. de Melo, M. Krishna, I., Paull, Shkurti, and Tor- ralba. Conceptfusion: Open-set multimodal 3d mapping. arXiv, 2023.
[39] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.0955. 2023.
[40] R. Ding, J. Yang, C. Xue, W. Zhang, S. Bai, and X. Qi. Language-driven open-vocabulary 3d scene understanding. arXiv preprint arXiv:2211.16312 2022.
[41] K. He, G. Gkioxari, Dollár, and R. Girshick. Mask r-emn. In Proceedings of the IEEE international conference computer vision, pages 2961-2969, 2017.
[42] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[43] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, I.. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.02643. 2023.
[44] S. Liu, 7. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. L. Yang, H. Su, J. Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 2023.
[45] E. Schubert, Sander, M. Ester, H. Kriegel, and X. Xu. Dbscan revisited, revisited: why and how you should (still) use dbscan. ACM Transactions Database Systems (TODS), 42 13)
[46] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer EEcvn 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740- 755. Springer, 2014.
[47] A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmenta- tion. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356-5364, 2019.


Reviews:
[{"review_id": "aELiDfzQkHB", "reviewer": "Reviewer_dbTB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel approach for semantic alignment and retrieval of 3D object instances, termed as an effective open-vocabulary method that does not rely on explicit training with 3D data. The primary objective of this method is to provide a ranked set of 3D instance segments based on their semantic similarity to a given text query. \n\nThe proposed pipeline consists of several stages. A 2D Open-Vocabulary Instance Segmentation Method is pre-trained to extract text-aligned features and generate 2D region proposals. To enable accurate matching between the 2D regions and 3D instances, a two-stage object detection architecture is employed. The matching process relies on cosine similarity to establish the correspondence between 2D and 3D representations, followed by fusion of the matched instances in the 3D space. Furthermore, post-processing steps involving filtering and merging of the fused results are applied to enhance the overall accuracy and coherency. ", "summary_of_recommendation": "Useful and straightforward pipeline towards 3D open vocubulary scene understanding. But some important questions need to be clarified.\n\n\nUpdate: Thanks for the efforts of authors. After reading the feedback from the authors and going through the comments of other reviewers. The reviewer agree's that this work is a good “engineered” solution to achieve open vocab 3d segmentation, which is good to the community but not that novel. As a result, the reviewer would like to keep to the original scores.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "Q1: Running time is missing.\n\nQ2: If no ground truth depth information is available, how to conduct 2D-3D projection? Will it affect the quality of the overall pipeline?\n\nQ3: How to deal with small objects and occluded objects in 3D filtering and association?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a novel methodology for the semantic alignment and retrieval of 3D object instances, presenting an effective open-vocabulary approach that does not necessitate explicit training with 3D data\n\n1. Quality\n\nThe paper is nicely written and structured. The problem well exposed and easy to understand. The method clearly exposed, and results are provided for the main standard tracking benchmarks. However, the 2D to 3D projection may miss some information.\n\n2. Clarity\n\nThis paper is clearly written and easy to read.\n \n3. Originality\n\nThis paper represents an incremental advancement in the field, distinguishing itself from existing works such as OpenScene through its direct projection of 2D results into the 3D space. By accomplishing the seamless integration of both 2D and 3D modalities, this paper pioneers a distinctive approach that relies on fundamentally distinct algorithms and methodologies. This originality arises from the novel fusion of 2D and 3D information, resulting in an innovative framework for semantic alignment and retrieval of 3D object instances\n\n4. Significance\n\nIt has certain significance to introduce 2D algorithm into 3D task.\n\nStrengths:\n1. Straightforward but good pipeline to obtain 3D semantic scene representation.\n2. Do not need any 3d labels.\n\nWeakness:\n1. 2D-3D projection is tricky? What if the depth information is noisy?\n2. 3D filtering and association is also problematic for small objects and occluded objects", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "KmiKdJppfi", "reviewer": "Reviewer_Qx8o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Developing on the work of OpenScene [14] which pioneered open-vocabulary 3D segmentation without training on 3D data, this work extends that idea by adding the capability of instance segmentation, thus allowing tasks such as robot manipulation. Despite the similarity in method formulation with [14], the resulting properties and performed tasks (real robot experiments) are sufficiently novel and present a useful contribution.", "summary_of_recommendation": "I believe this paper presents a useful research contribution and it's findings/proposed-method is likely to have high impact. I have noted a couple minor weaknesses but I am willing to recommend accepting this paper despite those (as the strengths far outweigh the weaknesses).\n\nPost rebuttal update:\nI will be keeping my original rating for the following reasons:\n- I feel that the authors response to my review is adequate but not strong.\n- I have taken the other reviews into account; which noted additional concerns regarding this work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1) Please comment on the adaptability of this method to different domains (in terms of the many manual thresholds used).\n\n2) line 207: \\theta_{recall} is undefined.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n1) Open vocabulary instance segmentation (as pioneered in this paper) is a powerful tool that will give robots the ability to find and manipulate arbitrary objects present in a scene. This paper highlights the need for instance segmentation for this purpose (as opposed to the heatmaps produced by previous work). \n\n2) The method is technically sound and is backed up by adequate experimental results. \n\n3) Real world robot experiments. The demonstration of the ability to identify grasp regions of arbitrary objects with language specifications will likely have high impact. \n\n4) The paper is well written, well organized and was a pleasure to read. \n\n\nWeaknesses:\n\n1) The authors complain of a related work [14] needing a single manual threshold to generate it's output (line 39,40), presenting it as a weakness. However, the method proposed in this paper consists of many manual thresholds: at least 10 (lines 171, 172, 178, 195, 200, 203, 206, 207, 213, 215). It is likely that all these thresholds will need to be re-tuned once the domain of the data changes, thus limiting it's adaptability / ease-of-use. This paper does not present results on different domains (eg. outdoors as presented in [14]).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "8ReVAu-XEv", "reviewer": "Reviewer_MRDv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an approach for open-vocabulary 3D object detection that is based on a straightforward lifting of 2D open-vocabulary object detections (bounding boxes) to 3D (assuming known depth and camera poses). The fused objects in 3D each have a CLIP embedding, which enables open-vocabulary (text-based) querying. The approach is evaluated on ScanNet-200 and YCB-Video.", "summary_of_recommendation": "While the paper presents an open-vocabulary 3D detection scheme of potential interest to the robot perception community, there still remain a number of design choices and evaluation-related concerns that need to be addressed before I can recommend acceptance. While I will, for now, score this a weak reject, I am very much open to revising my rating should my concerns be adequately addressed.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "I would like to see \"Weaknesses\" [W1], [W2], and [W3] addressed.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\nThe paper is well-written and easy to understand. The approach is simple, and explained in a concise manner, with sufficient detail to enable a skilled practitioner to implement it. From a sequence of input images, an open-vocabulary object detector (here, Detic) is used to obtain 2D bounding boxes. Each bounding box is tagged with a text label (one of 21k ImageNet categories), which is passed through CLIP to produce a text embedding for that label. Assuming depth images and camera poses to be known, the 2D proposals may be lifted to 3D using criteria such as bounding box overlap and CLIP feature similarity; and postprocessed to remove duplicate 2D (or 3D) detections. This results in a 3D map where each instance is additionally tagged with a CLIP embedding. Given a text query, the query is encoded using the CLIP text encoder, and compared with the stored text embeddings of each 3D instance via cosine similarity. The system then returns a ranked list of object instances that best match the text query.\n\nThis system is quite simple, yet performant. The system achieves better performance than OpenScene -- a recent method that computes a per 3D point CLIP descriptor (and consequently only performs semantic, not instance, segmentation) for open-vocabulary 3D detection. It also performs better than Fusion++ and PanopticFusion, which are object-centric mapping approaches that lift 2D object detections to 3D.\n\nThe qualitative results in the supplemental video demonstrate the crisp object instances and accurate retrievals across one small apartment and another tabletop scene.\n\n\n## Weaknesses\n\nThere are a number of issues with the current edition of the manuscript that I would like to see discussed/addressed before I can recommend acceptance.\n\n[W1] **Dependence on detection and description schemes**: The proposed method seems to be fairly agnostic to the exact open-vocabulary object detector and the feature descriptor used. Yet somehow, the current manuscript describes the method as being only applicable to DETIC detections and CLIP text feature descriptors; because of which the core contribution (i.e., a lightweight 3D lifting mechanism for 2D bounding box detections) does not seem to get its due credit (and also makes it very difficult to attribute credit to each aspect of the pipeline).\n\n* The presented 3D lifting scheme is lightweight and performant, regardless of whether the object detector is closed- or open-vocabulary; and is agnostic to the choice of open-vocabulary detector. To establish this claim, one would need to evaluate the proposed method with a closed-vocabulary object detctor; and also with other open-vocabulary object detectors (popular open-source examples include Grounded-DINO, ViLD).\n\n* The description scheme used in the proposed method is to take the Detic text label for each bounding box and encode the label using the CLIP text encoder. However, this seems to defeat the purpose of using image-text aligned embeddings (like CLIP) in the first place; since the image content is never used at subsequent stages in the pipeline. Only the text labels end up being used; and only text queries are submitted to the system at inference time. If only text queries are being used, one could alternatively try out embeddings like RoBERTa and GloVe, which will likely provide similar or better similarity measures, while being more lightweight. (on the other hand, if the usage of image-language aligned features is to be justified, the text CLIP embeddings will need to be replaced by the image-level CLIP embeddings).\n\n\n[W2] **Evaluation on open-set categories**: Another key issue with the current evaluation scheme is that since the object detector Detic is trained using LVIS, CC, and ImageNet-21K, which collectively include most (if not all) of the ScanNet 200 categories. This effectively reduces the problem addressed in this paper to a closed-set / long-tailed detection problem setting, opposed to a true open-set recognition one. Apart from ScanNet-200, the only other dataset evaluated is YCB-Video, but there are only 12 videos in that dataset (the validation split). Evaluating on truly open-set scenarios is indeed a challenge, as pointed out by ConceptFusion [39] and LERF [40] (both cited here-in; and proposing to collect open-set data of their own). It is therefore important to evaluate the proposed method on open-set recognition scenarios (i.e., categories that the underlying object detection model has not seen during training time). While Table 3 attempts to evaluate \"ImageNet21k - ScanNet200\" categories, that settting too, is not truly open-set (because the underlying model used, Detic, has still been trained on the LVIS, CC, and ImageNet-21K datasets). One concrete example for an open-set scenario is another CoRL paper from last year \"Semantic Abstraction\" (https://semantic-abstraction.cs.columbia.edu/), which evaluates on the ARKitScenes dataset.\n\n[W3] **Evaluation concerns with using only mAP**: Table 1 and Table 2 compare mAP scores across the proposed method, Fusion++, and PanopticFusion. Each of these methods reconstructs a different pointcloud or voxelgrid; owing to their customized noise-removal and filtering strategies. How is consistency ensured when comparing mAP metrics across these different pointclouds? (e.g., if the pointclouds from the proposed approaches end up conservatively removing noisy points; while the baselines do not; the mAP numbers for the proposed approach will likely end up being better) What I'm trying to get at, is that the evaluation protocol ends up conflating errors due to incorrect map construction with errors induced due to imperfect segmentation. This can be mitigated, in part, if reacalls and F1 scores were to be computed and reported.\n\n\n# Post-rebuttal update\n\nThe author response addresses my concerns [W1] and [W3] adequately. In terms of [W2], however, a few concerns still remain, necessitating a more nuanced discussion. Since the author response only came in towards the very end of the discussion phase, this did not allow for reviewer-author interaction. That said, I appreciated the thoroughness and thought that went into the author response, and would like to revise my score to a \"weak accept\", but with certain caveats.\n\n* The evaluation in the paper does not sufficiently demonstrate open-set properties, compared to the queries demonstrated in other recent relevant methods such as LERF, ConceptFusion, CLIP-Fields, and variants. Even in the additional experiments showcased in the rebuttal, *evaluation is restricted to synonyms of classes that appear in the ScanNet200 dataset*, limiting the \"open-setness\" claims of the system. I would strongly recommend adding a few results (qualitative results could be okay too) on scenes from NYUv2 or ARKitScenes, following SemanticAbstraction -- or demonstraing stronger open-set results using other appropriate datasets. The authors agree with part of my characterization of their open-set claims, and also say in the rebuttal that they do.\n\n* I agree with the authors' characterization that the proposed approach will attempt to preserve DETIC's performance, and *lift it to 3D*.\n\n* I would like this discussion/caveat on \"open-set\" evaluation to feature at a prominent location in the revised manuscript.\n\nTo summarize, I find [W1] and [W3] adequately addressed. [W2] is not completely addressed, but with the amends suggested in the author response, that to me tips the scales just about enough in favor of a weak accept rating.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "3st6meJKEAt", "reviewer": "Reviewer_dozd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents OVIR-3D, a 3D instance retrieval from a text input. Given an explored 3D environment and a open-vocabulary query, the proposed method is able to rank all likely 3, without training on 3D data. The method uses an 2D object detector (Detic)  features that are first aligned with text features using CLIP.  The region obtained from Detic are then projected in 3D and compared (with cosine similarity features and overlapping) to existing 3D instances.", "summary_of_recommendation": "The paper is well written and clear. Although contribution is not technically novel, it is well-executed, with convincing results.\nNonetheless, there are still some technical points to clarify before recommending this paper for CoRL.\n\nUpdate:\nGiven the clarification and additional evaluation provided by the authors, the reviewer recommends a Weak Accept for this paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1) The authors did not clearly motivate their work. Why is it useful to do 3D instance retrieval when there are a lot of existing 2D methods? \n\n2) The proposed matching is unclear. The reviewer understand the cosine similarity but what about the region overlapping? How is the potential rotation  (depending on the point of view) of the 3D objects addressed since it seems that there is no registration?\n\n3) It is not clear for the reviewer how to separate the 3D instances from the proposed approach. Let's say that there are 2 instances of the same objects in different areas of the environment. Their features may certainly match as well as their overlapping region.\n\n4) The reviewer does not understand the 3D features  $f_i^{3D}$. Since the memory bank is initially empty, it can only be filled with $f_i^{2D}$. How does $f_i^{2D}$ become $f_i^{3D}$?\n\n5) Please detailed, the different steps required to transfer to a real robot as shown in Section 7. From the reviewer understanding, on a fixed arm, there is no exploration (or is there a scanning phase?), thus less redundant region proposal. Is the proposed method robust enough to work with a single view? \n\n6) Related to 5),  it would be interesting to quantify the robustness of the proposed approach given the number of views.\n\n7) Minor typo: line 261: It's -> It is", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well written and mostly easy to understand. The paper is validated on standard datasets and with an extensive ablation studies to assess the benefits of each part of OVIR-3D. Despite these undeniable strengths, the reviewer considers OVIR-3D as an \n\"engineered\" solution combining several existing approaches. Thus, the technical novelty and contribution of this paper are somehow limited.\nOn the other hand, in OVIR-3D, the different existing approaches are combined in a clever and efficient way. This paper might be of interest for CoRL community. \nThe paper should nonetheless be improved and clarified on certain aspects as detailed in the \"Questions for Rebuttal\" section.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Summary, strengths and weaknesses
This paper introduces a method for open-vocabulary 3D object retrieval without 3D training. This approach combines 2D-to-3D lifting with CLIP embeddings for open-vocabulary querying, demonstrating a novel method for semantic alignment and retrieval of 3D object instances. Although the method is simple, the experimental results using standard datasets show that the method is performant. 

The reviewers initially raised a few concerns regarding the method and the experimental comparison. The authors provided a detailed response to each of these points. The reviewers agree that most of the concerns have been addressed. 

In their final scores, all the reviewers agree on the quality and acceptance. I agree with their consensus.
Decision: Accept (Poster)