CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection
Authors: Yang Cao
Yihan Zeng
Hang Xu
Dan Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D vision
open-vocabulary perception
multi-modal learning
point cloud
3D object detection

Abstract: Open-vocabulary 3D Object Detection (OV-3DDet) aims to detect objects from an arbitrary list of categories within a 3D scene, which remains seldom explored in the literature. There are primarily two fundamental problems in OV-3DDet, *i.e.*, localizing and classifying novel objects. This paper aims at addressing the two problems simultaneously via a unified framework, under the condition of limited base categories. To localize novel 3D objects, we propose an effective 3D Novel Object Discovery strategy, which utilizes both the 3D box geometry priors and 2D semantic open-vocabulary priors to generate pseudo box labels of the novel objects. To classify novel object boxes, we further develop a cross-modal alignment module based on discovered novel boxes, to align feature spaces between 3D point
cloud and image/text modalities. Specifically, the alignment process contains a class-agnostic and a class-discriminative alignment, incorporating not only the base objects with annotations but also the increasingly discovered novel objects, resulting in an iteratively enhanced alignment. The novel box discovery and crossmodal alignment are jointly learned to collaboratively benefit each other. The
novel object discovery can directly impact the cross-modal alignment, while a better feature alignment can, in turn, boost the localization capability, leading to a unified OV-3DDet framework, named **CoDA**, for simultaneous novel object localization and classification. Extensive experiments on two challenging datasets (*i.e.*, SUN-RGBD and ScanNet) demonstrate the effectiveness of our method and also show a significant mAP improvement upon the best-performing alternative method by 80%. Codes and pre-trained models are released on [the project page](https://yangcaoai.github.io/publications/CoDA.html).

CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D
Object Detection
Yang Cao²
Yihan Zeng² Hang Xu² Dan Xu
Hong Kong
University Science and Technology Huawei Noah's Ark Lab?
Abstract
Open-vocabulary 3D Object Detection (OV-3DDet) aims detect objects from arbitrary list of categories within 3D scene, which remains seldom explored fundamental problems OV-3DDet, i.e.,
the literature. There are primarily localizing and classifying novel objects. This paper aims at addressing the two problems simultaneously via a unified framework, under the condition of limited base categories. To localize novel 3D objects, we propose an effective 3D Novel Object Discovery strategy, which utilizes both the 3D box geometry priors and 2D semantic open- nen--ooabularry priors to generate pseudo box labels of the novel objects. To classify novel object boxes, we further develop a cross-modal alignment module based on discovered novel boxes, to align feature spaces between 3D point cloud and image/text modalities. Specifically, the alignment process contains a plass-agnostic and a class- dlass-iicrimiiativee alignment, incorporating not only the base objects with annotations but also the increasingly discovered novel objects, resulting in iteratively enhanced alignment. The novel box discovery and cross- modal alignment are jointly learned to collaboratively benefit each other. The novel object discovery directly impact the cross-modal alignment, while better feature alignment can, in turn, boost the localization capability, leading unified OV-3DDet framework, named CoDA, for simultaneous novel object localization and classification. Extensive experiments on two challenging datasets (i.e., SUN-RGBD and ScanNet) demonstrate the effectiveness of our method and also show a significant mAP improvement upon the best- pessforeformnn alternative method by 80%. Codes and pre-trained models released on the project page!)
Introduction
3D object detection (3D-Det) 19 21 20 is a fundamental task in computer vision, with wide applications in self-driving, industrial manufacturing, and roboties. Given point cloud scenes, 3D-Det aims to localize and classify objects in the 3D scenes. However, most existing works heavily rely on datasets with fixed and known categories, which greatly limits their potential applications in the real world due to the various and unbounded categories. open-vocabulary 3D object detection (OV-3DDet) is on the verge appearing to detect novel objects. In OV-3DDet, the model typically trained on datasets with limited base categories while evaluated on scenes full of novel objects, which brings additional challenges. In the problem OV-3DDet, the key challenge how to localize and classify novel objects using only annotations of very limited base categories. There are only very few works in the literature targeting this challenging problem so far.
"Corresponding author
Sicchu....//bbllcattioooo//A.A. html
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

To obtain novel object boxes. 15 utilizes large-scale pre- pre-trained 2D open- vocen-uuby object detection (OV-2DDet) model B35 as prior knowledge localize a large number of 2D novel object boxes, and then generates pseudo 3D object box labels for corresponding 3D) novel objects. Instead of directly relying on an external OV-2DDet model to obtain novel object localization capability, we aim to learn a discovery of novel 3D object boxes, based on limited 3D annotations of base categories on the target data. With the discovered novel object boxes, we further explore joint class-agnostic and class-specific cross-modal alignment, using the tuman-annotated base object boxes, and the discovered novel object boxes. The novel object box discovery and cross-modal alignment are collaboratively learned to achieve simultaneous novel object localization and classification.
Specifically, to localize novel objects, we first propose 3D Novel Object Discovery (3D-NOD) training strategy that utilizes information from both the 3D and 2D domains to discover more novel objects the training process. detail, the model learns class-agnostic 3D box prediction using 3D) box annotations of the base categories, which can provide 3D geometry priors of 3D boxes based on geometric point cloud features. To confirm discovered novel boxes, the pretrained rision-language model CLIP [22] is also utilized to provide 2D semantic category priors. By considering both 3D and 2D cues, novel objects can be effectively discovered during iterative training.
Building upon 3D-NOD, to achieve novel object classification, we further propose Discovery-Driven Cross-Modal Aligmment (DCMA) module that aligns 3D object features with 2D object and text features in a large vocabulary space, using both and the discovered 3D boxes of novel categories. DCMA comprises two major components. i.e.. category-agnosti box-wise feature distillation and class- class-specific feature alignment. The former aligns 3D point cloud features and 21) CLIP image features at the box level, without relying on information about specific categories. The distillation pushes the 3D point cloud and 2D image features closer wherever a 3D box covers. On the other hand, DCMA aligns 3D object features with CLIP text features in large category vocabulary space, using category information from both annotated base objects and discovered novel objects, in a cross-modal contrastive learning manner.
3D object boxes of base categories,
The proposed 3D-NOD and DCMA can be jointly optimized in a unified deep framework named CoDA. They collaboratively learn to benefit each other. The discovered novel object boxes through 3D-NOD can greatly facilitate the cross moosl feature alignment, while better feature alignment through DCMA can further boost the model localization capability on novel objects. With our joint training of 3D-NOD and DCMA in CoDA to simultaneously localize and classify novel objects, our approach outperforms the best best-performing alternative method by 80% terms of mAP.
In summary, the main contribution of the paper is three-fold:
We propose an end-to-end open vocabulary 3D object detection framework named CoDA, which can learn simultaneously localize and classify novel objects, without requiring any prior information from external 2D open-vocabulary detection models.
We design an effective 3D Novel Object Discovery (3D-NOD) strategy that can localize novel objects by jointly utilizing 3D geometry and 2D semantic open-vocabulary priors. Based on discovered novel objects, we further introduce discovery-drive driven cross cross-modal alignment (DCMA) effectively align 3D point cloud features with 21) image/text features in a large vocabulary space.
We design mechanism to collaboratively learn the 3D-NOD and DCMA to benefit each other. The novel object boxes discovered from 3D-NOD are used in cross cross-modal feature alignment in DCMA, and better feature alignment in DCMA can facilitate novel object discovery.
2 Related Work
3D Object Detection. 3D object detection (3D-Det) 21 25 32 16 33 26, 3 has achieved great success in recent years. For instance, VoteNet 21 introduces a point voting strategy 3D object detection. It adopts PointNet (20] to process 3D points and then assigns these points to several groups, and object features are extracted from these point groups. Then the predictions are generated from these object features. Based on that, MLCVNet [25] designs new modules for point voting to
capture multi-level contextual information. Recently, the emergence of transformer- bassd methods has also driven the development of 3D-Det. For example, GroupFree [14] utilizes transformer
as the prediction head to avoid handcrafted grouping. 3DETR [19] proposes the first end-to-end transformer-bast based structure for 3D object detection. However, these methods mainly focus on the

---

Discovery-Drite Cros-MMuda Alignment (DCMA)
30 Novel Object Discovery (3D-NOD)
Cross-Priors
Base
Novel
30 Sorra Ph Prit
[Open Categories
Fry
Update
2D Object Features
Text Features
Class Aguostic
Clas--ppeiffle
Encoder
Divtillation
Contrastive Alignment
Novel
3D Goonetrry Priors
2D Object Regions
3D Objject Features from Object Queries
Crop
Classification
Encoder
Decoder
Projection
Lucalization Head
Object
Point Cloud
Predicted 3D Boxes
Prujected 2D Boxes
Figure I: Overview of the proposed open-vocabulary 3D object detection framework named CoDA. We consider 3DETR [19] our base 31D object detection framework, represented by the 'Encoder' and 'Decoder' networks. The object queries, together with encoded point cloud features, are input into the decoder. The updated object query features from the decoder are further input into 3D object classification and localization heads. We first propose a 3D Novel Object Discovery (3D-NOD) strategy, which utilizes both 3D geometry priors from predicted 3D boxes and 2D semantic priors from the CLIP model to discover novel objects during training. The discovered novel object boxes are maintained novel object box label pool. which is further utilized our proposed discovery- driven cross-modal alignment (DCMA). The DCMA consists of a class-agnostic distillation and class-specific contrastive alignment based on discovered novel boxes. Both 3D-NOD and DCMA collaboratively learn to benefit each other to achieve simultaneous novel object localization and classification in an end-to-end manner.
cooebboocaylary setting, i.e., the object categories in training and testing are the same and fixed. In contrast to these works, this paper targets open- vocabulary 3D-Det, and we design an end-to-ena deep pipeline called CoDA, which performs joint novel object discovery and classification.
Open-vocabulary 2D object detection. In open-vocabulary 2D object detection (OV-2DDet), some 6
works [7 1O, 22 11 27 18
29 28) push 2D object detection to the open-vocabulary pushes the CLIP from the image level to the regional level and learns region-text feature alignment. Detic (35) instead utilizes image-level supervision based on ImageNet (5). which assigns image labels to object proposals. GLIP [3] and MDETR 12] treat detection as the grounding task and adopt text query for the input image to predict corresponding boxes. All the methods mentioned above focus on 2D object detection and conduct the object detection from the 2D input image. However, in this work, we focus on open-wer world 3D object detection, which a challenging problem and remains seldom explored in the literature.
17 35
level. Specifically, RegionCLIP
Open-vocabulary 3D object detection. Recently, an open-set 3D-Det method [ has shown great potential in various practical application scenarios. It can successfully detect unlabeled objects by introducing ImageNet K.
(2). which classifies objects generated by a 3D-Det model [19 on the training set to obtain pseudo labels for the objects. Then the pseudo labels are adopted to retrain the open-set 3D-Det model
(5) to train 2D object detection model
[19]. Even though there are no labels for the novel objects during training, the names of novel categories are known and fixed that the category-specific pseudo labels can be generated, which is thus an open-set while dlose-vocabulary model. More recently, [15] propose a deep method tackle OV-3Ddet. They adopt the CLIP model 22 and large-scale pretrained external OV-2Ddet [35] model to generate pseudo labels of possible novel objects. Our work is closer to 15, but we target OV-3Ddet without relying on any additional OV-2Ddet model priors to obtain novel box localization capabilities. Our model CoDA can simultaneously learn to conduct novel object discovery and cross-modal alignment with a collaborative learning manner, to perform end-to-end novel object localization and classification.
Methods
3.1 Framework Overview
overview of our proposed open-world 3D object detection framework called CoDA is shown in Fig. [ Our method utilizes the transformer-basse point cloud 3D object detection model
3

---

3DETR [9] as the detection backbone, which is represented as an 'Encoder' and a 'Decoder'
shown in Fig. I It has localization and a classification head for 3D object box regression and box category identification, respectively. To introduce open-vocabulary detection capabilities to the 31) point cloud object detection task, we utilize the pretrained vision-language model CLIP [22] to bring us rich open-world knowledge. The CLIP model consists of both an image encoder (i.e., Cmag--mage) and text encoder (i.e., CLIP-Text) in Fig. [ To localize novel objects, we first propose the 3D Novel Object Discovery (3D-NOD) strategy, which jointly utilizes both 3D box geometry priors from base categories and 2D semantic priors from the CLIP model to identify novel objects. To classify novel objects. we further propose novel discovery-driven cross moos- alignment (DCMA) module, which aligns feature spaces between 3D point clouds and image/text modalities, using the discovered novel object boxes guidance. The 3D-NOD and DCMA are jointly optimized to collaboratively learn to benefit each other in the unified deep framework.
3.2
3D Novel Object Discovery (3D-NOD) with Cross-Priors
As shown Fig. [ our 3D Novel Object Discovery (3D-NOD) strategy allows for the discovery of novel objects during training. To achieve this goal, utilize cross-priors from both 3D and 2D domains. In terms of the 3D domain, we use 3D box geometry prior provided by the 3D object annotations of base categories, to train a class-agnostic box predictor. Regarding the 2D domain, we utilize the semantic priors from the CLIP model, to predict the probability of a 3D object belonging to novel category. We combine these two perspectives to localize novel object boxes.
Discovery based on 3D Geometry Priors. We begin with an initial 3D object box label pool O base for objects of base categories (i.e., seen categories) in the training set:
base 0
{o, = (g) cj E CSeen).
(I)
object labels. Based on Ofuse, we can train an initial 3D object detection model Wide by minimizing an object box regression loss function derived from 3DETR [9]] We consider a class-agnostic object detection for Wo, meaning that we train the model with class-agnostic box regression and binary objectness prediction, and do not utilize the class-specific classification loss. This is because the class- class-specific training with strong supervision from base categories hampers the model's capabilities of discovering objects from novel categories, which has also been observed from previous OV-2DDet frameworks [8]. Thus our modified backbone outputs object boxes, including the parameters for the box 3D localization, and their objectness probabilities. For the n-th object query embedding in 3DETR, we predict its objectness probability p%. based on Woet which is learned using 3D geometry priors from human-ane -nnotated 3D object boxes on base categories.
where CSeen represents the base category set of the training data, which has
Discovery based on 2D CLIP Semantic Priors. Then, we also project the 3D object box 1&D in the point cloud to the 2D box 12D on the color image with the corresponding camera intrinsic matrix M as follows:
2D = M M x (3
(2)
We then obtain the 2D object region 12 in the color image by cropping it with 12 Next, we obtain the 2D object feature FOb1 by passing the object region into the CLIP image encoder ECLIP As the testing categories are unknown during training, we adopt super category list Truppe following 9 to provide text semantic descriptions of rich object categories. We encode Tusper with the CLIP text encoder EGLIP to obtain the text embedding ESuper We then estimate the semantic probability distribution (i.e., P3DObiy of the 3D object using its corresponding 2D object feature as: n
padObj = (Pn,i:Ph Ph. ...., Ph,c) = Softmax(F I.n F SSSper)
(3)
where C' is the number of categories defined in the super category set Tupper, The symbol indicates the dot product operation. padOb) provides semantic priors from the CLIP model. We can obtain the object category c* for the object query by finding the maximum probability in P3d0bj as e" argmax, rrgmax.paD00, p3DO6,, We then combine both the objectness y from the 3D geometry priors and Phee. from the 2D semantic priors, and discover novel objects for the L-th training epoch as follows:
O Oftisec
foz
| Vol E Loouso(oU of) < <.255, pI. 0 .r... > 00 $ Seen), (4)
4

---

Update
Novel object
Discovered Novel Label Pool
discovery
Base Objocts
Novel Objects
3D Boxes
3D Object Prediction
CLIP
Image
Encoder
Obbeetness
Boxes
Cross
2D Ohject
Match
CLIP Text Encoder
Entropy Loss
3D Object Features
Object Regions
Features
Class-Modal Similarity
Class-Agnestic Distillation
Table
TV
Transformer Decoder
object
Base
Novel
Encoded
Object
Point Cloud
Queries
Features
Encoder
Class-Agnostic Distillation
Class-Specific
Alignment
Figure 2: Illustration of the proposed discovery-dri -rriver cross-modal alignment (DCMA) in our OV detection framework (CoDA). The DCMA consists of two parts, i.e., the class agnostic distillation and the class-specific contrastive feature alignment. The predicted 3D boxes from the detection head are projected obtain image object regions, which are passed into the CLIP image encoder to generate 2D object features. Then, the CLIP 2D object features and 3D point cloud object features are fed into the class-agnostic distillation module for feature alignment. Driven by discovered novel object boxes from the 3D-NOD strategy, the maintained novel box label pool that is updated during training can be used to match with the predicted 3D object boxes. We perform contrastive alignment for matched novel boxes to learn more discriminative 3D object features for novel objects. More discriminative 3D object features can turn facilitate the prediction novel object boxes.
where IoU3D means the calculation of the 3D loU two boxes; and are two thresholds for the semantic and the geometry prior, respectively; Cseen is the set of seen categories of the training data. Then, the discovered novel objects Octice are added to a maintained 3D box label pool Opovel as:
novel ++1
Opovel UO
(5)
The model is trained to iteratively update the novel object box label pool. By gradually expanding the box label pool with the novel objects Oponel we can broaden the model's capabilities to localize novel objects by using supervision from Opovel for the model W., With the semantic priors from CLIP and the 3D geometry prior from W,, the model and the 3D object box label pool can discover more and more novel objects following the proposed object discovery training strategy.
3.3 Discovery-Driven Cross-Modal Alignment (DCMA)
To bring oocabbacy capabilities from the CLIP model, we propose a discovery-driven cross- modal alignment (DCMA) module, align the 3D point cloud object features with 2D- 2D-image/CLIP. text features, based on the discovered novel object boxes Oeniet For the proposed DCMA, we utilize the discovered novel object boxes conduct types of cross-modal feature aligament, i.e., class-agnostic distillation and class-specific contrastive alignment.
Object-Box Class- Distillation. We introduce class-agnostic distillation strategy to align the 3D point cloud object features and 2D image object features, to transfer more open-world knowledge from the CLIP model. For the n-th object query, we can obtain its updated 3D feature F3dObj at the last layer of the decoder. Based on the 3D object feature, the model predicts its 3D box parameterization 13?, and then have 2D-projected object features F2DObj by inputting CLIP with the cropped 2D region based on D2D push the feature spaces of both the 3D and 2D objects as close as possible, if a object box predicted from the 3D object feature, we align them by using class-agnostie L1 distillation loss,
obtain its corresponding 2D box 129 by Eqn. (2). Now
5

---

which is defined as follows:
N
2 distill
13D F2D0bi|
(6)
where N is the number of object queries. As shown in Fig. 2 even if the box covers background region, the class-agnostic distillation still narrows the distance between the two modalities, leading to a more general alignment on different modalities and scenes. As we can see, class-agnostic distillation eliminates the requirement of category labels of GT boxes. Thus, the computation of distill does not need category annotations.
Discovery-driven class-specific contrastive alignment. As we discussed in Sec. 3.2 we modify the 3DETR detection backbone, by removing the fixed-category class-specift classification head. We can extract 3D point cloud object features from the updated object query embeddings. For the n--h object query embedding, we can obtain a corresponding 3D object feature FsDO at the last layer of the decoder. With the 3D object feature F3D0bjj we expect the feature to be discriminative for both base categories and novel categories defined in the superset Tsuper, We, therefore, perform class-specific alignment with the CLIP text features. Specifically, we calculate the normalized similarities using the dot product operation between F&DObj and the text features F7 as follows:
S = Softmax(FF2 F (uSpper)
Then, we introduce the point cloud and text contrastive loss to provide the supervision signal for learning the 3D object features of the object queries. We have already measured the similarity between the 3D object point cloud feature and the text features. To construct cross-modal contrastive loss, we need to identify the most similar pair between these two modalities as a ground-truth label. We adopt the Bipartite Matching from [19] to match the predicted 3D) object boxes with the discovered novel boxes in the box pool Olabe! This is to guarantee that the class-specific alignment is conducted on foreground novel objects, instead of on non-object background boxes, as the object query embeddings also lead to the prediction of noisy 3D object boxes because of the limitation of supervision on novel objects during learning. If we perform class-specific alignment on noisy object boxes, it makes the 3D object feature less discriminative. When we have matching from the box label pool O O'abel, we then obtain the corresponding category label from the CLIP model by finding the maximum similarity score and construct one-hoo hot label vector h, for the 3D object feature F3DOb of the n-th object query. A detailed illustration can be seen in Fig. 2 By utilizing the discovered foreground novel object boxes, we can construct discovery-driven cross-modal contrastive loss function as follows:
N
Tert
1F 3DObj llabel). CE(SS,, hn): Σ n=1
(8)
where N the number of object queries; 1(.) is an indicator function that returns 1 if box matching is identified between the query and the box label pool else 0; CE(-) is a cross-entropy loss function. In this way, the object queries that are associated with novel objects are involved in the cross-modal alignment, leading to better feature alignment learning. The 3D features of novel objects can learn to be more discriminative, further resulting in more effective novel object discovery. In this way, the 3D novel object discovery and cross- modaalty feature alignment jointly boost the localization and classification for open- ppccbuocayllary 3D object detection.
4 Experiments
4.1
Experimental Setup
Datasets and Settings. We evaluate our proposed approach on two challenging 3D indoor detection datasets, i.e., SUN-RGBD 24 and ScanNetV2 4. SUN-RGBD has 5,000 training samples and oriented 3D bounding box labels, including 46 object categories, each with more than 100 training samples. ScanNetV2 has 1, 1,200 training samples containing 200 object categories 23 Regarding the training and testing settings, we follow the strategy commonly considered in 2D open-vocabulary object detection works [8] 29]. It divides the object categories into seen and novel categories based on the number of samples of each category. Specifically, for SUN-RGBD, the categories with the top-10 most training samples are regarded as seen categories, while the other 36 categories are treated as novel categories. For Sca the categories with the top- 10 most training samples are regarded

---

as seen categories, other 50 categories are taken as novel categories. Regarding the evaluation metrics, we report the performance on the validation set using mean Average Precision (mAP) at an IoU threshold of 0.25, as described in 19,, denoted as AP25.
Implementation Details. The number of object queries for both SUN-RGBD and ScanNetv2 is set to 128, and the batch size is set to 8. We trained base 3DETR model for 1080 epochs using only class-agnostic distillation
base model. Subsequently, we train the base model for an additional 200 epochs with the proposed 3D novel object discovery (3D-NOD) strategy and discovery-driver cross-modal feature alignment (DCMA). Note that, to ensure fair comparison, all the compared methods are trained using the same number of epochs. For 3D-NOD, the label pool is updated every 50 epochs. For feature alignment, considering that the discovered and GT boxes in scene are limited, and a larger number of object queries can effectively facilitate contrastive learning, aside from the queries that match GT boxes, we choose an additional 32 object queries from the 128 queries, which are involved in the alignment with the CLIP model. The hyper-parameter settings used in the training process follow the default 3DETR configuration as described in [9].
4.2 Model Analysis
In this section, we provide a detailed analysis of our open- open-vocabulary detection framework and verify each individual contribution. The different models are evaluated on three different subsets of categories, Le., novel categories, base categories, and all the categories. These subsets are denoted as AP Novels AP Base. and M respectively. All the evaluation metrics, including mean Average Precision (mAP) and recall (AR), are reported at an IoU threshold of 0.25. The ablation study experiments are extensively conducted on the SUN-RGBD dataset.
Methods
AP Noeeel
AP Base
AP Mean
AR
AR
AR Mean
3DETR + CLIP Distillation
3.61
30.56
9.47
21.47
63.74
30.66
3,28
33.00
9.74
19.87
64.31
29.53
3D-NOD + Distillation
5,48
32.07
11.26
33.45
66.03
40.53
3D-NOD Distillation & PlainA
0.85
35.23
8.32
34.44
66.11
41.33
3D-NOD + DCMA (full)
6.71
38.72
13.66
33.66
66.42
40.78
Table I: Ablation study to verify the effectiveness of the proposed 3D novel object discovery (3D-
NOD) and discovery-driven driven cross-modaliy alignment (DCMA) in our 3D OV detection framework CoDA.
Effect of class-agnostic distillation. To demon-
35
strate the effecttivnesss of class-agnostic distilla-
30-
AR
tion in discovery-driven alignment, we first train a class-agnostic 3DETR detection model that pro- vides basic 3D localization for novel objects. The 3D bounding boxes are then projected onto 2D color images to crop the corresponding 2D ob- ject regions. By feeding the object regions into the CLIP model, we conduct open- vocabulary classification to create an open-vocayulary 3D de- tection framework that requires both point cloud and image inputs. This baseline method is de- noted as "3DETR+CLIP" in Tab. [1 We also train 3DETR model with only the class-agnostic distilla- tion, which serves as our base model and is denoted as 'Distillation' in Tab. D As shown in the table, when tested with only pure point cloud as input, the "Distillation' model achieves comparable results to the '3DETRCCLIP' model in terms of AP Novet (3.28 vs. 3.61). indicating that class agnnosic distil- lation provides preliminary open- vocabulary detec- tion ability for 3DETR. Moreover, given the fact that the class-agnostic distillation pushes the 3D point cloud object features and 2D image features closer wherever a 3D bounding box is predicted, and the base categories have more training object
25
20
10
5
Wo 3D-NOD
|w33D-NO
200
400
600
800
1000
1200
Training
Figure 3: The influence of 3D Novel Object Discovery (3D-NOD) during training. The blue curve in the graph represents AP APnooct while the orange curve represents AR The left side of the black dashed line corresponds the training process of our base model 'Distillation, which is trained for 1080 epochs. After that, we apply our 3D-NOD on the base model and continue to train for an additional 200 epochs, which is represented by the right side of the black dashed line. By applying our 3D-NOD approach, we observe significant increase in both AP nowd and ARnovdt, which breaks the limitation of base category annotations.
7

---

samples, the distillation on the base categories is more effective, leading to a higher performance in terms of AP Base (33.00 vs. 30.56).
Effect of 3D Novel Object Discovery (3D-NOD). Building upon the base model 'Distillation', we apply our 3D novel object discovery (3D-NOD) strategy to enhance the open-vocahulary de- tection capability, denoted as "3D-NOD+Distilattion in Tab. [ As can be seen in the table, '3D- NOD+Distillation achieves a higher recall than both 'Distillation' and '3DETR+CLIP" (33.45 vs. 19.87/21.77). This demonstrates that our 3D-NOD can discover more novel objects during training, leading to higher Novo (5.48 vs. 3,28/3.61). To further analyze the influence of 3D novel object discovery during training, we monitor the entire training process and regularly evaluate the model at intermediate checkpoints. As shown in Fig. [3] the blue curve represents AP novel while the orange curve represents AR The left side of the black dashed line corresponds to the training process of our base model 'Distillation' which is trained for 1080 epochs. After that, we apply our 3D-NOD strategy on the base model and continue to train it for an additional 200 epochs, represented by the right side of the black dashed line. We observe that limited by the base category labels, the AP of the base model nearly stops growing at 1000 epochs. Moreover, ARnovel begins to decrease as training continues, which meets the category forgetting phenomenon due to the lack of annotations for novel categories. However, by applying our 3D-NOD approach, we observe significant increase in both AP novel and AR which demonstrates the effectiveness of our 3D-NOD approach.
Effect of Discovery-Driver Cross-Modal Alignment (DCMA). To explore the impact of different alignment methods on the performance of the "DD-NDDDDisiilattion model. Firstly, we introduce the plain text-point alignment to the model, denoted as HID-NOD+Distillatton & PlainA'. The plain alignment involves aligning the category list of base category texts without using the discovered boxes by 3D-NOD, As shown in Tab. [
the Base is higher than that of 3DD-NOD+Distillation (35.23 vs. 32.07), indicating that the base category texts help to improve the cross-modal feature alignment. Additionally, 3D-NOD still boosts the NRNouet, indicating that it can still facilitate the discovery of novel objects under this comparison. However, the Nowel is lower at 0.85, which shows that the model loses its discriminative ability on novel categories if without utilizing the box discovery driven by 3D-NOD. In contrast, after applying our discovery-driven class-speciien method, the "3D-NOD+DCMA' model in Tab. [T achieves the best performance on both AP Noeel and AP Base- This suggests that compared with the plain alignment, our discovery-d alignment method brings more discriminative object features by using larger category vocabulary driven by the 3D-NOD.
cross-modal alignment
Effect of the collaborative learning of 3D-NOD and DCMA. As shown in Tab. I by comparing with the alignment method that is not driven by 3D-NOD (i.e., '3D-NOD + DCMA' vs. '3D-NOD + Distillation & PlainA'), our 3D-NOD method improves the cross-model alignment, resulting in better performance on the novel category classification. To investigate the effect DCMA on the novel object discovery, we apply our 3D-NOD method to the "3DETR+CLIP pipeline without utilizing DCMA, which is denoted as ADDER+CLIP++D-NNEE "3D-NOD+DCMA (full)' outperforms "3DETR+CLIN +CLIP+3D-NOD in terms of AP and AR for both novel and base categories. This demonstrates that the proposed DCMA can effectively help to learn more discriminative feature representations of the object query embeddings, thus improving the localization capability. Overall, these results show that 3D-NOD and DCMA together can improve the detection performance by 26% on AP Novel and 48% on AP Base.
in Tab. 2 As shown in Tab. [
Methods
AP Novel
AP Buse
AP Meun
Noee
AR Base ARMe
3DETR CLIP 22 3DETR + CLIP + 3D-NOD
3.61
30.56
9.47
21.47
63.74
30.66
5.30
26.08
9.82
32,72
64.43
39.62
3D-NOD DCMA (full)
6.71
38.72
13.66
33.66
66.42
40.78
Table 2: Ablation study on SUN-RGBD [4] to verify the effectiveness of the collaborative learning of DCMA and 3D-NOD. Compared with together can effectively improve the open-vocabulary detection performance.
utilizing 3D-NOD and DCMA
The Sensitivities of Thresholds in 3D-NOD. In 3D novel object discovery, the utilization of the 2D semantic priors and 3D geometry priors requires two thresholds. We perform experiments to evaluate their sensitivities, and the results are presented in Tab. 3. The first row denoted by '0.0' represents the base model without our contributions. As we can see, our 3D-NOD method can produce stable
8

---

Semantic
Geometry
Novel
AP Buse
AP Mean
AR Novel
AR Base
MRMean
0.0
0.0
3.28
33.00
9.74
19.87
64.31
29.53
0.3
0.3
6.71
38.72
13.66
33.66
66.42
40.78
0.3
0.5
6.35
39.57
13.57
31.93
66.91
39.53
0.5
0.3
5.70
38.61
12.85
27.05
63.61
35.00
0.5
0.5
5.70
39.25
13.00
28.27
65.04
36.26
Table 3: Ablation study the thresholds used for the 2D semantic and 3D geometry priors in the 3D Novel Object Discovery (3D-NOD) strategy. Disabling these priors leads to significant performance drops (see the first row), and the model achieves the best AP performance with thresholds of 0.3.
improvements with different thresholds from a wide range, demonstrating that its effectiveness not reliant on carefully selected thresholds. As shown in Tab. despite the performance variability, all our models trained with different threshold settings can consistently outperform the model trained without 3D-NOD (i.e., '0.0&0.0" in the first row) by a significant margin of 70% or more, clearly verifying its advantage. Besides, the setting with a threshold of 0.3 yields the best performance as enables the discovery of more novel objects.
4.3 Comparison with Alternatives
Quantitative Comparison. Open-world vocabulary 3D detection still a very new problem. Since there very few works in the literature and our open-vocabulary setting is novel, no existing results can be directly compared with our approach. Therefore, we adapt the latest open-vocabulary point cloud classification method to generate pseudo boxes and employ PointCLIP 31, PointCLIPv2 [36 and Det-CLIP² 30 to conduct open vocabulayy object detection, which is denoted as Deet-PoitCCLIP.. 'Det- Peet-PointCLIPV2 and 'Det-CLIP2' in Tab, respectively. Additionally, we introduce a pipeline with 2D color images input, where use 3DETR generate pseudo 3D boxes. We then the camera intrinsic matrix to project 3D boxes onto 2D boxes, which are adopted to crop corresponding 2D) regions. Finally, the CLIP model is utilized to classify these regions. This pipeline enables
setting and evaluate its performance. Specifically,
utilize 3DETR
generate 3D detection results, denoted as '3D-CLIP' in Tab. 4 As can be observed in the table, the AP Novel and AR Noeel of our method are significantly higher than other methods, further demonstrating the superiority of our design in both novel object localization and classification.
Methods
Inputs
AP AP
AP Buse
AP Meun.
AR None!
AR Baa
AR Meuen
RUN-RGBD
Det-PointCLIP
31
Point Cloud
0.09
5.04
1.17
21.98
65.03
31.33
Det-PointCLIPP2
Point Cloud
0.12
4,82
1.14
21.33
63.74
30.55
Det-CLIP 30
Point Cloud
0.88
22.74
5.63
22.21
65.04
31.52
3D-CLIP
Image&Point Cloud
3.61
30.56
9.47
21.47
63.74
30.66
CoDA (Ours)
Point Cloud
6.71
38.72
13.66
33.66
66.42
40.78
ScanNetv2
Det-PointCL.PP
31
Point Cloud
0.13
2.38
0.50
33.38
54.88
36.96
Dot-PointCLIIPP2 36
Point Cloud
0.13
1.75
0.40
32.60
54.52
36.25
Det-CL.P2 30
Point Cloud
0.14
1.76
0.40
34.26
56.22
37.92
3D-CLIP 22
Image&Point Cloud
3.74
14.14
5.47
32.15
54.15
35.81
CoDA (Ours)
Point Cloud
6.54
21.57
9.04
43.36
61.00
46.30
Table 4: Comparison with other alternative methods on the SUN-RGBD and ScanNetv2 dataset. The 'inputs" in the second column refer to the testing phase. During testing, '3D-CLIP" needs 2D images as inputs to perform voccbuacayllary classification for the detected objects. While our method CoDA only utilizes pure point clouds testing inputs.
In Tab, 4 we present a comparison of our method with alternative approaches on ScanNetv2 4 In this evaluation, we consider the categories with the top 10 most training samples in ScanNetv2 as base categories, while the other first 50 categories presented in 23) are considered novel categories. To obtain image and point cloud pairs, we follow the approach described in [15] to generate point cloud scenes from depth maps. As shown in Tab. 4 our method significantly outperforms other approaches on AP and AR metrics for both novel and base categories, even with only point clouds as inputs during testing.

---

Comparison with OV-3DET [5] Note that the setting in our method is significantly different from OV-3DET [T5]. In our setting, we train the model using annotations from small number of base categories (10 categories) and learn to discover novel categories during training. We evaluate our model in a large number of categories (60 ScanNet and 46 in SUN-RGBD). However, [5] relies on a large-scale pretrained 2D open vocabulary detection model, i.e., OV-2DDet model [5 to generate pseudo labels for novel categories, and evaluate the model on only 20 categories. Thus, because the setting differences and the prior model utilized, direct comparison with [15] is not straightforward. To provide a comparison with [3], thanks to the released codes from the authors of [3] to generate pseudo labels ScanNet, we directly retrain method following the same setting of [15] on ScanNet, and provide a fair comparison with [15] on Tab, [|] As can observed, our method achieves clearly better mean AP (1.3 points improvement over all the categories), validating the superiority of our method.
Methods
Mean
chair
dresser
table
cabinet
bookshelf
pillow
OV-3DET [15]
18.02 19.32
57.29
42.26
27.06
31.50
8.21
14.17
2.98
5.56
23.00
31.60
CoDA (Ours)
68.09
44.04
28.72
44.57
3.41
20.23
5.32
0.03
27.95
45.26
Methods
bathtuh
refrigerator
desk
nightstand
counter
door
curtain
hox
lamp
hag
OV-3DET 15
56,28
10.99
19.72
0.77
0.31
9.59
10.53
3.78
2.11
2.71
CoDA (Ouns)
50.51
6.55
12.42
15.15
0.68
7.95
0.01
2,94
0.51
2.02
Table 5: Comparison of methods in the same setting of [15] on ScanNet. 'Mean" represents the
average value of all the 20 categories.
Qualitative Comparison. As depicted in Fig. our method demonstrates the strong capability discover more novel objects from point clouds. For instance, method successfully identify the dresser in the first scene and the nightstand in the second scene. The performance comparison indicates that our method generally exhibits superior open-vocabulary detection ability on novel objects.
3D CLIP
CoDA (Ours)
Figure 4: Qualitative comparison with 3D-CLIP (22). Benefiting from our contributions, our method CoDA can discover more novel objects, which are indicated by blue boxes the color images.
5 Conclusion
This paper proposes a unified framework named CoDA to address the fundamental challenges in OV-3DDet, which involve localizing and classifying novel objects. To achieve 3D novel object localization, we introduce the 3D Novel Object Discovery strategy (3D-NOD), which employs both 3D box geometry priors and 2D semantic open- geccbbocayulary priors to discover more novel objects during training. To classify novel object boxes, we further design a Discovery-driven Cross-Modis Alignment module (DCMA) that includes both class-agnostic and class-discrmiinative alignment to align features of 3D, 2D and text modalities. Our method significantly outperforms alternative methods by more than 80% in mAP.
10

---

References
Open-set 3d detection via image-level class and debiased cross-modal contrastive learning.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV. Springer, 2020.
[3] Bowen Cheng, Lu Sheng. Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for voting-based 3d object detection in point clouds. In CVPR, 2021.
[4] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-anne Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[5] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR. IEEE, 2009.
[6] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- voca-vocybllary object detection with vision-language model. CVPR, 2022.
[7] Chengjian Feng. Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei. Weidi Xie, Ma. Promptdet: Towards vpec-locab detection using uncurated images. In ECCV. Springer, 2022.
[8] Xiuye Gu, Tsung-Yi Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint (tii:2104.19222 2021.
[9] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[10] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow-detr: Open-world world detection transformer. In CVPR, 2022.
[11] Chao Jia, Yinfei Yang. Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, fun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and Jision-lanuuag representation learning with noisy text supervision. In JCMI.. PMLR, 2021.
[12] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In CVPR, 2021.
[13] Liunian Harold Li. Pengchuan Zhang. Haotian Zhang. Jianwei Yang. Chunyuan Li. Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre-training. In CVPR, 2022.
[14] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In ICCV, 2021.
[15] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shang- bang Zhang. Spen-vocabulary point-cloud object detection without 3d annotation. arXiv preprint arXiv:2304.0077 2023.
[16] Xinzhu Ma. Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular object detection. In CVPR, 2021.
[17] Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Opea--voabullary one-stage detection with hierarchical visual- langua-l knowledge distillation. In CVPR, 2022.
[18] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open- vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.00 062300 2022.
[19] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end -nd transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2906 2906-2917, 2021.
[20] Charles Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas, Pointnet: Deep lcarning on point sets for 3d classification and segmentation. In CVPR, 2017.
[21] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas, Deep hough voting for 3d object detection in point clouds. In ICCV, 2019.

---

[22] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021.
[23] David Rozenberszki, Or Litany, and Angela Dai. Language- grouuned indoor 3d semantic segmentation in the wild. ECCV. Springer, 2022.
[24] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: rgb-d scene understanding benchmark suite. In CVPR, 2015,
[25] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, and Jun Wang. Mlevnet: Multi-level context votenet for object detection. In CVPR, 2020.
[26] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang. Dening Lu, Mingqiang Wei, and Jun Wang. Venet: Voting enhancement network for 3d object detection. In ICCV, 2021.
[27] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dietionary-enriched visual-concept paralleled pre-training for open-world detection. In NeurIPS, 2022.
[28] Lewei Yao, Jianhua Han, Xiaodan Liang, Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In CVPR, 2023.
[29] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
[30] Yihan Zeng. Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung. Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive Language-imageppoiee pretraining from real-world point cloud data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15244 15253, 2023.
[31] Renrui Zhang, Ziyu Guo. Wei Zhang, Kunchang Li, Xupeng Miao. Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. In CVPR. 2022.
[32] Yinmin Zhang, Xinzhu Ma, Shuai Yi, Jun Hou, Zhihui Wang, Wanli Ouyang, and Dan Xu. Learn- ing geometry guided depth via projective modeling for monocular 3d object detection. arXiv preprint 1xXiv:2107.13911 2021.
[33] Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H13det: 3d object detection using hybrid geometric primitives. ECCV. Springer, 2020.
[34] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang. Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, al. Regionclip: Region- based language-image pretraining. In CVPR, 2022.
[35] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. arXiv preprint arXiv:2201.02660 2022.
[36] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, Peng Gao. Pointelip v2: Adapting clip for powerful 3d open-world learning, arXiv preprint arXiv:2211.11682, 2022.
12


Reviews:
[{"review_id": "PPzQ7xoXoO", "reviewer": "Reviewer_DwKh", "summary": "This paper is about addressing open-vocabulary 3D object detection, which describes a method aiming for two key challenges, localization and classification for novel objects in point-cloud. The key components are 3D novel-object discover module and the discovery-driven cross-modal alignment module. The first module tries to find as many as possible novel objects based on both geometry and semantic while the second module attempts to align the features of different modalities. The proposed method shows performance improvements compared with the designed baselines.", "questions": "My main concern is about the discussion to [15] and more necessary experimental analysis. Please refer the weakness part.", "limitations": "The work is only designed based on 3DETR, an indoor point-cloud detector. It should be evaluated based on more detectors.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and shows superior performance improvement compared with the designed baseline.", "weaknesses": "I have concerns about both the method novelty and the experiment details. Regarding the method novelty, the key module DCMA is very similar to [15], the difference seems that instead of using contrastive learning, the authors proposed to use distillation loss. However, there is neither insightful discussion nor experiments to analyze why the proposed method is better. \n\nRegarding experiments, [15] is discussed to be similar to this paper in the introduction while no more comparison in experiments, the authors should compare either the whole model or the key components.\nOn the other hand, I also have concerned about the fairness of the experiments based on 3DETR, as far as I know, 3DETR has much better performance on both SUNRGBD and ScanNet datasets. For the \"base\" category setting of Table 1, the performance is extremely low, does it mean using CLIP will degenerate the performance of 3DETR? How about directly using the same setting? Furthermore, it is necessary to evaluate the performance on top of different point-cloud detectors instead of only 3DETR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 Novelty of DCMA:**\n\nOur proposed Discovery-Driven Cross-Modal Alignment(DCMA) has two key differences from the contrastive learning in [15]:\n\n(i) Joint Learning with 3D Novel Object Discovery (3D-NOD): DCMA is simultaneously learned with our 3D-NOD, enabling generalization to a broader range of novel categories during the discovery process, which can be demonstrated by comparing '3D-NOD + DCMA (full)' over '3D-NOD + Distillation' and '3D-NOD + Distillation \\& PlainA' shown in Tab. 1 of the main paper. In contrast, [15] performs alignment for additional categories by utilizing an additional large pretrained 2D open-vocabulary detection model [34] to generate 3D pseudo labels, making a substantial difference in learning and obtaining open-world knowledge. To the best of our knowledge, the collaborative learning of classification and localization for novel objects clearly highlights a novel contribution in open-vocabulary 3D detection (OV-3DDet).\n\n(ii)\tClass-Agnostic Distillation during Discovery: [15] proposes a class-specific contrastive learning strategy, while we present a class-agnostic distillation that can more effectively align cross-modal features during the discovery process. The class-specific contrastive learning encourages features belonging to the same categories to be as similar as possible. However, during the discovery process, the discovered boxes may cover background regions. Since CLIP cannot classify the background, boxes on background with higher scores may be assigned to incorrect categories, thus leading to severe feature misalignment by class-specific contrastive learning. Conversely, our class-agnostic distillation aims at making the 3D object query features align with their corresponding projected 2D image features. The distillation can also cover 3D object boxes on the background. So the distillation can be effectively performed for the foreground and background simultaneously. To show our advantage compared to class-specific contrastive learning, we replace class-agnostic distillation with class-specific contrastive learning in our methods. As shown in Tab. A of the rebuttal PDF, the contrastive loss encourages more boxes to be classified as the foreground, resulting in a larger recall. However, as discussed, it also encourages more background boxes to be misclassified as foreground objects, leading to a lower overall mAP compared to our class-agnostic distillation.\n\n**Q2 Comparison with [15]:**\n\nWe would like to highlight that the setting in our methods is significantly different from [15]. In our setting, we train the model using annotations from a small number of base categories (10 categories) and learn to discover novel categories during training. We evaluate our model in a large number of novel categories (50 in ScanNet, and 46 in SUN-RGBD). However, [15] relies on an large-scale pretrained 2D open vocabulary detection model, i.e., OV-2DDet [34] model, to generate pseudo labels for novel categories, and evaluate the model on only 20 categories. Thus, because of the setting differences, and the prior model utilized, a direct comparison with [15] is not straightforward. To provide a comparison with [15] as requested, thanks to the releasing the codes from the authors of [15] to generate pseudo labels on ScanNet, we can directly retrain our method following the same setting of [15] on ScanNet, and provide a fair comparison with [15] on Tab. B of the rebuttal PDF. As can be observed, our method achieves clearly better mean AP (1.3 points improvement over all the categories), validating the superiority of our method.\n\n**Q3 Comparison with 3DETR [19]:**\n\nWe want to further clarify that, the settings on base categories between ours and 3DETR are different. Our method targets 3D open-vocabulary (OV) detection, and thus we follow a typical setting for base categories in 2D-OV detection frameworks, i.e., using top-10 categories with the highest number of training samples as the base categories. Because of the different settings, direct comparison on mAP of 10 categories (including 5 different categories) is not fair. To compare our method and 3DETR fairly, we can compare the mAP of the 5 shared categories. As shown in Tab. C of the rebuttal PDF, it can be clearly observed that our method achieves better performance than 3DETR in terms of the mean AP of the shared categories (58.51 vs. 57.36), which may confirm the benefits of using CLIP features for learning more discriminative features. Besides, according to the reviewer’s suggestion, we also conducted experiments to show the performance comparison between ours and 3DETR considering the same setting of the base categories. Limited by the computation resources, the model training is not finished yet. We will try to provide the comparison in the later discussion period once it is finished.\n\n**Q4 Different 3D detectors:**\n\nThanks for the suggestion. We considered 3DETR because it is a widely employed transformer-based 3D object detection backbone, which is also utilized by related work [15]. Our proposed method contributes a new technique for collaborative learning novel 3D object box discovery and cross-modal alignment, which is independent of different detection frameworks. Because of the time limit of the rebuttal, we need more time to run results using other 3D detection frameworks. We will do our best to provide new results regarding this point in the later discussion period."}, {"review_id": "lcJF5LwfoL", "reviewer": "Reviewer_Qxx5", "summary": "This paper proposes an end-to-end open-vocabulary 3D object detection (OV-3DDet) framework which can learn to localize and classify novel objects simultaneously.  To achieve OV-3Det, the author designed a 3D Novel Object Discover (3D-NOD) strategy which utilizes both the 3D box geometry priors and 2D semantic open-vocabulary priors to generate pseudo box labels of the novel objects and develop a Discovery-Driven Cross-Modal Alignment (DCMA) module based on discovered novel boxes, to align feature spaces between 3D point cloud and text modalities for better feature alignment. ", "questions": "1. In line 186, the author claims that \"even if the box covers a background region, the class-agnostic distillation still narrows the distance between the two modalities, leading to a more general alignment on different modalities and scenes.\" However, I didn't see any experiments to support such claim. \n2. In line 235, the paper states \"For feature alignment, we randomly selecte an additional 32 object queries to be involved in the alignment with the CLIP model.\"  Why do we need to select additional 32 object queries to conduct the feature alignment with CLIP model, not using the 128 object queries?\n3. Figure 3 shows that the AP_novel keeps stable and AR_novel even decreases after 1080 training epochs. Why are the long epoch training settings used for the final model ?\n4. The paper lacks some experiments to demonstrate the upper bound of the detection performance. For example,  the setting of training a class-agnostic 3DETR model on all the categories (base + novel) annotations and using CLIP to  classify the object during inference could be added to demonstrate the how well the cross-modal feature is aligned.\n5. Is it possible to show the cross-dataset evaluation performance, for example, training on ScanNet and test on SUN RGBD?", "limitations": "The generalization ability of this method is unknown. For example, the open-vocabulary ability remains unknown or limited when more categories (e.g., 200) are introduced.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The key difficulty of Open-vocabulary 3D Object detection lies in the localization of novel categories. The paper proposes a novel pseudo novel object annotation generation approach, 3D Novel Object Discovery (3D-NOD) by utilizing the cross-priors from both 3D and 2D domains. Different from previous work, the approach discards the reliance on any additional 2D detection model. The model can learn to localize more objects by updating the novel box label pool.\n2. The author provides a detailed analysis of the proposed open-vocabulary detection framework and verifies the benefits of each proposed module. The paper conducts relatively detailed ablation experiments for each proposed module and provides a clear introduction to the implementation details within each module.\n3. This paper is relatively clear and easy to follow. ", "weaknesses": "1. There are minor grammar and spelling issues in the paper, such as:\n    1. Line 161:  We then combine both the objections (-> objectness) p_n from the 3D geometry priors a xxxxx.\n    2. Line 235: For feature alignment, we randomly selected (-> select) an additional 32 object queries to be involved in the alignment with the CLIP model.\n\n2. Cross-modal feature alignment is common in previous open-vocabulary 3D object detection methods, such as OV-3DET. From my perspective, the DCMA method in the paper demonstrates that not only at the feature level for seen class objects but also on textual alignment for novel class objects should be considered in point-cloud-text feature alignment. From the overall description, it appears that this is not a novel solution but rather an integrated utilization of previous work.\n3. The novel box labels generated by the 3D-NOD approach originate from inaccurate 3D bounding boxes produced by the model during training, which are filtered based on certain conditions. These generated novel box labels still have issues in terms of accuracy in both position and size, which limits the final novel object localization accuracy.\n4. In Section 3.2, there is inconsistency in the use of symbols during the description of details, and it is rather complex. For example, $l_{n}^{3D}$ and $l_{n}^{2D}$ are unnecessary because people can understand what they are; the image encoder $E_{I,n}^{CLIP}$ can be simplified as $E_{img}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 Minor grammar issues:**\n\nThank you very much for your careful review. We will fix the minor issues in the revision.\n\n**Q2 Novelty of DCMA:**\n\nWe design the Discovery-Driven Cross-Modal Alignment (DCMA) in conjunction with the 3D-NOD to achieve joint learning of 3D object box discovery and cross-modal class-agnostic alignment. As far as we know, it is the first work that explores novel 3D object box discovery for open-world 3D object detection. The significant difference in terms of class-agnostic alignment with the related work OV-3DET [15] is elaborated in detail in the answer for Q1 of Reviewer DwKh.\n\n**Q3 Limitation from the inaccurate 3D bounding boxes:**\n\nAs shown in Tab. 3 of our main paper, compared with the model trained without 3D-NOD (i.e., the '0.0&0.0' baseline), the different thresholds for semantic and geometric priors for 3D-NOD all bring significant improvements (more than 70%) no matter the presence of noisy boxes generated, showing that our method is not sensitive to these filtering conditions. We also agree with the reviewer that if we can further reduce the noisy 3D boxes in the discovery stage, our 3D open-world detection performance can be further boosted since the cross-modal alignment will be more accurate and effective.\n\n**Q4 Inconsistency of the symbols:**\n\nThank you very much. we will carefully refine this part in the revision.\n\n**Q5 Experiments about the agnostic distillation:**\n\nTo evaluate this point, we further conducted experiments to compare the difference between the\nclass-agnostic and class-specific distillation, as shown in Tab. D of the rebuttal PDF. As discussed in the main paper, the key difference is whether the method considers the category labels for GT 3D boxes when constructing the distillation objective. In Tab. D of the rebuttal PDF, the $\\text{AP}_{Novel}$ of 'Class-agnostic Distillation' outperforms 'Class-specific Distillation', further proving that the class-agnostic distillation offers better generalization for novel categories. We will add the results and discussions in the updated version.\n\n**Q6 Unclear description of the additional 32 object queries:**\n\nWe are sorry for the unclear description. To clarify, the total number of object queries remains 128, and we do not increase the number of object queries of the detection backbone. As described in L211-L212 of the main paper, the contrastive alignment is conducted for the object queries that match the box labels from both GT and the discovered novel box pool. Considering that the discovered and GT boxes in a scene are limited, and a larger number of object queries can effectively facilitate contrastive learning, aside from the queries that match GT boxes, we choose an additional 32 object queries from the 128 queries. So 'Additional 32 object queries' are still included in the 128 object queries. Besides, the reason for not performing a contrastive alignment on all 128 queries is to avoid the contrastive learning being dominated by the background boxes, which harms the discovery of novel foreground objects.\n\n**Q7 The reason why the long epoch training settings are used:**\n\nFrom the observations, we noticed that in the early epochs, the training does not converge well, and the model tends to generate more boxes to cover more foreground objects, thereby increasing AR. However, it will also introduce more noisy boxes that cover the background, while later during the 3D-NOD stage, these noisy boxes cause incorrect discovery of novel object boxes, thus resulting in a worse performance during the 3D-NOD stage. To address this issue, before the 3D-NOD discovery process, we keep training the model for 1080 epochs to reduce the presence of noisy boxes. During these training epochs, both noisy and foreground boxes were reduced, leading to a decrease in AR while maintaining a stable AP. Despite the reduction in foreground boxes, we observed that with fewer noisy boxes, the 3D-NOD algorithm could quickly discover more foreground boxes, as demonstrated in Fig. 3 of our main paper. This led to a significant increase in both AP and AR. Besides, in 3DETR [19], they also train the model for 1080 epochs to make sure the training reaches a good convergence.\n\n**Q8 Upper bound of training a model on all the categories:**\n\nThank you for the suggestion. We conducted such an experiment to verify the performance upper bound. As shown in Tab. E of the rebuttal PDF, while our method demonstrates significant improvements in 3D novel object detection, compared with the upper bound, there is still room for further investigation.\n\n**Q9 Cross-dataset evaluation:**\n\nFollowing the suggestion, we have evaluated the cross-dataset performance. The comparisons between the cross-dataset and the same-dataset performances in terms of $AP_{Novel}$ are 3.88 vs. 6.12 on ScanNet and 2.64 vs. 6.71 on SUN RGBD. The lower performances of the cross-dataset evaluation can be due to the data domain gap from the utilization of different RGB-D sensors.\n\n\n**Q10 The generalization ability when more categories (e.g., 200) are introduced:**\n\nTo study the generalization ability of our model for more categories, we have conducted experiments shown in Fig. A of the Supplementary. From the results, a gradual decrease in terms of both $AP_{Mean}$ and $AP_{Novel}$ is observed, as the number of test categories increases. When it increases to 200 categories, $AP_{Mean}$ and $AP_{Novel}$ are reduced to 4.68% and 2.85%, respectively. This is a common phenomenon in 3D open-world detection [15].\n"}, {"review_id": "IVmxGjOPF3", "reviewer": "Reviewer_zhog", "summary": "This paper aims to address the problem of detecting objects from an arbitrary list of categories within a 3D scene, which is less explored area in the literature. It proposes a unified framework that simultaneously addresses the fundamental problems of localizing and classifying novel objects, under the condition of limited base categories. The framework incorporates a 3D Novel Object Discovery (3D-NOD) strategy to localize novel 3D objects, which utilizes both 3D box geometry priors and 2D semantic open-vocabulary priors to generate pseudo box labels of the novel objects. Besides, it introduces a cross-modal alignment module based on discovered novel boxes, to align feature spaces between 3D point cloud and image/text modalities for novel object box classification. Experiments on the SUN-RGBD and ScanNet demonstrate the effectiveness. ", "questions": "1. It would be interesting to see if the proposed model can detect objects beyond the 200 categories in ScanNet-200. Some visualizations or illustrations that demonstrate the upper bound of the proposed method would help readers understand its potential capabilities.", "limitations": "The paper doesn't explicitly discuss the limitations of the proposed method. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea makes sense to me. The framework includes a system that discovers new 3D objects and aligns them with other types of data. This helps the system learn better and eliminates the need for a 2D detector. The system also uses CLIP semantics to find objects in 3D scenes during training.\n2. The proposed framework performs much better than other methods on the ScanNet-200 and SUN RGBD datasets and achieves a significant improvement in mAP.", "weaknesses": "1. The paper doesn't discuss how the proposed framework's computational complexity compares to other state-of-the-art methods, or how efficient it is.\n2. The paper doesn't include an analysis of cases where the proposed framework fails to detect objects.\n3. It would be helpful to analyze how the convergence rate is affected when the novel object list is similar to the ScanNet-200 dataset.\n4. The experiments don't compare the proposed framework to a fully supervised ScanNet-200 method, which would help readers understand the gap in detecting novel objects. Additionally, comparing the proposed framework to an oracle-supervised method with the same model would be useful in understanding the usefulness and importance of the framework in real-world scenarios.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 Model complexity:**\n\nDuring testing, our method only adds the CLIP-text encoder and one more distillation head to 3DETR. Therefore, the parameters are just slightly more than 3DETR [19] backbone (6.99M vs. 5.81M), which demonstrates that our designs will not bring many extra computations.\n\n**Q2 Failure cases:**\n\nThanks for the suggestions. We will clarify the failure cases in the updated version. As shown Fig. C of the rebuttal PDF, failure cases in challenging scenes can be observed for similar semantic categories. For instance, in the second row, a 'coffee table' is misclassified as a 'table'. This issue may be further improved by introducing a stronger classifier than CLIP in future work. We will add the failure case analysis in the revision.\n\n**Q3 How the convergence rate is affected when the novel object list is similar to the ScanNet-200 dataset:**\n\nFollowing your suggestion, we conducted experiments on SUN-RGBD with the categories of Scan-Net200 as the super-category list for 3D-NOD. As shown in Fig. A of the rebuttal PDF, the training process of our base model, denoted as 'Distillation', is corresponding to the blue curve. The orange curve represents $AP_{Novel}$ of the model trained using 3D-NOD with the LVIS class list, while the green curve indicates the model trained using 3D-NOD with the ScanNet200 class list. From the figure, we can observe that the convergence rate is not significantly affected when replacing the category list with the ScanNet200 list. Notably, the green curve is slightly lower than the orange curve, which is probably because of the significant change in the number of classes, i.e., from a small number of classes (200 categories) to a much larger number of classes (1.2k categories) in the LVIS class list.\n\n**Q4 Comparison with a fully supervised ScanNet-200 method:**\n\nFollowing your suggestion, we trained a fully supervised model on ScanNet-200. The mAP achieved for all categories is 9.66. Our proposed method achieved mAP of 2.74 across all the categories. As discussed in Sec. 1.2 of the supplementary, the open-vocabulary 3D detection ability of our approach remains limited when evaluated on a larger number of categories (like 200 categories) despite a state-of-the-art performance.\n\n**Q5 Visualization to see if the proposed model can detect objects beyond the 200 categories in ScanNet-20:**\n\nAs shown in Fig. B of the rebuttal PDF, our methods can detect the objects belonging to the 'folder' category and beyond the 200 categories in ScanNet-200, which are labeled by blue boxes in the color image of the second row. It shows that our methods have the potential to extend the application to larger vocabularies.\n\n**Q6 Limitations:**\n\nAs we discussed in Q4 above, considering the pure point clouds are less discriminative than color images, the open-vocabulary detection ability for larger vocabulary is still limited, which is a common challenge for related 3D open-vocabulary scene understanding [R1, R2, R3]. \n\n[R1] Songyou Peng, Kyle Genova, Chiyu Jiang,Andrea Tagliasacchi, Marc Pollefeys, Thomas\nFunkhouser, etal. Openscene: 3d scene understanding with open vocabularies. arXiv preprint\narXiv:2211.15654,2022.\n\n[R2] Bo Liu,Shuang Deng,Qiulei Dong,and Zhanyi Hu. Language-level semantics conditioned 3d\npoint cloud segmentation. arXiv preprint arXiv:2107.00430,2021.\n\n[R3] Runyu Ding, Jihan Yang, Chuhui Xue,Wenqing Zhang,SongBai, and Xiaojuan Qi. Language-driven open-vocabulary 3d scene understanding. arXiv preprint arXiv:2211.16312,2022."}, {"review_id": "EJ1I6HRmOX", "reviewer": "Reviewer_goao", "summary": "This paper addresses the challenge of open-vocabulary 3D object detection (OV-3DDet). The proposed framework consists of two modules: the 3D Novel Object Discovery (3D-NOD) module and the Discovery-Driven Cross-Modal Alignment (DCMA) module. The 3D-NOD module utilizes a detector trained on annotations of base categories (without class-specific information) to localize potential objects. It then employs the pre-trained CLIP to discover the potential super categories of these objects. In the DCMA module, two types of alignments are performed. First, the 3D object features are aligned with their corresponding 2D object features using the CLIP image encoder. Second, the 3D object features are aligned with their corresponding text features using the CLIP text encoder. The proposed framework is mainly evaluated on the SUN RGB-D dataset to assess its performance and effectiveness.", "questions": "1. The correctness of Eq. 4 is questionable. It is unclear why a novel object must have an IoU over 0.25 with objects from base categories. Further clarification and justification from the authors would be helpful to understand the rationale behind this threshold.\n\n2. It is not explicitly mentioned whether the base objects are included in O^novel. If the base objects are not part of O^novel, it is important to clarify how the base objects contribute to the cross-modal alignment when only O^novel is used in computing the alignment loss (Eq. 8). \n\n3. In line 235, the meaning of \"additional 32 object queries\" is not explicitly explained. It is unclear whether it refers to the total number of queries in this alignment being 128+32.", "limitations": "The authors did not discuss its limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem of open-vocabulary 3D object detection addressed in this paper is highly relevant in real-world scenarios where the number of object classes is not predefined or limited. The ability to detect and classify objects in an open-world setting is crucial for applications such as autonomous driving, robotics, and augmented reality.\n\n2. The proposed method is technically sound. The approach of jointly training the novel object discovery module for localization and the cross-modal alignment module for classification is well-motivated.", "weaknesses": "1. The experimental results presented in the paper lack convincing evidence and could be improved in several aspects.\n\n+ First, in Table 4, it would be beneficial to compare the proposed method with the existing method [14], which is the first work on OV-3DDet, as well as [R1], which explores both open-vocabulary recognition and open-vocabulary localization. This would provide a clear assessment of the effectiveness of the proposed approach. Additionally, it is mentioned in line 150 and Figure 2 that the proposed method utilizes both images and point clouds, making an incorrect comparison with [22] in terms of inputs.\n\n+ Second, in Table 1, the detailed settings of the compared methods, such as distillation and 3D-NOD + Distillation, are unclear. It is essential to clarify how super category annotations for novel objects are obtained without 3D-NOD. Furthermore, it is unclear whether the compared methods are trained with the same number of epochs, which can significantly impact the performance comparison.\n\n+ Third, all experiments in the main paper are conducted solely on the SUN RGB-D dataset, it would be valuable to include evaluation results on the ScanNetV2 dataset,  which contains much more categories and is a more practical setting for open-vocabulary learning. Moreover, it is mentioned in line 227-228 that the remaining categories are treated as novel on the both datasets, indicating the presence of 190 novel categories on ScanNetV2. However, Table B in the appendix only considers 50 categories on ScanNetV2 as novel categories, creating a discrepancy.\n\n+ Lastly, according to Table 3, the performance seem to be sensitive to the two hyperparameters. It would be beneficial to provide more insights and analysis regarding the sensitivity of these hyperparameters.\n\n2. The paper's writing style tends to be wordy, leading to repeated chunks of information in the introduction, figure captions, and methodology sections. Streamlining the text and avoiding redundancy would enhance the overall readability and clarity of the paper.\n\n[R1] Zeng, Yihan, et al. \"CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 Comparison with existing method [15] and [R1]:**\n\nRegarding the comparison with [15], please refer to Tab. B of the rebuttal PDF. As for the comparison with [R1] (i.e., CLIP$^2$), we conducted comparison experiments on SUN-RGBD. More specifically, we utilize 3DETR to generate pseudo boxes and employ CLIP$^2$ to conduct open-vocabulary object detection, which is denoted as 'Det-CLIP$^2$'. As shown in Tab. F in the rebuttal PDF and Tab. 4 in the main paper, with pure point clouds as the testing input, Det-CLIP$^2$ [R1] achieves better AP than Det-PointCLIP [30] and Det-PointCLIPv2 [35], while our method still significantly outperforms them.\n\n**Q2 The comparison with [22] in terms of inputs:**\n\nThe inputs we discussed here refer to the testing phase, instead of the training phase. During testing, the method [22] needs 2D images as inputs to perform open-vocabulary classification for the detected objects. While our method only utilizes pure point clouds as testing inputs. This is made possible because DCMA aligns the 3D features with 2D image and text features during training. Thus, the comparison with [22] in terms of inputs is correct. We will update a clearer explanation.\n\n**Q3 How super category annotations for novel objects are obtained without 3D-NOD:**\n\nAs discussed in L177-L187 of the main paper, class-agnostic distillation eliminates the requirement of category labels of GT boxes. Thus, the training for our 'Distillation' does not need category annotations for novel objects. We will further clarify this point in the revision.\n\n**Q4 Whether the compared methods are trained with the same number of epochs:**\n\nTo ensure a fair comparison, they are trained using the same number of epochs. We will clearly describe this setting in the revision. We will also release all the codes and pre-trained models once the paper is accepted.\n\n**Q5 It would be valuable to include evaluation results on the ScanNet:**\n\nWe have included comparisons with alternative methods on the ScanNet, as presented in Tab. B of the supplementary material. Additionally, in this rebuttal, we compared our methods on ScanNet with the latest OV-3DDet work [15] following their data and experimental setups, and the results are shown in Tab. B of the rebuttal PDF. We achieve better performances (1.3 points mAP improvement) compared to [15].\nRegarding the ablation studies, we performed them on the SUN-RGBD dataset. This choice was made because SUN-RGBD also includes 200+ categories in the released official human annotation, and the training time on SUN-RGBD is shorter compared to ScanNet due to the smaller amount of training data. To further demonstrate that the conclusions from the ablation studies are consistent between SUN-RGBD and ScanNet, we also conducted the ablation study in Tab. 1 of the main paper on ScanNet in this rebuttal. The results are shown in Tab. G of the rebuttal PDF. Both Tab. G and Tab. 1 in the main paper indicate that our 3D-NOD and DCMA methods significantly improve performance over the baseline, further confirming the effectiveness of the proposed method.\n\nThanks for your suggestion. We will merge these results above into the updated version.\n\n**Q6 Unclear statement for the setting in ScanNet:**\n\nSorry for the unclear statement and thanks for pointing this out. To clarify, in the settings of ScanNet, the methods are evaluated on 60 categories (i.e., 10 base + 50 novel). Regarding the ablation study on the larger vocabulary (i.e., 200 categories), as shown in Fig. A in the Supplementary, we conducted it on the SUN-RGBD dataset as the other ablation study. We will make careful revisions.\n\n**Q7 Sensitivity to the two hyperparameters:**\n\nIn Tab. 3 of our main paper, we conducted an ablation study on the thresholds for the 2D semantic and 3D geometry priors in 3D-NOD. The entry '0.0&0.0' in the first row represents that the model is trained without 3D-NOD, which is our base model. When incorporating 3D-NOD with different thresholds for semantic and geometric information, all the trained models with different thresholds can consistently outperform '0.0&0.0' by a significant margin of 70% or more. This demonstrates that the effect of 3D-NOD is not sensitive to these hyperparameters.  Interestingly, the '0.3&0.3' threshold achieved the highest $AP_{Novel}$. This is because a lower threshold in 3D-NOD leads to the discovery of more novel objects, aligning with our initial intuition.\n\n**Q8 Writing style:**\n\nWe will carefully polish the paper in the revision.\n\n**Q9 Clarification for the threshold:**\n\nApologies for this typo and thanks for pointing this out. In Eq.4 of our main paper, the condition should be IoU < 0.25. This ensures that the discovered boxes do not overlap too much with the base objects, thereby guaranteeing as much as possible that they represent novel objects. We will revise accordingly for this in the updated version.\n\n**Q10 If only $O^{novel}$ is used in computing the alignment loss:**\n\nBoth the base and novel objects contribute to the cross-model alignment in our method. The symbol $O^{novel}$ in Eq. 8 should be $O^{label}$. We will update it in the revision.\n\n**Q11 Unclear description of the additional 32 object queries:**\n\nPlease refer to the answer for Q6 of Reviewer Qxx5.\n\n**Q12 Limitation:**\n\nPlease refer to the answer for Q6 of Reviewer zhog.\n\n\n[R1] Zeng, Yihan, et al. \"CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n"}]

Author rebuttal:
We thank all the reviewers for their detailly suggestions and encouraging comments: **"The idea makes sense"** and **"is well-motivated"** (Reviewer zhog, Reviewer goao). **"Discard the reliance on any additional 2D detection model"** (Reviewer Qxx5, Reviewer zhog). Bring **"superior performance improvements"** (Reviewer DwKh, Reviewer zhog). The paper is **"well-written"** and **"easy to follow"** (Reviewer DwKh, Reviewer Qxx5).

To make the discussion clearer, we will answer all the questions one by one for each reviewer, respectively. Note that all the tables and figures we prepared for this rebuttal are in **the PDF for the rebuttal**. Please have a check.



In this global response, we claim the potential negative societal impact, which will be updated in the new version.

**Potential negative societal impact:**


In our work, we train our methods only on public datasets [4, 24], ensuring that no private data is utilized. After releasing the codes and pre-trained models, some people may possibly consider using our method for large-scale open-world training involving personal data, causing the data privacy issue. We will explicitly cover this issue when we release the project and restrict any utilization of our project that may cause a potential negative societal impact.

Meta-review: The paper initially receives mixed ratings (two positive and two negative reviews). After the rebuttal and discussion phases, two negative reviews are upgraded to borderline acceptance. The AC took a close look at the paper, reviews, and rebuttals.

The tackled problem is of great interest and the authors have demonstrated a technically sound framework, especially the proposed way to discover and localize novel 3D objects via alignment between 2D and 3D spaces. Results are convincing and promising with extensive experiments. Overall, the AC finds that all the concerns are addressed well in the rebuttal and response, especially for questions from reviewer DwKh and goao who provided negative ratings initially. Therefore, the AC recommends the "acceptance" rating.
Decision: Accept (poster)