SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $N$  is the number of object queries;  $\mathbf{1}(\cdot)$  is an indicator function that returns 1 if a box matching is identified between the query and the box label pool else 0;  $\mathrm{CE}(\cdot)$  is a cross-entropy loss function. In this way, the object queries that are associated with novel objects are involved in the cross-modal alignment, leading to better feature alignment learning. The 3D features of novel objects can learn to be more discriminative, further resulting in more effective novel object discovery. In this way, the 3D novel object discovery and cross-modality feature alignment jointly boost the localization and classification for open-vocabulary 3D object detection.
Datasets and Settings. We evaluate our proposed approach on two challenging 3D indoor detection datasets, i.e., SUN-RGBD [24] and ScanNetV2 [4]. SUN-RGBD has 5,000 training samples and oriented 3D bounding box labels, including 46 object categories, each with more than 100 training samples. ScanNetV2 has 1,200 training samples containing 200 object categories [23]. Regarding the training and testing settings, we follow the strategy commonly considered in 2D open-vocabulary object detection works [8, 29]. It divides the object categories into seen and novel categories based on the number of samples of each category. Specifically, for SUN-RGBD, the categories with the top-10 most training samples are regarded as seen categories, while the other 36 categories are treated as novel categories. For ScanNetV2, the categories with the top-10 most training samples are regarded
as seen categories, other 50 categories are taken as novel categories. Regarding the evaluation metrics, we report the performance on the validation set using mean Average Precision (mAP) at an IoU threshold of 0.25, as described in [19], denoted as  $\mathrm{AP}_{25}$ .
Implementation Details. The number of object queries for both SUN-RGBD and ScanNetv2 is set to 128, and the batch size is set to 8. We trained a base 3DETR model for 1080 epochs using only class-agnostic distillation as a base model. Subsequently, we train the base model for an additional 200 epochs with the proposed 3D novel object discovery (3D-NOD) strategy and discovery-driven cross-modal feature alignment (DCMA). Note that, to ensure a fair comparison, all the compared methods are trained using the same number of epochs. For 3D-NOD, the label pool is updated every 50 epochs. For feature alignment, considering that the discovered and GT boxes in a scene are limited, and a larger number of object queries can effectively facilitate contrastive learning, aside from the queries that match GT boxes, we choose an additional 32 object queries from the 128 queries, which are involved in the alignment with the CLIP model. The hyper-parameter settings used in the training process follow the default 3DETR configuration as described in [19].
4.2 Model Analysis
In this section, we provide a detailed analysis of our open-vocabulary detection framework and verify each individual contribution. The different models are evaluated on three different subsets of categories, i.e., novel categories, base categories, and all the categories. These subsets are denoted as  $\mathrm{AP}_{\text{Novel}}$ ,  $\mathrm{AP}_{\text{Base}}$ , and  $\mathrm{AP}_{\text{Mean}}$ , respectively. All the evaluation metrics, including mean Average Precision (mAP) and recall (AR), are reported at an IoU threshold of 0.25. The ablation study experiments are extensively conducted on the SUN-RGBD dataset.
Methods	APNovel	APBase	APMean	ARNovel	ARBase	ARMean
3DETR + CLIP	3.61	30.56	9.47	21.47	63.74	30.66
Distillation	3.28	33.00	9.74	19.87	64.31	29.53
3D-NOD + Distillation	5.48	32.07	11.26	33.45	66.03	40.53
3D-NOD + Distillation & PlainA	0.85	35.23	8.32	34.44	66.11	41.33
3D-NOD + DCMA (full)	6.71	38.72	13.66	33.66	66.42	40.78
Table 1: Ablation study to verify the effectiveness of the proposed 3D novel object discovery (3D-NOD) and discovery-driven cross-modality alignment (DCMA) in our 3D OV detection framework CoDA.
Effect of class-agnostic distillation. To demonstrate the effectiveness of class-agnostic distillation in discovery-driven alignment, we first train a class-agnostic 3DETR detection model that provides basic 3D localization for novel objects. The 3D bounding boxes are then projected onto 2D color images to crop the corresponding 2D object regions. By feeding the object regions into the CLIP model, we can conduct open-vocabulary classification to create an open-vocabulary 3D detection framework that requires both point cloud and image inputs. This baseline method is denoted as '3DETR+CLIP' in Tab. 1. We also train a 3DETR model with only the class-agnostic distillation, which serves as our base model and is denoted as 'Distillation' in Tab. 1. As shown in the table, when tested with only pure point cloud as input, the 'Distillation' model achieves comparable results to the '3DETR+CLIP' model in terms of  $\mathrm{AP}_{\text {Novel }}$  (3.28 vs. 3.61), indicating that class-agnostic distillation provides preliminary open-vocabulary detection ability for 3DETR. Moreover, given the fact that the class-agnostic distillation pushes the 3D point cloud object features and 2D image features closer wherever a 3D bounding box is predicted, and the base categories have more training object
Figure 3: The influence of 3D Novel Object Discovery (3D-NOD) during training. The blue curve in the graph represents  $\mathrm{AP}_{\text {novel }}$ , while the orange curve represents  $\mathrm{AR}_{\text {novel }}$ . The left side of the black dashed line corresponds to the training process of our base model 'Distillation', which is trained for 1080 epochs. After that, we apply our 3D-NOD on the base model and continue to train it for an additional 200 epochs, which is represented by the right side of the black dashed line. By applying our 3D-NOD approach, we observe a significant increase in both  $\mathrm{AP}_{\text {novel }}$  and  $\mathrm{AR}_{\text {novel }}$ , which breaks the limitation of base category annotations.
samples, the distillation on the base categories is more effective, leading to a higher performance in terms of  $\mathrm{AP}_{\text {Base }}$  (33.00 vs. 30.56).
Effect of 3D Novel Object Discovery (3D-NOD). Building upon the base model 'Distillation', we apply our 3D novel object discovery (3D-NOD) strategy to enhance the open-vocabulary detection capability, denoted as '3D-NOD+Distillation' in Tab. 1. As can be seen in the table, '3D-NOD+Distillation' achieves a higher recall than both 'Distillation' and '3DETR+CLIP' (33.45 vs. 19.87/21.47). This demonstrates that our 3D-NOD can discover more novel objects during training, leading to a higher  $\mathrm{AP}_{\text {Novel }}$  (5.48 vs. 3.28/3.61). To further analyze the influence of 3D novel object discovery during training, we monitor the entire training process and regularly evaluate the model at intermediate checkpoints. As shown in Fig. 3, the blue curve represents  $\mathrm{AP}_{\text {novel }}$  while the orange curve represents  $\mathrm{AR}_{\text {novel }}$ . The left side of the black dashed line corresponds to the training process of our base model 'Distillation', which is trained for 1080 epochs. After that, we apply our 3D-NOD strategy on the base model and continue to train it for an additional 200 epochs, represented by the right side of the black dashed line. We observe that limited by the base category labels, the  $\mathrm{AP}_{\text {novel }}$  of the base model nearly stops growing at 1000 epochs. Moreover,  $\mathrm{AR}_{\text {novel }}$  begins to decrease as training continues, which meets the category forgetting phenomenon due to the lack of annotations for novel categories. However, by applying our 3D-NOD approach, we observe a significant increase in both  $\mathrm{AP}_{\text {novel }}$  and  $\mathrm{AR}_{\text {novel }}$ , which demonstrates the effectiveness of our 3D-NOD approach.
Effect of Discovery-Driven Cross-Modal Alignment (DCMA). To explore the impact of different alignment methods on the performance of the '3D-NOD+Distillation' model. Firstly, we introduce the plain text-point alignment to the model, denoted as '3D-NOD+Distillation & PlainA'. The plain alignment involves aligning the category list of base category texts without using the discovered boxes by 3D-NOD. As shown in Tab. 1, the  $\mathrm{AP}_{\text{Base}}$  is higher than that of '3D-NOD+Distillation' (35.23 vs. 32.07), indicating that the base category texts help to improve the cross-modal feature alignment. Additionally, 3D-NOD still boosts the  $\mathrm{AR}_{\text{Novel}}$ , indicating that it can still facilitate the discovery of novel objects under this comparison. However, the  $\mathrm{AP}_{\text{Novel}}$  is lower at 0.85, which shows that the model loses its discriminative ability on novel categories if without utilizing the box discovery driven by 3D-NOD. In contrast, after applying our discovery-driven class-specific cross-modal alignment method, the '3D-NOD+DCMA' model in Tab. 1 achieves the best performance on both  $\mathrm{AP}_{\text{Novel}}$  and  $\mathrm{AP}_{\text{Base}}$ . This suggests that compared with the plain alignment, our discovery-driven alignment method brings more discriminative object features by using a larger category vocabulary driven by the 3D-NOD.
Effect of the collaborative learning of 3D-NOD and DCMA. As shown in Tab. 1, by comparing with the alignment method that is not driven by 3D-NOD (i.e., '3D-NOD + DCMA' vs. '3D-NOD + Distillation & PlainA'), our 3D-NOD method improves the cross-model alignment, resulting in better performance on the novel category classification. To investigate the effect of DCMA on the novel object discovery, we apply our 3D-NOD method to the '3DETR+CLIP' pipeline without utilizing DCMA, which is denoted as '3DETR+CLIP+3D-NOD' in Tab. 2. As shown in Tab. 2, '3D-NOD+DCMA (full)' outperforms '3DETR+CLIP+3D-NOD' in terms of AP and AR for both novel and base categories. This demonstrates that the proposed DCMA can effectively help to learn more discriminative feature representations of the object query embeddings, thus improving the localization capability. Overall, these results show that 3D-NOD and DCMA together can improve the detection performance by  $26\%$  on  $\mathrm{AP}_{\text {Novel }}$  and  $48\%$  on  $\mathrm{AP}_{\text {Base }}$ .
Methods	APNovel	APBase	APMean	ARNovel	ARBase	ARMean
3DETR + CLIP [22]	3.61	30.56	9.47	21.47	63.74	30.66
3DETR + CLIP + 3D-NOD	5.30	26.08	9.82	32.72	64.43	39.62
3D-NOD + DCMA (full)	6.71	38.72	13.66	33.66	66.42	40.78
Table 2: Ablation study on SUN-RGBD [24] to verify the effectiveness of the collaborative learning of DCMA and 3D-NOD. Compared with 3DETR+CLIP+3D-NOD, utilizing 3D-NOD and DCMA together can effectively improve the open-vocabulary detection performance.
The Sensitivities of Thresholds in 3D-NOD. In 3D novel object discovery, the utilization of the 2D semantic priors and 3D geometry priors requires two thresholds. We perform experiments to evaluate their sensitivities, and the results are presented in Tab. 3. The first row denoted by '0.0' represents the base model without our contributions. As we can see, our 3D-NOD method can produce stable
Semantic	Geometry	APNovel	APBase	APMean	ARNovel	ARBase	ARMean
0.0	0.0	3.28	33.00	9.74	19.87	64.31	29.53
0.3	0.3	6.71	38.72	13.66	33.66	66.42	40.78
0.3	0.5	6.35	39.57	13.57	31.93	66.91	39.53
0.5	0.3	5.70	38.61	12.85	27.05	63.61	35.00
0.5	0.5	5.70	39.25	13.00	28.27	65.04	36.26
Table 3: Ablation study on the thresholds used for the 2D semantic and 3D geometry priors in the 3D Novel Object Discovery (3D-NOD) strategy. Disabling these priors leads to significant performance drops (see the first row), and the model achieves the best AP performance with thresholds of 0.3.
improvements with different thresholds from a wide range, demonstrating that its effectiveness is not reliant on carefully selected thresholds. As shown in Tab. 3, despite the performance variability, all our models trained with different threshold settings can consistently outperform the model trained without 3D-NOD (i.e., '0.0&0.0' in the first row) by a significant margin of  $70\%$  or more, clearly verifying its advantage. Besides, the setting with a threshold of 0.3 yields the best performance as it enables the discovery of more novel objects.
4.3 Comparison with Alternatives
Quantitative Comparison. Open-world vocabulary 3D detection is still a very new problem. Since there are very few works in the literature and our open-vocabulary setting is novel, no existing results can be directly compared with our approach. Therefore, we adapt the latest open-vocabulary point cloud classification method to our setting and evaluate its performance. Specifically, we utilize 3DETR to generate pseudo boxes and employ PointCLIP [31], PointCLIPv2 [36] and Det-CLIP² [30] to conduct open-vocabulary object detection, which is denoted as 'Det-PointCLIP', 'Det-PointCLIPv2' and 'Det-CLIP²' in Tab. 4, respectively. Additionally, we introduce a pipeline with 2D color images as input, where we use 3DETR to generate pseudo 3D boxes. We then use the camera intrinsic matrix to project 3D boxes onto 2D boxes, which are adopted to crop corresponding 2D regions. Finally, the CLIP model is utilized to classify these regions. This pipeline enables us to generate 3D detection results, denoted as '3D-CLIP' in Tab. 4. As can be observed in the table, the  $\mathrm{AP}_{\text{Novel}}$  and  $\mathrm{AR}_{\text{Novel}}$  of our method are significantly higher than other methods, further demonstrating the superiority of our design in both novel object localization and classification.
Methods	Inputs	APNovel	APBase	APMean	ARNovel	ARBase	ARMean
SUN-RGBD
Det-PointCLIP [31]	Point Cloud	0.09	5.04	1.17	21.98	65.03	31.33
Det-PointCLIPv2 [36]	Point Cloud	0.12	4.82	1.14	21.33	63.74	30.55
Det-CLIP2 [30]	Point Cloud	0.88	22.74	5.63	22.21	65.04	31.52
3D-CLIP [22]	Image&Point Cloud	3.61	30.56	9.47	21.47	63.74	30.66
CoDA (Ours)	Point Cloud	6.71	38.72	13.66	33.66	66.42	40.78
ScanNetv2
Det-PointCLIP [31]	Point Cloud	0.13	2.38	0.50	33.38	54.88	36.96
Det-PointCLIPv2 [36]	Point Cloud	0.13	1.75	0.40	32.60	54.52	36.25
Det-CLIP2 [30]	Point Cloud	0.14	1.76	0.40	34.26	56.22	37.92
3D-CLIP [22]	Image&Point Cloud	3.74	14.14	5.47	32.15	54.15	35.81
CoDA (Ours)	Point Cloud	6.54	21.57	9.04	43.36	61.00	46.30
Table 4: Comparison with other alternative methods on the SUN-RGBD and ScanNetv2 dataset. The 'inputs' in the second column refer to the testing phase. During testing, '3D-CLIP' needs 2D images as inputs to perform open-vocabulary classification for the detected objects. While our method CoDA only utilizes pure point clouds as testing inputs.
In Tab. 4, we present a comparison of our method with alternative approaches on ScanNetv2 [4]. In this evaluation, we consider the categories with the top 10 most training samples in ScanNetv2 as base categories, while the other first 50 categories presented in [23] are considered novel categories. To obtain image and point cloud pairs, we follow the approach described in [15] to generate point cloud scenes from depth maps. As shown in Tab. 4, our method significantly outperforms other approaches on AP and AR metrics for both novel and base categories, even with only point clouds as inputs during testing.
Comparison with OV-3DET [15]. Note that the setting in our method is significantly different from OV-3DET [15]. In our setting, we train the model using annotations from a small number of base categories (10 categories) and learn to discover novel categories during training. We evaluate our model in a large number of categories (60 in ScanNet and 46 in SUN-RGBD). However, [15] relies on a large-scale pretrained 2D open vocabulary detection model, i.e., OV-2DDet model [35], to generate pseudo labels for novel categories, and evaluate the model on only 20 categories. Thus, because of the setting differences and the prior model utilized, a direct comparison with [15] is not straightforward. To provide a comparison with [15], thanks to the released codes from the authors of [15] to generate pseudo labels on ScanNet, we can directly retrain our method following the same setting of [15] on ScanNet, and provide a fair comparison with [15] on Tab. 5. As can be observed, our method achieves clearly better mean AP (1.3 points improvement over all the categories), validating the superiority of our method.
Methods	Mean	toilet	bed	chair	sofa	dresser	table	cabinet	bookshelf	pillow	sink
OV-3DET [15]	18.02	57.29	42.26	27.06	31.50	8.21	14.17	2.98	5.56	23.00	31.60
CoDA (Ours)	19.32	68.09	44.04	28.72	44.57	3.41	20.23	5.32	0.03	27.95	45.26
Methods	bathtub	refrigerator	desk	nightstand	counter	door	curtain	box	lamp	bag
OV-3DET [15]	56.28	10.99	19.72	0.77	0.31	9.59	10.53	3.78	2.11	2.71
CoDA (Ours)	50.51	6.55	12.42	15.15	0.68	7.95	0.01	2.94	0.51	2.02
Table 5: Comparison of methods in the same setting of [15] on ScanNet. 'Mean' represents the average value of all the 20 categories.
Qualitative Comparison. As depicted in Fig. 4, our method demonstrates the strong capability to discover more novel objects from point clouds. For instance, our method successfully identify the dresser in the first scene and the nightstand in the second scene. The performance comparison indicates that our method generally exhibits superior open-vocabulary detection ability on novel objects.
Figure 4: Qualitative comparison with 3D-CLIP [22]. Benefiting from our contributions, our method CoDA can discover more novel objects, which are indicated by blue boxes in the color images.
This paper proposes a unified framework named CoDA to address the fundamental challenges in OV-3DDet, which involve localizing and classifying novel objects. To achieve 3D novel object localization, we introduce the 3D Novel Object Discovery strategy (3D-NOD), which employs both 3D box geometry priors and 2D semantic open-vocabulary priors to discover more novel objects during training. To classify novel object boxes, we further design a Discovery-driven Cross-Modal Alignment module (DCMA) that includes both a class-agnostic and a class-discriminative alignment to align features of 3D, 2D and text modalities. Our method significantly outperforms alternative methods by more than  $80\%$  in mAP.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. PMLR, 2021.
[23] David Rozenberszki, Or Litany, and Angela Dai. Language-grounded indoor 3d semantic segmentation in the wild. In ECCV. Springer, 2022.
[24] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR, 2015.
[25] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, and Jun Wang. Mlcvnet: Multi-level context voteret for 3d object detection. In CVPR, 2020.
[26] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Dening Lu, Mingqiang Wei, and Jun Wang. Venet: Voting enhancement network for 3d object detection. In ICCV, 2021.
[27] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. In NeurIPS, 2022.
[28] Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open-vocabulary object detection pre-training via word-region alignment. In CVPR, 2023.
[29] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
[30] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu. Clip2: Contrastive language-image-point pretraining from real-world point cloud data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15244-15253, 2023.
[31] Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointclip: Point cloud understanding by clip. In CVPR, 2022.
[32] Yinmin Zhang, Xinzhu Ma, Shuai Yi, Jun Hou, Zhihui Wang, Wanli Ouyang, and Dan Xu. Learning geometry-guided depth via projective modeling for monocular 3d object detection. arXiv preprint arXiv:2107.13931, 2021.
[33] Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In ECCV. Springer, 2020.
[34] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In CVPR, 2022.
[35] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krahenbuhl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. arXiv preprint arXiv:2201.02605, 2022.
[36] Xiangyang Zhu, Renrui Zhang, Bowei He, Ziyao Zeng, Shanghang Zhang, and Peng Gao. Pointclip v2: Adapting clip for powerful 3d open-world learning. arXiv preprint arXiv:2211.11682, 2022.