SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yang Cao $^{1}$  Yihan Zeng $^{2}$  Hang Xu $^{2}$  Dan Xu $^{1*}$  Hong Kong University of Science and Technology $^{1}$  Huawei Noah's Ark Lab $^{2}$
Open-vocabulary 3D Object Detection (OV-3DDet) aims to detect objects from an arbitrary list of categories within a 3D scene, which remains seldom explored in the literature. There are primarily two fundamental problems in OV-3DDet, i.e., localizing and classifying novel objects. This paper aims at addressing the two problems simultaneously via a unified framework, under the condition of limited base categories. To localize novel 3D objects, we propose an effective 3D Novel Object Discovery strategy, which utilizes both the 3D box geometry priors and 2D semantic open-vocabulary priors to generate pseudo box labels of the novel objects. To classify novel object boxes, we further develop a cross-modal alignment module based on discovered novel boxes, to align feature spaces between 3D point cloud and image/text modalities. Specifically, the alignment process contains a class-agnostic and a class-discriminative alignment, incorporating not only the base objects with annotations but also the increasingly discovered novel objects, resulting in an iteratively enhanced alignment. The novel box discovery and cross-modal alignment are jointly learned to collaboratively benefit each other. The novel object discovery can directly impact the cross-modal alignment, while a better feature alignment can, in turn, boost the localization capability, leading to a unified OV-3DDet framework, named CoDA, for simultaneous novel object localization and classification. Extensive experiments on two challenging datasets (i.e., SUN-RGBD and ScanNet) demonstrate the effectiveness of our method and also show a significant mAP improvement upon the best-performing alternative method by  $80\%$ . Codes and pre-trained models are released on the project page<sup>1</sup>.
3D object detection (3D-Det) [19, 21, 20] is a fundamental task in computer vision, with wide applications in self-driving, industrial manufacturing, and robotics. Given point cloud scenes, 3D-Det aims to localize and classify objects in the 3D scenes. However, most existing works heavily rely on datasets with fixed and known categories, which greatly limits their potential applications in the real world due to the various and unbounded categories. Open-vocabulary 3D object detection (OV-3DDet) is on the verge of appearing to detect novel objects. In OV-3DDet, the model is typically trained on datasets with limited base categories while evaluated on scenes full of novel objects, which brings additional challenges. In the problem of OV-3DDet, the key challenge is how to localize and classify novel objects using only annotations of very limited base categories. There are only very few works in the literature targeting this challenging problem so far.
*Corresponding author
<sup>1</sup> https://yangcaoai.github.io/publications/CoDA.html
To obtain novel object boxes, [15] utilizes a large-scale pre-trained 2D open-vocabulary object detection (OV-2DDet) model [35] as prior knowledge to localize a large number of 2D novel object boxes, and then generates pseudo 3D object box labels for corresponding 3D novel objects. Instead of directly relying on an external OV-2DDet model to obtain novel object localization capability, we aim to learn a discovery of novel 3D object boxes, based on limited 3D annotations of base categories on the target data. With the discovered novel object boxes, we further explore joint class-agnostic and class-specific cross-modal alignment, using the human-annotated base object boxes, and the discovered novel object boxes. The novel object box discovery and cross-modal alignment are collaboratively learned to achieve simultaneous novel object localization and classification.
Specifically, to localize novel objects, we first propose a 3D Novel Object Discovery (3D-NOD) training strategy that utilizes information from both the 3D and 2D domains to discover more novel objects in the training process. In detail, the model learns class-agnostic 3D box prediction using 3D box annotations of the base categories, which can provide 3D geometry priors of 3D boxes based on geometric point cloud features. To confirm discovered novel boxes, the pretrained vision-language model CLIP [22] is also utilized to provide 2D semantic category priors. By considering both 3D and 2D cues, novel objects can be effectively discovered during iterative training.
Building upon 3D-NOD, to achieve novel object classification, we further propose a Discovery-Driven Cross-Modal Alignment (DCMA) module that aligns 3D object features with 2D object and text features in a large vocabulary space, using both human-annotated 3D object boxes of base categories, and the discovered 3D boxes of novel categories. DCMA comprises two major components, i.e., category-agnostic box-wise feature distillation and class-specific feature alignment. The former aligns 3D point cloud features and 2D CLIP image features at the box level, without relying on information about specific categories. The distillation pushes the 3D point cloud and 2D image features closer wherever a 3D box covers. On the other hand, DCMA aligns 3D object features with CLIP text features in a large category vocabulary space, using category information from both annotated base objects and discovered novel objects, in a cross-modal contrastive learning manner.
The proposed 3D-NOD and DCMA can be jointly optimized in a unified deep framework named CoDA. They collaboratively learn to benefit each other. The discovered novel object boxes through 3D-NOD can greatly facilitate the cross-modal feature alignment, while better feature alignment through DCMA can further boost the model localization capability on novel objects. With our joint training of 3D-NOD and DCMA in CoDA to simultaneously localize and classify novel objects, our approach outperforms the best-performing alternative method by  $80\%$  in terms of mAP.
In summary, the main contribution of the paper is three-fold:
- We propose an end-to-end open vocabulary 3D object detection framework named CoDA, which can learn to simultaneously localize and classify novel objects, without requiring any prior information from external 2D open-vocabulary detection models.
- We design an effective 3D Novel Object Discovery (3D-NOD) strategy that can localize novel objects by jointly utilizing 3D geometry and 2D semantic open-vocabulary priors. Based on discovered novel objects, we further introduce discovery-driven cross-modal alignment (DCMA) to effectively align 3D point cloud features with 2D image/text features in a large vocabulary space.
- We design a mechanism to collaboratively learn the 3D-NOD and DCMA to benefit each other. The novel object boxes discovered from 3D-NOD are used in cross-modal feature alignment in DCMA, and a better feature alignment in DCMA can facilitate novel object discovery.
3D Object Detection. 3D object detection (3D-Det) [21, 25, 32, 16, 33, 26, 3] has achieved great success in recent years. For instance, VoteNet [21] introduces a point voting strategy to 3D object detection. It adopts PointNet [20] to process 3D points and then assigns these points to several groups, and object features are extracted from these point groups. Then the predictions are generated from these object features. Based on that, MLCVNet [25] designs new modules for point voting to capture multi-level contextual information. Recently, the emergence of transformer-based methods [2] has also driven the development of 3D-Det. For example, GroupFree [14] utilizes a transformer as the prediction head to avoid handcrafted grouping. 3DETR [19] proposes the first end-to-end transformer-based structure for 3D object detection. However, these methods mainly focus on the
Figure 1: Overview of the proposed open-vocabulary 3D object detection framework named CoDA. We consider 3DETR [19] as our base 3D object detection framework, represented by the 'Encoder' and 'Decoder' networks. The object queries, together with encoded point cloud features, are input into the decoder. The updated object query features from the decoder are further input into 3D object classification and localization heads. We first propose a 3D Novel Object Discovery (3D-NOD) strategy, which utilizes both 3D geometry priors from predicted 3D boxes and 2D semantic priors from the CLIP model to discover novel objects during training. The discovered novel object boxes are maintained in a novel object box label pool, which is further utilized in our proposed discovery-driven cross-modal alignment (DCMA). The DCMA consists of a class-agnostic distillation and a class-specific contrastive alignment based on discovered novel boxes. Both 3D-NOD and DCMA collaboratively learn to benefit each other to achieve simultaneous novel object localization and classification in an end-to-end manner.
close-vocabulary setting, i.e., the object categories in training and testing are the same and fixed. In contrast to these works, this paper targets open-vocabulary 3D-Det, and we design an end-to-end deep pipeline called CoDA, which performs joint novel object discovery and classification.
Open-vocabulary 2D object detection. In open-vocabulary 2D object detection (OV-2DDet), some works [7, 8, 10, 22, 11, 27, 18, 6, 17, 35, 29, 28] push 2D object detection to the open-vocabulary level. Specifically, RegionCLIP [34] pushes the CLIP from the image level to the regional level and learns region-text feature alignment. Detic [35] instead utilizes image-level supervision based on ImageNet [5], which assigns image labels to object proposals. GLIP [13] and MDETR [12] treat detection as the grounding task and adopt text query for the input image to predict corresponding boxes. All the methods mentioned above focus on 2D object detection and conduct the object detection from the 2D input image. However, in this work, we focus on open-world 3D object detection, which is a challenging problem and remains seldom explored in the literature.
Open-vocabulary 3D object detection. Recently, an open-set 3D-Det method [1] has shown great potential in various practical application scenarios. It can successfully detect unlabeled objects by introducing ImageNet1K [5] to train a 2D object detection model [2], which classifies objects generated by a 3D-Det model [19] on the training set to obtain pseudo labels for the objects. Then the pseudo labels are adopted to retrain the open-set 3D-Det model [19]. Even though there are no labels for the novel objects during training, the names of novel categories are known and fixed so that the category-specific pseudo labels can be generated, which is thus an open-set while close-vocabulary model. More recently, [15] propose a deep method to tackle OV-3Ddet. They adopt the CLIP model [22] and a large-scale pretrained external OV-2Ddet [35] model to generate pseudo labels of possible novel objects. Our work is closer to [15], but we target OV-3Ddet without relying on any additional OV-2Ddet model as priors to obtain novel box localization capabilities. Our model CoDA can simultaneously learn to conduct novel object discovery and cross-modal alignment with a collaborative learning manner, to perform end-to-end novel object localization and classification.
3.1 Framework Overview
An overview of our proposed open-world 3D object detection framework called CoDA is shown in Fig. 1. Our method utilizes the transformer-based point cloud 3D object detection model
3DETR [19] as the detection backbone, which is represented as an 'Encoder' and a 'Decoder' shown in Fig. 1. It has localization and a classification head for 3D object box regression and box category identification, respectively. To introduce open-vocabulary detection capabilities to the 3D point cloud object detection task, we utilize the pretrained vision-language model CLIP [22] to bring us rich open-world knowledge. The CLIP model consists of both an image encoder (i.e., CLIP-Image) and a text encoder (i.e., CLIP-Text) in Fig. 1. To localize novel objects, we first propose the 3D Novel Object Discovery (3D-NOD) strategy, which jointly utilizes both 3D box geometry priors from base categories and 2D semantic priors from the CLIP model to identify novel objects. To classify novel objects, we further propose a novel discovery-driven cross-modal alignment (DCMA) module, which aligns feature spaces between 3D point clouds and image/text modalities, using the discovered novel object boxes as guidance. The 3D-NOD and DCMA are jointly optimized to collaboratively learn to benefit each other in the unified deep framework.
3.2 3D Novel Object Discovery (3D-NOD) with Cross-Priors
As shown in Fig. 1, our 3D Novel Object Discovery (3D-NOD) strategy allows for the discovery of novel objects during training. To achieve this goal, we utilize cross-priors from both 3D and 2D domains. In terms of the 3D domain, we use 3D box geometry prior provided by the 3D object annotations of base categories, to train a class-agnostic box predictor. Regarding the 2D domain, we utilize the semantic priors from the CLIP model, to predict the probability of a 3D object belonging to a novel category. We combine these two perspectives to localize novel object boxes.
Discovery based on 3D Geometry Priors. We begin with an initial 3D object box label pool  $\mathbf{O}_0^{base}$  for objects of base categories (i.e., seen categories) in the training set:
$$
\mathbf {O} _ {0} ^ {\text {b a s e}} = \left\{o _ {j} = \left(l _ {j}, c _ {j}\right) \mid c _ {j} \in \mathbb {C} ^ {\text {S e e n}} \right\}, \tag {1}
$$
where  $\mathbb{C}^{Seen}$  represents the base category set of the training data, which has human-annotated object labels. Based on  $\mathbf{O}_0^{base}$ , we can train an initial 3D object detection model  $\mathbf{W}_0^{det}$  by minimizing an object box regression loss function derived from 3DETR [19]. We consider a class-agnostic object detection for  $W_0$ , meaning that we train the model with class-agnostic box regression and binary objectness prediction, and do not utilize the class-specific classification loss. This is because the class-specific training with strong supervision from base categories hampers the model's capabilities of discovering objects from novel categories, which has also been observed from previous OV-2DDet frameworks [8]. Thus our modified backbone outputs object boxes, including the parameters for the box 3D localization, and their objectness probabilities. For the  $n$ -th object query embedding in 3DETR, we predict its objectness probability  $p_n^g$ , based on  $\mathbf{W}_0^{det}$  which is learned using 3D geometry priors from human-annotated 3D object boxes on base categories.
Discovery based on 2D CLIP Semantic Priors. Then, we also project the 3D object box  $l_{n}^{3D}$  in the point cloud to the 2D box  $l_{n}^{2D}$  on the color image with the corresponding camera intrinsic matrix  $M$  as follows:
$$
l _ {n} ^ {2 D} = M \times l _ {n} ^ {3 D}, \tag {2}
$$
We then obtain the 2D object region  $I_{n}^{2D}$  in the color image by cropping it with  $l_{n}^{2D}$ . Next, we obtain the 2D object feature  $\mathbf{F}_{I,n}^{Obj}$  by passing the object region into the CLIP image encoder  $\mathbf{E}_I^{\mathrm{CLIP}}$ . As the testing categories are unknown during training, we adopt a super category list  $T^{super}$  following [9] to provide text semantic descriptions of rich object categories. We encode  $T^{super}$  with the CLIP text encoder  $\mathbf{E}_T^{\mathrm{CLIP}}$  to obtain the text embedding  $\mathbf{F}_T^{Super}$ . We then estimate the semantic probability distribution (i.e.,  $\mathbf{P}_n^{3DObj}$ ) of the 3D object using its corresponding 2D object feature as:
$$
\mathbf {P} _ {n} ^ {3 d O b j} = \left\{p _ {n, i} ^ {s}, p _ {n, 2} ^ {s}, p _ {n, 3} ^ {s}, \dots , p _ {n, C} ^ {s} \right\} = \operatorname {S o f t m a x} \left(\mathbf {F} _ {I, n} ^ {O b j} \cdot \mathbf {F} _ {T} ^ {S u p e r}\right), \tag {3}
$$
where  $C$  is the number of categories defined in the super category set  $T^{super}$ . The symbol  $\cdot$  indicates the dot product operation.  $\mathbf{P}^{3dObj}$  provides semantic priors from the CLIP model. We can obtain the object category  $c^*$  for the object query by finding the maximum probability in  $\mathbf{P}^{3dObj}$  as  $c^* = \operatorname{argmax}_c \mathbf{P}^{3DObj}$ . We then combine both the objectness  $p_n^g$  from the 3D geometry priors and  $p_{n,c^*}^s$  from the 2D semantic priors, and discover novel objects for the  $t$ -th training epoch as follows:
$$
\mathbf {O} _ {t} ^ {\text {d i s c}} = \left\{o _ {j} \mid \forall o _ {i} ^ {\prime} \in \mathbf {O} _ {0} ^ {\text {b a s e}}, \operatorname {I o U} _ {3 D} \left(o _ {j}, o _ {i} ^ {\prime}\right) <   0. 2 5, p _ {n} ^ {g} > \theta^ {g}, p _ {n, c ^ {*}} ^ {s} > \theta^ {s}, c ^ {*} \notin \mathbb {C} ^ {\text {S e e n}} \right\}, \tag {4}
$$
Figure 2: Illustration of the proposed discovery-driven cross-modal alignment (DCMA) in our OV detection framework (CoDA). The DCMA consists of two parts, i.e., the class-agnostic distillation and the class-specific contrastive feature alignment. The predicted 3D boxes from the detection head are projected to obtain 2D image object regions, which are passed into the CLIP image encoder to generate 2D object features. Then, the CLIP 2D object features and 3D point cloud object features are fed into the class-agnostic distillation module for feature alignment. Driven by discovered novel object boxes from the 3D-NOD strategy, the maintained novel box label pool that is updated during training can be used to match with the predicted 3D object boxes. We perform contrastive alignment for matched novel boxes to learn more discriminative 3D object features for novel objects. More discriminative 3D object features can in turn facilitate the prediction of novel object boxes.
where  $\mathrm{IoU}_{3D}$  means the calculation of the 3D IoU of two boxes;  $\theta^s$  and  $\theta^g$  are two thresholds for the semantic and the geometry prior, respectively;  $\mathbb{C}^{seen}$  is the set of seen categories of the training data. Then, the discovered novel objects  $O_{t}^{disc}$  are added to a maintained 3D box label pool  $\mathbf{O}_t^{novel}$  as:
$$
\mathbf {O} _ {t + 1} ^ {\text {n o v e l}} = \mathbf {O} _ {t} ^ {\text {n o v e l}} \cup \mathbf {O} _ {t} ^ {\text {d i s c}}, \tag {5}
$$
The model is trained to iteratively update the novel object box label pool. By gradually expanding the box label pool with the novel objects  $O_{t}^{novel}$ , we can broaden the model's capabilities to localize novel objects by using supervision from  $O_{t}^{novel}$  for the model  $W_{t}$ . With the semantic priors from CLIP and the 3D geometry prior from  $W_{t}$ , the model and the 3D object box label pool can discover more and more novel objects following the proposed object discovery training strategy.
3.3 Discovery-Driven Cross-Modal Alignment (DCMA)
To bring open-vocabulary capabilities from the CLIP model, we propose a discovery-driven cross-modal alignment (DCMA) module, to align the 3D point cloud object features with 2D-image/CLIP-text features, based on the discovered novel object boxes  $\mathbf{O}^{novel}$ . For the proposed DCMA, we utilize the discovered novel object boxes to conduct two types of cross-modal feature alignment, i.e., class-agnostic distillation and class-specific contrastive alignment.
Object-Box Class-agnostic Distillation. We introduce a class-agnostic distillation strategy to align the 3D point cloud object features and 2D image object features, to transfer more open-world knowledge from the CLIP model. For the  $n$ -th object query, we can obtain its updated 3D feature  $\mathbf{F}_n^{3dObj}$  at the last layer of the decoder. Based on the 3D object feature, the model predicts its 3D box parameterization  $l_n^{3D}$ , and then we can obtain its corresponding 2D box  $l_n^{2D}$  by Eqn. (2). Now we have 2D-projected object features  $\mathbf{F}_n^{2DObj}$  by inputting CLIP with the cropped 2D region based on  $l^{2D}$ . To push the feature spaces of both the 3D and 2D objects as close as possible, if a 3D object box is predicted from the 3D object feature, we align them by using a class-agnostic L1 distillation loss,
which is defined as follows:
$$
\mathcal {L} _ {\text {d i s t i l l}} ^ {3 D \leftrightarrow 2 D} = \sum_ {n = 1} ^ {N} \| \mathbf {F} _ {n} ^ {3 D O b j} - \mathbf {F} _ {n} ^ {2 D O b j} \|, \tag {6}
$$
where  $N$  is the number of object queries. As shown in Fig. 2, even if the box covers a background region, the class-agnostic distillation still narrows the distance between the two modalities, leading to a more general alignment on different modalities and scenes. As we can see, class-agnostic distillation eliminates the requirement of category labels of GT boxes. Thus, the computation of  $\mathcal{L}_{distill}^{3D\leftrightarrow 2D}$  does not need category annotations.
Discovery-driven class-specific contrastive alignment. As we discussed in Sec. 3.2, we modify the 3DETR detection backbone, by removing the fixed-category class-specific classification head. We can extract 3D point cloud object features from the updated object query embeddings. For the  $n$ -th object query embedding, we can obtain a corresponding 3D object feature  $\mathbf{F}_n^{3DObj}$  at the last layer of the decoder. With the 3D object feature  $\mathbf{F}_n^{3DObj}$ , we expect the feature to be discriminative for both base categories and novel categories defined in the superset  $T^{\mathrm{super}}$ . We, therefore, perform class-specific alignment with the CLIP text features. Specifically, we calculate the normalized similarities using the dot product operation between  $\mathbf{F}_n^{3DObj}$  and the text features  $\mathbf{F}_T^{Super}$  as follows:
$$
\mathbf {S} _ {n} = \operatorname {S o f t m a x} \left(\mathbf {F} _ {n} ^ {3 D O b j} \cdot \mathbf {F} _ {T} ^ {S u p e r}\right), \tag {7}
$$
Then, we introduce the point cloud and text contrastive loss to provide the supervision signal for learning the 3D object features of the object queries. We have already measured the similarity between the 3D object point cloud feature and the text features. To construct a cross-modal contrastive loss, we need to identify the most similar pair between these two modalities as a ground-truth label. We adopt the Bipartite Matching from [19] to match the predicted 3D object boxes with the discovered novel boxes in the box pool  $\mathbf{O}^{label}$ . This is to guarantee that the class-specific alignment is conducted on foreground novel objects, instead of on non-object background boxes, as the object query embeddings also lead to the prediction of noisy 3D object boxes because of the limitation of supervision on novel objects during learning. If we perform class-specific alignment on noisy object boxes, it makes the 3D object feature less discriminative. When we have a matching from the box label pool  $\mathbf{O}^{label}$ , we then obtain the corresponding category label from the CLIP model by finding the maximum similarity score and construct a one-hot label vector  $\mathbf{h}_n$  for the 3D object feature  $\mathbf{F}_n^{3DObj}$  of the  $n$ -th object query. A detailed illustration can be seen in Fig. 2. By utilizing the discovered foreground novel object boxes, we can construct a discovery-driven cross-modal contrastive loss function as follows:
$$
\mathcal {L} _ {\text {c o n t r a}} ^ {3 D \leftrightarrow T e x t} = \sum_ {n = 1} ^ {N} \mathbf {1} \left(\mathbf {F} _ {n} ^ {3 D O b j}, \mathbf {O} ^ {\text {l a b e l}}\right) \cdot \operatorname {C E} \left(\mathbf {S} _ {n}, \mathbf {h} _ {n}\right), \tag {8}
$$