Open-Set 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learning
Authors: Yuheng Lu
Chenfeng Xu
Xiaobao Wei
Xiaodong Xie
Masayoshi Tomizuka
Kurt Keutzer
Shanghang Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: open vocabulary
3d detection
contrastive learning

Abstract: Current point-cloud detection methods have difficulty detecting the open-set objects in the real world, due to their limited generalization capability. Moreover, it is extremely laborious and expensive to collect and fully annotate a point-cloud detection dataset with numerous classes of objects, leading to the limited classes of existing point-cloud datasets and hindering the model to learn general representations to achieve open-set point-cloud detection. Instead of seeking a point-cloud dataset with full labels, we resort to ImageNet1K to broaden the vocabulary of the point-cloud detector. We propose OS-3DETIC, an Open-Set 3D DETector using Image-level Class supervision. Specifically, we take advantage of two modalities, the image modality for recognition and the point-cloud modality for localization, to generate pseudo labels for unseen classes. Then we propose a novel debiased cross-modal cross-task contrastive learning method to transfer the knowledge from image modality to point-cloud modality during training. Without hurting the latency during inference, OS-3DETIC makes the well-known point-cloud detector capable of achieving open-set detection. Extensive experiments demonstrate that the proposed OS-3DETIC achieves at least 10.77 % mAP improvement (absolute value) and 9.56 % mAP improvement (absolute value) by a wide range of baselines on the SUN-RGBD dataset and ScanNet dataset, respectively. Besides, we conduct sufficient experiments to shed light on why the proposed OS-3DETIC works.

Under review as a conference paper at ICLR 2023
OPEN-SET 3D DETECTION VIA IMAGE-LEVEL CLASS AND DE-BIASED CROSS-MODAL CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Current point-cloud detectors have difficulty in detecting the open-set objects in the real world, due to their limited generalization capability. Moreover, collecting and fully annotating point-cloud detection dataset with large number of classes objects is extremely laborious and expensive, leading to the limited class size existing point-cloud datasets and hindering the model from learning general rep- resentations for open-set point-cloud detection. Instead of seeking well-annotated point-cloud datasets, resort to ImageNet1 K to broaden the vocabulary of point- cloud detectors. Specifically, we propose OS- 3DETIC, an Open-Set 3D DETector using Image-level Class supervision. Intuitively, we take advantage of two modal- ities, the image modality for recognition and the point-cloud modality for local- ization, to generate pseudo-labels for unseen classes. We then propose a novel de-biased cross-modal contrastive learning strategy to transfer knowledge from image to point-cloud. Without hurting the latency during inference, OS-3DETIC makes the well-known point-cloud detector capable of achieving open-set detec- tion. Extensive experiments demonstrate that the proposed OS- ---DETIC achieves at least 10.77% mAP improvement (absolute value) and 9.56% mAp improvement (absolute value) by a wide range baselines on the SUN-RGBD and ScanNerV2, respectively. Moreover, conduct sufficient experiments to shed light why the proposed OS-3DETIC works.
INTRODUCTION
3D point-cloud detection is defined as finding objects (localization) point-cloud and naming them (classification). Recently, deep learning based 3D detectors have achieved significant progress. However, most methods are developed on point-cloud detection datasets with limited classes, whereas the real world has cornucopia of classes. It is common for 3D detectors encounter objects that had never occurred during training, resulting in failure to generalize to real-life scenar- ios. Therefore, it is extremely important design an open-set point-cloud detector which is able to generalize to unseen classes. The key ingredient of open-set detection is that the model learns suf- ficient knowledge thus able to output general representations. To achieve this, in the image field, typical open-set classification and detection either require to introduce large-scale image-text pairs or image datasets with sufficient labels. For example, CLIP Radford et al. (202T) introduced 400 million image-text pairs for pre- pre-training to help visual models learn general representation. Detic Zhou et al. (2022) leverages ImageNet21K extend the knowledge of image detectors. OVR-CNN Zareian et al. (202T) uses the language pretrained embedding layer to broaden the vocabulary of the 2D detector.
However in the point-cloud field, as far as we know, there are few studies for the open-set point- cloud detection. The most notable hindrance is that we can hardly obtain large-scale point-cloud data and labels (or optionally, captions like the aforementioned image field), due to the difficulty of collection and annotation. The scarcity of point-cloud data and labels drastically restricts the point- cloud model from learning sufficient knowledge and obtaining general representations. Therefore, this limitation motivates us to ask Can we transfer the knowledge from images to point-cloud so that the point-cloud model is capable of learning general representations?

---

Under review as a conference paper at ICLR 2023
[bbox, class label)
AR. (8)
Can ImageNet Help?
30
Point-cloud includes seen classes and classes.
Raaeett) IiScaaNic Pand--Ceae ScuaNes Langs
Images from ImageNet
(a) ImageNet helps open-set 3D point-cloud detec- tion.
(b) AR25 w.r. label ration on the ScanNet dataset.
(c) ms.P3s w.r. categories on the ScanNet dataset.
Figure 1: (a). The left point-cloud includes the tables (green) and the chairs (red). The tables are labeled and denoted as seen objects, while the chairs are unseen objects. Right part shows exam- ples from ImageNetI which has sufficient labels. We aim at utilizing large-scale ImageNet K to broaden the vocabulary of point-cloud detector. (b). We train the 3DETR Misra et al. (02T)) and DETR Carion et al. (2020) on the ScanNet dataset. Under low-data regime, e.g.. with randomly sampling 10 % data, Al25 is still at high-level accuracy, even competitive to using 100 % data, which demonstrates that localization in point-cloud object detection generalizes well. (c). The per- formance comparison between the proposed OS- and the baseline 3DETR on the ScanNet dataset. OS-3DETIC significantly improves the baseline by large margin on all the categories.
Our work shows that the answer is positive. Intrinsically, images are dense RGB pixels while point- clouds consist of sparse xyz points. Although the large gap exists, both point-cloud and image are visual representations to the physical world and can express the same visual concepts Xu et al. (2021a); Park et al. (2022). Human-heings have no problem understanding both modalities. But two issues still exist: what kind of image data can we use and how to use the image data?
It is straightforward that we directly utilize the image detection dataset with fully bounding boxes and class labels, and transfer the knowledge from the 2D detector to 3D. However, bounding-bov level annotation is still laborious and difficult to scale, while open-set detection requires rich labels to help the detector learn sufficient knowledge Zhou et al. (2022). Therefore, in this paper, in- stead of seeking to build large-scale scale point-cloud dataset or use 2D detection datasets, we open-up another path by resorting to large-scale image datasets with image-level class supervision, Ima- geNet1 K Krizhevsky et af. (2112), to enable the point-cloud detector capable of learning general representations, thus broadening the vocabulary of the point-cloud detector, as shown in Fig. ((a)
Open-set 3D detection it to detect unseen categories without corresponding 3D labels. Note that, in this setting, open-set is defined in terms of the 3D detection, can make use of external knowledge such as ImageNet, which only provides Image-level category knowledge. Specifically, the proposed OS-3DETIC makes the 3D detector learning sufficient knowledge from image-leved supervision thus achieving open-set point-clous detection, and is a synergy of two components: 1) Make full use of knowledge learning from ImageNet and generalizability of localization point-cloud to gener- ate pseudo-labels for unseen class. 2) We design a de-biased cross-modal contrastive learning with listance-aware temperature to capture the shared low-dimensional space within and across modali- ties, thus better transferring the sufficient knowledge from image domain to point-cloud domain. It noteworthy to mention that during training, we introduce the paired images to narrow the gap be- tween the point-cloud data and the images from ImageNet, but we do not need any extra annotations except the
transformation matrix.
Extensive experiments show that OS-3DETIC outperforms a wide range of state-of thethe-rrt basclines by at least 10.77% mAP (absolute) and 9.56% mAP (absolute) without hurting the latency of the original 3D detector, on the unseen classes of SUN RGB-D et al.) (2015) and the ScanNet Dai (et al. (2017), respectively. An example on the ScanNet dataset shown in Fig. [(c] Sufficient ablation studies shed light why the OS-3DETIC works. Overall, our contribution as follows:
We propose open-set 3D detector with image-level class, termed OS OS-3DET, which is a synergy of two components: the pseudo-label generation from two modalities, and the de-biased cross-modal contrastive learning with distance-aware temperature.
2

---

Under review as a conference paper at ICLR 2023
OS-3DETIE can be regarded superior baseline in open-set 3D point-cloud detection.
Extensive experiments demostrate the effectiveness of OS-3DETIC, and also provide sufficient analysis to uncover why it works.
2 RELATED WORK
open-set object detection targets to detect novel classes that are never provided labels during the training Bansal et al. (2018); Gu et al. (202T) Rahman et al. (2020a bb: [Zhou et al. (2022); Zareian et al. (021). The most similar work to us is Detic Zhou et al. (0022) which utilizes ImageNet2IK
to broaden the classifier of the 2D detector. Yet, it is infeasible to directly use the same method to broaden the classifier of the point-cloud detector, due the large gap between image and point- cloud. Different from Detic that transfers knowledge within the same modality, we propose to trans- totally different modality. Point-Cloud Detection has been ac- tively researched Yang et al. g Chen et al. (2017); Wu et al. (2018) 2019); Xu u et al. (2020); Shi al.) (2019); Xu et al. (2018) 8 2021b): Zhou & Tuzel (2018); Yan al. (2018): Shi et (222bb a); Dosovitskiy et al. 20 Wu et al. (2020): Liu et al. (2021c) Misra et al. (202 Zhao et al. 2021
fer knowledge from ImageNett
yet to the best of our knowledge, open-set 3D point-clone
detection has not been widely explored, existing methods Cen et al. [2027] 2 Wong et al. (2020) mainly focus on identify the unknown
objects from known one, whereas our case, we further assign name to each unknown object,
which is similar to Xu el al. (202la); Zhang el al. (027); Cheraghian et al. 2009h 2027 but
these works mainly study classification.
3 METHOD
3.1 NOTATION AND PRELIMINARIES
We use E R3xH*W to represent image, and P = {pr E R², = 1,2, NN to rep- resent point-cloud, where N is the point number in the point-cloud. During training, we use 1) point-cloud dataset denoted as Drc ((P, (ban E RR, (RT,)))))) 1|27 with vocabulary size Oper where baD is the annotation of 3D bounding box, C3D is the corresponding classification la- bel: 2) paired image dataset denoted as Dimg (I, |2""" 3) ImageNetIK dataset denoted as Disn (II igny C2D ) }
IDiimn with vocabulary Cign- where 1gg denotes classfication label of the im- age in ImageNet K. During test, we evaluate on the vocabulary Crest, where Cign 2 Crest>Cpe-
A typical point-cloud detector deals with localization and classification, where the localization mod- ule outputs bounding boxes b3D E R and we could get its corresponding point-cloud ROI features faD. Similarly, we can project the 3D bounding box into 2D image via projection matrix K, i.e., ban E R2 and index the corresponding image ROI feature fan. Then we can use the f3n and fan to predict the class of each object.
3.2 3D-3DE::C: OPEN-SET 3D DETECTOR WITH IMAGE CLASSES
We use ImageNetlIK broaden the vocabulary of the point-cloud detector. To transfer the knowl- edge contained in ImageNetIK K to the point-cloud detector, paired image is introduced to bridge these two modalities. Specifically, We design two-phase training strategy to enable open-set point-cloud detection. The first-phase training is similar to Detic Zhou et al. (002)). which aims at leveraging ImageNet help the 2D detector be able learn sufficient knowledge. The second-phase is the core of the proposed OS-3DETIC, which aims at transferring the knowledge of the 2D detector to the 3D detector via customized pseudo-label strategy and de-biased cross-modal contrastive learn- ing, shown in Fig. 2 During inference, we only need the 3D detector without any extra models or modalities. OS-3DETIC is developed on top of 3DETR Misra et al. (202I), and we use DETR Carion et al. (2020) as the 2D detector. The details are illustrated below.
In the first stage, the point-cloud P in DRC is injected to the 3D detector, which is supervised by the ground truth (baD. tap} of seen objects. The paired images I from Dimg are input into the 2D detector, which is supervised by projected 3D boxes and the corresponding class, denoted as (K x bap, ). It is noteworthy to mention that the images I from from ImageNet1 K Dign
3

---

Under review as a conference paper at ICLR 2023
iseae Suzs InaaacN
S) Microwaaee Table
featurpe
RCH Suturn image Frees diracc
3D
Poin-cteed Frees 3D dizset
ROI fcatano Poinsi Good dian dissett Debiased Cress-modal Contrastive Learning
Figure 2: Overview of the phase 2 of OS-3DETIC, which is the core and synergy of two compo- nents: 1) We take advantage of two modalities, image modality for classification and point-cloud modality for localization, to generate pseudo-labels for unseen classes, and 2) we design de-biased cross-modal contrastive learning to transfer the knowledge from image to point-cloud. Note that before phase 2, we first train both the 2D detector and the 3D detector as similar to Detic Zhou al.) (2022). The green brackets denote ground truth of seen classes. For ImageNet1K, K, the "bbox" is not provided, so denoted as Ø. The red brackets denote pseudo-label, where bounding boxes come from the output of the 3D detector, and classes come from the output of the 2D detector.
also input into the same 2D detector with only image-level labels (category) provided. Following Detic Zhou et al. (2022), we choose the max-size proposal mmazsise and apply classification label 22 to supervised the classifier in the 2D detector. The loss in the phase training is given by
phosel =L 3n (baD. b&D) + LaR (CaD, W3DLLD)+
12 (K bąn, ban) + cls (C3D, W2nf2D) T. (CDD) WanLmarsise)
where L2D and 12 follow Misra et al. (2021) and Carion et al. (2020). L and L cross- entropy loss for classification. Wan and Wen denote the classifier in the 3D detector and the 2D detector, respectively.
we first generate pseudo-labels for unseen classes. The pseudo-lahels contain two parts: bounding box from the 3D detector, and class from the 2D detector. Specifically, since already exploit ImageNetIK to train the 2D detector in the phase 1, as similar to Detic Zhou et al. (2022). the 2D detector able classify unseen classes so that relatively accurate pseudo-labels. To leverage this, we crop the image region of the projected 3D detection and use the 2D detector's classifier to generate its class label. For the bounding box pseudo-labels, we take advantage of the generalizability of localization for the point-cloud detector, as mentioned in Fig. [(b) Besides using pseudo-labels, we also design de-biased cross-modal contrastive learning to better transfer the knowledge from 2D modality to point-cloud modality. Note that there is sigmificant synergy between this pseudo-label strategy and our proposed de-biased cross-modal contrastive learning. The pseudo-label is beneficial for true positive sampling of cross- modal contrastive learning, and cross-modal contrastive learning transfer the knowledge from image to point-cloud gradually, in turn helping generate better pseudo-labels. Thus the pseudo-labels are iteratively updated to be of higher quality. Overall, the total loss in phase 2 is given by
In phase 2,
can use its classification results as
T.phase2_ -DD (BaD, baD) 1.3D (CaD, WaDGAD) + T2 (K x baD ban)+
L2D Cap, W2Dfep) + L WaDfmazsise) + LDECC,
fgn
where BaD and Eyp come from either ground truth or pseudo-labels, and LDECe denotes the loss function of the de-biased mods- contrastive learning.
4

---

Under review as a conference paper at ICLR 2023
Negative sample
Postive sample
ImageNet provides positive samples and benefits pseudo label generation for unseen classes
Point-cloud from 3D datasct
ImageNet
Image from 3D dataset
Position-basee basee cross-modal
De-biased cross-modal
contrastive learming (Binsed)
contrastive learming (De-biused)
Figure 3: Typical biased cross-modal contrastive learning (left) and the proposed de-biased cross- modal contrastive learning (right). Position-basee cross-modal contrastive learning follows the po- sition correspondence with one-to-one match. This may result in assigning inaccurate negative sam- ple. The proposed de-biased cross-contrastive learning (DECC) leverages pseudo-labels to facilitate the biased issues. The red outline denotes it is unseen class and we assign a pseudo-label it, and the green outline denotes the seen class with ground truth.
3.3 DE-BIASED CROSS-MODAL CONTRASTIVE LEARNING
Before we de-thpth illustrate the proposed de-biased cross- modal contrastive learning, we first dis- cuss the typical biased contrastive learning in the following. Cross-modal contrastive learning has been demonstrated as a powerful method used for knowledge transferring Jiang al. (2019): Rad- ford et al. (2021): Minderer et al. (2022). and enabling zero-shot/open vocabulary classification and detection. Using instance discrimination as the pretext task, typical contrastive learning Radford al. 1)021); Chen et al. (2020); He et al. (2020); Khosla et al. (2020) pulls the semantically-cloo neighbors together and pushes away non-neighbors. In previous works Radford et al. (2021): Min- derer al. 2022), dissimilar (negative) points typically taken to be randomly sampled data- points, implicilly accepting that these points may actually express the same underlying concepts. For example, CLIP CLIP |Rafford et al. (202]) conducts contrastive learning on coupled text and image, while the text caption may describe a bunch of images that have similar semantics. This issue is studied as noisy matching Wu et al. (2021) biased contrastive learning Chuang al. (2020). In the image-poing poree--ooottoodoo field, existing studies mainly focused on matching the features from the same 3D positions Hou et al. (202T); Liu et al. (2021b); Park al. (2022). This results in biases, as shown in left part of Fig. 3 We can observe that both within and across modalities, some objects with the same classes are inaccurately assigned as negative samples.
In order to mitigate the biased issue in contrastive learning, we propose de-biased cross-modal contrastive learning (DECC), as shown in right part of Fig. We take the ROI features fan from 3D detector and f2D from the 2D detector, and use the ground truth or pseudo-labels Cap to assign the positive and negative samples. For mini-bated of faD and feD with total M features from two modalities, further use linear projection to transform these feature into hidden representations, which is denoted as hG where = 1, 3, M. Then the loss is given by
Σm G. he/T(diat..) LDECC M log
Σ5 e hTT//To
where m is the number of positive sample corresponding to h; (m <= M), Fy is the base temper- ature. It is noteworthy to mention that different from previous contrastive learning with constant temperature for each sample, we here adopt a distance-aware temperature rddist,,) = To x x dist,, for positive sample, where 7 are hyperparameters. In particular, calculate the Fuclidean distance
5

---

Under review as a conference paper at ICLR 2023
of the two samples in 3D space. Note that if the of the sample from ImageNet, consider the distance between ImageNet and any of other sample as 1. Intuitively, the correlation between two close-by objects is greater than that of two distant object, thus scale the temperature with distance-aware strategy to facilitate this correlation. Specifically, If y < 1.0, then LDECc pays more attention to connect close-by objects, and if y 1.0, DDCCC degenerates de-biased cross- modal contrastive learning with constant temperature, while 7 < 1.0, LDECe focuses on bridge two distant objects. In practices, we set 7 > 1.0 is to enforce the alignment of close-by objects.
EXPERIMENTS
In this section, we compare the proposed OS-3DETIC with popular baselines on two widely used 3D detection datasets, SUN RGB-D Song et al. (2013) and ScanNe(Dai et al. (2017). Then we conduct sufficient analysis and ablation studies to explore why OS-3DETIC works. The details are illustrated the below.
4.1 DATASETS AND EVALUATION METRIC
2015 and ScanNetV2 Dai al. (2017) are two widely used 3D detection datasets. We follow the data preprocessing procedure from VoteNet Qi et al. (2019b). except for the sampling of unseen classes. We randomly select 10 unseen classes for both two datasets. Specifi- cally, toilet, bed, chair, bathtub, sofa, dresser, scanner, fridge, lamp and desk are unseen classes for SUN RGB-D, while table, night stand, cabinet, counter, garbage bin, bookshelf, pillow, microwave, sink and stool are seen classes, toilet, bed, chair, sofa, dresser, table, cabinet, bookshelf and pillow are unseen classes for ScanNet, and bathtub, fridge, desk, night stand, counter, door, curtain, box, lamp and bag seen classes.
SUN RGB-D Song
The metrics we use in the experiments are Average Precision (AP), mean Average Precision (mAP) and Average Recall (AR) at IoU thresholds of 0.25, denotes AP25. mAP25. ARg5. respectively. mAP takes both classification and localization in to consideration, the large the mAP means the better the detection, whereas AR mainly focuses localization. Therefore, compared to AR, mAP is a better metric for evaluating detection result.
4.2 EXPERIMENTAL. SETUP AND IMPLEMENTATION DETAILS
Implementation The proposed OS-3DETIC is mainly based on 3DETR Misra et al. (202I), yet it can also generalize to other point-cloud detectors. During training, minibatch of data consists of the point-cloud with its paired image and several images that from ImageNet. 3DETR as the 3D detector consumes the point-cloud, and DETR Carion et al. (2020) the 2D detector deals with images. Note that only classification loss used in the DETR when the input is images from ImageNet. We select the max-size ROI features detected by DETR and apply the classfication loss on it, as following Detic Zhou et al. (2022). To and 7 are two hyperparameters in the distance-aware temperature strategy, in practice, To set to 0.2, y is set 1.1, and the distance between image from ImageNet and any other sample is 1.0. For both 3DETR and DETR, a unified learning rate is set to 2 x 10- 5, with batch size of 4 for each GPU, and we train our model on 8 RTX 2080 TI gpus. We train 200 epochs for the phase 1, and 200 epochs for the phase 2.
Pseudo-Label Generation Pseudo-lahel consists of two elements: 3D bounding box and category. We first generate initial pseudo-labels after finishing phase 1, and update them iteratively. As we discussed in Section 3.2 the 3D detector performs generalizable object localization, while the 2D detector is trained on the ImageNet dataset with sufficient knowledge. Therefore, the bounding box comes from the 3D detector and the class label comes from the 2D detector. Furthermore, we only keep proposals with high confidence and filter out the proposals that are duplicated with ground truth or no point in it. Finally, only top-k proposals are kept for each unseen class, and à increases linearly every 50 epochs in phase 2. is default as 50, and increases 10 (default) every 50 epochs.

---

Under review as a conference paper at ICLR 2023
Table 1: Detection results (AP25) unseen classes of SUN RGB-D.
Mathod
toilet
hed
chair
bathtab
sofa
dresser
scanner
fridge
lamp
desk
mean
GroupFroc5 ot al. 2021d
0.23
0.04
1.25
0.03
0.21
0.21
0.14
0.10
0.03
3.02
0.53
VoteNet Qrera 20 1a
0.12
0.05
1.12
0.03
0.09
0.15
0.06
0.11
0.04
2.10
0.39
HRDNei Zhonng
1.28
0.22
0.03
6.09
3DETR MISTa
1,57
0,23
0.77
0.24
0.04
0.61
0.32
0.36
0.01
8.92
1.31
OS 3S.PPinnCTIIP Zhans
2021
7.90
2.84
3.28
0.14
1.18
0.39
0.14
0.98
0.31
5.46
2.26
OS Image al
10.19
2.14
0.09
3.25
0.01
0.15
0.55
0.04
0.27
0.02
5.48
1.20
Detie-ModelNet Zhou
2022
3,56
1.25
2,98
0.02
1.02
0.42
0.03
0.63
0.12
5.13
1.52
Detic-ImageNe Ziku aral
2022
0.01
0.02
0.19
0.00
0.00
1.19
0.23
0.00
7.23
0.91
Ours
43.97
6.17
0.89
45.75
2.26
8.22
0.02
8.32
0.07
14.60
13.03
Improvement
+36.07
+3.33
-2.39
+45.51
+1.08
+7.03
40.30
+7.34
-0.24
+5.68
+10.77
Table 2: Detection results (AP25) unseen classes of ScanNet.
Method
toilet
hed
chair
sofa
dresser
table
cahinet
bankshelf
pillow
sink
meam
GroupFree SD Läu
20
0.63
0.52
1.52
0.52
0.20
0.59
0.52
0.25
0.01
0.15
0.49
VoteNet Orer Oreral
2019
0.02
0.12
0.02
0.11
0.07
0.05
0.00
0.04
H.DNNe Zhany et al 20:4
0.55
0.29
1.70
0.31
0.18
0.76
0.49
0.40
0,01
0.10
0.48
3DETR Misau
2.25
1.11
OS-PoiniCLIP Zianng
2021
6.31
0.66
0.68
2,05
0.79
3.09
0S-Imag:2PPin//X Porngg:2Ponn/uuee Xurei A02 11
0.24
0.77
0.96
1.39
0.24
2,82
0.95
0.91
0.00
0.08
0.84
Detic- ModelNee Zhou
4.25
0.98
4.56
1.20
0.21
3.21
0.56
1.25
0.00
0.65
1.69
Detie-ImagoNel Zirou ot
2024
0.04
0.01
0.16
0.01
0.52
1.79
0.54
0.28
0.04
0.70
0.41
2.63
7.27
18.64
2.77
14.34
2.35
4.54
21.08
12.65
Improvement
+42.44
+0.34
+0.96
+ 14.76
+2.11
47.17
11,36
+2.29
+3.38
+20.29
+9.56
4.3 MAIN RESULTS
As there is no baseline directly solving the problem of both open-set 3D point-cloud localization and classification, we mainly compare OS-3DETIC with state-of-the-arr 3D point-cloud detectors Liu et al. d2) Qi et al. 20119a; Zhang et al. (2020); Misra et al.] (202T) and some well-known works Zhang et al. (2021): Xu et al. (202 Zhou et al. (2022) that study either transferability in point-cloud 2D open-set detection. Specifically, the baselines use include:
3Do Liu et al. (201d),, VoteNet Qi et al. (2019)). H3DNet Zhang et al. (2020), 3DETR Misra et al. (2021) are well-known and representative 3D point-cloud detectors that are chosen as our baselines. Specifically, these four baselines are trained on the seen classes while being tested on the unseen.
The second is PointCLIP Zhang et al. (202I) which bridges the point-cloud and text do- main. We use it directly pre-trained open-set 3D classifier and replace the classifier of 3DETR with PointCLIP. This baseline is denoted as OS-Point DS-PointCL.IP. which is similar to well-known 2D open-set detection works Bansal et al. (2018): : Gu al. (2021): )): /hhou et al. (2022) that replace the classifier in the detector with the generalizable classifier.
Besides, Xu et al Xu et al. (2021a) transfer the image pre-trained transformer to the point- cloud by copying or inflating the weights. Similarly, we copy the weights of the trans- former and the classifier from pre- pre-trained DETR (pre-trained on COCO Lin et al. (2014) 3DETR and finetune the set aggregation module and the 3D box head. We term this baseline as OS-mmaae2ooint.
Moreover, Detic Zhou et al. (2022) leverages large-scale classification dataset (ImageNet) to broaden the 2D detector, here we directly extend the idea to 3D open-set detection. Specifically, we consider two manners, extend the classifier via ModelNet ImageNet, and term them as Detic-ModelNet and Detic- Imagee,, respectively.
The results are presented in Tables I and 2 We can observe that OS-PointCLIP achieves mAP2s of 2.26% and 3.09%, outperforming the other baselines on both SUN RGB-D and ScanNet. Fur- thermore, Detic-ModelNet reaches mAPPss of 1.52% and 1.69% on SUN RGB-D and ScanNet, respectively. It can be observed that both OS-PointCLIP and Detic-ModelNet outperforms OS- Image2Point and Netge Indeed, OS-PointCLIP and Detic-ModelNet enhances the classifier of the detector via introducing the pretraining on ModelNet, which is a 3D classification
7

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study different components.
Augmentation Based CL
Position Based CL
Class Based CL
Distance-Aware Temperature
Baseline Pseudo-L.abe
m.AP2s
A Ags
1.31
27.00
12.55
38.35
10.48
36.44
12.36
37.35
12.92
42.48
13.03
10
14
10
12
82
30
ARg (90
m.44%.
2
8
cloud mdP
Trage i&
Polneelou Allys
Insage ARRs
A/la
70 Thats (55)
(a) Performance Data Ratio.
(b) Performance Iteration.
Figure 4: (a). illustrates relation between mAP25. AB25 and the data regime of both 2D and 3D detection, the green, orange, solid and dashed lines represent point-cloud, image, m.AP25 and AR25. respectively. (b). illustrates the relation between mAP2s, Altss and pseudo-label iteration, the blue and red lines represents mAP2s and AR25, respectively.
dataset, while OS- OS-Image2Point and Petic-ImageNet try to transfer knowledge from image (COCO and ImageNet) to point-cloud. This contrast shows that the modality gap between 2D and 3D does indeed hinder knowledge transfer. The results also demonstrate that directly plugging and playing the Detic method on 3DETR with introducing ImageNet is infcasible to transferring the knowledge from 2D image to 3D. Nonetheless, RGB-D and ScanNet, respectively, which proves the proposed OS-3DETIC can indeed make use of the image knowledge, to achieve open-set 3D detection.
method achieves m mAP25 2, of 13.03% and 12.65% SUN
4.4 ANALYSIS AND ABLATION STUDY
Ablation Study on Different Components We conduct an ablation study on SUN RGB-D, the results of unseen classes are reported in Table Our baseline 3DETR which trained only on seen categories. "Pseudo-Label" denotes to train with pseudo-label. "Augmentation Based CL" de- notes we follow SimCLR Chen al. (222)); He al. (2020) that utilize data augmentation strategy in contrastive learning, which however, is biased contrastive learning setting as well. Position- based contrastive learning takes only the same instance from the paired image/point-cloud as pos- itive. Class based contrasitve learning takes positive long two samples belong to the same category. "Distance- Temperature" refer to the strategy that is proposed in Section 3.3 We observe that, first, pseudo-label brings the largest improvement, actually, pseudo-label implic- itly transfers the knowledge contained ImageNet to a totally different modality, the point-cloud modality, via providing useful class labels. This only validates our hypothesis that the 3D de- tector functions as a general region proposal network, but also demonstrates the effectiveness of introducing large-scas scale image-level supervision for classification. Second, class-based contrastive learning and distance temperature further significantly improve the performance, while tra- ditional augmentation-
contrastive learning and position- contrastive learning hurt the performance, which indicates the weakness of biased contrastive learning, and demonstrates the pro- posed de-biased cross-modal contrastive learning indeed benefits the point-cloud detector learn general representations.
Analysis of Performance on Different Training Data Ratio 4() illustrate relation between mA Pas, AR2s and the different training data ratio. the left y axis denotes m.P the right axis
((a)
8

---

Under review as a conference paper at ICLR 2023
denotes AR25, and the axis represents the data ratio that used during training. We draw both mAP25. AR25 results of the point-cloud and the paired image branch. On one hand, compared with 22P25, AR25 converges with relatively less training data, which exists in both the image and the point-cloud branches. A partially reflects localization ability, which means we can train a generalizable bounding box detector with few annotations. On the other hand, compared with A R25 of image branch, the AR25 of point-cloud branch converges with less training data, which means the localization ability of 3D detector are better than 2D detector, verifying the strategy of using the point-cloud detector to generate bounding box pseudo-labels. Moreover, the converged m.AP25 of the image branch better than the point-cloud branch. This may be because texture and detail account a lot for classification, which however is the weakness of the point-cloud detector.
Analysis on Pseudo-Label Effects Fig. 4(b) illustrates the relation between m.A22s. AR2s and pseudo-label iteration. During the second phase, we iteratively update the pseudo-label every 50 epochs. The results show that the more the iteration, the better the pseudo-labels should be, thus leading to the better the performance.
4.5
QUALITATIVE RESULTS
Fig. 5 provides the qualitative results. We can observe that Baseline is able to predict the relatively accurate locations compared with ground truth, but the size, center, and direction are incorrect, especially the unseen class bounding boxes (chair in Scene I, sofa in scene 2, bed in scene 3 and sofa in scene 4). Compare baseline with OS-3DETIC, OS-3DETIC performs much better the unseen classes.
Buseline
OS: 3DETIC
Ground Truth
(a) Scene
(b) Scene
(c) Scene
(d) Scene
Figure 5: Visualization of detection result. 4 column represent 4 different scenes, the comparison conducts among Baseline, OS-3DETIC and Ground Truth. All the bouning boxes in Baseline and OS-3DETIC predicted bounding boxes, And the red bounding boxes in Ground Truth represent unseen classes samples, and the green represent seen classes.
CONCLUSION
In this paper, we study a new problem of open-set 3D detection. The proposed method OS-3DETIC introduces ImageNet1 to help open-set point-cloud detection. OS-3DETIC consists of two com- ponents: 1) we take advantage of two modalities - the image modality for classification and the point-cloud modality for localization, to generate pseudo-labels for unseen classes, and 2) de-biased cross-modal contrastive learning transfer the knowledge from images point-clouds. Extensive experiments show that we improve a wide range of baselines by a large margin, demonstrating the effectiveness of the proposed method. We also provide explanations of why it works via ablation studies and analyzing the representations. We do hope our work could inspire the research commu- nity further explore this field.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 384-400, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp.
Springer, 2020.
Jun Cen, Peng Yun, Junhao Cai, Michael Yu Wang, and Ming Liu. Open-set 3d object detection. In 2021 International Conference on 3D Vision (3DV), pp. 869-878. IEEE, 2021.
Jun Cen, Peng Yun, Shiwei Zhang, Junhao Cai, Di Luan, Michael Yu Wang, Ming Liu, and Minggian Tang. Open-world semantic segmentation for lidar point clouds. In European Conference on Computer Vision (ECCV). Springer. 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. pp. 1597-1607. PMLR, 2020.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3D object detection network for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
pp.
Ali Cheraghian, Shafin Rahman, Dylan Campbell, and Lars Petersson. Mitigating the hubness problem for zero-shot learning of 3d objects. arXiv preprint arXiv: 1eXXiv1907.06333 1907.06377, 2019a.
Ali Cheraghian, Shafin Rahman, and Lars Petersson. Zero-shot learning of 3d point cloud objects. In 2019 16th International Conference on Machine Vision Applications (MVA), pp. 1-6. IEFE, 2019b.
Ali Cheraghian, Shafinn Rahman, Townim Chowdhury, Dylan Campbell, and Lars Petersson. Zero-shot learmingg 3d point cloud objects and beyond. arXiv preprint arXiv:2104.04988 2021.
Ching Yao Chuang, Joshua Robinson, Yen-Cher Lin, Antonio Torralba, and Stefanic Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8775, 2020.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11999 2020.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Zero-shot detection via vision and language knowledge distillation. arXiv e-prints. pp. arXiv-2104, 2021.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Hou, Saining Xie, Benjamin Graham, Angela Dai, and Matthias Nießner. Pri3d: Can 3d priors help 2d representation learning? In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5693-5702, 2021.
Huajie Jiang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Transferable contrastive network for generalized zero-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9765-9774, 2019.
10

---

Under review as a conference paper at ICLR 2023
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 13:18661--1877 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp- 1097-1105, 2012.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision (ECCV), pp. 740-755, 2014.
Bo Liu, Shuang Deng, Qiulei Dong, and Zhanyi Hu. Segmenting 3d hybrid scenes via zero-shot learning. arXiv preprint 1xXiv:217700043:: 2021a.
Yueh-Cheng Liu, Kai Huang. Hung- Yueh Chiang, Hung-Ting Su, Zhe- Yu Liu, Chin- Chin-Tang Chen, Ching-Yu Yu Tseng, and Winston H Hsu. Learning from 2d: Pixel-to-point knowledge transfer for 3d pretraining. arXiv e-prints, pp. arXiv-2104, 2021b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021c
Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via trans- formers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2929-2938, 2021d.
Björn Michele, Alexandre Boulch, Gilles Puy, Maxime Bucher, and Renaud Marlet. Generative zero-shot learning for semantic segmentation of 3d point clouds. In 2021 International Confer- ence on 3D Vision (3DV), pp. 992-1002. IEEE, 2021.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann. Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Amab, Mostafa Dehghani. Zhuoran Shen, et Sim- ple vocabuaayy object detection with vision transformers. arXiv preprint arXiv:2205.06230, 2022.
Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEFF/CVF International Conference on Computer Vision pp- 2917, 2021.
Jinhyung Park, Chenfeng Xu, Yiyang Zhou, Masayoshi Tomizuka, and Wei Zhan. Detmatch: Two teachers are better than one for joint 2d and 3d semi-supervised object detection. arXiv preprint arXiv:2203.000 00000.09910,. 2022.
C. Qi, Or Litany, Kaiming He, and Leonidas J. Guibas. Deep hough voting for 3d object detection in point clouds. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9276- 9285, 2019a.
Charles Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In proceedings ofe the IEEE/CVF International Conference Computer Vision, pp. 9277-9286, 2019b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. -7633 PMLR, 2021.
Shafin Rahman, Salman Khan, and Nick Barnes. Improved disual-semante alignment for zero-shot object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11932-1 1939, 2020a.
Shafin Rahman, Salman H Khan, and Fatih Porikli. Zero-shot object detection: joint recognition and localization of novel concepts. International Journal of Computer Vision 128(12):2979- 2999, 2020b.
11

---

Under review as a conference paper at ICLR 2023
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Point-RCNN: 3d object proposal generation and detection from point cloud. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-779, 2019.
Shaoshuai Shi. Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN: Point-voxel feature set abstraction for 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10529-10555 2020a.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts:
3d object detection from point cloud with part-aware and aut--aggregation network. IEEE Trans- actions on Pattern Analysis and Machine Intelligence (PAMI), 2020b.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 367-576, 2015.
Kelvin Wong, Shenlong Wang, Mengye Ren, Ming Liang, and Raquel Urtasun. Identifying unknown instances for autonomous driving. In Conference on Robot Learning, pp. 384-393, PMLR, 2020.
Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer. SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3d lidar point cloud. In IEEE International Conference on Robotics and Automation (ICRA). pp. 1887-1893, 2018.
Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer. Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-ohject segmentation from a lidar point cloud. In ICRA, 2019.
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arXiv preprint arXiv:2006.03677 2020.
Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, and Joseph Gonzalez. Data efficient language-supprviseed zero-shot recognition with optimal transport distillation. arXiv preprint arXiv:2112.09445, 2021.
Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In European Conference on Computer Vision (ECCV), pp. 1-19. Springer, 2020.
Chenfeng Xu, Shijia Yang. Bohan Zhai, Bichen Wu, Xiangyu Yue, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Image2point: 3d point-cloud understanding with pretrained 2d convnets. arXiv preprint arXiv:2106.041188 2021a.
Chenfeng Xu, Bohan Zhai, Bichen WU, Tian Li, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. You only group once: Efficient point-cloud processing with token represen- tation and relation inference module. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4589-4596. IEEE, 2021b.
Danfei Xu, Dragomir Anguelov, and Ashesh Jain. PointFusion: Deep sensor fusion for 3D bounding box estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018
Bin Yang, Wenjie Luo, and Raquel Urtasun. PIXOR: Real-time 3d object detection from point clouds. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7652- 7660, 2018
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 14393-14402, 2021.
12

---

Under review as a conference paper at ICLR 2023
Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, and Hongsheng Li. Pointelip: Point cloud understanding by clip. arXiv preprint arXiv:211..02413, 2021.
Zaiwei Zhang, Bo Sun, Haitao Yang, and Qi-Xing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In ECCV, 2020.
Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259-16268, 2021.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Phillip Krähenbühl, and Ishan Misra Detecting twenty-thousand classes using image-level supervision. arXiv preprint (eXiv:2201.02603 2022.
Yin Zhou and Oncel Tuzel. VoxeINet: End-to-end learning for point cloud based 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
13


Reviews:
[{"review_id": "ufzU_hTfqtO", "reviewer": "Reviewer_fcKh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way to transfer semantic information from Imagenet1K to train a 3D detector on point clouds to detect classes for which there are no 3D labels at all. \n\nThe method trains a 2D detector (DETR) and a 3D detector (3DETR) in two stages. \n1) The 2D and 3D detector are co-trained, in line with Detic (Zhou 2022), with joint losses. The 3D labels (Sun/ScanNet) train the 3D detector, as well as 2D detector on 2D boxes derived from the 3D GT. The Imagenet labels train the DETR classifier head (applied on the largest detected box in the image). \n2) In Phase 2, an augmented dataset is created by running 3D detector to find \"unseen classes\". The 2D crops obtained from the detections are classifier by the 2D detector. Training continues also incorporating this augmented dataset, but adding contrastive training losses for the 2D crops, that take into account ImageNet classification results (crops with same class are considered positives). \nThe process of generating the augmented dataset is iterated several times, taking more and more positive examples from the classifier. \n\nThe method is evaluated on Sun and Scannet 3D datasets on a number of withheld classes. \n\n\n\n", "summary_of_the_review": "The application is interesting and novel. The system is a combination of mostly known components, but they are put together reasonably well. Related work section could be improved. I think it's important to show the gap between the OS3-DETIC system and full supervision. ", "strength_and_weaknesses": "Strengths: \n+ Seemingly novel use case of transferring 2D classification knowledge to a point cloud 3D detector. The choice of Imagenet1K seems suitable as a source of supervision, as the classes in question are present in Imagenet. \n+ The method is shown to ourperform several reasonable baselines on Sun and Scannet for a number of classes, in some cases by significant margins.  \n+ Suitable ablations and studies of the effects of iterating the process are done. \n\nPotential weaknesses: \n- The start of related work section seems missing. It also claims \"to the best of our knowledge, there has been no work on open-set 3D object detection\". There is some related work that should be cited. For example Cen, J., Yun, P., Cai, J., Wang, M.Y., Liu, M.: Open-set 3d object detection. 3DV, 2021. Or Wong, K., Wang, S., Ren, M., Liang, M., Urtasun, R.: Identifying unknown instances for autonomous driving. In: CoRL. PMLR (2020). These are not quite the same, as they don't transfer class labels from a trained image classifier, however. \n- It is unclear how well the method does compared to full supervision. It would help to share how well DETR3 does on the classes when there is actually supervision. \n- Ablation shows that the first step of generating pseudolabels (mostly following the Detic paper) yields most of the gains. The application of \"class-unbiased\" contrastive training, or \"distance-aware\" temperature, which are technical details of some minor novelty, yield gains that seems quite small. \n- Sun RGB-D dataset contains also images, yet the transfer there is done purely on the 3D point cloud data. It would have been interesting to see how well we can do when also using the RGB data. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gMpm-3eYl_C", "reviewer": "Reviewer_rTqP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel method for open-set 3D detection using image-level class supervision. The core idea is leveraging each of the image and point-cloud modalities to generate pseudo labels for unseen classes. To improve the positive and negative sample matching, the authors propose debiased cross-modal contrastive learning. The proposed approach takes two phases to train, where the first aims to train classification and localization models to generate pseudo labels and the second aims to further facilitate information transferring with the proposed contrastive learning strategy. \n\nThe authors have evaluated their method on the SUN RGBD and ScanNet datasets.", "summary_of_the_review": "Overall, I think the idea of generating pseudo labels and debiased cross-modal contrastive learning is interesting. During the rebuttal, the authors have addressed my concerns. I would like to raise my rating to acceptance. ", "strength_and_weaknesses": "Strength:\n1. The paper is well motivated to address an important topic in computer vision.\n\n2. The debased cross-modal contrastive learning is interesting. \n\n\nWeakness:\n1. Is the paper trying to localize and assign semantics simultaneously or just assign semantics, by assuming the localization is naturally generalizable to open-set? Please clarify.\n\n2. What is the justification for the statement that the 3D detector is generalizable to unseen categories? \n\n3. How to weigh different terms in Eq. (1) and Eq. (2)?\n\n4. The experiment setup is questionable. The authors propose to randomly select unseen and seen classes, which however unavoidably exists resemblance in semantics and shape. For example, for both SUN-RGBD and ScanNet, semantically similar concepts like desk and table are leaked in unseen categories. \n\n5. Is the unseen / seen category sampling strategy consistent with other referenced open-set methods, e.g., Zhang et al., Xu et al., Zhou et al.? \n\n6. The proposed method underperforms previous works in lamp, scanner and chair on the SUN -RGBD dataset.\n\n7. In the ablation study, distance aware temperature seems to have marginal improvement in mAP but cause significant regression on AP. Please explain further. \n\n8. I would like to see the ablation study on the proposed DECC idea. \n\n9. Writing quality should be much improved. Typos and grammar errors are throughout the manuscript. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "adhKuzRTaPB", "reviewer": "Reviewer_6Lrw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a framework for performing open vocabulary 3D object detection. The proposed framework mostly follows the 2D open vocabulary detector Detic[1] with 2D localizer replaced by 3D localizer. Different from Detic, the proposed framework introduced a contrastive feature learning module named debiased cross-modal contrastive learning(DECC) between 2D and 3D RoI features. Experiments on ScanNet and SUN RGB-D showed that the proposed method achieved superior results over existing methods and several baselines.", "summary_of_the_review": "Overall I think the proposed framework, while lacking technical novelty, could serve as a much stronger baseline compared with existing open vocabulary 3D detection methods. I am leaning to recommend for its acceptance.", "strength_and_weaknesses": "Pros:\n1. The paper is generally easy to follow and proposed framework could serve as a much better new baseline.\n2. The results are strong compared with existing works.\n\nCons:\n1. Some baselines settings are not quite fair. Taking Detic-ModelNet and Detic-ImageNet as examples, it is hard to understand why only the classifier of Detic are extended, as the proposed method itself extended both classifier and regressor from Detic.\n2. The technical novelty is somewhat limited. The proposed framework is almost a Detic detector with 2D localizer replaced by 3D localizer. The proposed DECC improves only marginally to its pseudo label baseline as shown in Table 3.\n\nMiscs:\n1. Figure 4 has jammed y-axis legend texts for both sub-figures.\n\n[1] Detecting Twenty-thousand Classes using Image-level Supervision, Zhou et al, ECCV 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WrWbhqygdk", "reviewer": "Reviewer_2qoz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors addressed an open-set 3D detection problem to broaden the vocabulary of the point-cloud detection without laborious and expensive data annotation. Inspired by previous open-set works, the authors proposed OS-3DETIC consisting of two main functions as classification using image-based model and localization using point-cloud model. Furthermore, to maximize the capability of transferring knowledge between two modalities, the authors proposed a de-biased cross-modal contrastive learning as an auxiliary task in the open-set detection problem. In experimental section, the proposed method showed the significant improvement in 3D unseen benchmark and is validated by various ablation studied provided.", "summary_of_the_review": "I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper.", "strength_and_weaknesses": "[+] First of all, the motivation of the paper seems to be meaningful and pragmatic for 3D object detection in the perspective of the generalized 3D object detection and difficulty of the 3D annotation procedure. The key idea is very intuitive how to obtain the pseudo label for 3D object detection and the overall framework seems to be well-designed to embody the author’s purpose. Even if the technical novelties is limited, the idea of de-bias contrastive learning approach seems to be reasonable for overcoming the limitation of position-based contrastive learning, which is a hard semantic relationship between defined pairs.\n\n[-] One concern is an influence of de-bias learning to 3D localization. Unlike 2D localization problems, the orientation is included in 3D localization task. The orientation is known to bias to the part or certain positions of the object. As mentioned in the paper, if the ROI features is used for de-bias learning, it might conflict with the problem to estimate the orientation. I pretty agree that debias learning is helpful to recognize the class, but I am not sure that this learning strategy is also beneficial to 3D localization task.\n\n[-] According to Table.3, the authors argued that the 3D detector can be seen as general region proposal network. In other words, the proposed method can be highly affected by the quality of localization results from 3D detection model. Compared with images, point-cloud has many object shapes that are difficult to understand and define to the object due to occlusion, truncation etc. Moreover, since 3D object detection estimates the abstract scale of the object, the scale of the box could be biased for a certain class. Therefore, I doubt whether the 3D detector can play the same role as RPN of Faster-RCNN, which is trained by various scaled boxes and appearance or 2D object proposal methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject