Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel method for open-set 3D detection using image-level class supervision. The core idea is leveraging each of the image and point-cloud modalities to generate pseudo labels for unseen classes. To improve the positive and negative sample matching, the authors propose debiased cross-modal contrastive learning. The proposed approach takes two phases to train, where the first aims to train classification and localization models to generate pseudo labels and the second aims to further facilitate information transferring with the proposed contrastive learning strategy. 

The authors have evaluated their method on the SUN RGBD and ScanNet datasets.
strength_and_weaknesses: Strength:
1. The paper is well motivated to address an important topic in computer vision.

2. The debased cross-modal contrastive learning is interesting. 


Weakness:
1. Is the paper trying to localize and assign semantics simultaneously or just assign semantics, by assuming the localization is naturally generalizable to open-set? Please clarify.

2. What is the justification for the statement that the 3D detector is generalizable to unseen categories? 

3. How to weigh different terms in Eq. (1) and Eq. (2)?

4. The experiment setup is questionable. The authors propose to randomly select unseen and seen classes, which however unavoidably exists resemblance in semantics and shape. For example, for both SUN-RGBD and ScanNet, semantically similar concepts like desk and table are leaked in unseen categories. 

5. Is the unseen / seen category sampling strategy consistent with other referenced open-set methods, e.g., Zhang et al., Xu et al., Zhou et al.? 

6. The proposed method underperforms previous works in lamp, scanner and chair on the SUN -RGBD dataset.

7. In the ablation study, distance aware temperature seems to have marginal improvement in mAP but cause significant regression on AP. Please explain further. 

8. I would like to see the ablation study on the proposed DECC idea. 

9. Writing quality should be much improved. Typos and grammar errors are throughout the manuscript. 
clarity,_quality,_novelty_and_reproducibility: The paper has good clarity and quality. The proposed idea is novel. Given the introduction of the method and the experimental setup, the results are reproducible. 
summary_of_the_review: Overall, I think the idea of generating pseudo labels and debiased cross-modal contrastive learning is interesting. During the rebuttal, the authors have addressed my concerns. I would like to raise my rating to acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a way to transfer semantic information from Imagenet1K to train a 3D detector on point clouds to detect classes for which there are no 3D labels at all. 

The method trains a 2D detector (DETR) and a 3D detector (3DETR) in two stages. 
1) The 2D and 3D detector are co-trained, in line with Detic (Zhou 2022), with joint losses. The 3D labels (Sun/ScanNet) train the 3D detector, as well as 2D detector on 2D boxes derived from the 3D GT. The Imagenet labels train the DETR classifier head (applied on the largest detected box in the image). 
2) In Phase 2, an augmented dataset is created by running 3D detector to find "unseen classes". The 2D crops obtained from the detections are classifier by the 2D detector. Training continues also incorporating this augmented dataset, but adding contrastive training losses for the 2D crops, that take into account ImageNet classification results (crops with same class are considered positives). 
The process of generating the augmented dataset is iterated several times, taking more and more positive examples from the classifier. 

The method is evaluated on Sun and Scannet 3D datasets on a number of withheld classes. 




strength_and_weaknesses: Strengths: 
+ Seemingly novel use case of transferring 2D classification knowledge to a point cloud 3D detector. The choice of Imagenet1K seems suitable as a source of supervision, as the classes in question are present in Imagenet. 
+ The method is shown to ourperform several reasonable baselines on Sun and Scannet for a number of classes, in some cases by significant margins.  
+ Suitable ablations and studies of the effects of iterating the process are done. 

Potential weaknesses: 
- The start of related work section seems missing. It also claims "to the best of our knowledge, there has been no work on open-set 3D object detection". There is some related work that should be cited. For example Cen, J., Yun, P., Cai, J., Wang, M.Y., Liu, M.: Open-set 3d object detection. 3DV, 2021. Or Wong, K., Wang, S., Ren, M., Liang, M., Urtasun, R.: Identifying unknown instances for autonomous driving. In: CoRL. PMLR (2020). These are not quite the same, as they don't transfer class labels from a trained image classifier, however. 
- It is unclear how well the method does compared to full supervision. It would help to share how well DETR3 does on the classes when there is actually supervision. 
- Ablation shows that the first step of generating pseudolabels (mostly following the Detic paper) yields most of the gains. The application of "class-unbiased" contrastive training, or "distance-aware" temperature, which are technical details of some minor novelty, yield gains that seems quite small. 
- Sun RGB-D dataset contains also images, yet the transfer there is done purely on the 3D point cloud data. It would have been interesting to see how well we can do when also using the RGB data. 
clarity,_quality,_novelty_and_reproducibility: The paper is very clear, the supplement contains additional results and code has been provided, so the work is reproducible. 

In terms of novelty, the idea to transfer ImageNet1K label knowledge to a 3D detector seems novel. The approach itself is mostly a combination of existing approaches (DETR, DETR3, Detic, Contrastive learning) but the combination makes sense. 

Some things could be made clearer: 
- Description of phases and related notation is a bit cumbersome. 
- A little more clarity on how you generate 3D boxes for unseen classes in phase 2 would be helpful. If you trained a 3D detector classifier head for the seen classes in scene1, do you just take any objects where classification is anything other than 'background'? 
- "max-size proposal fmaxsize " is not particularly clear without having the Detic paper context. 

There are a few minor language issues: 
“Real world owns a cornucopia of classes” 
“Or using 2D detection dataset” 
“Ote that” 


summary_of_the_review: The application is interesting and novel. The system is a combination of mostly known components, but they are put together reasonably well. Related work section could be improved. I think it's important to show the gap between the OS3-DETIC system and full supervision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a framework for performing open vocabulary 3D object detection. The proposed framework mostly follows the 2D open vocabulary detector Detic[1] with 2D localizer replaced by 3D localizer. Different from Detic, the proposed framework introduced a contrastive feature learning module named debiased cross-modal contrastive learning(DECC) between 2D and 3D RoI features. Experiments on ScanNet and SUN RGB-D showed that the proposed method achieved superior results over existing methods and several baselines.
strength_and_weaknesses: Pros:
1. The paper is generally easy to follow and proposed framework could serve as a much better new baseline.
2. The results are strong compared with existing works.

Cons:
1. Some baselines settings are not quite fair. Taking Detic-ModelNet and Detic-ImageNet as examples, it is hard to understand why only the classifier of Detic are extended, as the proposed method itself extended both classifier and regressor from Detic.
2. The technical novelty is somewhat limited. The proposed framework is almost a Detic detector with 2D localizer replaced by 3D localizer. The proposed DECC improves only marginally to its pseudo label baseline as shown in Table 3.

Miscs:
1. Figure 4 has jammed y-axis legend texts for both sub-figures.

[1] Detecting Twenty-thousand Classes using Image-level Supervision, Zhou et al, ECCV 2022

clarity,_quality,_novelty_and_reproducibility: The writing is general clear.
I checked codes come with the supplementary materials but found no README files & config files which makes the reproduction of this work hard.
summary_of_the_review: Overall I think the proposed framework, while lacking technical novelty, could serve as a much stronger baseline compared with existing open vocabulary 3D detection methods. I am leaning to recommend for its acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors addressed an open-set 3D detection problem to broaden the vocabulary of the point-cloud detection without laborious and expensive data annotation. Inspired by previous open-set works, the authors proposed OS-3DETIC consisting of two main functions as classification using image-based model and localization using point-cloud model. Furthermore, to maximize the capability of transferring knowledge between two modalities, the authors proposed a de-biased cross-modal contrastive learning as an auxiliary task in the open-set detection problem. In experimental section, the proposed method showed the significant improvement in 3D unseen benchmark and is validated by various ablation studied provided.
strength_and_weaknesses: [+] First of all, the motivation of the paper seems to be meaningful and pragmatic for 3D object detection in the perspective of the generalized 3D object detection and difficulty of the 3D annotation procedure. The key idea is very intuitive how to obtain the pseudo label for 3D object detection and the overall framework seems to be well-designed to embody the author’s purpose. Even if the technical novelties is limited, the idea of de-bias contrastive learning approach seems to be reasonable for overcoming the limitation of position-based contrastive learning, which is a hard semantic relationship between defined pairs.

[-] One concern is an influence of de-bias learning to 3D localization. Unlike 2D localization problems, the orientation is included in 3D localization task. The orientation is known to bias to the part or certain positions of the object. As mentioned in the paper, if the ROI features is used for de-bias learning, it might conflict with the problem to estimate the orientation. I pretty agree that debias learning is helpful to recognize the class, but I am not sure that this learning strategy is also beneficial to 3D localization task.

[-] According to Table.3, the authors argued that the 3D detector can be seen as general region proposal network. In other words, the proposed method can be highly affected by the quality of localization results from 3D detection model. Compared with images, point-cloud has many object shapes that are difficult to understand and define to the object due to occlusion, truncation etc. Moreover, since 3D object detection estimates the abstract scale of the object, the scale of the box could be biased for a certain class. Therefore, I doubt whether the 3D detector can play the same role as RPN of Faster-RCNN, which is trained by various scaled boxes and appearance or 2D object proposal methods.

clarity,_quality,_novelty_and_reproducibility: The paper introduced a new open-set 3D object detection problem by cross-modal learning. Through technical novelty in the individual parts are limited, the overall framework seem to be interesting and perform well in provided experiments. Most of description seems to clear and be detailed for reproducing the proposed method.
summary_of_the_review: I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The paper proposes to use ImageNet1K to broaden the vocabulary of 3D object detector to tackle open-set object detection in point clouds.  A two phase training strategy is proposed for training a 2D detector and 3D detector.  In the first phase, the two detector are both trained together, with classification labels ImageNet used as weak supervisory signal (it is used as the label for the largest detected box in the image) for training the 2D detector (following Detic, Misra et al.).  Then in the second phase, the trained 2D detector is used to generate pseudo category labels for bounding boxes detected by the 3D detector for unseen classes.  The two detectors are again trained together, with a de-biased cross-modal contrastive loss.  Experiments on ScanNet and SUN-RGBD show that the proposed method can successfully detect unseen classes.

Strengths:
- The paper investigates the important setting of leveraging 2D data for training 3D object detectors where some classes do not have 3D labeled data
- The proposed method to transferring label knowledge from ImageNet combines prior approaches in a novel and practical way 
- The experiments show that proposed approach outperforms baselines

Weaknesses:
- The related work section is poor and hard to read (it consist mostly of a list of citations) without providing much information about the prior work and how this work is different from the prior work.
- The writing for the paper is poor in general.  The paper has a bunch of typos (Figure 1: ration => ratio), some grammar issues, and lots of places where parenthetical citation should be used (e.g. "3DETR Misra et al. (2021)" => "3DETR (Misra et al. 2021)".
justification_for_why_not_higher_score: The related work section is very poor and not really understandable.

Reviewers also had a lot of questions during the review cycle.  While the authors provided convincing responses, many of the clarification points did not seem to have made it into the updated revision.

The copyediting of the paper is just poor.  The paper has:
1. Tons of typos 
- Figure 1: ration => ratio, 
- Figure 3: Postive => Positive
- "from from" => "from" 
2. Grammar errors
- "..are also input into the same 2D detector with only image-level labels (category) are provided"
3. Lots of places where parenthetical citation should be used.    While this is a minor issue that could be fixed in a revision pass, it does impact the readability of this paper.  Currently some sections are just a long list of citations that are not properly parenthesized, and in many places, the name of the method just flows into the citation as well.

I would also recommend the equations be more properly typeset.
Decision: Reject