Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the problem of novel class discovery and detection. The authors propose a two-stage object detection network called Region-based NCDL. The proposed approach uses region proposals and is trained to classify them as belonging to either the seen classes or one of the unseen/novel classes. 

Post-rebuttal - I have decided to update the rating to 7 after the authors' rebuttal to the initial reviews and updates to the paper.
strengths_and_weaknesses: The paper is mostly well written and the proposed approach achieves good results. I have the following questions/comments which the authors should address in the rebuttal and the updated submission.

1. In lines 114-115, the authors claim that in ZSL all the target classes are known and pre-determined. However, this is not true. Zero-shot learning methods do not need to assume knowledge of target classes apriori. 

2. Something that's not clear to me currently is how exactly are the semantic class assignments made under clustering. Without using additional language information, how do the authors assign a semantic meaning to the discovered classes? The authors should provide the details of this in their rebuttal and add it to the final version. 

3. In lines 185-187, the authors mention following [1] and [2] to generate cluster assignments. The authors should add some more details about these methods to make the paper more self-contained and easier to understand. 
questions: Please see the section above. 
limitations: The authors have added a very brief discussion on the limitations of the proposed approach.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes the RNCDL network for novel class discovery, detection, and localization (NCDL). The model is firstly supervised and trained on the source domain for known classes and then self-supervised trained on the target domain to discover unknown classes. During discovering, the self-supervision is guided by the feature of region proposals and a constraint for the novel class assignments to follow a long-tail distribution. Extensive experiments on COCO, LVIS, and Visual Genome demonstrate its strong generalizability.
strengths_and_weaknesses: Strength:
1. This work studies the problem of novel class discovery in the wild, which is insightful and valuable in terms of spurring future work.
2. The overall framework is intuitive and has good soundness, and extensive experiments demonstrate its effectiveness and good generalizability.

Weakness:
1. The idea of secondary classification head architecture is not novel. In [1] [2] they adopt a similar secondary classification head that parallels the known classification head and novel classification head. Can you clarify the difference between this work and theirs, and demonstrate the significance of the difference?
2. Table 3 shows the memory module contributes most to the performance, but the idea is still not very novel and similar to [3] [4], and the explanation about it (line 211 to line 218) and ablation experiment (line 308 to line 311) is too coarse, can you explain more? Line 215 says only the proposal feature from the last batch is stored in the memory, how about storing earlier proposal features?

[1] Fan Z, Ma Y, Li Z, et al. Generalized few-shot object detection without forgetting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[2] Cao Y, Wang J, Jin Y, et al. Few-Shot Object Detection via Association and DIscrimination[J]. Advances in Neural Information Processing Systems, 2021

[3] Caron M, Misra I, Mairal J, et al. Unsupervised learning of visual features by contrasting cluster assignments[J]. Advances in Neural Information Processing Systems, 2020

[4] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

questions: Please to refer the weakness section:
1. Please demonstrate the significance of the difference between several modules of this work and other similar works
2. Please add a more detailed explanation about the memory module and ablation study about the stored proposal features 

limitations: N/A
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a well-designed object detection pipeline to discovery novel objects given known objects annotations. The key ingredient is training the classification branch in roi-head with a combination of supervised loss and self-supervised loss, where the self-supervised loss is to force the object features into a set of clusters and these cluster ids could be used as the novel object category id. The experiment are well-organized, the quantitative and visual results are pretty promising.
strengths_and_weaknesses: Strengths:
- As far as I know, this work is one of the first works to study novel object detection by self-supervised technology and a well-qualified pipeline is set up accordingly.
- The novel object discovery is modeled as a clustering problem and solved by Sinkhorn Clustering algorithm. This is a good application of self-supervision on object detection. 
- The experiment settings are very convincing, including cross-category(COCO-LVIS) and cross-dataset(LVIS-Visual Genome) generalization. The visualization looks amazing. 

Weaknesses:
- The biggest weakness of this paper is its writing quality in the section of method. The authors didn't explain important details (see Questions below) and rely on many reference works. This is definitely wrong, because a good paper should be self-contained. 
- In Line 165, "we freeze all the layers apart from the primary (known) classification head". Does it mean the whole novel object discovery phase is transformed to a simple linear problem given a set of fixed object features? 
questions: As the statement in weakness, this paper misses some important details:
- In Line 9-10, "distribution of class assignments should follow natural long-tail distributions". In Line 191, "current-batch cluster assignments to follow Gaussian distribution" and in Line of 35 of supplementary material, the marginals distribution is set as N(1, 0.5)·M/N. Why is a Gaussian distribution equal to a long-tail distribution? I find a similar implementation in reference [1], and read [1] for many times, but still can't understand. Could the author provide some more clear explanations?
- In Line 195, "cluster assignments are then used to calculate cross-entropy loss". Is the cluster assignment the hard version (0/1) or soft version as in SwAV[10]?  Is the cross-entropy loss a softmax version or a sigmoid version?
- In Line 230, "a standard forward RCNN pass based on concatenated logits from both classification heads". Does the logits include background class? Since the background logit is removed and no background proposals are trained during the discovery phase, how does the network know which proposal is background in inference?
- Although the method section "self-supervision via swapping cluster assignments" is based on reference[1,10], the authors are supposed to use some formulas to set up the problem, as method section of reference[1,10]. 
- In Line 256, "we follow [...] to match predicted cluster assignments with annotated semantic classes". Is this a Hungarian matching? I suggest the authors to provide more details about how to match here, since [...] reference works are mostly about image classification, and this paper is about object detection.
-  By the way, do the authors have a plan to release the code? Novel object discovery is a very important problem in object detection, releasing the code will be a good contribution to the community,

I think this paper is an obviously-accept paper, but my initial score to this paper is 6. If the authors could clarify these details in rebuttal, I will increase my score to 7. Further, if the authors could further re-write the section of method and make it self-contained, I will increase my score to 8. 

References:
- [1] Yuki M. Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. In NeurIPS, 2020.
- [10] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
limitations: The authors didn't discuss the limitations of their work. However, it is ok for me, since this paper is one of the pioneering works in discovering novel object by self-supervised technology in object detection field.

This paper studies on object detection problem, which is a neutral computer vision technology and has no specific negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper investigates the problem of novel class discovery, detection, and localization, which trains a model with a labeled source dataset and unlabeled target datasets that can discover, classify and localize novel classes. This paper proposed a two-stage object detection network and outperforms traditional clustering algorithms and self-supervised contrastive learning methods.
strengths_and_weaknesses: **## Strengths ##**

1. The setting investigated in this paper is interesting and practical.
2. The proposed method achieves good performance on complex datasets.
3. Extensive ablation studies demonstrate the effectiveness of each component.

**## Weaknesses ##**

1. The novelty of the method is limited since the techniques introduced in this paper have been proposed by other NCD and clustering methods. For example, self-supervised discovery [17], second classification head [17], and swapping cluster assignments [1,2]. This paper only replaces the image features with RoI features in the detection task.

2. In Sec 4.2, this paper investigates the number of novel classes, COCO+LVIS setting only contains about 1200 classes, while 3000 novel classes are used. How to assign the 3000 classes to specific classes?

3. This paper leverages [1,2] for generating pseudo-labels. Kmeans is only compared as the baseline. What if using kmeans to generate pseudo-labels while still using other techniques?

4. In the supplement, the memory size is only 100 batches per GPU, but there exist 3000 novel classes. Is such a small memory enough?

5. What does the fully-supervised method in Tab. 5 denote? Is it trained on LVIS or VisualGanome? It is kind of unconvincing if the fully-supervised method is trained on VisualGanome.

questions: See weaknesses.
limitations: Yes.
review_title: null

Meta-review: metareview: The meta reviewer has carefully read the paper, reviews, rebuttals, and discussions. The meta reviewer appreciates the authors' efforts to respond and revise the paper. The authors did a good job of convincing the reviewers. The meta reviewer believes that open-world object detection using self-supervised learning is of interest to the community. And the authors clearly explain the motivation that object discovery is modeled as a clustering problem and solved by Sinkhorn clustering. The authors are suggested to polish the paper considering the reviewers' comments.
confidence: Certain
Decision: Accept