Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a new method to discover novel visual categories from unlabelled images. The experimental results demonstrate the effectiveness of the proposed method and achieve the state-of-the-art result on fine-grained visual recognition and object classification.
main_review: The paper utilizes dual ranking statistics to transfer knowledge from known classes in the labeled data to the unlabelled data and can generate a more robust pseudo label for learning on the unlabelled data. 

Besides, a mutual knowledge distillation method is proposed to allow information exchange and encourage agreement between the local and global branches.

The paper is well organized and easy to follow. The proposed model consists of two branches. Although the usage of the existing binary pseudo label limits the novelty of the model, the dual structure and the distillation to encourage agreement between branches are innovative. The global branch originates from [1] and the local branch includes a FIFO queue as a memory bank to query the ranking statistics as a dynamic object part dictionary.

Typo: a redundant close parenthesis in the rear of Eqn (8).

[1] Han K, Rebuffi S A, Ehrhardt S, et al. Automatically Discovering and Learning New Visual Categories with Ranking Statistics[C]//International Conference on Learning Representations. 2019.
limitations_and_societal_impact: Authors should consider adding more pictures to make the paper easier to read. And add some case studies of the classification results.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tackles novel class discovery of unlabeled data via transfer learning-based two-brach learning with pairwise pseudo-labels by two different criteria: similarity on global descriptor and local features. The distinct criteria generate more robust pseudo labels than the single ones and alleviate the negative impact from false pseudo-labels in a complementary manner. The proposed mutual knowledge distillation further facilitates effective mutual learning between two branches. Extensive experiments on standard classification datasets and more challenging fine-grained datasets show that their method has significant advantages over SOTA owing to robust pseudo-labels by dual rank statistics.

main_review: * The paper is overall well-written and well-organized with clear motivation.

* The setting of the novel class discovery is interesting and should be practically important.

* The experiments are comprehensive and the results show clear improvement over previous SOTA.

limitations_and_societal_impact: 1. I wonder why the authors do not conduct fine-tuning on labeled data before training with unseen unlabeled data. Does initialization with self-supervised learning generate more robust pseudo-labels than a fine-tuned model in the early training stage? and why? In a similar vein, what initialization is used for the *Ours w/o Self-sup* in Table 4?

2. Some experimental issues.
    * How did the authors conduct validation? Due to the existence of unseen classes in unlabeled data, how to conduct validation is not trivial.
    * Have different distillation approaches been investigated for mutual training? Such as the l2-regularization between $z_i$ and $z^\prime_i$.

3. Equation of Jensen Shannon Divergence (Eq. 6) is wrong. Jensen Shannon Divergence is $\operatorname{JSD}(p, q)=\frac{1}{2} D_{K L}\left(p \| \frac{p+q}{2}\right)+\frac{1}{2} D_{K L}\left(q \| \frac{p+q}{2}\right)$.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper considers the problem of novel class discovery (NCD). Specifically, the authors propose a Dual Ranking Statistics method that can leverage both global and local factors for learning novel classes. In addition, a Mutual Knowledge Distillation approach is proposed to explore the relationship between the global and local branches, which can further improve the results. Experiments on several datasets, especially on fine-grained cases, show the benefit of the proposed method.
main_review: Pros:

+ Writing: This paper is well written and easy to follow. The overall paper is well-organized. Figures and Tables are clear to show the methods and comparisons.

+ Novelty: This paper considers an interesting problem, i.e., novel class discovery. In addition, a fine-grained setting is introduced in this paper, which is not studied in previous NCD works. For the method, this paper introduces two simple but effective approaches, which explicitly take the advantage of both the global and local factors and obtain consistent improvement.

+ Experiments: This paper presents extensive experiments to show the effectiveness of the components of the proposed method. The comparisons with other methods are comprehensive and fair. In addition, the authors also provide the experiments on more difficult settings: unknown number of novel classes and open-set unlabeled data.

Cons/Concerns:

I have some questions about the method and the experiment.

- Did you use the proposed losses for the labeled data, especially the local loss? If no, how about the results of using them for the labeled data. Will this improve the discrimination of the representation?

- For the part dictionary memory bank, why not save all the parts of a sample into the memory? Could you provide the difference between using a random part of a sample and using all parts of a sample?  

- Saving random/all part features into the memory is a bit wild? Is it possible to first find high-response regions (for example the activations) and only save them into the memory? 

- The proposed has two branches, which one is used for the testing stage? With JSD loss, I think these two branches may have similar results. However, when removing JSD loss, which one is used? 

- In Table 6, it seems that a larger memory size brings higher results. Why not using the larger memory size for all other experiments? Will it lead to a very large computation cost?

- For the self-pretraining, how about the results of removing the self-pretraining? In addition, why use RotationNet for CIFAR while MOCOV2 for others? How about the results of using RotationNet/MOCO for others/CIFAR?

- Which dataset is evaluated in Table 1 of Supple. Is it CIFAR-100?

- Open questions: (1) In the open-set setting (Table 1 of Supple), did you use any technique to recognize the samples of known and unseen classes? If no, will the model assign all the samples to the known classes and give very few values/probabilities to the novel classes? (2) It would be nice that the authors can provide a visualization for the local comparison process. For example, we save many local features in the memory bank, when computing the similarity between each part of a sample and parts in the memory, we visualize the image rejoins of corresponding highly ranked parts. 

Overall, this paper considers an interesting problem and presents a novel and effective approach, which achieves the SOTA results. During rebuttal, please mainly answer the first 7 concerns. The authors can also selectively provide more discussions and visualizations for the open questions.

**Post Rebuttal** After reading the response of the authors, my concerns are well-addressed. I also read the comments of other reviewers and I agree with them that this paper presents a novel approach for novel class discovery. Thus, I decided to keep my original rating and tend to give 'accept' to this paper.


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new technique for visual class discovery, which relies on soft label assignment via ranking statistics, computed from both global features and local/part patterns. One of the most novel components is a JSD loss to align global/part features; other components include ranking statistics to compute soft pseudo labels, and MSE for consistency of predictions under different transformations. 
main_review: The approach seems sound. Some components are more novel (ranking statistics for labels, global/local alignment) and others are less novel. The contribution of each part seems properly verified through ablations. Results overall are strong enough, although margins to the baselines are not huge. In particular, both ranking statistics and global/local alignment are shown to have an impact. The particular manner of doing distillation, local-to-global, is also ablated. Multiple evaluation settings are shown, i.e. standard (coarse) classification as well as fine-grained.

Minor questions: 
- the s_{ij} notation is a bit strange--usually labels are for one sample?
- what is z_i^{'u} i.e. what is the ' here? (above Eq. 4)
- there shouldn't be a sentence break on L134
limitations_and_societal_impact: Not discussed, not necessary
review_title: null

Meta-review: This paper presents a novel class discovery technique based on dual ranking statistics and mutual knowledge distillation. The main idea is reasonable and shows superior performance compared to existing methods. All reviewers are positive about this paper and there are no particularly negative comments. However, the proposed method is lacking in theoretical support and the experiments are limited to small datasets and/or relatively a small number of unlabeled classes. Overall, I recommend accepting this paper for a poster presentation.
Decision: Accept (Poster)