Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose a methodology to discover new categories in an unlabeled dataset with the help of a label one. The authors propose the following methodology. First bootstrap some features using self-supervised learning on labeled and unlabeled data. Then transferring the knowledge of the labeled data to the unlabeled one by supposing that the representations of both are similar, the similarity being a rank statistic. Then using this knowledge a joint supervised-unsupervised objective.

Q1. The paper is about discovering new visual classes. Section 2 mention that the number of classes C^u must be known "a priori". How do you tackle this limitation? Is there any heuristic, similar to the one found in the clustering literature, that could help?

Q2. Have other losses been investigated for the clustering head? Such as the triplet loss, or deep clustering loss rather than BCE?

I would suggest to report standard deviations as the experiments were repeated 10 times on random train-test splits.

I propose a weak accept. The paper is well written, the methodology is original, the experiments are convincing and the authors will release publicly the code. My main concern is Q1, which is eluded. Second, it would be nice to have an experiment illustrating the impact of the rank k when transferring knowledge. Is k=5 always good? If not, what could influence the best value?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper addresses the problem of clustering unseen classes. To learn a robust feature extractor, this paper proposes a multi-stage training framework, which leverages different supervised manners in each stage. Specifically, they initialize the network using the self-supervised learning on the union of all available data and then further finetune it using labelled data. Based on this, they propose the rank statistics which leverages the activation knowledge on labelled classes and rank the activated dimensions. Unseen data having similar rank results are clustered to obtain the initial pseudo labels. Finally, the network is jointly optimized with the ground-truth and generated pseudo labels (the pseudo ones will be updated during training). Extensive experiments on 5 datasets show that their method has significant advantages over SOTA owing to the learned robust feature extractor.

+Strengths:
1. The writing of this paper is satisfactory. Both the related works, motivations and technical details are clearly introduced. 
2. The experiments are solid. They evaluate their method on five popular object datasets and both ablation of each components and comparison with SOTA are shown in the paper.
3. This paper also shows that their method has good ability of avoiding forgetting of old (seen) classes, which may provide some insights about feature extraction for improving incremental learning. 

-Weaknesses:
1. Except for the experimental evaluation, what is the advantage of rank statistics over directly comparing feature vectors? Why robust?
2. Some experimental issues. a) how will the choice of k in top-k rank influence the performance? b) why the advantages of incorporating incremental learning on SVHN and CIFAR-10 are not obvious? c) why evaluation of incremental learning on CIFAR-100 is not well (acc difference between old and new is larger than other datasets). Besides, what is acc. performance of old classes with only labeled data for training.
3. Typos. In Sec.4, the writing of KCL (KLC) and MCL (MLC) is not consistent.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 

  This paper tackles the problem of unsupervised object discovery, whereby a labeled dataset must be leveraged in order to then cluster an unlabeled dataset with a set of unknown categories. The paper contributes three main ideas to succeed at this task, namely 1) use of self-supervised learning to initialize the representations in a way that doesn't bias them to the labeled data, 2) a robust rank-based metric to generate estimates of similarity/dissimilarity along with consistency-based regularization to improve optimization, and 3) Joint optimization/refinement using a combination of labeled/unlabeled losses, as well as ability to learn incrementally without forgeting the original labeled classes. Results are shown on a range of datasets including OmniGlot, ImageNet, CIFAR-10, CIFAR-100, and SVHN. The results demonstrate improvement over the current state of art for this task. 

  Overall, my current vote for this paper is a weak reject. The main reason is that the paper really combines a set of known methods (self-supervised learning, consistency-based regularization, and an ad-hoc training regimen. On the other hand, the paper is well-written and provides nice rigorous experiments showing clear improvements over state of art by porting these known techniques from different domains (self/semi-supervised learning). However, if satisfactory answers to questions below are given, I am willing to change my rating.

Main Argument

Strengths

  - Overall the paper tackles an interesting problem, and does so in a way that achieves good results beyond state of art. 

  - An ablation study is provided which shows the contribution of different parts of the method. 

  - The paper is very well-motivated, written, and methods are described nicely and succinctly.

Weaknesses

  - Clearly the methods employed are, by themselves, not novel and have been used for a range of other ML problem formulations. What is the clear contribution/novelty of the work?

  - While the paper raises an interesting motivation about not biasing feature learning by using self-supervised learning, it's not clear to me that this claim is justified. What is the evidence for this, besides better performance? While the ablation w/o self-supervised learning performs more poorly than everything combined, the other parts of the ablation (with self-supervised learning) also perform poorly. Clearly, there is some interaction between the different aspects of the method, but I am not sure what that is. Why is the full combination so much better than if any one thing is removed?

  - The method is very similar to KCL (e.g. loss (3) is the same and justified via a graphical model formulation in that paper). The paper does not really read like it is building off of that though, which seems a bit misleading. Do the authors believe there is additional novelty, or is it a matter of adding the three contributions to KCL? What is the major reason that self-supervision with KCL still doesn't do as well? As far as I can tell, the only difference is self-supervised learning, the fact that you have a manual 3-stage curriculum, and the robust ranking method/consistency loss (i.e. the three contributions).

  - You assume that the number of clusters is known; this is one of the advantages of all of the prior work in that they can estimate this. How well does the method work if the number of clusters is not known? 

Additional comments not related to final vote:

  - The paper seems to start out in a way that implies the problem is new; citations should be provided to the three main compared works in the intro to make it clear that this is a follow-on to an existing problem. Further, as mentioned above MCL/KCL (especially the latter) are very similar in nature and introduced this problem (including [1] which also used unsupervised feature learning in one condition). These should be included as "work most related to ours" given that they came before Han et al.

  [1] Deep Image Category Discovery using a Transferred Similarity Function, https://arxiv.org/abs/1612.01253.

  
review_title: Official Blind Review #1

Meta-review: The paper defines a methodology to discover unknown classes in a semi-supervised learning setting, based on: i) defining a proper representation based on self-supervision on all samples; ii) defining equivalence classes on the unlabelled samples, based on ranking statistics; iii) training supervised heads aimed to predict the labels (when available) and the equivalence class indices (when unlabelled). 

All reviewers agree that the ranking statistics-based heuristics is a quite innovative element of the paper. The extensive and careful experimental validation, with the ablation studies, establishes the merits of all ingredients. 

Therefore, I propose acceptance of this paper.

Decision: Accept (Poster)