Automatically Discovering and Learning New Visual Categories with Ranking Statistics
Authors: Kai Han
Sylvestre-Alvise Rebuffi
Sebastien Ehrhardt
Andrea Vedaldi
Andrew Zisserman
Venue:  (2020)
Keywords: deep learning
classification
novel classes
transfer learning
clustering
incremental learning

Abstract: We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. This setting is similar to semi-supervised learning, but significantly harder because there are no labelled examples for the new classes. The challenge, then, is to leverage the information contained in the labelled images in order to learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data. In this work we address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labeled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning to train the representation from scratch on the union of labelled and unlabelled data; (2) we use rank statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) we train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data. We evaluate our approach on standard classification benchmarks and outperform current methods for novel category discovery by a significant margin.

Published as a conference paper at ICLR 2020
AUTOMATICALLY DISCOVERING AND LEARNING NEW VISUAL CATEGORIES WITH RANKING STATISTICS
Kai Han" Sylvestre-Alvise Rebuffi
Sebastien Ehrhardt"
Andrea Vedaldi Andrew Zisserman
Visual Geometry Group, Department of Engineering Science, University of Oxford (khan, srebuffi. hyenal, vedal dii, az a2 robotts.........oo exx.ac.u
ABSTRACT
We tackle the problem of discovering novel classes in image collection given labelled examples of other classes. This setting is similar to semi-supervised learning, but significantly harder because there are no labelled examples for the new classes. The challenge, then, is to leverage the information contained in the labelled images order to learn a purposs clustering model and use the latter to identify the new classes in the unlabelled data. In this work we address this problem by combining three ideas: (1) we suggest that the common approach of bootstrapping an image representation using the labeled data only introduces an unwanted bias, and that this can be avoided by using self-supervised learning train the representation from scratch the union of labelled and unlabelled data; (2) we use rank statistics to transfer the model's knowledge of the labelled classes to the problem of clustering the unlabelled images; and, (3) train the data representation by optimizing a joint objective function on the labelled and unlabelled subsets of the data, improving both the supervised classification of the labelled data, and the clustering of the unlabelled data. We evaluate our approach standard classification benchmarks and outperform current methods for novel category discovery by significant margin.
INTRODUCTION
Modern machine learning systems can match or surpass human-level performance in tasks such as image classification (Deng et al. (209). but at the cost of collecting large quantities of annotated training data. Semi-supervised learning (SSL) (Oliver et al. 2018) can alleviate this issue by mixing labelled with unlabelled data, which is usually much cheaper to obtain. However, these methods still require some annotations for each of the classes that one wishes to learn. We argue this is not always possible in real applications. For instance, consider the task of recognizing products in supermarkets. Thousands of new products are introduced in stores every week, and it would be very expensive to annotate them all. However, new products do not differ drastically from the existing ones, it should be possible to discover them automatically as they arise in the data. Unfortunately, machines are still unable to effectively learn new classes without manual annotations.
In this paper, we thus consider the problem of discovering new visual classes automatically, assuming that certain number of classes are already known by the model (Hsu al., 2018 2019 Han et al.) 2019). This knowledge comes in the form of a labelled dataser of images for certain set of classes. Given that this data is labelled, very effective classifier for the known classes, particularly if Convolutional Neural Networks (CNNs) are employed. However, this does not mean that the learned features are useful as representation of the new classes. Furthermore, even if the representation transfers well, one still has the problem of identifying the new classes in an unlabelled dataset, which a clustering problem.
supervised learning techniques can be used to train
We tackle these problems by introducing novel approach that combines three key ideas (section and fig. [].. The first idea is to pre-train the image representation (a CNN) using all available images, both labelled and unlabelled, using self-supervised learning objective. Crucially, this objective does not leverage the known labels, resulting in features that are much less biased towards the labelled
indicates equal contribution

---

Published as a conference paper at ICLR 2020
STEP 1
STEP
STEP 3
Learning low-lovel ures Seif Self-superiiiion
Leacuing higher lovel feetures: Full-supeviiioon
Leveraging soquired knowledge Joint optimization
Figure I: Overview of our three step learning pipeline. The first step of the training consists in learning unbiased image representation via self-supervision using both labelled and unlabelled data, which learns well the early layers of the representation; in the second step, we fine-tune only the last few layers of the model using supervision on the labelled set; finally, the fine-tuned representation is used, via rank statistics, to induce clusters in the unlabelled data, while maintaining a good representation on the labelled set.
classes. Labels are used only after pre-training to learn classifier specific to the labelled data as well as to fine-tune the deepest layers the CNN, for which self-supervision not as effective.
The second idea is to the problem of clustering the unlabelled ones. Information is transferred by sharing the same representation between labelled and unlabelled images, motivated by the fact that the classes are often similar to the known ones. In more detail, pairs of unlabelled images are compared via their representation vectors. The comparison is done using robust rank statistics, by testing if two images share the same subset of & maximally activated representation components. This test is used decide if unlabelled images belong the same (new) class or not, generating set of noisy pairwise pseudo-labels. The pseudo-labels are then used to learn similarity function for the unlabelled images.
approach to transfer the information contained in the labelled images
The third and final idea is, after bootstrapping the representation, optimise the model by minimizing a joint objective function, containing terms for both the labelled and unlabelled subsets, using respectively the given labels and the generated pseudo-labels, thus avoiding the forgetting issue that may arise with a sequential approach. A further boost is obtained by incorporating incremental learning of the discovered classes in the classification task, which allows information flow between the labelled and unlabelled images.
We evaluate our method several public benchmarks (section 3), outperforming by large margin all existing techniques (section 4 that can be applied to this problem, demonstrating the effectiveness of our approach. We conclude the paper by summarizing our findings (section (5). Our code can be found at http: //pp://wwwwww// robots ox ac wt/p:////////eeeacc//////////
novel
METHOD
Given an unlabelled dataset D [ i automatically cluster the images into number of classes C', which we assume to be known priori. We also assume have a second labelled image dataset D' {( yf), i 1..... ... where vl E .1,..., , C²) } is the class label for image a/e- We also assume that the set of Ct labelled classes is disjoint from the set of C unlabelled ones. While the statistics of D' and h thus differ, hypothesize that a general notion of what constitutes a "good class" can be extracted from D and that the latter can be used better cluster D
I,..., M) of images x. E xx our goal is to
We approach the problem by learning an image representation & x r + (xx E Idd in the form of a CNN. The goal of the representation to help recognize the known classes and discover the new ones. In order to learn this representation, we combine three ideas, detailed in the next three sections.
2.1 SELF-SUPERVISED LEARNING
Given that we have a certain number of labelled images D at our disposal, the obvious idea is to these labels to bootstrap the representation & by minimizing standard supervised objective such as the cross-entropy loss. However, experiments show that this causes the representation to overly-specialize for the classes in D', providing poor representation of the new classes in D".
2

---

Published as a conference paper at ICLR 2020
Thus resist the temptation of using the labels right away and use instead self self-supervised learning method to bootstrap the representation O. self-supervised learning has been shown (Kolesnikov et al. 2019; Gidaris et al., 2018) to produce robust low-level features, especially for the first few layers of typical CNNs. It has the benefit that no data annotations are needed, and thus it can be applied to both labelled and unlabelled images during training. In this way, we achieve the key benefit of ensuring that the representation is initialized without being biased towards the labelled data.
In detail, we first pre-train our model & with self-supervision on the union of D' and D (ignoring all labels). We use the RotNet (Gidaris et al. 2018) approact due to its simplicity and efficacy, but any self-supervised method could be used instead. We then extend the pre-trained network with a classification head n/ IRd + Re implemented as a single linear layer followed by a softmax layer. The function r/' o is fine-tuned on the labelled dataset D' in order to learn classifier for the C known classes, this time using the labels yi and optimizing the standard cross-entropy (CE) loss:
LCE = N
log my (2()
(1)
where 2 = ((xt) Rd is the representation of image at. Only n/' and the last macro-block of +b (section are updated in order to avoid overfitting the representation to the labelled data.
2.2 TRANSFER LEARNING VIA RANK STATISTICS
Once the representation & and the classifier n/' have been trained, we are ready to look for the new classes in D Since the classes in D are unknown, we represent them by defining a relation among pairs of unlabelled images (z. x²)) ). The idea is that similar images should belong to the same (new) class, which we denote by the symbol 8t = 1, while dissimilar ones should not, which we denote by Sij- 0. The problem is then to obtain the labels 8ij-
Our assumption is that the new classes will have some degree of visual similarity with the known ones. Hence, the learned representation should be applicable to old and new classes equally well. As a consequence, we expect the descriptors z = (xx") and x. = ((x)) of two images 2². x from the new classes to be close if they are from the same (new) class, and to be distinct otherwise.
Rather than comparing vectors =²", 2 directly (e-g., by a scalar product), however, we use a more robust rank statistics. Specifically, we rank the values in vector 2 by magnitude. Then, if the rankings obtained for two unlabelled images x." and z are the same, they are very likely to belong to the same (new) class, so we set šij 1. Otherwise, we set S-y O. In practice, it is too strict to require the two rankings to be identical if the dimension of 2 is high (otherwise we may end up with relax this requirement by only testing if the sets of the top-k ranked dimensions are the same (we use k = in our experiments), i.e.:
sij 0 for all pairs (i, + j). Therefore,
Sij = (topp(((z)) = top, ( (*x))))
(2)
where tops Rd + P([1..... d}) associates to vector the subset of indices {1...., d} of its top-k elements.
Once the labels Siy have been obtained, we use them pseudo-labels to train comparison function
for the unlabelled data. In order to do this, we apply a new head 17" Rd + C to the image representation 2!¹ ((x(() to extract a new descriptor vector 17 (),") optimized for the unlabelled data. As in section 2.T the head is composed of linear layer followed by softmax. Then, the inner
product 17" (21)) (2 is used as a score for whether images x. and zy belong to the same class or
not. Note that 17" (z,") ) is normalized vector due to the softmax layer in 17" This descriptor is trained by optimizing the binary cross-entropy (BCE) loss:
MM
M
LBCE = M
1
Σ 1=1
Ss log n7" TT"")""T"""""""""""""""""""""" (2])) + (1 s.y) log(1 (1(1-10 (x()) (<("""""((())) (=])))
(3)
1=
We present to the network & randomly-rotated versions Hx of each image and task it with predicting H. The problem is formulated as a 4-way classification of the rotation angle, with angle in (0",90", 180", 270" }. The model 4b( Rx) is terminated by single linear layer with 4 outputs each scoring hypothesis. The parameters of 1 and & are optimized by minimizing the cross-entropy loss on the rotation prediction.
3

---

Published as a conference paper at ICLR 2020
Furthermore, we structure 1) in particular manner: We set its output dimension to be equal to the number of new classes C In this manner, we can use the index of the maximum element of each vector 1/" argmax, (n" (b(a")ly as prediction D. for the class of image xy (as opposed to assigning labels via a clustering method such as k-means).
2.3 JOINT TRAINING ON LABELLED AND UNLABELLED DATA
We now have two losses that involve the representation Φ: the CE loss LCE for the labelled data D' and the pairwise BCE loss LBCE for the unlabelled data D They both share the same image embedding 4. This embedding can be trained sequentially, first on the labelled data, and then on the unlabelled data using the pseudo-labels obtained above. However, in this way the model will very likely forget the knowledge learned from the labelled data, which is known catastrophic forgetring in incremental learning (Rebuffi et al., 2017 Lopez-Paz & Ranzato 2017 Shmelkov et al. 2017 Aljundi et al. 2018).
In contrast, we jointly fine-tune our model using both losses at the same time. Note that most of the model & is frozen; we only fine-tune the last macro-block of together with the two heads 11" and ny. Importantly, as we fine-tune the model, the labels Sij are changing at every epoch as the embedding yf is updated. This in turn affects the rank statistics used to determine the labels 8ij as explained in section 2.2 This leads "moving target" phenomenon that can introduce some instability in learning the model. This potential issue is addressed in the next section.
2.4 ENFORCING PREDICTIONS TO BE CONSISTENT
In addition to the CE and BCE losses, we also introduce consistency regularization term, which is used for both labelled and unlabelled data. In semi-supervised learning (Oliver et al. 2018 Tarvainen & Valpola| 2017 Laine & Aila, 2017), the idea of consistency is that the class predictions image J and on a randomly-trans ferran counterpart lx (for example an image rotation) should be the same. In case, as will be shown in the experiments, consistency is very important to obtain good performance. One reason is that, as noted above, the pairwise pscudo-labels for the unlabelled data are subject to change on the fly during training. Indeed, for an image x/ and randomly-transsfn transfomeed counterpart tx:", if we do not enforce consistency, we can have (opx)) F top (& (tx)") )). According to eq. (2) defining Sij, it could result in different Sty for xN depending on the data augmentation applied to the image. This variability of the ranking labels for given pair could then confuse the training of the embedding.
an
Following the common practice in semi-supervised learning, we use the Mean Squared Error (MSH) as the consistency cost. This is given by:
N
M
LMSSE
Σ n/ +
1 M
Σ (1(()))
(4)
where 2 is the representation of tæ.
The overall loss of our model can then be written as
LCB + LBCE + w()
(5)
where the coefficient w() is a ramp-up function. This is widely used in semi-supervised learn- ing (Laine & Aila 2017, Tarvainen & Valpola 2017). Following (Laine & Aila, 2017, Tarvainen & Valpola 2017). we use the sigmoid-shaped function w(t) = Ae -(155(111 4)" where is current time step and T is the ramp-up length and X R...
2.5 INCREMENTAL. LEARNING SCHEME
We also explore a setting analogous to incremental learning. In this approach, after tuning on the labelled set (end of section 2. we extend the head n to C" new classes, so that v/ Rd - CCC +o" The head parameters for the new classes are initialized randomly. The model is then trained using the same loss eq. (5). but the cross-entropy part of the loss is evaluated on both labelled and unlabelled data D' and D Since the cross-entropy requires labels, for the unlabelled data we use the pseudo-labels 0y" which are generated on-the-fly from the head 17" at each forward pass.
The advantage is that this approach increments 'f' to discriminate both old and new classes, which is often desirable in applications. It also creates a feedback loop that causes the features 2ū to be
4

---

Published as a conference paper at ICLR 2020
Table I:
Ablation study. "woo Con." means without consistency constraints;
CIFAR-10
CIFAR-100
SVHN
Ours w/o Con
82.6412..0%
61.8:1.66%
1.93
"w/o CE" means without the cross en- tropy loss for training on labeled data.
Ours w/o CE
84.711..4%
58.41...%%
59.716.6%
Ours w/o BCE
26.2.12.0%
6.6.10.7%
24.510.5%
"w/o BCE" means without binary cross
Ours w/o S.S.
89.4=1.4%
67.4+200%
72.9=5.0%
entropy loss for training on unlabeled
Ours full Ours w/ L.L.
90.4.1055% 0.5%
73.2.12.1%
95.0.1
data; "w/o S.S." means without self- supervision.
91.7.10.9%
75.2.14.2%
95.2.1
refined, which in turn generates better pseudo-labels Df for D" from the head ny". In this manner, further improvements can be obtained by this cycle of positive interactions between the two heads during training.
EXPERIMENTS
3.1 DATA AND EXPERIMENTAL DETAILS
We evaluate our models on a variety of standard benchmark datasets: CIFAR-10 (Krizhevsky & Hin-
ton 2009), CIFAR-100 (Krizhevsky & Hinton 2009), SVHN (Netzer et al. 2011), OmniGlot (Lake al. 2013), and ImageNet (Deng et al. 2009). Following Han et al. (2019). we split these to have classes respectively in the unlabelled set. In addition, for OmniGlot and ImageNet we use 20 and different splits respectively, as in Han et al. (2019), and report average clustering
accuracy. More details on the splits can be found in appendix
Evaluation metries. We adopt clustering accuracy (ACC) evaluate the clustering performance of our approach. The ACC is defined as follow:
N Σ - (m)) N
max RESym
where V; and y; denote the ground-truth label and clustering assignment for each data point a. E D respectively, and Sym L) is the group permutations of L elements (this discounts the fact that the cluster indices may not be in the same order as the ground-truth labels). Permutations are optimized using the Hungarian algorithm (uuhn 1955).
Implementation details. We use the ResNet-18 (He et al. 2016) architecture, except for OmniGlot for which we use VGG-like network (Simonyan Zisserman 2015) with six layers to make our setting directly comparable to prior work. We use SGD with momentum (Sutskever et al. 2013) as optimizer for all but the OmniGlot dataset, for which we use Adam (Kingma & Ba (2014). For all experiments we use batch size of 128 and k = 5 which we found worked consistently well across datasets (see appendix D More details about the hyper-parameters can be found appendix
3.2 ABLATION STUDY
We validate the effectiveness of the components of our method by ablating them and measuring the resulting ACC the unlabelled data. Note that, since the evaluation is restricted to the unlabelled data, we are solving clustering problem. The same unlabelled data points are used for both training and testing, except that data augmentation (i.e. image transformations) is not applied when computing the cluster assignments. As can be seen in table [ all components have significant effect as removing any of them causes the performance to drop substantially, Among them, the BCE loss by far the most important one, since removing it results in a dramatic drop of 40-60% absolute ACC points. For example, the full method has ACC 90.4% on CIFAR-10, while removing BCE causes the ACC to drop to 26. 2%. This shows that that our rank- embedding comparison can indeed generate reliable pairwise pseudo labels for the BCE loss. Without consistency, cross entropy, or self-supervision, the performance drops by more modest but still significant 7.8%, 5.7% and 1.0% absolute ACC points, respectively, for CIFAR-10. It means that the consistency term plays a role as important as the cross-entropy term by preventing the "moving target" phenomenon described in section 2.4 Finally, by incorporating the discovered classes in the classification task, we get a further boost of 1.3%, 2.0% and 0.2% points CIFAR-10, CIFAR- 100 and SVHN respectively.
5

---

Published as a conference paper at ICLR 2020
Table 2: Novel category discovery results on CIFAR-10, CIEAR-100, and SVHN. ACC on the unlabelled set. "w/ S.S." means with self-supervised learning.
No
CIFAR-10
CIFAR-100
SVHN
k-means (MacQueen
1967)
65.5++0.0 %
56.6+
42.6%
(2) KCL [su el al.
2018)
66.5113.9%
14.3111.3%
21.4%:00.6
(3) MCL Hssu el al.
2019
64.2:10.19
21.3:13.4%
38.6%
(4) DTC lan et al.
2019)
87.51..3%
5..711..2%
60.9%
(5) k-means (MacQueen
1967) w/ S.S.
72.5.10.0%
1..311
46.7.10.0%
(6) KCL [Isu el al.
2018)
w/ S.S.
72.3110.2%
42.11
65.661..9%
MCL Hssu el al.
2019)
w/ S.S.
70.9:10.1%
21.5:1...3%
53.1110.3%
(8) DTC Han et al.
2019
w/ S.S.
88.7+:0.3%
1.23%
75.7=0.4%
Ours
90.4:1:0.5%
73.2.1.2.1%
95.0=10.2%
(10)
Ours w/ I.L.
9..7:+0.9%
4..2%1...2%
95.2 0.2%
Figure 2: Evolution of the t-SNE
during the training of CIFAR-10. Performed on unlabelled data (i.e.. instances of dog, frog, horse, ship, truck). Colors data points denote their ground-truth labels.
(a) init
(b) epoch 30
(c) epoch 90
3.3 NOVEL CATEGORY DISCOVERY
We compare our method to baselines and state-of-the-art methods for new class discovery, starting from CIFAR-10, CIFAR- 100, and SVHN in table 2 The first baseline (row 5 in table 2 amounts to applying k-means (Maquueen 1967) to the features extracted by the fine-tuned model (the second step in section 2.1) for which use the k-means++ (Arthur & Vassilvitskii 2007) initialization. The second baseline (row 1 in table 2 is similar, but uses as feature extractor a model trained from scratch using only the labelled images, which corresponds to standard transfer learning setting. By comparing rows 1, 5 and 9 in table 2 we can see that our method substantially outperforms k-means. Next, compare with the KCL Hsu et al. 2018), MCL (Hsu al., 2019) and DTC (Han al.) 2019) methods. By comparing rows 2 to 9, we see that our method outperforms these by large margin. We also try improve KCL, MCL and DTC by using the same self-supervised initialization we adopt (section 2.1. which indeed results in an improvement (rows 2-4 vs 6-8). However, their overall performance still lags behind ours by a large margin. For example, our method of section [2.4 achieves 95.0% ACC on SVHN, while "KCI. w/ S.S.". "MCL. w/ S.S." and "DTC w/ S.S." achieve only 65.6%, 53.1% and 75.7% ACC, respectively. Similar trends hold for CIFAR-10 and CIFAR- 100. Finally, the incremental learning scheme of section 2.5 results in further improvements, as can be seen by comparing rows and 10 of table
In fig. 2 show the evolution of the learned representation the unlabelled data CIFAR- 10 using t-SNE (van der Maaten & Hinton 2008). As can be seen, while the clusters overlap in the beginning, they hecome more and more separated as the training progresses, showing that model can effectively discover novel visual categories without labels and learn meaningful embeddings for them.
We further compare our method to others on two more challenging datasets, OmniGlot and ImageNet, in table 3 For OmniGlot, results are averaged over the 20 alphabets in the evaluation set (see ap- pendix Aj; for ImageNet, results are averaged over the three 30-class unlabelled sets used in (Hsu al. 2018: Since we have a relatively larger number of labelled classes in these two datasets, we follow (Han et al. 2019) and use metric learning on the labelled classes to pre- pre-train the feature extractor, instead of the self-supervised learning. We empirically found that self-supervision does not provide obvious gains for these two datasets. This is reasonable since the data in the labelled sets of these two datasets are rather diverse and abundant, so metric learning can provide good feature initialization as there is less class-specific bias due to the large number of pre-training classes. However, by comparing rows and in table 3 is clear that metric learning alone is not sufficient for the task of novel category discovery. Our method substantially outperforms the k-means results obtained using the features from metric learning by 11.9% and 10.6% on OmniGlot and ImageNet respectively. Our method also substantially outperforms the current state-of-the-an achieving 89.1%

---

Published as a conference paper at ICLR 2020
No
OmniGlot
ImageNet
(1)
k-means (MaQQueen
1967)
77.2%
71.9%
Table Novel category discovery
(2)
KCL Hsü al.
2018)
82.4%
73.8%
results on OmniGlot and ImageNet.
(3)
MCL Hss ot al.
2019
83.3%
74.4%
ACC on the unlabelled set.
(4)
DTC (Han et al.
2019
89.0%
78.3%
(5)
Ours
89.1%
82.5%
Figure 3: t-SNE on CIFAR-10: impact of incre- mental Learning, (a) representation on the la- belled and (b) unlabelled CIFAR classes Colors of data points denote their ground-truth labels. We observe a bigger overlap in (a) between the "old" class 3 and the "new" class 5 when not incorporat- ing Incremental Learning.
(a) Ours
(b) + incr. learning
and 82.5% ACC on OmniGlot and ImageNet respectively, compared with 89.0% and 78.8% of HHan
al. 2019), thus setting the new state-of-the aftt
3.4
INCREMENTAL LEARNING
Table 4: Incremental Learning with the novel categories. "old" refers the ACC on the labelled classes while "new" refers to the unlabelled classes in the testing ser. "all" indicates the whole testing set. should be noted that the predictions are not restricted to their respective subset. Standard
deviation can be found in appendix C
CIFAR-IN 10
CIFAR- 100
SVHN
Classes
old
new
all
old
new
all
old
new
all
KCL w/ S.S.
79.4%
60.1%
69.8%
23.4%
29.4%
24.6%
90.3%
65.0%
81.0%
MCL w/ S.S. 81.4%
64.8%
73.1%
18.2%
18.0%
18.2%
94.0%
48.6%
77.2%
DTC w/ S.S.
58.7%
78.6%
68.7%
47.6%
49.1%
47.9%
90.5%
72.8%
84.0%
Ours w/ LL.
90.6%
88.8%
89.7%
71.2%
56.8%
68.3%
96.3%
96.1%
96.2%
Here, we further evaluate our incremental scheme for novel category discovery as described in sec- tion 2.5 Methods for novel category discovery such as (Han et al.) 2019 Hss al., [219]] 2018) focus on obtaining the highest clustering accuracy for the new unlabelled classes, but may forget the existing labelled classes in the process. In practice, forgetting not desirable as the model should be able to recognize both old and new classes. Thus, we argue that the classification accuracy on the labelled classes should be assessed as well, as for any incremental learning setting. Note however that our setup differs substantially from standard incremental learning (Rebuffi et al. 2017] Lopez-Paz & Ranzato 2017 Shmelkov et al., 2017) Aljunni et al., 2018) where every class is labelled and the focus is on using limited memory. In our case, memory constraints, but the new classes are unlabelled, which is often encountered in applications.
store and access the original data without
By construction (section 2.5 ) our method learns the new classes on top of the old ones incrementally, out of the box. In order to compare to methods such as KCL, MCL and DTC that do not have this property, we proceed as follows. First, the method runs as usual to cluster the unlabelled portion of the data, thus obtaining pseudo-labels for it, and learning feature extractor as byproduct. Then, the feature extractor is used to compute features for both the labelled and unlabelled training data, and linear classifier is trained using labels and pseudo- labels, jointly on all the classes, old and new.
We report in table the performance of the resulting joint classifier networks on the testing set of each dataset (this is now entirely disjoint from the training set). Our method has similar performances on the old and new classes for CIFAR-10 and SVHN as might be expected as the split between old and new classes is balanced. In comparison, the feature extractor learned by KCL and MCL. works much better for the old classes (e.g., the accuracy discrepancy between old and new classes is 25.3% for KCL on SVHN). Conversely, DTC learns features that work better for the new classes, as shown by the poor performance for the old classes on CIFAR-10. Thus, KCL, MCL and DTC learn representations that are biased to either the old or new classes, resulting overall in suboptimal performance. In contrast, our method works well on both old and new classes; furthermore, drastically outperforms existing methods both.
7

---

Published as a conference paper at ICLR 2020
4 RELATED WORK
Our work draws inspiration from semi-supervised learning, transfer learning, clustering, and zero-shot learning. We review below the most relevant contributions.
In semi-supervised learning (SSL) (Chapelle et al., 2006), a partially labelled training dataset is given and the objective is to leam a model that can propagate the labels from the labelled data to unlabelled data. Most SSL methods focus on the classification task where, usually both labelled and unlabelled points belong to the same set of classes. On the contrary, our goal to handle the case where the unlabelled data classes differ from the labelled data. Oliver et al. (2018) summarizes the state-of-the-are SSL methods. Among them, the consistency-badt -based methods appeared to be the most effective. |Rasmus et al. (2013) propose ladder network which is trained on both labelled and unlabelled data using a reconstruction loss. [Line & Aila (2017) simplifies this ladder network by enforcing prediction consistency between data point and its augmented counterpart. As an alternative to data augmentation, they also consider a regularization method based on the exponential moving average (EMA) of the predictions. This idea is further improved by Tarvainen & Valpola (2177): instead of using the EMA of predictions, they propose to maintain the EMA of model parameters. The consistency then measured between the predictions of the current model (student) and the predictions of the EMA model (teacher). More recently (and closer to our work) practitioners have also combined SSL with self=supervision(Reeuif et al.. 2019 Zhai et al., 2019) to leverage dataset with very few annotations.
Transfer learning (Pan & Yang 2010 Weiss et al., 2016, Tan et al.) 2018) is an effective way to reduce the amount of data annotations required to train model by pre-training the model on an existing dataset. In image classification,, for example, it is customary to start from model pre-trained on the ImageNet (Deng al. 2009) dataset. In most transfer learning settings, however, both the source data and the target data are fully annotated. In contrast, our goal is to transfer information from labelled dataset to unlabelled one.
Many classic (e.g.. Aggarwal & Reddy (2013) MacQueen 1967)): ): Comaniciu & Meer (1979): Ng
et al. (200T) and deep learning (e.g.. Xie al. 2016) Chang al. (2017): Dizaji et al. (0117) Yang
et al. 2017 2016); Hsu et al. (2018) 2019)) c clustering methods have been proposed to automatically
partition an unlabelled data collection into different classes. However, this task is usually ill ill-posed as there are multiple, equally valid criteria to partition most datasets. We address this challenge by learning the appropriate criterion by using a labelled dataset, narrowing down what constitutes a proper class. We call this setting "transfer clustering".
To the best of our knowledge, the work most related ours are (Hsu et al. 2018 2019 Han et al. 2019). Han et al. (2019) also consider discovering new classes as transfer clustering problem. They first learn a data embedding by using metric learning on the labelled data, and then fine-ture tune the embedding and learn the cluster assignments on the unlabelled data. In (Hsu et al. 2018 2019). the authors introduce KCL and MCL clustering methods. In both, a similarity prediction network (SPN), labelled dataset. Afterwards, the pre-trained SPN
also used in (Hsu et al. 2016), is first trained is used to provide binary pseudo labels for training the main model on an unlabelled dataset. The overall pipelines of the two methods are similar, but the losses differ: KCL uses Kullback-Leibler divergence based contrastive loss equivalent to the BCE used in this paper (eq. (3) ). and MCL uses the Meta Classification Likelihood loss. Zero-shot learning (ZSL) (Xian al. 2018 Fu al., 2018) can also be used to recognize new classes. However, differently from our work, ZSL also requires additional side information (e.g.. class attributes) in addition to the raw images.
Finally, other works (Dean et al. 2013 Yagnik et al. 2011) discuss the application of rank statistics to measuring the similarity vectors; however, to the best of our knowledge, we are the first apply rank statistics to the task of novel category discovery using deep neural networks.
CONCLUSIONS
In this paper, we have looked at the problem of discovering new classes in an image collection, leveraging labels available for other, known classes. We have shown that this task can be addressed very successfully by a few new ideas. First, the use of self superviedd learning for bootstrapping the image representation trades off the representation quality with its generality, and for problem this leads to a better solution overall. Second, we have shown that rank statistics are an effective method compare noisy image descriptors, resulting in robust data clustering. Third, we have
8

---

Published as a conference paper at ICLR 2020
shown that jointly optimizing both labelled recognition and unlabelled clustering in incremental learning setup can reinforce the two tasks while avoiding forgetting. On standard benchmarks, the combination of these ideas results in much better performance than existing methods that solve the same task. Finally, for larger datasets with more classes and diverse data (e.g., ImageNet) we note that self-supervision enough representation. In such cases, we still show that the rank statistics for clustering gives drastic improvement over existing methods.
be bypassed as the pretraining labelled data already provides powerful
6 ACKNOWLEDGMENTS
This work is supported by the EPSRC Programme Grant Seebibyte EP/M013774/1, Mathworks/DTA DFR02620, and ERC IDIU-638009.
REFERENCES
Charu C. Aggarwal and Chandan K. Reddy. Data Clustering: Algorithms and Applications. CRC Press, 2013.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018.
David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. In ACM-SIAM symposium on Discrete algorithms, 2007.
Jianlong Chang, Lingfeng Wang, Gaofeng Meng. Shiming Xiang, and Chunhong Pan. Deep adaptive image clustering. In ICCV, 2017.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-Supervised Learning. MIT Press, 2006.
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE TPAMI, 1979.
Thomas Dean, Mark A. Ruzon, Mark Segal, Jonathon Shlens, Sudheendra Vijayanarasimhan, and Jay Yagnik. Fast, accurate detection of 100,000 object classes single machine. In CVPR, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jua Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In ICCV, 2017.
Yanwei Fu, Tao Xiang, Yu-Gang Jiang. Xiangyang Xue, Leonid Sigal, and Shaogang Gong. Recent advances in zero-shot recognition: Toward data-efficient understanding of visual content. IEEE Signal Processing Magazine, 2018
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning discover novel visual categories via deep transfer clustering. In ICCV, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Deep image category discovery using a transferred similarity function. arxiv, 2016.
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In ICLR, 2018.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. In ICI.R, 2019.

---

Published as a conference paper at ICLR 2020
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representa- tion learning. In CVPR, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 2015.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In NeurIPS, 2017.
James MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1967.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Andrew Y. Ng. Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and algorithm. In NeurlPS, 2001.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep supe-vupe learning algorithms. In NeurlPS, 2018.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering. 2010.
Rasmus, Mathias M. Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi-supervised learning with ladder networks. NeurlPS, 2015.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
Sylvestre-Alviss Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Semi-supervised learning with scarce annotations. arxiv, 2019.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. In ICCV, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. ICLR, 2015.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In ICML, 2013.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey deep transfer learning. In International Conference on Artificial Neural Networks, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- consistency targets improve semi-supervised deep learning results. In NewrIPS, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In NeurlPS, 2016.
Karl Weiss, Taghi M. Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big Data, 2016.
10

---

Published as a conference paper at ICLR 2020
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE TPAMI, 2018.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, 2016.
Jay Yagnik, Dennis Strelow, David A. Ross, and Ruei sung Lin. The power of comparative reasoning. In ICCV, 2011.
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. Towards --means-friendee spaces: Simultaneous deep learning and clustering. JCML, 2017.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In CVPR, 2016.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer, S41: Self-supervised semi- supervised learning. 2019.

---

Published as a conference paper at ICLR 2020
A DATASET SPLITS
For CIFAR-10 and SVHN we keep the labels of the five first categories (namely airplane, automobile, bird, cat, deer for CIFAR-10 10, 0-4 for SVHN) and keep the rest of the data as the unlabelled set. For CIFAR-100 100 we use the first 80 categories as labelled data while the rest are used for the unlabelled set. Following Hsu et al. (2018) 2019), for OmniGlot, each of the 20 alphabets in evaluation set (with 20-47 categories, 659 characters/class) is used as unlabelled data, and all the 30 alphabets in background set are used as labelled set (964 characters/class) For ImageNet, we follow |Hsu et al. (2018) 2019) to use the 882/1 18 classes split proposed in /inyals et al. (2016), and use the three 30-class subsets sampled from the 1 118 classes as unlabelled sets.
B IMPLEMENTATION DETAILS
In the first sule-super training step, otherwise mentioned, we trained our model with the pretext task of rotation predictions (i.e., a four-class classification: 0%, 90", 180%, and 270") for 200 epochs and step-wise decaying learning rate starting from 0.1 and divided by at epochs 60, 120, and 160.
In the second step of our framework (i.e., supervised training using labelled data), we fine-tuned our model on the labelled set for 100 epochs and step-wise decaying learning rate starting from 0.1 and halved every 10 epochs. From this step onward we fix the first three convolutional blocks of the model, and fine-tuned the last convolutional block together with the linear classifier.
Finally, in the last joint training step, we fine-tuned our model for 200/ epochs for (CIFAR-10 10, CIFAR-10O, SVHN SVHNN/OmniGHa /OmniGoo//maaeeeee which was randomly sampled from the merged set of both labelled and unlabelled data. The initial learning rate was set to 0.1 for all datasets, and was decayed with a factor of 10 at the 170th/(30th, 60th epoch for {CIFAR-10 10, CIFAR- 100, S SVHN) /lmageNet. The learning rate of 0.01 was kept fixed for OmniGlot. For the consistency regul gagizaaittioon term, used the ramp-up function as described in section 2.4 with À = (5.0, 50.0, 50.0, 100.0, 10.0). and T (50, 150, 80, 50) for CIFAR-10, CIFAR-100, SVHN, OmniGlot, and ImageNet respectively.
In the incremental learning setting, all previous hyper parameters remain the same for our method. We only add ramp-up on the cross entropy loss unlabelled data. The ramp-up length is the same as the one used for eq. (4) and we use for all experiments a coefficient of 0.05. For all other methods we trained the classifier for 150 epochs with SGD with momentum and learning rate of 0.1 divided by 10 at epoch 50.
We implemented our method using PyTorch 1.1.0 and ran experiments on NVIDIA Tesla M40 GPUs. Following (Han et al.) 2019). our results were averaged over 10 runs for all datasets, ex- cept for ImageNet which was averaged over the three 30-class subsets. In general, we found the results were stable. Our code is publicly available at htht::: //t:://wwww www. robots ox. ww.acc.uk uk. -vgg/
research/aute aattoccchaattooo _
C STANDARD DEVIATION OF INCREMENTAL LEARNING EXPERIMENT IN TABLE
Table 5: Incremental Learning with the novel categories. "old" refers to the standard deviation ACC on the labelled classes while "new" refers to the unlabelled classes in the testing set. "all" indicates the whole testing set. It should be noted that the predictions not restricted to their respective subset.
CIFAR-10
CIFAR-100
SVHN
Classes
old
new
all
old
new
all
old
new
all
KCL w/ S.S.
0.6%
0.6%
0.1 %
0.3%
0.3%
0.2%
0.3%
0.5%
0.1
MCL w/ S.S. 0.4%
0.4%
0.1%
0.3%
0.1%
0.2%
0.2%
0.3%
0.1 0.1%
DTC w/ S.S.
0.6%
0.2%
0.3%
0.2%
0.2%
0.2%
0.3%
0.2%
0.1%
Ours w/ L.L.
0.2%
0.2%
0.1%
0.1%
0.3%
0.1%
0.1%
0.0%
0.1%
12

---

Published as a conference paper at ICLR 2020
Performance of our method w.r.t A
SVHN CIFARIO
ACC
10
20
30
Figure 4: Performance evolution with respect to k. We report results for k (1,2,3,5, 10, 15, 20, 50}.
D IMPACT OF l OVER RESULTS
We provide additional study of the evolution of performances of our method with respect to k. We results on SVHN/CIFAR 10CIFARROO in fig. 4 We found that = (5, 7} gave the best results overall. We also found that for all values of & except results in general stable.
E RESULTS WITH AN UNKNOWN NUMBER OF CLASSES
While in our work we assume the number of new classes C to be known priori, this hypothesis can be restrictive practice. Instead, one can estimate the number classes in D using recent methods such as DTC (Han et al.) 2019). In t we compare ACC of KCL (Hsu et al. (2018), KCL (Hsu jet al. 2019), DTC et al.) 2019) and our method on unlabelled splits of OmniGlot and ImageNet datasets with C computed from DTC. We note that our method again reaches the state-of-the-art on ImageNet and on par with the state-of-the-an on OmniGlot.
Table 6: Novel category discovery results with unknown C
No
OmniGlot
ImageNet
(1) KCL
Hsu et al.
2018
80.3%
71.4%
(2) MCL Hsu et al.
2019
80.5%
72.9%
(3)
DTC
Han ct al.
2019
87.0%
77.6%
(4)
Ours
85.4%
80.5%
13


Reviews:
[{"review_id": "SJx82Wa0YH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper addresses the problem of clustering unseen classes. To learn a robust feature extractor, this paper proposes a multi-stage training framework, which leverages different supervised manners in each stage. Specifically, they initialize the network using the self-supervised learning on the union of all available data and then further finetune it using labelled data. Based on this, they propose the rank statistics which leverages the activation knowledge on labelled classes and rank the activated dimensions. Unseen data having similar rank results are clustered to obtain the initial pseudo labels. Finally, the network is jointly optimized with the ground-truth and generated pseudo labels (the pseudo ones will be updated during training). Extensive experiments on 5 datasets show that their method has significant advantages over SOTA owing to the learned robust feature extractor.\n\n+Strengths:\n1. The writing of this paper is satisfactory. Both the related works, motivations and technical details are clearly introduced. \n2. The experiments are solid. They evaluate their method on five popular object datasets and both ablation of each components and comparison with SOTA are shown in the paper.\n3. This paper also shows that their method has good ability of avoiding forgetting of old (seen) classes, which may provide some insights about feature extraction for improving incremental learning. \n\n-Weaknesses:\n1. Except for the experimental evaluation, what is the advantage of rank statistics over directly comparing feature vectors? Why robust?\n2. Some experimental issues. a) how will the choice of k in top-k rank influence the performance? b) why the advantages of incorporating incremental learning on SVHN and CIFAR-10 are not obvious? c) why evaluation of incremental learning on CIFAR-100 is not well (acc difference between old and new is larger than other datasets). Besides, what is acc. performance of old classes with only labeled data for training.\n3. Typos. In Sec.4, the writing of KCL (KLC) and MCL (MLC) is not consistent.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lWqw86FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n  This paper tackles the problem of unsupervised object discovery, whereby a labeled dataset must be leveraged in order to then cluster an unlabeled dataset with a set of unknown categories. The paper contributes three main ideas to succeed at this task, namely 1) use of self-supervised learning to initialize the representations in a way that doesn't bias them to the labeled data, 2) a robust rank-based metric to generate estimates of similarity/dissimilarity along with consistency-based regularization to improve optimization, and 3) Joint optimization/refinement using a combination of labeled/unlabeled losses, as well as ability to learn incrementally without forgeting the original labeled classes. Results are shown on a range of datasets including OmniGlot, ImageNet, CIFAR-10, CIFAR-100, and SVHN. The results demonstrate improvement over the current state of art for this task. \n\n  Overall, my current vote for this paper is a weak reject. The main reason is that the paper really combines a set of known methods (self-supervised learning, consistency-based regularization, and an ad-hoc training regimen. On the other hand, the paper is well-written and provides nice rigorous experiments showing clear improvements over state of art by porting these known techniques from different domains (self/semi-supervised learning). However, if satisfactory answers to questions below are given, I am willing to change my rating.\n\nMain Argument\n\nStrengths\n\n  - Overall the paper tackles an interesting problem, and does so in a way that achieves good results beyond state of art. \n\n  - An ablation study is provided which shows the contribution of different parts of the method. \n\n  - The paper is very well-motivated, written, and methods are described nicely and succinctly.\n\nWeaknesses\n\n  - Clearly the methods employed are, by themselves, not novel and have been used for a range of other ML problem formulations. What is the clear contribution/novelty of the work?\n\n  - While the paper raises an interesting motivation about not biasing feature learning by using self-supervised learning, it's not clear to me that this claim is justified. What is the evidence for this, besides better performance? While the ablation w/o self-supervised learning performs more poorly than everything combined, the other parts of the ablation (with self-supervised learning) also perform poorly. Clearly, there is some interaction between the different aspects of the method, but I am not sure what that is. Why is the full combination so much better than if any one thing is removed?\n\n  - The method is very similar to KCL (e.g. loss (3) is the same and justified via a graphical model formulation in that paper). The paper does not really read like it is building off of that though, which seems a bit misleading. Do the authors believe there is additional novelty, or is it a matter of adding the three contributions to KCL? What is the major reason that self-supervision with KCL still doesn't do as well? As far as I can tell, the only difference is self-supervised learning, the fact that you have a manual 3-stage curriculum, and the robust ranking method/consistency loss (i.e. the three contributions).\n\n  - You assume that the number of clusters is known; this is one of the advantages of all of the prior work in that they can estimate this. How well does the method work if the number of clusters is not known? \n\nAdditional comments not related to final vote:\n\n  - The paper seems to start out in a way that implies the problem is new; citations should be provided to the three main compared works in the intro to make it clear that this is a follow-on to an existing problem. Further, as mentioned above MCL/KCL (especially the latter) are very similar in nature and introduced this problem (including [1] which also used unsupervised feature learning in one condition). These should be included as \"work most related to ours\" given that they came before Han et al.\n\n  [1] Deep Image Category Discovery using a Transferred Similarity Function, https://arxiv.org/abs/1612.01253.\n\n  ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJlZg2lpqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a methodology to discover new categories in an unlabeled dataset with the help of a label one. The authors propose the following methodology. First bootstrap some features using self-supervised learning on labeled and unlabeled data. Then transferring the knowledge of the labeled data to the unlabeled one by supposing that the representations of both are similar, the similarity being a rank statistic. Then using this knowledge a joint supervised-unsupervised objective.\n\nQ1. The paper is about discovering new visual classes. Section 2 mention that the number of classes C^u must be known \"a priori\". How do you tackle this limitation? Is there any heuristic, similar to the one found in the clustering literature, that could help?\n\nQ2. Have other losses been investigated for the clustering head? Such as the triplet loss, or deep clustering loss rather than BCE?\n\nI would suggest to report standard deviations as the experiments were repeated 10 times on random train-test splits.\n\nI propose a weak accept. The paper is well written, the methodology is original, the experiments are convincing and the authors will release publicly the code. My main concern is Q1, which is eluded. Second, it would be nice to have an experiment illustrating the impact of the rank k when transferring knowledge. Is k=5 always good? If not, what could influence the best value?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper defines a methodology to discover unknown classes in a semi-supervised learning setting, based on: i) defining a proper representation based on self-supervision on all samples; ii) defining equivalence classes on the unlabelled samples, based on ranking statistics; iii) training supervised heads aimed to predict the labels (when available) and the equivalence class indices (when unlabelled). 

All reviewers agree that the ranking statistics-based heuristics is a quite innovative element of the paper. The extensive and careful experimental validation, with the ablation studies, establishes the merits of all ingredients. 

Therefore, I propose acceptance of this paper.

Decision: Accept (Poster)