Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation
Authors: Bingchen Zhao
Kai Han
Venue: NeurIPS 2021 Poster (2021)
Keywords: novel category discovery
transfer learning
clustering
deep learning

Abstract: In this paper, we tackle the problem of novel visual category discovery, i.e., grouping unlabelled images from new classes into different semantic partitions by leveraging a labelled dataset that contains images from other different but relevant categories.  This is a more realistic and challenging setting than conventional semi-supervised learning. We propose a two-branch learning framework for this problem, with one branch focusing on local part-level information and the other branch focusing on overall characteristics. To transfer knowledge from the labelled data to the unlabelled, we propose using dual ranking statistics on both branches to generate pseudo labels for training on the unlabelled data. We further introduce a mutual knowledge distillation method to allow information exchange and encourage agreement between the two branches for discovering new categories, allowing our model to enjoy the benefits of global and local features. We comprehensively evaluate our method on public benchmarks for generic object classification, as well as the more challenging datasets for fine-grained visual recognition, achieving state-of-the-art performance.

Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation
Bingchen Zhao
Kai Han2.3,4.
TToggi University The University of Hong Kong zhaobc, wwweeeigmmmmm.mmmmmmmm
Google Research UUniversity of Bristol
com com kaihanx0hku, hk
Abstract
In this paper, we tackle the problem of novel visual category discovery, i.e., group- ing unlabelled images from new classes into different semantic partitions by lever- aging a labelled dataset that contains images from other different but relevant categories. This more realistic and challenging setting than conventional semi-supervised learning. We propose two-branch learning framework for this problem, with one branch focusing on local part-level information and the other branch focusing on overall characteristics. To transfer knowledge from labelled data unlabelled data, we propose using dual ranking statistics on both branches to generate pseudo labels for training on the unlabelled data. We further introduce a mutual knowledge distillation method to allow information exchange and encour- age agreement between the two branches for discovering new categories, allowing our model to enjoy the benefits of global and local features. We comprehensively evaluate method on public benchmarks for generic object classification, well as the more challenging benchmarks for fine-graing grained visual recognition, achieving state-of tatte-of-thealle art performance.
1 Introduction
Superior performance on many problems has been achieved by recent machine learning models, especially the ones based on deep learning. While the success comes at the cost of large-scale human annotation, which prohibitively expensive practice. For example, modern convolutional neural networks (CNNs) can surpass human-level recognition performance on ImageNet after training with over one million labelled images [22] On the one hand, is not possible annotate all possible classes in the real world, as there are way more classes than the 1, 000 classes in ImageNet and new classes keep growing over time. the other hand, annotating specific data, such as the medical data, may require specific expertise, which renders the large-scale annotation extremely difficult, if not impossible. Therefore, it desired to enable the machine learning systems to deal with unlabelled data automatically.
which aims at discovering new visual categories on unlabelled data by transferring knowledge from labelled data. The labelled data is assumed to contain similar but different categories those in the unlabelled data. learning in the sense that both labelled and unlabelled data are used to learn the model. While novel category discovery much harder because supirsuper learning assumes that every class contains labelled instances, while for novel category discovery there are no labels available the new classes the unlabelled data. This setting also relevant to
Recently, the problem of novel category discovery was formalized in 20 18).
This problem is similar to semi-supervic
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

unsupervised clustering. But differently, novel category discovery makes use of the labelled data to extract a specific class prior (i.e.. the properties that delineate class) for partitioning the unlabelled data, while the unsupervised clustering may produce multiple different but equally valid clustering results by adopting different properties (e.g., color, shape, pose, lighting, etc). In this paper, we introduce a simple and effective two-branch framework for novel category discovery, with one branch focusing on local part-level information and the other branch focusing on overall characteristics. Our contributions are as follows.
First, we propose to apply dual ranking statistics for transferring knowledge from the known classes in the labelled data to the unlabelled data, resulting in more robust pseudo label generation for learning on the unlabelled data. We maintain a dynamic object part dictionary and apply part-level ranking statistics on the similarity distribution of each instance over the dictionary to obtain pseudo labels, which are complementary to the pseudo labels obtained by simply examining the ranking statisties of global descriptors.
Second, we introduce mutual knowledge distillation method to allow information exchange and encourage agreement between the local and global branches. The dual ranking statistics provide global and part-level information for learning in two branches separately. The mutual knowledge distillation further allows information exchange between the two branches and makes them benefit from each other. Unlike conventional methods that distill between teacher and student models with known labels, our method distills between two branches of the same model without any manual annotations.
Third, we comprehensively evaluate our method on public benchmarks for generic object classi- fication, including CIFAR10, CIFAR100, and ImageNet, obtaining state-of-the-art results. Fur- thermore, we also validate our approach on the more challenging fine-grained datasets CUB-200, Stanford-Cars, and FGVC-Aircraf,, in which the local details are more important to distinguish different classes. Our method outperforms existing methods by substantial margin, thanks to the ability of our model subtly exploit both local and global information. Our code can be found at htttttts//githuut com
wwwpeo/ono/DDnanniiii. Dennantt////nnnenn dual- maal-rank-uan ncd.
2 Related work
Our work is relevant to novel category discovery, knowledge distillation, and part-level feature learning. We briefly review the most relevant work below.
Novel category discovery is relatively new problem setting recently formalized by 20 18 with the task being automatically discovering new object categories in the unlabelled data by making use of a labeled dataset containing different but relevant object classes. The purpose of using the extra labelled data to learn a category prior to reduce the ambiguity of class definition. This task is closely related to unsupervised clustering and semi- superviide learning, but also significantly different from them. Unsupervised clustering has been studied for decades with many classical approaches [35] 3 [ and deep learning based solutions [50, 16 40) being proposed. Due to the lack of proper class prior, multiple equally valid clustering results can be achieved by different criteria. Therefore, it can not be directly applied to discovery new classes as we expect the model to follow unique class definition. 6 38 46), unlabelled data are used together with labelled data to train a more robust model, with the assumption that all classes in the unlabelled data have labelled instances. However, this is unlikely to be true for real applications where unlabelled data may come from new classes. Thus, novel category discovery is more realistic setting. The DTC method introduced in (00 approaches this problem in two steps. The model is firstly trained with supervised learning on the labelled data to capture high-level semantic class information and then trained on the unlabelled data with a clustering loss to discover new categories. In [18] [9]. Han et al. proposed to use sul--sueevisioo bootstrap feature learning and introduced ranking statisties on the feature embedding to provide pseudo labels for training on unlabelled data. The KCL (25] and MCL [26] methods were designed for general transfer learning across domains and tasks, which can also be applied for novel category discovery. These two methods maintain two models for training and testing, a pretrained binary classifier for pseudo label generation and a clustering model. The binary classifier pretrained on the labelled data is used to provide pseudo labels to train the clustering model on unlabelled data. Concurrent to our work, several methods 28
In semi-supervised learning 43 39
58 57 are proposed to improve the performance of novel category discovery from different perspectives, showing promising results.

---

The existing methods only consider the global feature descriptors while igmoring the local object parts which are essential distinguish classes that look similar. In our approach, we jointly consider both and allow information exchange between them for more reliable new category discovery.
Knowledge distillation is often employed to learn compact student model using the knowledge distilled from a larger teacher model by enforcing the agreement of outputs or representations between the two models (e.g.. [24]]]]] 41 54
52 27 29 48)). Apart from distilling larger teacher model to a smaller student model, self-distillation methods show that distilling between two identical models can also improve the performance [S Is Mutual learning [56] is a similar technique that trains two models of the same architecture simultancously but with different initialization and encourages them to learn collaboratively from each other. It has been shown that different initialization leads the model to focus on different regions of the input and thus the trained model can better capture the holistic structure of the data, resulting better performance. Though effective, the above methods require the labels for training. Thus they can not be applied for novel category discovery. Recently, [3] introduced a suferviuee distillation method called SEED to improve the representation learning on a small model (student) by distilling from pretrained large model (teacher). The student model is trained to predict the same similarity score distribution inferred by the frozen teacher model over queue of instances. We draw inspiration from [ to design the mutual knowledge distillation module in our method for novel category discovery between the local and global branches.
Part-level features have been shown to be effective for tasks involving image verification such as few-shot learning [O, 55 and image retrieval 7 5
In [O], the part-level visual concepts are generated by clustering feature vectors from the feature maps obtained using a trained neural network. These visual concepts are used as matching primitives at test time for few-shot learning. Zhang et al. (55) used the Farth-Mover's Distance (EMD) between local part features to measure the distance between two images for matching. Doersch et al. [12] proposed CrossTransformers
model the cross-attention between each local part of a query image and a set of support images for few-shot learning. Part-level features have been more widely used in the domain
of image retrieval with both classical [34] 5 and deep learning based 2 37
45 7 methods.
Intuitively, the part-level features provide better precision for the retrieved results because the local matching is more restricted, and global features yield better recall [7]. In our work, we leverage both local and global features to enjoy the benefits of both through the dual ranking statistics and the mutual knowledge distillation, for more robust novel category discovery.
3 Method
The goal of novel category discovery is to automatically partition unlabelled instances x.' E D" into C semantic clusters by transferring the knowledge from the labelled instances (a4, v/) E D', where v E {I...., C²). The key assumption is that the classes in D' and D" are relevant though different, so that the properties that constitute a class learned from D can be transferred to D" Following (18). we assume C is known a priori. When it is unknown, we can employ off-thc-shelf methods such as (2 to get an estimate.
To tackle this challenge, we introduce a framework with mutual knowledge distillation between a global branch and local branch (see fig. [ The global branch is designed to capture the overall feature, while the local branch is designed to focus on each individual spatial local part. They have shared feature extractor fe. Each branch has feature projection layer and two linear heads. We denote the feature projection layer as & and the two linear heads as 1/ and 17" respectively. We also denote v followed by the average pooling operation as v.. Unless stated otherwise, we use subseripts and p to differentiate feature projection layers and linear heads of global and local branches respectively in the rest of the paper. The two linear heads are responsible for classifying O' labelled categories and clustering C unlabelled categories respectively. To transfer knowledge from the labelled data to the unlabelled data, we adopt global image level and local object pert-level ranking statistics on the feature representations. In this way, the model will have reliable pseudo labels for training on the unlabelled data. Moreover, we also incorporate mutual knowledge distillation between the two branches to enforce global and local agreement, leading to more reliable novel category discovery. Next, we will introduce each component of our framework in more detail. In section 3.1
we first introduce the our knowledge transfer method using dual ranking statistics. In section 3.2 we describe our mutual knowledge distillation method for novel category discovery. Lastly, we summarize the overall training loss in section 3.3
3

---

RS
4
AVB
Z
11y
C
BCE
B3
16
1
Global
4
Fj
So
Local
B
P
vp
AVG
2/
7p
BCE
V (b,,...,
RS
Figure 1: Overview of our proposed method. An input image x; is firstly processed by the feature extractor fe. The resulting feature is sent to the global and local branches. The global branch uses ranking statistics on the global feature embedding 2; to generate pseudo labels the LE loss for new class discovery, while the local branch uses ranking statistics on the similarity score vector o; between each of the local parts and dynamic part dictionary generate pseudo-labels. The two branches mutually distill knowledge from each other to allow information exchange and encourage agreement through two auxiliary memory banks with LILD loss. We omit the supervised linear head, cross-entropy loss Ler and consistency loss LMSE in the plot for the sake of clarity.
3.1 Dual ranking statistics for knowledge transfer
It is proven that ranking statistics is robust to noise, especially in nigh-dimensional space (51). Han al. [18] proposed to use ranking statistics for novel category discovery. The idea is to generate pair-wise pseudo labels by comparing the top-k ranks of two feature vectors via examining the feature magnitude. Specifically, the binary pseudo label sij is determined by
Sij (top, (2,) t2px(2;))
where and 2j are feature vectors of two unlabelled images. With the binary pseudo labels, the model then be trained using the binary cross-entropy loss the unlabelled data.
Unlike [8]] which only considers global image feature descriptors, we also explore each individual object part for novel category discovery. Verification based pert-level features has been shown to be effective to match two images (e.g. [2)). The pair-wise wise verification using global features focuses the holistic structure of the object and thus may introduce more false positives with high recall (but low precision). While pair-wise verification using local part features focuses on local details, which is more strict, and thus may introduce more false negatives with high precision (but low recall) [7]] Hence, local part features and global features are complementary to each other for verification and should be considered jointly for more robust category discovery, as ideally to have both high precision and recall. Therefore, to achieve this goal, we propose to have one branch using global features and another branch using part-level features. Ranking statistics is applied each branch to generate pseudo labels. For the global one, we simply apply soft extension of the can be
expect
hard ranking statistics [18]. In particular, instead of forcing sty in eq. (I) to be either replaced by continuous value Siy = & E [0, 1] where c is the number of shared elements in (2p). and top, (z,). Hence, the BCE loss for the global branch can be written as
M
M
CICE
Σ
log n (=])) + (1 2..) log(1 (=+)) (=
M
1=1 1=1
where M is the number of unlabelled images, 2 (fo E Rd is global feature vector of the image x", and sy, is the soft ranking statistics score between x. and zy..
For the local one, propose maintain memory bank to act dynamic object part dictionary and obtain the ranking statisties by comparing each object part with the collection of part descriptors in the memory bank. Concretely, the memory bank is a First-In First-Out (FIFO) queue V [e1...., Ve]

---

V fir u2, Te)
AVG
0
Figure 2: Design of the local comparison process. Each part of an image is compared against all parts in the memory bank V. forming one similarity vector for each part. All resulting similarity vectors are then fused to single vector by average pooling, which used to generate pair-wise pseudo labels.
storing the part-level features, where e is the number of part-level features in the queue. Each part feature in e is the d-dimensional feature vector from a randomly selected spatial location in the feature map (x((()))) Ruxhxw of an randomly sampled image v; D U Du We select one part feature vector from each image in current mini-batch. Other part sampling methods like using all parts or selecting based on activation maps can also be applied (as will be seen section 4.3). while we found that the simple random sampling is on par with other more complicated methods. Following MoCo (2T), when the current mini-batch of part features enqueued into V. the oldest mini-batch in V is dequeued. For the feature vector 9. drawn at every spatial location of the feature map ())())) in current mini-batch we can then obtain a e-dimensional vector representing the similarity between qj and all part features the dictionary V. All qj for x across w locations are then fused together by average-pooling Let us denote by of E TR the fused similarity vector (see fig. D. For any pair of unlabelled images x. and x; in current mini-batch, their soft ranking statistics score s/j can then be obtained by comparing of¹ and of as described above for the global feature descriptors. Similar to eq. ( we can define the BCE loss LEE to train the local branch. The BCE loss for both branches can then be written as
LBCE
CICEE
CE
3.2 Mutual knowledge distillation for novel category discovery
So far, our model considers local and global information in two branches independently. To make the two branches directly benefit from each other, we propose mutual knowledge distillation method for novel category discovery to allow information exchange and encourage agreement between the local and global branches.
Existing mutual knowledge distillation methods are normally designed for two models (i.e., teacher model and student model or two peer models) 124 (6),, while we distill between two branches of the same model with each branch having a different focus. More importantly, unlike the conventional mutual learning and knowledge distillation methods that have the same class assignment for the same input due to the availability of labels, in our setting, the cluster assignments of the unlabelled linear heads ny and nn for the same unlabelled data point may be completely different for the two branches. Therefore, these knowledge distillation methods can not be applied for novel category discovery. Recently. [13] introduced the knowledge distillation between the teacher and student models by comparing the similarity score distribution between each instance and a queue of features from the larger teacher model. Inspired by [13], develop mutual knowledge distillation method between the local and global branches for novel category discovery without using labels.
Instead of using the softmax output of ni and no for mutual learning, which is invalid in our case, we mutually distill the two branches via the similarity score distribution over a memory bank per branch. Specifically, maintain two FIFO feature banks to store the features extracted by Vi and big. denoted B, bb. .b...... by] and B, 1b?.... bz]. Both B, and B, contain features
5

---

of zj E D". Again, similar to MoCo [2I]. when the current mini-batch is enqueued, the oldest mini-batch dequeued in both memory banks. For each unlabelled image z we first obtain the feature representations using z Vi (x(())) and 2!" = ())((()))) The similarity score distribution pd (x Bg) and pd (x Bp) can be defined as
p] exp(22" b 69/TT)
(2 it. ................................ pll,
LB exp(a;" b//TT
and
exp(z!" 1/51T)
(, = -Ir......r1 P3
(5)
EV--E exp(z ///TT)
where 7 is scalar temperature to control the sharpness of the similarity score distribution.
To encourage agreement between the similarity score distributions of unlabelled image from two branches, we apply mutual learning on the score distribution using the symmetric Kullback-Lebller Divergence (sKLD) loss
1 (DRI (p") + DEC Dk((p?l)r)) LKLDD 2
(6)
where DEL I. is the Kullback- (KL) divergence with Dx (Pillp2) = P1 log In this way, the global and local branches of the model can learn from each other, while maintaining their own merits. Our mutual knowledge distillation method differs from SEED [13 in several aspects. First, SEED aims improving high level visual representation, while we aim at enforcing the agreement between local and global features for novel category discovery; second, SEED distills between two different models while we distill between two branches of the same model; third, SEED maintains queue derived from the global features of the teacher model, while maintain two queues as dictionaries for global and local features; last, in SEED the larger teacher model is frozen during distillation and the student model is trained with cross-entropy loss, whereas in method both branches are jointly trained with the sKLD loss.
3.3 Overall training loss
Apart from the BCE and sKLD losses introduced above, we also apply the standard cross-entropy loss on both branches, which is written as
N yi log T/ (+)) + ya log (=(")
1 N
LCB
where is the number of labeled images, =f = (fo and x/ = (p Similar to [8], the feature extractor fe is pretrained with self supervueee learning and is frozen during the training for novel category discovery.
Following 18 20
we also include the consistency regularization loss to enforce the predictions of the same data point under different transformations be the same. Specifically, let à be a randomly transformed counterpart of the input image z,. The consistency loss is then defined as
LMSE
Σ (2) - (r((+1) (2
(8)
M
- - M Σ
+ - (2
(9)
where 2; is the feature embedding of the transformed image ã. Without enforcing the consistency, we may have different ranking statistics for é; and 2,, which will lead to diffferent Sij for the same images under different augmentation, confusing the training.
In summary, the overall loss function used to train our model is
L LECE + LOKLD + LcE + w(t)
(10)

---

where w(t) = Ae A is ramp-up function as widely used in the literature |46 33 18 R,, t and r are the current time step and the ramp-up length respectively.
with
After training, as the two branches already agree with each other, we simply take the output of the global branch the final prediction. In particular, for each unlabelled image, we take the index of the max value in the softmax output as its cluster assignment. It is also worth noting that the memory banks and ranking statistics are no longer needed inference time.
4 Experimental results
4.1 Experimental setup
Benchmark and evaluation metric. We follow [8] validate our method on variety of benchmark datasets for generic image classification including CIFAR-10 [31], CIFAR-100 [31) and ImageNet-1K K [T.. For ImageNet-IK, three 30-class unlabelled splits are used in the ex- periments and the average performance is reported. We further experiment with ImageNet- 100 [47 which has less number of labelled classes but the same number of unlabelled classes as ImageNet-IK. IK. In addition, we also validate our method on the more challenging fine-grained datasets including CUB-200 49], Stanford-Cars [30], and FGVC aircraft [36]. One key assump- tion for novel category discovery is that the labelled classes are relevant to the unlabelled ones. Due to the diversity of classes in generic image recog- nition datasets, it is not easy to tell the relevance be-
Table 1: Data splits in the experiments.
tween classes, though it is implicitly contained in the rules used to identify classes during data cura- tion. contrast, the relevance of the fine-grained datasets is explicitly determined by the fact that all classes in line-grained dataset belong to the same entry level class, e.g., birds, cars, and airplanes for the above line-grained datasets. Besides, the fine-grained datasets pose more challenges for novel category dis- covery because of the high inter-class similarity. In this case, spatial local details become crucial clues distinguish them. The labelled and unlabelled splits are summarized in table D
labelled
unlabelled
CIFAR-10
5
5
CIFAR-100
80
20
ImageNet- 1K
882
(30, 30, 30)
ImageNet-
70
30
CUB-200
160
40
Stanford-Cars
156
40
FGVC-aircraft
81
21
We follow the standard practice in the literature to adopt clustering accuracy (ACC) on the unlabelled data the metric for evaluation. It is defined as Σ. 1 (v) h(v:)) where h is the optimal permutation obtained by Hungarian algorithm the ground-truth label yf, and N, is the number of unlabelled images.
[32] that can match the clustering assignment y/ with
Implementation details. We follow [18] to initialize our model with parameters pretrained by self-supervised learning. Our default model is realized based on ResNet50 (23] Namely, the first three macro blocks of ResNet50 are the feature extractor, fa, which is initialized with MoCov2 [ pretrained on ImageNet via self-supervision and are frozen during training for novel category discovery. The last macro block is duplicated into two as the projection layers ub, and v/g for local and global branches respectively. Each of them is followed by two linear heads, 7/' and 17", for labelled and unlabelled data respectively. For a fair comparison with prior methods, we also experiment using ResNet18 with RotNet [17] initialization. For all our experiments, we use SGD with momentum 44 as the optimizer and use batch size of 128 for labelled data and 64 for unlabelled data. The temperature parameter 7 set to 0.07, and the size of the three memory-bank. are all set 2048 for memory efficiency and also considering the relatively limited number of images in the fine-grained datasets. For the dual ranking statistics, we set k 5 for the global branch following [18) and k = 30 for the local branch to include more local parts into consideration. Our experiments are performed using GTX 1080Ti GPUs.
4.2 Comparison to the state-of-the-art
Comparison on generic image classification datasets. In table we compare our method with baselines and taate-of-the-aad methods for novel category discovery on generic image classification datasets CIFAR-10, 10, CIFAR-100, and ImageNet following the same protocol as RankStat [18] for
7

---

fairness. Our method achieves state-of-the-and results on all datasets. It can be seen that our method
significantly outperforms k-means, KCL [25 MCL [26]. and DTC [20]. Notably, our method outperforms the previous state-of-the-art 18 on the most challenging ImageNet-1K K dataset by significant margin of 6.4%. The improvements on CIFAR-10 and CIFAR-100 are smaller. We hypothesize that this is due to the spatial resolution discrepancy. The image resolution of CIFAR-10 and CIFAR- 100 only 32 x 32, which leads to very small spatial local features. Thus, the local branch cannot offer much more useful information than the global branch. In contrast, the input resolution of ImageNet-II IK is 224 224. Hence, larger spatial feature map can be obtained to take more advantages of the local branch.
Table 2: Comparison of novel category discovery on generic classification datasets. For fair comparison, our method uses ResNet18 (23) backbone initialized with RotNet [17] following [18].
No Method
CIFAR-10
CIFAR 100
ImageNet-
(1) k-means 35
7..5+
56.3+:1.7%
71.9%
(2) KCL 25
66.5+3.9%
14.3+1.3%
73.8%
(3) MCL [26
64.2:0.1%
21.3=3.4%
74.4%
(4) DTC 20
87.5+
56.7=11.2%
78.3%
(5) RankStat 18
90.4++0.5%
73.2++2.1%
82.5%
(6) Ours
91.6++0.%%
75.3+
88.9 %
Comparison on fine-grained image classification datasets. In table 3 we compare with other methods on fine-grained image classification datasets. In these datasets, the difference of holistic structure between classes is small, and the classes are mostly differentiated by the local details. Thus, only looking at the global feature descriptors is far from enough for the fine-grained scenario, where the local information plays a vital role. Comparing rows 2-3 with row can be observed that our proposed method substantially outperforms previous statc-of-the-ar models that only use global features. For example, our method shows 8. 5.3%, 8.1% and 4. 4.1% improvements respectively on CUB-200 Stanford-Cars and FGVC-Aireral over RankStat [88. To further verify the effectiveness of our local branch, we carry out another experiment by dropping the global branch (row 4 table Dropping the global branch will disable the mutual knowledge distillation accordingly. It can be seen that using the local branch alone already yields a notable improvement over RankStat [18 (row 3 vs row 4). Our full method establishes the new state-of-the-arr.
Table 3: Comparison of novel category discovery on fine-grained classification datasets. "Ours w/o global" means our proposed method without global branch and mutual distillation.
No
Method
CUB-200
Stanford-Cars
FGVC-Aireraft
(1) k-means 35
20.4 = 1.1%
31.4 + 0.9%
44.7 + 1.3%
(2) DTC 20
33.6 0.7%
46.5 + 2.4%
58.7 1.2%
RankStat 18
39.5 + 1.7%
53.8 + 2.0%
66.3 + 0.7%
(4) Ours w/o global
43.1 2.3%
56.8 2.3%
67.3 1.0%
(5) Ours full
47.8 + 2.4%
61.9 + 2.5%
70.4 + 0.9
4.3 Ablation study
Effectiveness of different components. In table we ablate different components of our method. is clear that all components in our method are effective as removing any of them will cause the performance drop. Without BCE loss, our dual ranking statistics is disabled and no pair-wise pseudo labels are transferred from the labelled data. Therefore, the parameters within 17" remain untrained, leading to poor performance equivalent to random baseline. The sKLD loss also has a strong impact on the final performance. Without it, the performance drops 8.0-11.3% absolute ACC. This performance drop shows that the information exchange between the global and the local branches are crucial. The consistency loss also plays an important role, as the performance without consistency drops 9.9-12.2% absolute ACC. Similar to [18 20 the cross-entropy loss, consistency loss and seif-supervision are also important for our method. Having these components a unified framework, our full method obtains the best performance.
8

---

Table 4: Effectiveness of different components of our method. "MSE" means MSE consistency loss; "CE" means cross-entropy loss for training on labelled data; "BCE" means binary cross-enrrppy loss for training both global and local branches on unlabeled data; "sKLD" means the sKI.D loss for mutual distillation between the two branches; "Self-sup." means self-supervised pre-training.
CUB-200
Stanford-Car
FGVC-Aircraft ImageNet-100
Ours w/o BCE
2.2 + 1.3%
3.1 + 0.4%
5.1 = 0.4%
3,0 + 0.3%
Ours w/o sKIDD
39.8 + 1.8%
50.6 2.1%
60.8 + 1.5%
58.2 + 1.2%
Ours w/o CE
41.2 + 2.4%
52.4 + 4.3%
60.2 + 2.7%
59.1 + 2.7%
Ours w/o MSE
37.9 + 4.5%
50.6 6.2%
58.9 + 5.7%
57.2 + 3.6%
Ours w/o Self-sup.
44.3 3.5%
58.2 = 1.8%
67.4 +: 1.3%
65.3 + 1.3%
Ours full
47.8 + 2.4%
61.9 + 2.5%
70.4 =: 0.9%
69.4 + 2.1%
Different configurations of the two-branch design. Our two-branch design allows mutual knowl- edge distillation between local and global branches. In S] we compare different branch config- urations. Row and row represent the global and local single-branch baselines. As can be seen, the local branch configuration performs better than the global one which is the configuration of [18), demonstrating the local ranking statistics with the part dictionary is more effective than that with global features. By introducing another branch of the same type (row 1 + row 3, row 2 + row 4) to incorporate mutual knowledge distillation, the performance can be consistently boosted, which further verifies the effectiveness of our mutual knowledge distillation method for novel category discovery. Mutual knowledge distillation between two local branches is more effective than the counterpart between two global branches. Row 5 presents the results of mutual distillation between global branch and local branch, showing best performance, which corroborates that mutual knowledge distillation between local and global branches enables them complement each other.
Table 5: Different configurations of the two-branch design.
No
Configuration
CUB200
Stanford-Cars
ImageNet-100
(1)
global
39.5 = 1.7%
53.8 + 2.0%
62.5 + 1.2%
local
43.1 0.9%
56.8 + 1.7%
64.2 + 1.6%
global
global
41.2 0.8%
54.6 + 0.7%
63.2 + 0.9%
local
local
44.7 1.1%
57.9 + 0.5%
65.7 + 1.4%
global
local
47.8 + 2.4%
61.9 + 2.5 %
69.4 + 2.1 %
Effects of larger memory banks. It has been shown in 21 that increasing the size of the memory bank can improve sulf-supervised representation learning. In our experiments, we set the memory sizes of Bp. B, and V to 2048 for training efficiency. In table we show the results by increasing the memory sizes. As can be seen, similar to (21), the memory size and the performance is positively correlated. With more instances or parts in the memory banks, more useful information can be used for mutual knowledge distillation (B, and By) and local part ranking statistics measure (V), thus improving the results. However, with a relatively small memory size of 2048, our method already achieves promising results. Note that due to the limited number of images in CUB-200 and Stanford-Cart the performance boost with the increase of bank size for B, and B, quickly reaches plateau. While for CIFAR-10 and ImageNet- more instances can be enqueued to foster the knowledge distillation. In terms of V. the performance is consistently improved with the increase of the size of V for all datasets. Due to the larger image resolution in CUB-200 and Sauford-Carrs more useful part features can be extracted from each image for local ranking statistics. Therefore, the performance is less affected by the smaller number of images. This further demonstrates the effectiveness of our local pert-level ranking statistics.
Different sampling methods for memory bank V. The part memory bank V stores the leve- features for the local-compan comparion branch, and is updated in a FIFO manner. Here we compare different ways of maintaining V. Our default choice is to randomly sample one part-level feature from each image
training mini-batch. However, random sampling may not select the most informative part-level features from the image, so an alternative is to select part-level features based on alas-aactiattion-map (CAM) [59]. Instead of random sampling, the part with the highest response is selected by CAM. Rather than sampling only single part from each image, we can simply use all

---

Table 6: Increasing the size of memory bank. "SCars" denotes Stanford- Cars dataset; "IM-100"
denotes ImageNet- 100 dataset.
(a) ACC with varying size for Bp and Bg.
(h) ACC with varying size for V.
Bank size 2048
4096
8192
16384
32768
Bank size
2048
4096
8192
16384
32768
CIFAR-10
91.6%
92.3%
92.4%
92.7%
93.0%
CIFAR-10
91.6%
91.7%
91.7%
91.9%
92.0%
CUB-200
47.8%
48.3%
48.4%
48.5%
48.4%
CUB-200
47.8%
48.5%
48.7%
49.2%
49.3%
SCars
61.9%
62.4%
62.8%
62.9%
62.9%
SCars IM-100
61.9%
62.7%
63.4%
63.6%
63.7%
IM-100
69.4%
70.2%
70.8%
71.1%
71.3%
69.4%
70.5%
71.2%
71.3%
71.4%
the parts for each image. We compare these three different choices in table 7 can seen that using all the parts causes slight performance drop. This is reasonable because using all parts introduces redundancy in V. making the local comparison more prone to noise. On the other hand, selecting the most activated part using CAM shows a better performance than the random selection, indicating that more informative parts the memory bank can enable better local comparison. As the gap is not significant, use the random sampling our default choice for its simplicity and efficacy.
Table 7: Effects of using different way to update V. "Random" means random part selection; "All"
means using all parts; "CAM" means selecting the most activated parts using CAM [59]
CIFAR-100
ImageNet-100 100
CUB-200
Stanford-Can
Random
75.3 + 2.3%
69.4 + 2.1%
47.8 + 2.4%
61.9 + 2.5%
All
76.0 + 2.6%
68.7 : 2.3%
46.5 + 1.9%
61.0 + 2.4%
CAM
76.5 + 2.4%
70.3 + 2.4%
48.5 + 1.8%
62.4 + 2.1%
Unknown number of novel classes. In a more realistic setting, the unlabelled class number C may not be known. In this case, can apply existing methods to estimate category number in
the unlabelled data, such as the algorithm from DTC [20). before adopting our method. In table S
we provide the results of novel category discovery using the estimated number of classes on CUB-
200/ImageNet-100 by DTC [20]. We denote the estimated class number C",, As shown in table 8
with the estimated class number by DTC, our model also obtains significantly better performance
than RankStat [8]
Table 8: Results with estimated class numbers and ground truth class numbers.
C0B-200
ImageNet- 100
Class numbers
C 40
Cn 43
Ca 30
Cn 32
RankStat 18
1.75%
38.2+ %
66.3++0.7%
65.0 1.2% 0.99
Ours
47.8 2.4%
44.6+t2.6 2.6 %
68.8 + 1.5%
Please refer to the supplementary for more results and analysis. Our code can be found at https //gi thub. http://///eemmatttaaaaaaaaaaaaaa
Conclusion
We have introduced two-branch learning framework for novel category discovery, with one branch focusing on local part-level information and the other branch focusing on overall characteristics. To transfer knowledge from labelled data unlabelled data, we proposed to use local ranking statistics by maintaining a local part dictionary build on-the-ft for training, together with the global ranking statistics. We further introduced mutual knowledge distillation method for information exchange and encouraging the agreement between the two branches. We thoroughly evaluated our method on generic image classification datasets as well as fine frninned recognition datasets, achieving sate--ofttheatt performance.
10

---

References
[1] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Varia- tional information distillation for knowledge transfer. In CVPR, 2019.
[2] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivie. Netvlad: Cnn architecture for weakly supervised place recognition. In CVPR, 2016.
[3] David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding, In ACM-SIAM Symposium on Discrete Algorithms, 2007.
[4] Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi. Label refinery: Improving imagenet classification through label progression. arXiv preprint 105.61180,.06611, 2018.
[5] Herbert Bay, Tinne Tuytelaars, and Lue Van Gool, Surf: Speeded up robust features. In ECCV, 2006.
[6] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach semi-supervised learning. In NeurIPS, 2019.
[7] Bingyi Cao, André Araujo, and Jack Sim. Unifying deep local and global features for image search. In ECCV, 2020.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020.
[9] Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE TPAMI, 2002.
[10] Boyang Deng, Qing Liu, Siyuan Qiao, and Alan Yuille. Few-shot learning by exploiting visual concepts within cnns. In arXiv preprint arXiv:1711.08277 2017.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[12] Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformerr: spatially-awar few-shot transfer. In NeurlPS, 2020.
[13] Zhiyuan Fang, Jianfeng Wang. Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. Seed: Self-supervised distillation for visual representation. In ICLR, 2021.
[14] Enrico Fini, Enver Sangineto, Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricei. A unified objective for novel class discovery. In ICCV, 2021.
[15] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. ICML, 2018
[16] Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimiza- tion. In ICCV, 2017.
[17] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018
[18] Kai Han, Sylvestre-Alvis Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. In ICLR, 2020.
[19] Kai Han, Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEEE TPAMI, 2021.
[20] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In ICCVC,

---

[21] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[24] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS 2014 Deep Learning Workshop, 2015.
[25] Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In ICLR, 2018.
[26] Yen-Cham Cen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. In ICLR, 2019.
[27] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint 1.07:11777.02199 2017.
[28] Xuihui Jia, Kai Han, Yukun Zhu, and Bradley Green. Joint representation learning and novel category discovery on single-and multi-modal data. In ICCV, 2021.
[29] Jangho Kim, SK Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. In NeurlPS, 2018.
[30] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13), 2013.
[31] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
[32] Harold Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 1955.
[33] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
[34] David G Lowe. Distinctive image features from scale-invariant keypoints. In IJC,, 2004.
[35] James MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1967.
[36] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.515/1 2013.
[37] Hyconwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. In ICCV, 2017.
[38] Avital Oliver, Augustus Odena, Colin Raffel, Ekin D Cubuk, and lan J Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS, 2018
[39] Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Semi-supervised learning with scarce annotations. In CVPR Deep- Viston workshop. 2020.
[40] Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Lsd-c: Linearly separable deep clusters. In ICCV Workshop on Visual Inductive Priors for Data-Efficient Deep Learning, 2021.
[41] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In /CLR, 2015.
[42] Oriane Siméoni, Yannis Avrithis, and Ondrej Chum. Local features and visual words emerge in activations. In CVPR, 2019.
12

---

[43] Kihyuk Sohn, David Berthelot, Chun- Li, Zizhao Zhang. Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NewrIPS, 2020.
[44] Iya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. JCML, 2013.
[45] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Mare Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii. Inloc: Indoor visual localization with dense matching and view synthesis. In CVPR, 2018.
[46] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NeurIPS, 2017.
[47] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
[48] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[49] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. Caltech- UCSD Birds 200. Computation & Neural Systems Technical Report, 2010.
[50] Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICML, 2016.
[51] Jay Yagnik, Dennis Strelow, David A. Ross, and Ruei sung Lin. The power of comparative reasoning. In ICCV, 2011.
[52] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation:
Fast optimization, network minimization and transfer learning. In CVPR, 2017.
[53] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
[54] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
[55] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Differentiable earth mover's distance for few-shot learning. In CVPR, 2020.
(56) Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018.
[57] Zhun Zhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighbor- hood contrastive learning for novel class discovery. In CVPR, 2021.
[58] Zhun Zhong, Linchao Zhu, Zhiming Luo, Shaozi Li, Yi Yang, and Nicu Sebe. Openmix: Reviving known knowledge for discovering novel visual categories in an open world. In CVPR, 2021.
[59] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Auda Oliva, and Antonio Torralba. Learning Deep Features for Discriminative Localization. In CVPR, 2016.
13


Reviews:
[{"review_id": "z36c1SmYQB0", "reviewer": "Reviewer_uqsh", "summary": "The paper tackles novel class discovery of unlabeled data via transfer learning-based two-brach learning with pairwise pseudo-labels by two different criteria: similarity on global descriptor and local features. The distinct criteria generate more robust pseudo labels than the single ones and alleviate the negative impact from false pseudo-labels in a complementary manner. The proposed mutual knowledge distillation further facilitates effective mutual learning between two branches. Extensive experiments on standard classification datasets and more challenging fine-grained datasets show that their method has significant advantages over SOTA owing to robust pseudo-labels by dual rank statistics.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* The paper is overall well-written and well-organized with clear motivation.\n\n* The setting of the novel class discovery is interesting and should be practically important.\n\n* The experiments are comprehensive and the results show clear improvement over previous SOTA.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. I wonder why the authors do not conduct fine-tuning on labeled data before training with unseen unlabeled data. Does initialization with self-supervised learning generate more robust pseudo-labels than a fine-tuned model in the early training stage? and why? In a similar vein, what initialization is used for the *Ours w/o Self-sup* in Table 4?\n\n2. Some experimental issues.\n    * How did the authors conduct validation? Due to the existence of unseen classes in unlabeled data, how to conduct validation is not trivial.\n    * Have different distillation approaches been investigated for mutual training? Such as the l2-regularization between $z_i$ and $z^\\prime_i$.\n\n3. Equation of Jensen Shannon Divergence (Eq. 6) is wrong. Jensen Shannon Divergence is $\\operatorname{JSD}(p, q)=\\frac{1}{2} D_{K L}\\left(p \\| \\frac{p+q}{2}\\right)+\\frac{1}{2} D_{K L}\\left(q \\| \\frac{p+q}{2}\\right)$.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "I read the paper thoroughly, and spent a few days for reviewing."}, "extra_text": {}}, {"review_id": "M2CDkdaxhC", "reviewer": "Reviewer_wP3g", "summary": "This paper introduces a new method to discover novel visual categories from unlabelled images. The experimental results demonstrate the effectiveness of the proposed method and achieve the state-of-the-art result on fine-grained visual recognition and object classification.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper utilizes dual ranking statistics to transfer knowledge from known classes in the labeled data to the unlabelled data and can generate a more robust pseudo label for learning on the unlabelled data. \n\nBesides, a mutual knowledge distillation method is proposed to allow information exchange and encourage agreement between the local and global branches.\n\nThe paper is well organized and easy to follow. The proposed model consists of two branches. Although the usage of the existing binary pseudo label limits the novelty of the model, the dual structure and the distillation to encourage agreement between branches are innovative. The global branch originates from [1] and the local branch includes a FIFO queue as a memory bank to query the ranking statistics as a dynamic object part dictionary.\n\nTypo: a redundant close parenthesis in the rear of Eqn (8).\n\n[1] Han K, Rebuffi S A, Ehrhardt S, et al. Automatically Discovering and Learning New Visual Categories with Ranking Statistics[C]//International Conference on Learning Representations. 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors should consider adding more pictures to make the paper easier to read. And add some case studies of the classification results.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "6KX2Yzfrxed", "reviewer": "Reviewer_iVtL", "summary": "This paper considers the problem of novel class discovery (NCD). Specifically, the authors propose a Dual Ranking Statistics method that can leverage both global and local factors for learning novel classes. In addition, a Mutual Knowledge Distillation approach is proposed to explore the relationship between the global and local branches, which can further improve the results. Experiments on several datasets, especially on fine-grained cases, show the benefit of the proposed method.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ Writing: This paper is well written and easy to follow. The overall paper is well-organized. Figures and Tables are clear to show the methods and comparisons.\n\n+ Novelty: This paper considers an interesting problem, i.e., novel class discovery. In addition, a fine-grained setting is introduced in this paper, which is not studied in previous NCD works. For the method, this paper introduces two simple but effective approaches, which explicitly take the advantage of both the global and local factors and obtain consistent improvement.\n\n+ Experiments: This paper presents extensive experiments to show the effectiveness of the components of the proposed method. The comparisons with other methods are comprehensive and fair. In addition, the authors also provide the experiments on more difficult settings: unknown number of novel classes and open-set unlabeled data.\n\nCons/Concerns:\n\nI have some questions about the method and the experiment.\n\n- Did you use the proposed losses for the labeled data, especially the local loss? If no, how about the results of using them for the labeled data. Will this improve the discrimination of the representation?\n\n- For the part dictionary memory bank, why not save all the parts of a sample into the memory? Could you provide the difference between using a random part of a sample and using all parts of a sample?  \n\n- Saving random/all part features into the memory is a bit wild? Is it possible to first find high-response regions (for example the activations) and only save them into the memory? \n\n- The proposed has two branches, which one is used for the testing stage? With JSD loss, I think these two branches may have similar results. However, when removing JSD loss, which one is used? \n\n- In Table 6, it seems that a larger memory size brings higher results. Why not using the larger memory size for all other experiments? Will it lead to a very large computation cost?\n\n- For the self-pretraining, how about the results of removing the self-pretraining? In addition, why use RotationNet for CIFAR while MOCOV2 for others? How about the results of using RotationNet/MOCO for others/CIFAR?\n\n- Which dataset is evaluated in Table 1 of Supple. Is it CIFAR-100?\n\n- Open questions: (1) In the open-set setting (Table 1 of Supple), did you use any technique to recognize the samples of known and unseen classes? If no, will the model assign all the samples to the known classes and give very few values/probabilities to the novel classes? (2) It would be nice that the authors can provide a visualization for the local comparison process. For example, we save many local features in the memory bank, when computing the similarity between each part of a sample and parts in the memory, we visualize the image rejoins of corresponding highly ranked parts. \n\nOverall, this paper considers an interesting problem and presents a novel and effective approach, which achieves the SOTA results. During rebuttal, please mainly answer the first 7 concerns. The authors can also selectively provide more discussions and visualizations for the open questions.\n\n**Post Rebuttal** After reading the response of the authors, my concerns are well-addressed. I also read the comments of other reviewers and I agree with them that this paper presents a novel approach for novel class discovery. Thus, I decided to keep my original rating and tend to give 'accept' to this paper.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "3FB3qAUglVr", "reviewer": "Reviewer_72SS", "summary": "The paper proposes a new technique for visual class discovery, which relies on soft label assignment via ranking statistics, computed from both global features and local/part patterns. One of the most novel components is a JSD loss to align global/part features; other components include ranking statistics to compute soft pseudo labels, and MSE for consistency of predictions under different transformations. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The approach seems sound. Some components are more novel (ranking statistics for labels, global/local alignment) and others are less novel. The contribution of each part seems properly verified through ablations. Results overall are strong enough, although margins to the baselines are not huge. In particular, both ranking statistics and global/local alignment are shown to have an impact. The particular manner of doing distillation, local-to-global, is also ablated. Multiple evaluation settings are shown, i.e. standard (coarse) classification as well as fine-grained.\n\nMinor questions: \n- the s_{ij} notation is a bit strange--usually labels are for one sample?\n- what is z_i^{'u} i.e. what is the ' here? (above Eq. 4)\n- there shouldn't be a sentence break on L134", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not discussed, not necessary", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel class discovery technique based on dual ranking statistics and mutual knowledge distillation. The main idea is reasonable and shows superior performance compared to existing methods. All reviewers are positive about this paper and there are no particularly negative comments. However, the proposed method is lacking in theoretical support and the experiments are limited to small datasets and/or relatively a small number of unlabeled classes. Overall, I recommend accepting this paper for a poster presentation.
Decision: Accept (Poster)