A Flexible Framework for Discovering Novel Categories with Contrastive Learning
Authors: Xuhui Jia
Kai Han
Yukun Zhu
Bradley Green
Venue:  (2021)
Keywords: deep learning
novel classes
clustering
self-supervised learning
unsupervised learning

Abstract: This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we proposed using category discrimination on labelled data and cross-modal discrimination on multi-modal data to augment instance discrimination used in conventional contrastive learning approaches. We further introduce Winner-Take-All (WTA) hashing algorithm on the shared representation space to generate pairwise pseudo labels for unlabelled data to better predict cluster assignments. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFAR10, CIFAR100 and ImageNet, obtaining state-of-the-art results.

Under review as a conference paper at ICLR 2021
A FLEXIBLE FRAMEWORK FOR DISCOVERING NOVEL CATEGORIES WITH CONTRASTIVE LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
This paper studies the problem of novel category discovery on single- and multi- modal data with labels from different but relevant categories. We present generie, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particu- lar, we proposed using category discrimination discrimination on multi-modal data to augment instance discrimination used conventional contrastive learning approaches. We further employ Honner-Take--e All (WTA) hashing algorithm on the shared representation space to generate pair- wise pseudo labels for unlabelled data better predict cluster assignments. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFARIO, CIFARI00 and ImageNet, obtaining state-of-the-ar results.
labelled data and cross-modal
INTRODUCTION
With the tremendous advances in deep learning, recent machine learning models have shown superior performance on many tasks, such image recognition (Deng et al. 20099 Kuznetsova et al. 2020).
object detection (Zhou et al.) 2019; Tan et al., 2020). image segmentation (Cheng et al. 2019). etc. While the state-of-the-as these models heavily relies on the huge amount of data with human annotations under the closed- world assumption. Applying deep learning in real (open) world brings many new challenges: is cost-inhibitive to identify and annotate all categories, and new categories could keep emerging. Conventional methods struggle on handling unlabelled data from new categories (Fontanel et al.) 2020). On the flip side, real world provides rich unlabeled data, which are often multi-modal (e.g.. video and audio), allowing more possibilities for machine learning models to learn in a similar way as human. Indeed, human learn from modal data everyday with text, videos, audios, etc.
models might even outperform human in these tasks, the success of
In this paper, we focus on automatically learning to discover new categories in the open world setting. Similar to recent work (Han et al. 2019) 2020) which transfer knowledge from labelled images of few classes to other unlabelled image collections, we formulate the problem as partitioning unlabelled data from unknown categories into proper semantic groups, while some labelled data from other categories are available. This is a more realistic setting than pure unsupervised clustering which may produce equally valid data partitions following different unconstrained criteria (e.g., images can be clustered by texture, color, illumination, etc) and closed-word recognition which can not handle unlabelled data from new categories without any labels. Meanwhile, our setting more similar the human cognition process where humans can easily learn the concept of new object by transferring knowledge from known objects.
Specifically, we introduce a flexible end-to-end framework to discover categories in unlabelled data, with the goal of utilizing both labelled and unlabelled data to build unbiased feature representation, while transferring more knowledge from labelled to unlabelled data. In particular, we extend the conventional contrastive learning (Chen et al.) 2020b; He et al. 2020) to consider both instance discrimination and category discrimination to learn reliable feature representation labelled and unlabelled data. We also demonstrate that the cross- modsl discrimination would further benefit representation learning data with multi-modalities. To leverage more of unlabelled data,

---

Under review as a conference paper at ICLR 2021
introduce the Winner-Take- (WTA) hashing Yagnik et al. 2011) on the shared representation spaces to generate pair-wise pseudo labels on-the-fly, which is the key for robust knowledge transfer from the labelled data to unlabelled data. With the weak pseudo labels, the model can be trained with a simple binary cross-entropy loss on the unlabelled data together with the standard cross-entropy loss on the labelled data. This way our model can simultaneously learn feature representation and performs the cluster assignment using an unified loss function.
The main contributions of the paper can be summarized as follows: (1) we propose a generie, end-to- end framework for novel category discovery that can be trained jointly on labelled and unlabelled data; (2) to the best of our knowledge, we are the first to extend contrastive learning in novel category discovery task by category discrimination on labelled data and cross rrosslmodal discrimination on multi- modal data; (3) we propose strategy to employ WTA hashing on the shared representation space of both labelled and unlabelled data to generate additional (pseudo) supervision on unlabeled data; and (4) we thoroughly evaluate our end-to-end framework on challenging large scale multi-modal video benchmarks and single-modal image benchmarks, outperforming existing methods by significant margin.
RELATED WORK
Our method is related to self-supervised learning, semi-supervised learning, and clustering, while different from each of them. We review the most relevant works below.
Self-supervised learning aims at learning reliable feature representations using the data itself to provide supervision signals during training. Many pretext tasks (e.g., relative position (Doersch et al. 2015), colorization (Zhang et al. 2017), rotation prediction (Gidaris et al. (018)) have been proposed for self-supervised learning, showing promising results. Recently, the constrastive learning based methods, such (He et al. (220)) and (Chen et al. 2020b), have attracted lots of attention by its simplicity and effectiveness. The key idea of contrastive learning is instance discrimination, i.e., pulling similar pairs close and pushing dissimilar pairs away in the feature space. (Khosla et al., 2020) studied the supervised contrastive learning on labelled data as an alternative of cross-entropy. With the labels, more positive pairs can be generated from the intra-class instances, enabling category discrimination.
Estimation (NCE) (Gutmann & Hyvärinen 2010; van den Oord ct al.] 2018) is an effective contrastive loss widely used in these methods. When handling multi-modal data like videos, different self-supervised learning methods have been proposed to ex expioit data of different modalities, such as (Patrick et al., 2020 Alayrac et al.) 2020 Asano et al. 2020: Alwassel et al. 2019 Morgado et al.] (2020). Among them. (Morgado et al. 2020) suggests that cross-modal discrimination can be adopted to improve the representation learning for downstream tasks like image recognition and object detection, which implies that good representations are shared between multiple views of the world. TTan et al. (2019) shows that cross-view prediction outperforms conventional alternatives in contrastive learning images, depth, video and flow, and more views can lead to better representation. In this paper, we consider the visual and audio modalities for cross-modal learning in videos, and present a new way to incorporate contrastive learning for both labelled and unlabelled data to bootstrap representation learning for novel category discovery.
Semi-supervised learning (Chapelle et al. (006) considers the setting with labelled and unlabelled data. Specifically, the unlabelled data are assumed to come from the same classes as the labelled data. The objective is learn a robust model making use of both labelled and unlabelled data to avoid over-fitting to the labelled data. While this problem is well studied in the literature (e.g.. OOliver al. 2018 Tarvainen & Valpola| 2017 Rebuffi et al., 2020a)), existing methods can not handle unlabelled data from new classes. In contrast, our method is designed to discover new categories in the unlabelled data automatically.
Clustering, which aims at automatically partitioning the unlabelled data into different groups, has long been studied in the machine learning community. There are many classic methods (c.g., k- means (MacQueen) 1967). mean-shift (Comaniciu & Meer 1979)) and deep learning based methods (e.g., (Xie et al. 2016 Dizaji et al. 2017 Rebuffi et al. (020b)) showing promising results. However, the definition of a cluster can be intrinsically ambiguous, because different criteria can be used to cluster the data. For example, objects can be clustered by color, shape or texture, and the clustering results will be different by taking different criteria, while these criteria cannot be predefined in the
2

---

Under review as a conference paper at ICLR 2021
clustering methods. In this paper, we aim to learn these criteria implicitly from the labelled data and transfer them to the unlabelled data on-the-fly.
Until recently, the problem of discovering data starts to be considered. (Han et al. 2019) introduces a method to first pretrain the model on labelled data followed by fine-tuning with unsupervised clustering loss. (Han al.. 2020) presents a three-stage method for new image category discovery by transferring labels from the labelled data the unlabelled data via feature ranking statistics. The model needs be pretrained sequentially for rotation prediction GGdarrs et al. 2018) and supervised classification. (Hsu et al. 20188) and (Hsu al. 2019) can also be applied to discover new categories in image datasets. Both methods require to maintain two models separately. One model is for binary label prediction and the other is for clustering. In contrast, our method is a flexible end-to-end framework which can be applied to single- and multi-modal data. As will be shown in the experiments, our method substantially outperforms others in all benchmarks.
categories in unlabelled data by exploiting the labelled
3 METHOD
Given an unlabelled collection of instances x E D our objective is automatically partition these instances into C" different semantic groups. We also assume that there is a labelled collection of instances (ad, v/) E D' where sa E (I....,C' from which we want to transfer the knowledge to the unlabelled data that the unlabelled data can be grouped into proper classes, while the classes in D' and D are different but relevant. Each Te E D U D' can be either an image or a multi-modal video consisting of visual stream z. and the corresponding audio stream 2ç-
Our approach is an end-to-end trainable framework that can jointly learn the representation and clustering assignment for the unlabelled data. Figure [ shows the overview of our approach. Consider a video clip x; = (x" ..)), the visual encoder f. and the audio encoder fo first encode the visual and audio streams into two feature vectors = and z which are then concatenated to form the global representation z; = [= = for the input video clip. A projection function then project z; to ž; to fuse multi-modal feature in compact representation space. Note that 1j is an identity mapping function for single-modal data. If 2; is from labelled data, &, will be sent to the linear head ø' for supervised learning. Otherwise, it will be sent to the linear head d² for new category discovery. In addition, = and 2 are further encoded to E and 2n by h2 and ha respectively for constrastive learning to bootstrap the representation learning on both labelled and unlabelled data. hg and ha are two MLP functions followed by le normalization, which is common practice in contrastive learning.
The end-to-end training of the model consists of several important components including training on labelled data with full supervision, constrative learning with instance and category discrimination on both labelled and unlabelled data, and training on unlabelled data with pair-wise pseudo labels transferred from the representation jointly learned by the previous two. To effectively transfer knowledge from the labelled to unlabelled data, we employ the winner-take-all (WTA) hashing algorithm on Σ, Next, we will introduce our extended contrastive learning for novel category discovery by considering both instance and category discrimination, and knowledge transfer via the WTA hashing in more details.
3.1 UNIFIED CONTRASTIVE LEARNING ON LABELLED AND UNLABELLED DATA
Given batch of N data points that are randomly drawn from both D and D i.c., x; Unif(D"u D') for all the elements 2; in batch B, our objective is to extract as much information as possible from B to learn a representation that can be used to cluster unknown classes. Recent developments in contrastive learning focus on learning image representation under self-supervised scenario, where no labels are available, therefore it can be thought of as contrasting instance representations from a pair of data augmentation (positive) to those of other samples (negative). In our case, we have a mixed setting with both labelled and unlabelled data. We adopt the contrastive learning to jointly learn from the mixed dataset for novel category discovery. Next, we first demonstrate our unified contrastive learning single-modal data, and then extend to the more flexible multi-modal scenario.
3

---

Under review as a conference paper at ICLR 2021
Contrastive learning
fr
fa
Visual
Audio
WTA
Sy
1. L,BEE
Figure I: Overview of our end-to-end framework. For multi-modal videos, our framework consists of two feature encoders Ju and fa. two MLPs hi and ha for contrastive learning, one fusion layer and two linear heads o' and 6² for classification and clustering. The training signal for Ø" is obtained by WTA on-the-fly. For single-modal images, the audio encoder fe is omitted and 17 turns to be an identity mapping function.
3.1.1
BAGGEE-ODAL LEARNING
Let æ; be a single-modal data point and E W .1...., 2N be the index of pairs of augmented samples from the batch B. With the embedded representation & of 2,. adopt NCE (Gutmann & Hyvärinen 2010) as our contrastive loss function for instance discrimination, which can be written
exp (2: ///TT)
CNCE-I - log Σn [n exp (2; /n/T)
where 2 is the augmented counterpart of 2, 1 [nyts) is an indicator function evaluating iff # i a scalar temperature. This loss widely used in conventional contrastive learning approaches.
In our case, some data points in the mini-batch B are accompanied with labels. If 2; is labelled data point in the batch, besides considering only x; and its transformed counterpart as a positive pair, other data points from the same class 2G also be paired with x; to form more positive pairs to be pulled together, allowing category discrimination. The contrastive loss for category discrimination with these additional positive pairs can be written as:
exp (2; /g/TT Σn, 1 inzti exp (2: 2n/T))
C NCE-C
Σ log gEQ(1)
(2)
1Q(1)
where Q(i) (())=19EN) E /:: g:
ys} denotes the indices of other data points which have the same label as I; in the batch B. Note that this is effective only for labelled data, as Q(i) 0 for unlabelled data. Therefore, the unified contrastive loss can be written as:
2N
CCL
1
Σ
+ (
2N
Note that there is only one positive pair in C.¹ LNCE-I which aims instance discrimination, whereas multiple other samples from same class considered positive in L C.CCEC which aims at class discrimination. C.NCE-I and C. C.NCE-C have exactly the same denominator, which is the summation of 2N - scores for both positive and negative pairs. In this way, both labelled and unlabelled data are used for representation learning while making full use of the labels contained in the labelled data. C. NCE--C is critical complementary C CNCE-1 resulting more discriminative representation space for robust clustering will be seen in the experiments.
3.1.2 MOLAI-MODAL LEARNING
For multi-modal data like videos, besides the conventional within-modal contrastive learning for single-modal data like images, also have the additional cross-modal option. As noted
4

---

Under review as a conference paper at ICLR 2021
by (Alwassel et al. 2019: Morgado et al. 2020). cross-modal agreement leads to better representation
than within-modas agreement in self-supervised representation learning for supervised downstream tasks like object recognition and detection. However, setting contains both labelled data and unlabelled data, resulting in a mixture of instance discrimination and class discrimination. It is not immediately obvious whether the within-modal or cross-modal choice is more effective under such setting. For within-modal case, we can either discriminate visual or audio samples. Let the embedded representation for a multi-modal data point 2; be & = { ". We define modality selecting functions Go and 91 to allow either within- modal discrimination (e.g., 90(24) Σ and Φ1. (&:) 2() or cross- mossal discrimination (e.g.. gu (2) 2 and 91 (2,) = 2 ). The instance discrimination and category discrimination objectives in Eq. ① and Eq. (2 can then be rewritten as
exp (90(2)) 91 (2x))//T
C, CNCR-I log Σn 1 [mzzil exp (g)(()) g1 (1(
(4)
and
exp (gu (3)( 91 (2g)/T) [n exp (00 (2:) 9n (2,))/T)
CNCE-C 4 = [Q(1)
log
(5)
(EQ())
As will be shown in the experiments, after investigating different strategies, we find that the cross- modal contrastive learning can produce better representation for novel category discovery. This might be due to the fact that cross eross-modal representations are intrinsically different, thus reducing the chance of obtaining trivial solutions such as discriminating objects by only verifying the most salient features in visual or audio modality alone. Therefore, unless stated otherwise, the modality selecting functions go and 91 choose visual and audio representations respectively in our formulation.
3.2 KNOWLEDGE TRANSFER USING WINNER-TAKE-ALL HASH
To leverage the labelled data to help novel category discovery in unlabelled data, we transfer knowl- edge from labelled data to unlabelled data by adopting the Winner-Take-All (WTA) hash (Yagnik) et al., 2011) during training. WTA is sparse embedding method that maps the feature vectors into integer codes. In this work, we employ WTA to measure the similarity between each pair of unlabelled data points from new categories in the shared embedding space of both labelled and unlabelled data, so that we can transfer knowledge from the labelled categories to the unlabelled ones.
The idea of WTA Yagnik et al. 2011)) is to measure similarity between nigh-dimensional feature vectors by comparing multiple partial ranking statistics. The WTA algorithm works as follows. First, randomly generate set of H permutations P {Pr,.... pt1) )- For an unlabelled sample x; D", we extract its feature vector ž, We then apply each Ph. E P on ž, to obtain a transformed feature vector ph (æ,), i.e., shuffled version of 2, Let c be the index of maximum value in the first elements of ph (Z.). We can then obtain the WTA hash code by c, (c....,, ..., .......)))) for each 24.
In our case, we employ WTA hash code to measure the similarity Siy between D; and x to generate pairwise pseudo labels for novel category discovery, by simply comparing their WTA hash codes c, and ey:
1. (c >=
(6)
O, otherwise
where j is an empirical scalar threshold. Note that WTA is only applied during training to generate binary pseudo labels, and it is not needed test time.
As discussed in (Yagnik et al. 201T), the precise values in the high-dimensional embedding is often not important, and the relative magnitude matters more. While requiring the total orderings to be identical is too strict for real application, as noise inevitably exists even using ranking statistics, therefore, WTA introduces multiple partial order statistics. Besides introducing more resilience to noise, we further emphasize that the partial orders for local rank correlation also captures the relative structural information of the objects. Intuitively, the global ranking statistics may only consider the most salient features in the embedding space, while the multiple partial orders spread more in the embedding space, thus capturing more structural information. As shown in (Zhou et al. 2020). modern CNNs very likely make decisions by focusing on the salient patterns while overlooking the holistic structural composition. As will be seen in the experiments, WTA outperforms other alternatives for generating pairwise pseudo labels to discover categories.
5

---

Under review as a conference paper at ICLR 2021
After applying the WTA algorithm, data points with Eq. ( Assume we have M unlabelled samples in a mini-batch. By using these pairwise similarities pseudo labels, we can then train the model with binary cross-entropy loss to simultaneously learn representation and cluster assignments on the unlabelled data from new classes:
obtain pairwise similarity St for each pair of unlabelled
M
1 BCC
Sij log 6" (2,) 8" (2,) + (1 -sy) log (1 (1- 6" (zi) (1))"(z)) 8" (z)) (7)
M
where o² Rd RC is non-linear function mapping ž; into an embedding space with the same dimension as the number of classes in the unlabelled data followed by softmax normalization. In this way, we can obtain the cluster assignment for each unlabelled sample by indexing the location of the maximum value in 6 (2x) after training, without requiring another offline clustering procedure for class assignment.
3.3 JOINT LEARNING OBJECTIVE
Inspired by the literature of semi-supervised learning (Tarvainen & Valpola 2017), ) we include a consistency regularization loss on both labelled and unlabelled data. The purpose of such consistency loss is to enforce the class predictions on a data point z; and its transformed counter part t(x;) to be the same. This is especially important for unlabelled data samples. By enforcing the consistency, Ij and t(x;) will be treated as positive pair regardless of the WTA hash code, as the WTA hash code of z; might be different from that of (anz), thus smoothing the training. The consistency loss is commonly implemented as the mean squared error (MSE) between the class predictions. Let LMSE be the consistency loss and _CE be the cross-entropy loss on labelled data. The consistency loss between I; and its transformed version z; is defined as
CSE (d(z)) - o())))
(8)
where @ is o' or d" depending on the input. The overall training loss of our end-to-end framework can then be written as
C C- + BCE + -1- (1-wwr))cCL + (
where (r) is ramp-up function slowly increasing from 0 to 1 along with the training. In our experiment, we follow (Laine& & Aila, 2017) to use w(r) = Ae 55--5) where r, T and À are current epoch number, total number of epochs and a positive scalar factor. We set (1 (r)) as the weight for contrastive learning. At the early stages, the cluster assignment predictions are noisy and we expect the model to focus more on representation learning, thus a higher weight is set to representation learning and lower weight is set for consistency, In the late stages, the representation is good enough and we would like the model to focus on novel category discovery, therefore the weight is higher for consistency loss and lower for contrastive learning.
EXPERIMENTS
Benchmarks and evaluation metric. We comprehensively evaluate our approach for novel cate- gory discovery large-scale image benchmarks including ImageNet (Deng et al.) (009)
10 (Krizhevsky & Hinton| 2009) 100 (Krizhevsky & Hinton 2009) and video benchmarks
including Kinetics-400 (Kay al. 2017V VGG-Sound (Chen et al. 2020ap. We follow (Han et al.) 2020) to split the ImageNer/CIFAR 1O/CIO///IRRII 100 to have 30/5/20 classes in the unlabelled classes. For fair comparison with (Han et al. 2019: 2020 Hsu et al. 2018b) three 30-class splits used for ImageNet, and the results are averaged over the three splits. We split Kineties-400/GGG-Sooe to have 50/39 classes in the unlabelled data, which are much more challenging than the image benchmarks. As only URLs are publicly available for video datasets, by the time of our experiments, we have 170k video clips with sound in Kinetics-400 and 183k videos with sound VGG-Sound. To measure the novel category discovery accuracy, we adopt the widely used average clustering accuracy defined P(C. = Σ 1sy." e. where y.¹ and ÿ.' denote the ground-truth -ruth label and predicted cluster assignment for each unlabelled data point, is the total number of unlabelled instances in the whole dataset, P(C") denotes the set of all possible permutations of C" elements,

---

Under review as a conference paper at ICLR 2021
is arbitrary permutation in P(C"). We obtain the optimal permutation c" by Hungarian algo- rithm (Kuhn [955). Note that as no supervision is used for unlabelled data, the same data are used for both training and evaluation following standard practice
(Ji ct al.
2019 Han et al. 2019).
Implementation details. We use R3D-18 (Tran et al. 2018 Ji et al. ResNet-18 (He al., 2016) as the image and audio encoder. The feature vector dimension is 512 for both encoders. The MLPs for contrastive learning consist a hidden layer of size 512, a linear layer of size 128, and an 6-2 normalization layer (Wu et al. 2018). The output dimension of the hidden layer is 512. On image benchmarks, we follow SimCLR (Chen et al. 2020b) to randomly apply cropping, resizing, horizontal flip, color distortion, and Gaussian blur for data augmentation. On video benchmarks, we follow (Szegedy et al.) [015) to use the input video and audio clips of 1 and 2 second duration respectively. Video frames are resized such that the shorter side has a size of 128, followed by random cropping with size 112. We preprocess the audio by randomly sampling within 0.5 seconds of the video and compute a log mel bank features with 257 filters and 199 time- frames, followed by SpecAugment (Park al.) 2019). During evaluation, we follow (Patrick et al. 2020) to uniformly sample 10 clips from each video and take the mean score for prediction. For WTA, we set H to be equal the feature dimension (i.e., 512), and follow (Yagnik et al. 2011) to set l as 4. For the threshold μ., we empirically set to 260 in our experiments. We train our models with SGD (Sutskever al.) (203)) and use batch size of 1024 for all benchmarks except IEFFAR-O/CIFAR-- we use a batch size of 256. The models are trained with 8 x 8 TPUv2 Dragonfish devices.
2012) as the video encoder and
which
4.1 ABLATION STUDY
Component analysis. We validate the effectiveness of each component of our method during training both image and video benchmarks. The results are reported in table As can be seen, all components are important to our method. Removing the BCE loss causes the most performance drop. Namely, the ACC drops from 93.4% 32.2% CIFAR-10, 76.43% 15.5% CIFAR-100 and 56.5% + 5.8% on Kinetics-400, suggesting that the WTA-hashing comparison can indeed generate reliable pairwise pscudo labels for the BCE loss. Removing the contrastive learning leads to significant performance drop (row 4 vs row 6). By comparing rows 5 and 6, we can see that instance discrimination (NCE-I) alone is not good enough for the task of novel category discovery, and incorporating the category discrimination (NCE-C) is very effective. Please refer to the supplementary for comparison between WTA and other alternatives like cosine similarity.
Table 1: Ablation study on image and video benchmarks. MSE: consistency constraint; CE: cross entropy loss; BCE: binary cross entropy loss; NCE-I: NCE for instance discrimination; NCE-C: NCE for category discrimination. Results on CIFAR- 1--/OO//IIRR-III 100 are averaged over 10 runs.
No MSE
CE
BCB
NCEI
NCE-C
CIFAR-10
CIFAR-100
Kinetics-400
(I)
X
/
/
91.3 + 1.4%
74.7 + 2.9%
50.9%
(2)
/
X
/
77.4 + 5.4%
68.5 + 4.7%
24.5%
(3)
/
/
32.2 = 1.8%
15.5 1.2%
5.8%
(4)
/
/
x
89.8
0.9%
70.1 1: 4.5%
47.6%
(5)
/
/
X
91.6 1.2%
75.2 : 3.1%
51.7%
(6)
/
4
V 93.4 0.6%
76.43 2.8%
56.5%
Multi-modal contrastive learning. As mentioned in section 3.1 for multi-modal data like videos, we can have multiple contrastive options. Namely, we can conduct within-modal and cross-modal contrastive learning for each of NCE-I and NCE-C. We present the results of different options in table 2 By comparing rows 1 and 4, we can see that cross cross-modal is more effective than within- modal instance discrimination. can be seen from rows 1-3 and rows 4-6 that NCE-C can effectively improve the representation for novel category discovery. Also, it is important to consistently use either within-modal or cross-modal discrimination for NCE-I and NCE-C (row 2 vs row 3; row 5 vs row 6). Overall, cross-modal contrastive learning for NCE-I and NCE-C (row 6) performs notably better than all other cases. We hypothesize this is because enforcing the cross- moossl agreement can avoid the modal falling into the trivial solution, such as simply identifying the most salient visual or audio features. The modality discrepancy can effectively avoid such cases, thus improving the quality of learned representation.
7

---

Under review as a conference paper at ICLR 2021
Table Multi-modal contrastive learning. "(a, a)" denotes the audio-modal contrastive; "(v, 8)" denotes visual-modal contrastive; and "(a, v)" cross-modal contrastive.
No NCE-I
NCE-C Kinetics-400
(1)
(a,a) + (v,0)
46.2%
(2)
(a,a) + (0,0)
(a, + (v,v)
51.4%
(3) (a,a) + [e,0)
(a,v)
49.1%
(a,
51.7%
(a,e)
(a,a) + (v)
56.1%
(6) (a,e)
(a.
56.5 %
4.2 NOVEL CATEGORY DISCOVERY ON IMAGE BENCIMARKS
In table 3 we compare our approach with the k-means baseline and the previous state-of-the-art methods. By comparing row 10 with rows 1-6, we can clearly see that our method substantially outperforms the h-means baseline and existing methods. For example, our method outperforms the previous statc-of-the- of-thh-art method (Han et al.) 2020), denoted as RS in 5, by 3%/3.2%/4.22 CIFAR-IOCIIAAR 100010000/meeeettt We also report the incremental learning result of RS (Han et al. 2020) in row 6, where they introduce additional step to create interactions between labelled heads and unlabelled heads. The k-means baseline row is evaluated on the features of unlabelled data extracted using the model pretrained on labelled data by cross-entropy loss. We further enhance the k-means baseline, KCL (Hsu et al., 201 8a) and MCL (Hsu et al. (019) by introducing the same constrative learning used in our framework (see rows 7-9). The contrastive learning successfully boosts the performance for all of them. but the results still largely lag behind our method. For example, after the enhancement, the ACC of KCL increased to /5. 4% 74.3% on CIFAR- 10/CIFAR- m0//mmg while the results of our method are 93.4 / 7.477664446 //6 /86.7% 7%, demonstrating the effectiveness of our approach on image benchmarks.
Table 3: Novel category discovery on image benchmarks. "CL" denotes the contrastive learning introduce in section 3.1 which performs both instance discrimination and category discrimination.
Method
CIFAR-10
CIFAR-100
ImageNet
(1)
k-means (MacQueen.
1967
65.5 + 0.0%
56.6 + 1.6%
71.9%
(2)
KCL (Hsu ct al.
2018a
66.5 + 3.9%
14.3 + 1.3%
73.8%
MCL
Hsu et al.
2019
64.2 :: 0.1%
21.3 + 3.4%
74.4%
DTC
Han
2019
w/ S.S.
88.7 + 0.3%
67.3 + 1.2%
RS
Han
2020
90.4 0.5%
73.2 2.1%
82.5%
RS
Han al.
2020
w/ I.L.
91.7 0.9%
75.2 + 4.2%
(7) k- means (MacQueen,
1967 w/ CL
73.2
58.8 t 1.5%
73.1% 1%
(8)
KCL Hsu ct al.
2018a
w/ CL
73.9 0.1%
57.4 + 3.6%
74.3%
(9) MCL. HSu ct al.
2019
w/ CI.
72.3 + 0.2%
50.8 + 3.1%
75.2%
(10) Ours
93.4 0.6%
76.4 2.8%
86.7 %
4.3 NOVEL CATEGORY DISCOVERY ON VIDEO BENCHMARKS
In table 4 we compare our framework with k-means baseline and (Han et al. 2020) Kinetics-400 and VGG-Sound benchmarks. In rows 1-3, we train the models on labelled data with cross-entropy loss, and extract the features on unlabelled data for k-means. We validate features from multi- modalities as well of each single-modality. The multi-modal features are more effective than the single-modal features. To compare with the previous state-of-the-an method (Han al.) 2020). which is designed for image category discovery, implement its multi-modal counterpart (row 4) and also enhance it by replacing the original RotNet pretraining with our improved contrastive learning (row 5). It can be that the contrastive learning introduced in framework can further boost the performance of (Han et al.) 2020), and our approach consistently outperforms the k-means baseline and (Han et al. 2020 by large margin. For example, method achieves 56.5%/50.0% ACC on Kinetic-400/VGGG-SO
while Han et al. 2020) only gives 31 2%/38.6% ACC.
Please refer the appendix for more results.
8

---

Under review as a conference paper at ICLR 2021
Table 4: Novel category discovery on video benchmarks.
Method
audio
video
Kinetics-400
VGG-Sound
(1) k-means
MacQueen 1967
x
40.9%
32.4%
k-means
MacQueee
1967
18.7%
41.7%
k-means
MacQueen
1967
4
41.1%
43.4%
(4)
RS (Han et al.
2020
w/ RotNet
/
31.2%
38.6%
(5)
RS
(Han et al.
2020
w/ CL
V
33.5%
42.2%
Ours
56.5%
50.0%
CONCLUSION
In this paper, we have presented flexible end-to-end framework tackle the challenging problem of novel category discovery. First, we extended the conventional contrastive learning to perform instance discrimination as well as category discrimination jointly by making full use the labelled data and unlabelled data. In this way, the learned representation is not biased towards labelled or unlabelled data, thus can be effectively used to discover new categorics. Second, to successfully transfer knowledge from the labelled data to the unlabelled data, employed the WTA hashing algorithm to generate pair-wise weak pseudo labels for training on unlabelled data, which is the key automatically partition the unlabelled data into proper groups after training. Third, for multi-modal data, we investigated different ways contrastive learning and we empirically found that cross-modal noise contrastive estimation performs consistently better than other options. Last, thoroughly evaluated our approach on challenging image and video benchmarks and obtain superior results in all cases, demonstrating the effectiveness of our approach.
REFERENCES
Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. arXiv prepreint arXiv:2006. 16228, 2020.
Humam Alwassel, Dhruv Mahajan, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self- supervised learning by cross cross-modal modal audio-vidco clustering. arXiv preprint arXiv:1911.1266 2019.
Yuki M. Asano, Mandela Patrick, Christian Rupprecht, and Andrea Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-supervision. arXiv prepreint arXiv:2006. 13622, 2020.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-Supervised Learning. MIT Press, 2006.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML,, 2020b.
Bowen Cheng, Maxwell Collins, Yukun Zhu, Ting Liu, Thomas S Huang. Hartwig Adam, and Liang-Chich Chen. Panoptic-decplab. arXiv preprint
2019.
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE TPAMI, 1979.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Kamran Ghasedi Dizaji, Amirhossein Herandi, Cheng Deng, Weidong Cai, and Heng Huang. Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization. In ICCV, 2017.

---

Under review as a conference paper at ICLR 2021
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Dario Fontanel, Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulò, Elisa Ricci, and Barbara Caputo. Boosting deep open world recognition by clustering. arXiv preprint a004v:200..13849 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ICLR, 2018.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning discover novel visual categories via deep transfer clustering. In ICCV, 2019.
Kai Han, Sylvestre-Alvis -Aliise Rebuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automatically discovering and learning new visual categories with ranking statistics. ICLR. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning to cluster in order to transfer across domains and tasks. In ICLR, 2018a.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. A probabilistic constrained clustering for transfer learning and image category discovery. In CVPR Deep- Visiinnsion workshop, 2018b
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classification without multi-class labels. In ICLR, 2019.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE TPAMI, 2012.
Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, pp. 9865-9874, 2019.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint
mete::0055095590 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.11330 11362, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, pp. 1-26, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In ICLR, 2017.
James MacQueen. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium Mathematical Statistics and Probability, 1967.
10

---

Under review as a conference paper at ICLR 2021
Pedro Morgado, Nuno Vasconcelos, and Ishan Misra. Audio-visual instance discrimination with cross-modal agreement. arXiv prepreint arXiv:2004.12945 2020.
Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS, 2018.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheme Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.088 087779, 2019.
Mandela Patrick, Yuki Asano, Ruth Fong, João Henriques, Geoffrey Zweig, and Andrea Vedaldi. Multi-modal supersupeo from generalized data transformations. arXiv preprint arXiv:2003.04298, 2020.
Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Semi-supervised learning with scarce annotations. In CVPR Deep Vision Workshop. 2020a.
Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Lsd-c: Linearly separable deep clusters. arXiv preprint arXiv:2006. 10039, 2020b.
lly Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In JCML, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In CVPR, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 1785v 2019.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. CVPR, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv prepreint arXiv: 1rXiv:8807. 807.03348, 2018.
Zhirong WU, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In ICMI., 2016.
Jay Yagnik, Dennis Strelow, David A. Ross, and Ruei sung Lin. The power of comparative reasoning. In ICCV, 2011.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In CVPR, 2017.
Mohan Zhou, Yalong Bai, Wei Zhang, Tiejun Zhao, and Tao Mei. Look-into-object: Self-supervised structure modeling for object recognition. In CVPR, 2020.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. arXiv:1904.07850, 2019.
Objects as points.
arXiv preprint
11

---

the unlabelled classes in the Under review as a conference paper at ICLR 2021
A REPLACE STATISTICAL RANKING WITH WTA IN RS
To better understand the role of WTA in our framework, investigate RS Han al. (2020) furhter by replacing the ranking statistics with the WTA used in our framework. The results are reported in Table 5 It can bee seen that simply replacing ranking statistics with WTA cannot boost the performance. Other components of our approach are also important, as shown in Table [ into a unified framework, it achieves superior performance.
Integrating
Table 5: Replacing ranking statistics in Han et al. (2020) by WTA
Dataset
RS Han et al. (2020)
RS w/ WTA
CIFAR-10
90.4
90.1
CIFAR-100
73.2
67.3
B PARAMETER ANALYSIS FOR WTA
We obtain the two hyperparamters of WTA, i.e. threshold and window size k. by examining different values on the labelled data. We further split the labelled data into smaller labelled subset and unlabeled subset (i.e. pretending part of the labelled data to be unlabelled), and find and k that give the best results on the unlabelled subset. More specifically, for Kinetics-400, we split 350- class labelled set into 300-class labelled subset and 50-class unlabelled subset. For we split the 270-class labelled subset into 23 -class labelled subset and 39-class unlabelled subset respectively. The number of unlabelled classes used here are the main experiments.
Taking the empirical WTA window size k 4 in Yagnik et al. (2011), experiment with different threshold and report the results in table We find that the performance is generally stable for greater than 200. We set y to 240 in our experiments. Note that the feature vectors have dimension of 512.
Table 6: Performance of different WTA threshold on video benchmarks.
Dataset
130
180
200
240
260
300
Kinetcis-400
19.4
39.7
55.2
54.5
54.8
54.5
VGG-Sound 21.5
41.6
49.6
51.3
50.9
47.9
Given μ 240, we sweep different & and report the results in table 7 We find that the k = 4 and perform comparably well, and they are both better than 2 and i: 16. Hence, we simply use the & = 4 in experiments.
Table 7: Performance of different WTA window size on video benchmarks.
Dataset
k =
2
k
=
4
k =
k == 16
Kinetcis-400
52.6
56.7
55.9
49.4
VGG-Sound 46.2
51.8
51.0
48.3
C COMPARING WTA WITH OTHER ALTERNATIVES
We compare WTA and other alternatives to transfer pseudo labels on-the-fly in framework. We compare WTA with cosine similarity, ranking statistics (Han et al.) (020). and nearest- neiress-neighbour. Shows the results. We can see that WTT performs significantly better than all other alternatives.
12

---

Under review as a conference paper at ICLR 2021
Table WTA other alternatives.
Method
Kinetics-400
VGG-Sound
cosine
35.4%
29.8%
nearest-neighbour
19.6%
11.5%
ranking statistics (Han et al.
2020
37.4%
38.6%
WTA
56.5%
50.0%
D UNKNOWN CLASS NUMBER IN UNLABELLED DATA
Following Han al. (2020) assume the number of the classes, C in the unlabelled data is known a-priori. When C not known, we can use the method introduced in Han et al.) (2019) to estimate C first, and then substitute the estimated number into our framework. We evaluate the performance of our approach on ImageNet using the unknown category numbers estimated by DTC. The estimates 34/32/31 and the ground-truth numbers are 30/30/30 on the three unlabelled subsets. The average accuracy over three subsets is 84.1 1% which outperforms Han et al. 2020) by 3.6%.
E UNSUPERVISED CLUSTERING
We further experiment with our approach for pure unsupervised clustering on the unlabelled subset of CIFARIO and CIFARI00, which contains 5 and 20 classes respectively, by simply dropping the labelled data. Our method achieves 84.6% and 61.5% on the two datasets respectively, while the results by k-means baseline (using features extracted by the model trained on the labelled subset) are 65.5% and 56.5% respectively (see t)ble showing the superiority of our approach. This reveals that our method also an effective clustering method.
F QUALITATIVE RESULTS
We show the qualitative results on images and videos. fig. 2 and fig. 3 we show t-SNE projection for the features of data from the 5 unlabelled classes in CIFAR-10. The features are extracted using the model pretrained on the labelled data fig. ( and using our model fig. respectively. We can that the embedding is rather cluttered using the model pretrained on the labelled data, while our model can successfully partition the unlabelled data into tight semantic groups. Similarly, we compare the features videos in fig. 4 and fig. For visualisation purpose, randomly choose unlabelled instances from 10 classes from Kinetics-400, we can see that our model can successfully separate novel classes into compact groups, while the novel categories are projected very close to each other for the baseline model trained with full supervision on the labelled data.
3
13

---

Under review as a conference paper at ICLR 2021
-20
-40
-60
20
Figure 2: t-SNE visualization of features from unlabelled data on CIFAR-10, using the model pretrained on the labelled data with fully supervised training.
20
-40
-60
-50
-40
-20
20
40
60
Figure 3: 1-SNE visualization of features from unlabelled data CIFAR-10, using model after end-to-end training on both labelled and unlabelled data.
14

---

Under review as a conference paper at ICLR 2021
60
0
-40
-60
-40
-20
20
40
Figure t-SNE visualization features from unlabelled data on Kinetics-400, using the model pretrained on the labelled data with fully supervised training.
80
60
40
-20
40
-60
-40
-20
20
Figure 5: t-SNE visualization of features from unlabelled data Kinetics-400, using our model after end-to-end training on both labelled and unlabelled data.
15


Reviews:
[{"review_id": "lgwIKzg-vqU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThis paper addresses the problem of clustering unseen classes with the help of labeled data from an exclusive set of classes. It proposes an end-to-end learning strategy that combines a set of optimization objectives for training a deep neural network that can output a cluster index for the unlabeled data. The objectives include modified BCE for clustering, instance-wise contrastive loss, class-wise contrastive loss, consistency loss for augmented data pairs, and standard cross-entropy for the labeled data. The objectives utilize labeled and unlabeled data jointly. Extensive experiments on multiple image datasets show its advantage over SOTA in the class-discovery benchmark. The extended setup with multi-modality data demonstrates the capability of gaining additional performance when richer information is available.\n\n[Strengths]\nThe experiments have good coverage on four popular datasets. The ablation of each component clearly points out the import factors to achieve good performance. The results show a significant advantage over the SOTA method.\n\n[Weaknesses]\nIt is unclear what the novelty is in the proposed method compared to (Han et al. 2020). The primary difference is replacing the self-supervised learning method (rotation) with contrastive learning. The remaining learning objectives are highly similar to (Han et al. 2020). Although this paper optimizes all objectives jointly, it is unclear what the necessary change is to make this possible (or not possible in (Han et al. 2020)). Lastly, the winner-take-all-hash in the L_BCE is highly similar to the rank statistics (Han et al. 2020). The author should give an in-depth comparison against  (Han et al. 2020) and discuss what issues are mitigated by the proposed changes.\n\n[Questions]\n1. The performance of DTC and RS in Table 3 is significantly lower than the original paper. Why?\n2. How will the choice of mu in eq-4 influence the performance?\n\n-------------------------------------------\nPOST REBUTTAL\n\nThanks for the author's swift responses. I appreciate that. However, the paper still needs to provide more evidence and precise comparisons to clarify the mentioned concerns. Three major concerns are listed below:\n1. Why does the proposed pipeline perform better than RS? The paper should add the proposed modifications one-by-one into the RS method and show/discuss how these individual design choices affect the performance.\n2. WTA is worse than rank statistics in Appendix Table 5. Is it a counterexample of WTA? It needs more experimental supports to show WTA is a better method.  The hyper-parameters used in all experiments should be provided in a table and explain how the parameters are selected. Please make sure the same tuning budget is given to all the methods for a fair comparison.\n3. What are the necessary changes to make end-to-end training possible (or not possible in RS)? The paper should summarize RS's design and explain why it can not do end-to-end training, then explain what modifications are made to make this possible. This comparison is better to be illustrated in figures.\n\nOverall, this is an interesting paper, and the community will be benefit from it if well presented. Please consider revising and submitting to another venue.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "kNeeAGDm-ys", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an end-to-end framework for new class discovery. There are several key steps in the framework. (1)First is that they use separate losses, for instance, and class discrimination, so that even the unlabeled data can be trained jointly. (2)The second one is they use WTA hashing other than clustering algorithms to assign pseudo labels for unlabelled data. (3)The last one is they extend the framework to multi-modality settings. \n\nPros:\n+ The paper is easy to follow. The authors have a very clear structure for the motivations, problems with the current approaches, and contributions.\n+ The idea to generalize/unified instance and class discrimination in this semi-supervised setting are pretty simple and effective. \n+ The final experimental results are impressive compared to previous approaches. Especially adding multi-modality and video in evaluation is appreciated.\n\nCons:\n- The presentation of contribution two is not convincing in two ways. First, at the algorithm level, if WTA is not the paper’s contribution, I recommend authors either weaken the way they describe it in abstract/intro and shorten sec 3.2; or explain in details why not other LSH variants or even learning to hash variants for unsupervised/semi-supervised setting? The reasoning in 3.2 is mostly adapted from the original WTA paper and also the follow-up best paper in CVPR[1]. Second, in experiments, I can see when changing from WTA to Simhash, the performance drops sharply (even worse than the baseline). Does it mean that WTA is the real component that makes the framework working? I can also infer it from table 1 that when BCE is omitted, the performance is affected most.\n- The framework is ad hoc. There are many losses and design options making the practicality of the framework doubtful.\n\nMinor comments that will not affect scores:\n- In the experiments section, I recommend first presenting the final results first and then ablations. \n\n[1] Dean, Thomas, et al. \"Fast, accurate detection of 100,000 object classes on a single machine.\"  CVPR2013\n\n===============================================================================\n\nAfter reading the rebuttals and comments from other reviewers, I decided to keep my original score.\n\nMy biggest concern hasn't been addressed. I understand there is a comparison between different alternatives in appendix A, but only WTA is outperforming RS in the existing work by a large margin. All the other reviewers and I are curious why this is the case.   \nPerforming an A/B testing type of analysis can only tell us WTA is the best in some limited benchmarks. However, what would really make this framework principle and shine are the insights and messages we can learn from your solution. \n\nI further thought about the phenomenon and came up with some hypotheses. For example, WTA is an LSH that the collision probability of h(x) and h(y) corresponds to the ranking/ order alignment of $x_0...x_{d-1}$ and $y_0...y_{d-1}$. Because the choice of K is usually 2,4,8,16, it is more like local ranking information within a few random dimensions, while RS in Han et al is comparing the ranking of x and y globally (across d dimensions). Also, if you see your table 7,  when k goes larger, your performance saturates and degrades. If you start by analyzing the fundamental difference between RS and WTA along with your appendix B and C, you can maybe hypothesize that local ranking (among random dimensions) is more effective than global ranking and verify it with further experiments.  E.g. Intuitively with simple calculation, the half of the pair-wise order information among 4 elements (K=4) could be preserved with WTA hash code ( it is the index of the max so if the hash code is 0, it records 3 out of 6 relations: element 0 > elem1, >elem 2, >elem 3). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "NibAhlR98J", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY\n\nThe paper proposes a multi-task loss for semi-supervised representation learning and category discovery (i.e. clustering unlabeled examples). At a high level, this paper adds a contrastive loss term to the approach from Han et al. 2020. The loss has five components: (i) an InfoNCE loss where positive pairs are transformations of the same image, (ii) an InfoNCE loss where positive pairs are images with the same class label, (iii) an MSE loss between the representations of transformations of the same image, (iv) a categorical cross-entropy loss for labeled data, and (v) a binary cross-entropy loss (\"same class\" vs. \"different class\") where class labels are based on thresholding a measure of similarity between representations (distance between Winner-Take-All hash codes). The loss supports multi-modal data in that (i) and (ii) may each be computed based on representations of any modality (intra-modal, e.g. anchor and positive are both audio) or any pair of modalities (inter-modal, e.g. anchor is audio and positive is video). The method is evaluated on category discovery benchmarks based on image and video datasets.\n\n\nSTRENGTHS\n\nI think one can make a good argument that there has been too much focus on purely unsupervised methods lately - certainly the semi-supervised scenario this paper considers is far more realistic. \n\nThe paper is generally readable and the figures and tables are clear. The proposed method is reasonable and the empirical performance is strong. Most of the reported numbers are based on multiple runs, and an ablation study is included.\n\nThe analysis of multi-modal configurations in Table 2 is interesting and makes a clear case for cross-modal representation learning. \n\nWEAKNESSES\n\nIt looks like the baseline methods were not re-implemented (the numbers were just copied over from previous work). Are all hyperparameters, models, etc. similar enough that these numbers can be fairly compared? Please clarify.\n\nOn the topic of fair comparisons, Han et al. 2020 seems to report stronger performance than is credited to them in Table 3. Also, the proposed method is very closely related to Han et al. 2020 (this paper adds a term to the loss proposed in that paper) but that is not clear from the text. Please make that more explicit. \n\nThere doesn't seem to be any description of validation sets or hyperparameter tuning procedures. Without this information, it's difficult to evaluate the significance of the quantitative results.\n\nThe paper would benefit from a bit more editing - see \"minor comments\" section below. In addition, the general flow of the writing could be improved. For instance, phrases \"the accuracy is increased to X/Y/Z on dataset A/B/C\" are used often and are a bit hard to parse.\n\nOVERALL\n\nThe paper addresses an interesting and important question and presents strong results. However, there are some concerns regarding (i) how hyperparameters were tuned and (ii) the difference between the contributions of this paper and others, notably Han et al. 2020. \n\nMINOR COMMENTS\n\n**Abstract**\n\nThe terminology \"We further introduce Winner-Take-All (WTA) hashing...\" makes it sound a bit like WTA is something introduced in this paper.\n\n**Section 3.1**\n\nThe notation $B \\sim D^u \\cup D^l$ is a bit unclear. Consider writing something like \"$x_i \\sim \\mathrm{Unif}(D^u \\cup D^l)$ for all elements $x_i$ in batch $B$.\"\n\nPerhaps $Q(i)$ should be defined as $\\{ q \\in \\mathcal{N} \\backslash i : y_q = y_i\\}$? Is $Q(i)$ meant to exclude only example $i$, or does it also exclude $z_{i'}$? Perhaps $p$ is meant to denote that?\n\nThe phrase \"Without loss of generality\" is a little out of place since no theoretical claim is being made.\n\n**Section 3.2**\n\nIn the paragraph after Equation 4, \"uisng\" should be \"using\". \n\nIt's not really clear what motivates the use of WTA here - one could imagine all sorts of schemes for deciding whether two feature vectors are \"similar enough\" and this one seems a bit unusual. While there are additional comparisons in the appendix, it's hard to evaluate those results without knowing more about the hyperparameter selection procedure.\n\nIt's also not clear why the binary cross-entropy loss is chosen over e.g. yet another contrastive loss, where the pseudo-labels are used to determine whether two examples are to be considered similar or different.\n\nPresumably $\\hat{z_i}$ should be $\\bar{z_i}$ after Equation 5.\n\nWhy limit this step to just unlabeled examples in each batch? It doesn't seem like there's any reason it couldn't also be applied to the labeled examples.\n\n**Section 3.3**\n\nPlease explicitly define $\\mathcal{L}^\\mathrm{MSE}$. Does \"the predictions\" mean \"the representations of $x_i$ and $t(x_i)$\" in this context?\n\nIt seems like there's some missing text in the phrase \"We set $(1-\\omega(r))$ for contrastive learning.\" \n\n**Section 4**\n\nIs $e$ an arbitrary element of $\\mathcal{P}(C^u)$ or is it the optimal permutation? The notation and text contradict each other. \n\nBoth $M$ and $U$ have been used to denote \"the number of unlabeled instances\" - it might be helpful to choose just one. \n\nHow is $C^u$ set? Is to chosen to be the correct number of classes? It seems like it would be appropriate to examine mis-specification of this parameter since it is generally not known in the \"open world\" setting the authors discuss. This is particularly important since $C^u$ is a hyperparameter of the proposed method (not just used for an evaluation metric).\n\n**Table 1** \n\n\"Kinetcis-400\" should be \"Kinetics-400\"\n\n**Conclusion**\n\nI think \"imperially\" should be \"empirically\".\n\n**Update: After considering the other reviews and subsequent discussion, I have decided to maintain my score. As covered elsewhere in the discussion, the key shortcoming of the paper in its current form is that it provides little insight into why the proposed method outperforms the highly similar method in Han et al. For instance, the fact that WTA is crucial for the method proposed in the paper but does nothing for Han et al. is rather mysterious. I think it is necessary to provide more insight into these differences before readers can have full confidence in the proposed method.**", "extra_scores": {}, "extra_text": {}}, {"review_id": "I7hcbZiUdkI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a contrastive learning method for novel category discovery. The authors propose loss functions that can utilize labeled data of known categories and unlabelled data like semi-supervised learning, whereby they leverage both category discrimination and instance discrimination. They also extend cross-modal discrimination for multi-modal data. But these are the side loss functions. The key component to discover novel classes is a clustering loss, for which the authors propose to use Winner-Take-All hashing to generate pseudo labels of cluster partitions.\n\nThe proposed method is evaluated on CIFAR-10/-100 and ImageNet for single modality, and Kinetics-400 and VGG-Sound for audiovisual modality, and shows notable performance improvement over the state-of-the-art.\n\n\nReasons for score: \n\nThe proposed method shows significant performance improvement for the target task, but the manuscript seems not ready for publication to this reviewer in that: 1) it is unclear specifically what makes the proposed method better than the competing method [Han et al. 2020], 2) the motivation of the proposed method sounds hand-wavy (in particular, why should the multi-modal extension be considered?), and 3) there is room to improve the paper organization further (it comes and goes).\nThis reviewer would like to ask for clear and logical responses to the comments. \n\nPros:\n- Noticeable performance gain over the state-of-the-art\n- The authors demonstrate that WTA hashing based clustering is effective for novel category discovery.\n- Reasonable ablation study\n\nCons:\n- Lack of motivation of the designs (in particular, multi-modal)\n- The paper is written in a confusing way to identify the most significant contribution (I deem it is WTA hashing based clustering).\n- The experiments are not designed to clearly reveal a key contribution.\n- There is room to improve the paper organization further.\n\n\nDetail comments and questions:\n- After going over all the experiment including the experiments in the supplmentary, this reviewer thinks that WTA hashing based clustering is the source of the most significant gain the authors obtain. Is this a correct parse? If yes, then both the description and experiments are not presented to effectively emphasize this part. In particular, the experiment could be designed to show more clearly about this point.\n\t- Specifically, Tables 3 and 4 show that, even with contrastive learning (CL), the competing methods do not show notable performance gain. Then, the key difference that leads to the performance gain may come from the WTA hashing based supervision. It would have been effective to present the combination of RS [Han et al., 2020] + WTA hashing based clustering, whereby the source of the performance gain becomes clearer.\n\t- The ablation study in Table 1 does not replace the above suggestion, because it is obvious to have detrimental performance when omitting the closest loss to the target task (the novel class discovery) which is the WTA hashing based clustering loss. Thus, to show the effectiveness of WTA hashing based clustering in the target task regime, it should have been compared with its counterparts from the state-of-the-arts.\n\n- Why is multi-modal important or effective for novel category discovery? It is barely discussed.\n\n- The paper structure and organization should be improved.\n\t- It would have been clearer to put a separate subsection for describing the multi-modal extension case from Sec. 3.1, because the current description is written in a back and forth way. The current description would make the readers distracted to concentrate on the cores.\n\t- The main and key experiments should come first, and the other side experiments including the ablation study go later. Thereby, the reader can get a clear goal and achievement of the development first before going to the details.\n\t- The current two paragraphs right above Eq. (4) are wordy and hard to take the true intention of the context. For example, in the second paragraph, $k$ is mentioned but the discussion is already made before introducing the algorithm in the previous paragraph; thus, not smoothly linked well what the authors want to convey in the first paragraph. It would be improved by rewriting those paragraphs to clearly link between the algorithm and the design principles.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A2E3UpbjbRQ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new framework for novel category discovery, i.e., assigning novel class labels of an unlabeled dataset from a given (different but relevant) labeled dataset. Specifically, the framework (a) combines SimCLR, supervised contrastive learning, supervised learning, and consistency loss to learn a better representation from a mixed dataset of labeled and unlabeled samples, and (b) proposes a loss for learning a cluster assignment based on the winner-take-all coding of the learned representation from (a). Experimental results on image and video datasets confirm the effectiveness of the proposed framework compared to prior works, in terms of the average clustering accuracy on samples with unseen classes.  \n\nOverall, the paper is well-written. The method is clearly explained given the complexity of the method itself. I liked their strong experimental results, and a thorough ablation study performed to verify the individual effects of each component of the method. It is also inspiring to see their another ablation study to explore different modality pairs possible in multi-modal contrastive learning, which concludes that contrasting against different modalities leads a better representation (at least) for the target task.\n- p5, \"Concretely, we randomly generate a set of H permutations ...\": It would be nice if the paper could clarify what each permutation actually permutes, i.e., the domain. Also, are those permutations kept unchanged for both training and inference, or else?\n- Eq. 6: I feel L^CE is somewhat less motivated than the others - I found (possibly) L^NCE-C already forces the representation to be class-discriminative, then why should L^CE be crucial for the method to work, as shown in the ablation study? I think more rationale behind this could be provided in the paper.\n- Eq. 6: Should L^MSE and L^CL be always jointly ramped up and down? What happens if L^MSE is not ramped-up, i.e., not weighted by w(r)?\n- Could the proposed framework still work when there is no available labeled sample, i.e., for pure unsupervised clustering?\n- Table 3: Why DTC+CL and RS+CL are not considered?\n\n---\nPost-rebuttal update \n\nI have read other reviews and responses, and I am willing to decrease my score to 5.\nMy initial review was based on assumption that RS (Hen et al., 2020) is significantly different to this paper, but after reading other reviews and responses now I can see that it is indeed questionable. Also, It was somewhat surprising to me that the paper copied wrong values from the RS paper, as pointed out by R5, considering that RS is the closest related work of the paper.\n\nAlthough I still think the proposed method is reasonable and the empirical results are impressive, I agree with other reviewers that the paper should have done more analysis on why the proposed method works to convince the reviewers (and the future readers), and to give more insights on why the combination of losses (and the use of WTA) is a right way to go.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously raised concerns on the lack of insights on why the proposed method works better than Han et al., 2020, and why WTA brings significant gains only to the proposed method and not to Han et al. I think the paper is promising but providing these insights are critical to making the work convincing to the readers. The reviewers have made excellent points to improve the paper; I'd recommend the authors to incorporate them in their future submission.
Decision: Reject