No Representation Rules Them All in Category Discovery
Authors: Sagar Vaze
Andrea Vedaldi
Andrew Zisserman
Venue: NeurIPS 2023 poster (2023)
Keywords: Category discovery
semi-supervised learning
self-supervised learning
classification
TL;DR: We present a synthetic dataset for Generalized Category Discovery, as well as a new method for the task

Abstract: In this paper we tackle the problem of Generalized Category Discovery (GCD). Specifically, given a dataset with labelled and unlabelled images, the task is to cluster all images in the unlabelled subset, whether or not they belong to the labelled categories. Our first contribution is to recognise that most existing GCD benchmarks only contain labels for a single clustering of the data, making it difficult to ascertain whether models are leveraging the available labels to solve the GCD task, or simply solving an unsupervised clustering problem. As such, we present a synthetic dataset, named 'Clevr-4', for category discovery. Clevr-4 contains four equally valid partitions of the data, i.e based on object 'shape', 'texture' or 'color' or 'count'. To solve the task, models are required to extrapolate the taxonomy specified by labelled set, rather than simply latch onto a single natural grouping of the data. We use this dataset to demonstrate the limitations of unsupervised clustering in the GCD setting, showing that even very strong unsupervised models fail on Clevr-4. We further use Clevr-4 to examine the weaknesses of existing GCD algorithms, and propose a new method which addresses these shortcomings, leveraging consistent findings from the representation learning literature to do so. Our simple solution, which is based on `Mean Teachers' and termed $\mu$GCD, substantially outperforms implemented baselines on Clevr-4. Finally, when we transfer these findings to real data on the challenging Semantic Shift Benchmark suite, we find that $\mu$GCD outperforms all prior work, setting a new state-of-the-art.

No Representation Rules Them All in Category Discovery
Sagar Vaze
Andrea Vedaldi
Andrew Zisserman
Visual Geometry Group University of Oxford
www.robbss.ox.aa.uukk
Abstract
In this paper tackle the problem of Generalized Category Discovery (GCD). Specifically, given a dataset with labelled and unlabelled images, the task is to cluster all images in the unlabelled subset, whether or not they belong the labelled categories. Our first contribution is to recognize that most existing GCD benchmarks only contain labels for single clustering of the data, making difficult to ascertain whether models are using the available labels to solve the GCD task, or simply solving an unsupervised clustering problem. As such, we present a synthetic dataset, named 'Clevr-4', for category discovery. Clevr-4 contains four equally valid partitions of the data, i.e. based on object shape, texture, color or count. To solve the task, models are required to extrapolate the taxonomy specified by the labelled set, rather than simply latching onto a single natural grouping the data. We use this dataset to demonstrate the limitations of unsupervised clustering in the GCD setting, showing that even very strong unsupervised models fail on Clevr-4, We further use Clevr-4 to examine the weaknesses of existing GCD algorithms, and propose a new method which addresses these shorteomings, leveraging consistent findings from the representation learning literature to do Our simple solution, which is based on 'mean teachers' and termed μGCD, substantially outperforms implemented baselines on Clevr-4. Finally, when we transfer these findings real data on the challenging Semantic Shift Benchmark (SSB), we find that μGCD outperforms all prior work, setting a new state-of-the-an
1 Introduction
Developing algorithms which can classify images within complex visual taxonomies, i.e. image recognition, remains fundamental task in machine learning [1-3]. However, most models require these taxonomies to be pre-defined and fully specified, and are unable to construct them automatically from data. The ability to build taxonomy is not only desirable in many applications, but is also considered core aspect of human cognition [46]. The task of constructing a taxonomy is epitomized by the Generalized Category Discovery (GCD) problem [7, 8]: given dataset of images which is labelled only in part, the goal is to label all remaining images, using categories that occur in the labelled subset, or by identifying new ones. For instance, 'spaghetti" and 'penne' pasta products, a model must understand the concept of 'pasta shape' well enough generalize "macaroni" and 'fusilli'. It must not cluster new images based on, for instance, the color of the packaging, even though the latter also yields valid, but different, taxonomy.
supermarket, given only labels for
GCD is related to self-supervise learning [9] and unsupervised clustering [10], which can discover some meaningful taxonomies automatically [1I]. However, these cannot solve the GCD problem, which requires recovering any of the different and incompatible taxonomies that apply to the same data. Instead, the key to GCD is in extrapolating a taxonomy which is only partially known. this paper, objective is to better understand the GCD problem and improve algorithms' performance.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

stribute-based Clustering
Cluster Exemplars
Raw Data
Count:
Count:
Cuuster
Cluster
calato
Techer new
Object count
Shape: ophere
Shape:
Cluster
Cluster
mile daupe
Shape
A
Dlaoe sphios
Texture: chosseree
Texture: notal
Clustor
6 Cluster 2
Texture
A
Obiec
Figure 1: What is the key difference between Generalized Category Discovery (GCD) and tasks like supervid learning unsupervised clustering? GCD's key challenge is extrapolating the desired clustering of data given only subset of possible category labels. We present a synthetic dataset, Clevr-4, which contains four possible clusterings of the same images, and hence can be used to isolate the GCD task. Above, one can cluster the data based on object count, shape or texture.
To this end, in section 3, we first introduce the Clevr-4 dataset. Clevr-4 is synthetic dataset where each image fully parameterized by a set of four attributes, and where each attribute defines an equally valid grouping of the data (see fig. 1). Clevr-4 extends the original CLEVR dataset [12] by introducing new shapes, colors and textures, as well as allowing different object counts to be present in the image. Using these four attributes, the same set of images can be clustered according to four statistically independent taxonomies. This feature sets apart from most existing GCD benchmarks, which only contain sufficient annotations to evaluate single clustering of the data.
Clevr-4 allows us to probe large pre-trained models for biases, i.e., for their preference to emphasize a particular aspect of images, such as color or texture, which influences which taxonomy can be learned. For instance, contrary to findings from Geirhos al. [13], we find almost every large model exhibits strong shape bias. Specifically, in section 4, we find unsupervised clustering even with very strong representations like DINO [14] and CLIP [15] - fails on many splits of Clevr-4, despite CLEVR being considered 'toy' problem in other contexts [16]. As a result, we find that different pre-trained models yield different performance traits across Clevr-4 when used as initialization for category discovery. We further use Clevr-4 to characterize the weaknesses of existing category discovery methods; namely, the harms of jointly training feature-space and classifier losses, as well as insufficiently robust pseudo-labellin strategies for 'New' classes.
Next, in section 5, we leverage our findings on Clevr-4 develop simple but performant method for GCD. Since category discovery has substantial overlap with self-supervised learning, we identify elements of these methods that are also beneficial for GCD. In particular, mean-teacher based algorithms [17] have been very effective in representation learning [14, 18], and we show that they can boost GCD performance well. Here, a 'teacher' model provides supervision through pseudo- labels, and is maintained as the moving average of the model being trained (the 'student'). The slowly updated teacher is less affected by the noisy pseudo-labels which it produces, allowing clean pseudo- labels to be produced for new categories. Our proposed method, 'YGCD' ("mean-GCD") extends the existing state-of-ther the-art [19], substantially outperforming on Clevr-4. Finally, in section 6, we compare μGCD against prior work on real images, by evaluating the challenging Semantic Shift Benchmark (SSB) [20]. We substantially improve state-of-the -frthe-r on this evaluation.
In summary, we make the following key contributions: (i) We propose new benchmark dataset, Clevr-4, for GCD. Clevr-4 contains four independent taxonomies and can used to better study the category discovery problem.
(ii) We use Clevr-4 to garner insights on the biases of large pre- treitrined models as well as the weaknesses of existing category discovery methods. We demonstrate that even very strong unsupervised models fail on this 'toy" benchmark. (iii) We present novel method for GCD, ,GCD, inspired by the mean-teacher algorithm. (iv) We show μGCD outperforms baselines on Clevr-4, and further sets new state-of-the on the challenging Semantic Shift Benchmark.

---

2 Related Work
Representation learning. The common goal of self-, semi- and unsupervised learning is to learn representations with minimal labelled data. A popular technique is contrastive learning [21, 22], which encourages representations of different augmentations of the same training sample to be similar. Contrastive methods are typically either based on: InfoNCE [23] (e.g.. MoCo [24] and SimCLR [21];; or online pseudo-labelling (e.g.. SWaV [9] and DINO [14]. Almost all contrastive learning methods now adopt a variant of these techniques [25-28]. Another important component in many pseudo- labelling based methods is 'mean-teachers' [17] (or momentum encoders [24]), in which a 'teacher' network providing pscudo-labels maintained as the moving average of a 'student' model. Other learning methods include cross-stitch [29], context- prrexictttiiiiionn [30], and reconstruction [31]. In this work, we use mean-teachers to build a strong recipe for GCD.
Attribute learning. We propose a new synthetic dataset which contains multiple taxonomies based on various attributes. Attribute learning has a long history in computer vision, including real-world datasets such as the Visual Genome [32], with millions of attribute annotations, and VAW, with 600 attributes types [33]. Furthermore, the disentanglement literature [34-36] often uses synthetic attribute datasets for investigation [37, 38.. We find it necessary to develop a new dataset, Clevr- 4, for category discovery as real-world datasets have either: noisy/incomplete attributes for each image [32, 39]; or contain sensitive information (e.g. contain faces) [40]. We find existing synthetic datasets unsuitable as they do not have enough categorical attributes which represent 'semantic' factors, with attributes often describing continuous 'nuisance' factors such as object location or camera pose [37, 38, 41].
Category Discovery. Novel Category Discovery (NCD) was initially formalized in [42]. It differs from GCD as the unlabelled images are known to be drawn from a disjoint set of categories to the labelled ones [43-47]. This is different from unsupervised clustering [10, 48], which clusters unlabelled data without reference labels at all. It is also distinct from semi-supervised learning [17, 26, 491, where unlabelled images come from the same set of categories as the labelled data. GCD [7, 8] was recently proposed as a challenging task in which assumptions about the classes in the unlabelled data are largely removed: images in the unlabelled data may belong to the labelled classes or to new ones [19, 50-52]. We particularly highlight concurrent work in SimGCD [19], which reports the best current performance on standard GCD benchmarks. Our method differs from SimGCD by the adoption of a mean-test teacher [17] to provide more stable pseudo-labels training, and by careful
consideration of model initialization and data augmentations. [52] also adopt a momentum- encoder, though only for a set of class prototypes rather than in mean-teacher setup.
3 Clevr-4: a synthetic dataset for generalized category discovery
Generalized Category Discovery (GCD). GCD [7] is the task of fully labelling a dataset which only partially labelled, with labels available only for a subset of the categories. Formally, we are presented with a dataset D. containing both labelled and unlabelled subsets, Dz = {<,,, y/) 1 e X x Vic and Dr, = {(x; ) E X x Yuu The model is then tasked with categorizing all instances in Du, The unlabelled data contains instances of categories which do not appear in the labelled set, meaning that V2 C 3/2, Here, we assume knowledge of the number of categories in the unlabelled set, í.e. k Dixl Though estimating i is interesting problem, methods for tackling it are typically independent of the downstream recognition algorithm [7, 42]. GCD is related to Novel Category Discovery (NCD) [42], but the latter makes the not-so-realistic assumption that Ve O Yu 0.
The category discovery challenge. The key to category discovery (generalized or not) is to use the labelled subset of the data to extrapolate a taxonomy and discover novel categories in unlabelled images. This task of extrapolating a taxonomy sets category discovery apart from related problems. For instance, unsupervised clustering [10] aims to find the single most natural grouping of unlabelled images given only weak inductive biases (e.g., invariance to specific data augmentations), but permits limited control which taxonomy is discovered. Meanwhile, semi-supervised learning [26] assumes supervision for all categories in the taxonomy, which therefore must be known, in full, a-priori.
A problem with many current benchmarks for category discovery is that there is no clear taxonomy underlying the object categories (e.g., CIFAR [53]) and, when there is, it is often ill-posed to understand it given only few classes (e.g., ImageNet-100 100 [42]. Furthermore, in practise, there
3

---

are likely to be many taxonomies of interest. However, few datasets contain sufficiently complete annotations to evaluate multiple possible groupings of the same data. This makes difficult to ascertain whether model is extrapolating information from the labelled set (category discovery) or just finding its own most natural grouping of the unlahelled data (unsupervised clustering).
Clevr-4. In order to better study this prob- lem, we introduce Clevr-4, a synthetic benchmark which contains four equally valid groupings of the data. Clevr-4 extends the CLEVR dataset [12], using Blender [54] to render images of multiple objects and place them in a static scene. This is well suited for category discovery,
Table 1: Clevr-4 statistics for the different splits of the dataset. Note that the same data must be classified along independent taxonomies in the different splits.
Texture
Color
Shope
Count
Examples
Jowcal, rubiber|
Fed, bluel
[Lorus, cabe)
(-,2)
3%
D4
10
10
De
2.18
233
2.18
2.18
6.8K
B.I.K
6.4%
6.8%
PPWl
8.4K
5.4K
8:4
8.48
as each object attribute defines a different taxonomy for the data (e.g., it enables clustering images based on object shape, color etc.). The original dataset is limited as it contains only three shapes and two textures, reducing the difficulty of the respective clustering tasks. We introduce 2 new colors, 7 new shapes and 8 new textures to the dataset, placing between I and 10 objects in each scene.
Each image is therefore parameterized by object shape, texture, color and count. The value for each attribute is sampled uniformly and independently from the others, meaning the image label with respect to one taxonomy gives us no information about the label with respect another. Note that this sets Clevr-4 apart from existing GCD benchmarks such as CIFAR-100 [53] and FGVC-Aircraft [55]. These datasets only contain taxonomies at different granularities, and as such the taxonomies are highly correlated with each other. Furthermore, the number of categories provides no information regarding the specified taxonomy, as all Clevr-4 taxonomies contain = 10 object categories.
Finally, we create GCD splits for each taxonomy in Clevr-4, following standard practise and reserving half the categories for the labelled set, and half for the unlabelled set. We further subsample 50% of the images from the labelled categories and add them to the unlabelled set. We synthesize 8.4K images for GCD development (summarized in table I), and further make a larger 100K image dataset available. The full generation procedure is detailed in appendix A.
Performance metrics. We follow standard practise [7, 47] and report clustering accuracy for evaluation. Given predictions, ÿt, and ground truth labels, yi, the clustering accuracy is ACC = No
1. maxnes, N. Σ/= 1 I[,)), where S₁ is the symmetry group of order k. Here N, is the number of unlabelled images and the max operation is performed (with the Hungarian algorithm [56]) to find the optimal matching between the predicted cluster indices and ground truth labels. For GCD models, we also report ACC on subsets belonging to 'Old' (y/ E V/c) and 'New' (ys E Yu \ 32c) classes. The most important metric the 'All' accuracy (overall clustering performance), as the precise 'Old' and 'New' figures are subject to the assignments selected in the Hungarian matching.
4 Learnings from Clevr-4 for category discovery
In this section, we use Clevr-4 to gather insights into the category discovery problem. In section 4. 1, we assess the limitations of large-scale pre-training for the problem, before using Clevr-4 to examine the weaknesses of existing category discovery methods in section 4.2. Next, in section 4.3, we use our findings to motivate stronger method for GCD (which we describe in full in section 5), finding that this substantially outperforms implemented baselines on Clevr-4.
4.1 Limitations of large-scale pre-training for category discovery
Here, we show that pre-trained representations develop certain 'biases' which limit their performance when used directly or as initialization for category discovery.
Unsupervised elustering of pre-trained representations (table 2). We first demonstrate the limita- tions of unsupervised clustering of features as an approach for category discovery (reporting results with semi-supervised clustering in fig. 11). Specifically, we run k-means clustering [57] on top of
features extracted with self- [9, 14, 21J, weakly- [15], and fully-supervised [2, 3, 58] backbones, reporting performance on each of the four taxonomies Clevr-4. The representations are trained on up to 400M images and are commonly used in the vision literature.

---

Table 2: Unsupervised clustering accuracy (ACC) of pre-trained models Clevr-4. We find
most models are strongly biased towards shape, while MAE [31] exhibits a color bias.
Pre-training Method
Pre-training Data
Backbone
Texture
Shape
Color
Count
Average
SWaV [9]
ImageNet-1K
ResNer50
13.1
63.5
12.1
18.9
27.4
MoCoV2 [22]
ImageNet-IK
ResNet50
13.0
77.5
12.3
18.8
30.4
Supervised [2]
ImageNet-IK
ResNet50
13.2
76.8
15.2
12.9
29.5
Supervised [3]
ImageNet-IK
ConyNeXT-B
13.4
83.5
12.1
13.1
30.5
DINO [14]
ImageNet-1K
ViT-
16.0
86.2
11.5
13.0
31.7
MAE [31]
ImageNet-1K
ViT-B/16
15.1
13.5
64.7
13.9
26.8
iBOT [59]
ImageNet-IK
ViT-B/16
14.4
85.9
11.5
13.0
31.2
CLIP [15]
WIP-400M
VIT-B/16
12.4
78.7
12.3
17.9
30.3
DINOv2 [60]
LVD-142M
ViT-B/14
11.6
98.1
11.6
12.8
33.5
Supervised [58]
ImageNet-21K
ViT-B/16
11.8
96.2
11.7
13.0
33.2
Table 3: Effects of large-scale pre-training on category discovery accuracy (ACC) on Clevr-4. Contrary to dominant findings in the vision literature, we find that large-scale pre-training provides inconsistent gains on Clevr-4. For instance, on count, training a lightweight model (ResNet18) from
scratch substantially outperforms initializing from DINOv2 (ViT-B/14) trained on 142M images.
Method
Backbone
Pre-training (Data)
Textuee
Shape
Color
Count
Average
Average Rank
SimGCD
ResNet18
58.1
97.8
96.7
67.6
80.5
2.0
SimGCD VIT-B/16
MAE [31] (ImageNet-1k)
54.1
99.7
99.9
53.0
76.7
2.0
SimGCD
ViT-B/14
DINOv2 [60] (LVD-142M)
76.5
99,9
87.4
51.3
78.8
2.0
We find that most models perform well on the shape taxonomy, with DINOv2 almost perfectly solving the task with 98% accuracy. However, none of the models perform well across the board. For instance, on some splits (e.g., color), strong models like DINOv2 perform comparably to random chance. This underscores the utility of Clevr-4 for delineating category discovery from standard representation learning. Logically, it is impossible for unsupervised clustering on any representation to perform well on all tasks. After all, only a single clustering of the data is produced, which cannot align with more than one taxonomy. We highlight that such limitations are not revealed by existing GCD benchmarks; on the CUB benchmark (see table 5), unsupervised clustering with DINOv2 achieves 68% ACC (x 140x random).
Pre-trained representations for category discovery (table 3). Many category discovery methods use self-supervised representation learning for initialization in order to leverage large-scale pre- training, in the hope of improving downstream performance. However, as shown above, these representations are biased. Here, we investigate the impact of these biases on a state-of-the-ar method in generalized category discovery, SimGCD [19]. SimGCD contains two main loss components: (1) a contrastive loss on backbone features, using self-supervised InfoNCE [23] on all data, and supervised contrastive learning [61] ] on images with labels available; and (2) contrastive loss to train a classification head, where different views of the same image provide pseudo-labels for each other. For comparison, we initialize SimGCD with a lightweight ResNet 18 trained scratch; ViT-B/16 pre-trained with masked auto-encoding [31]: and a ViT-B/14 with DINOv2 [60] initialization. For each initialization, we sweep learning rates and data augmentations.
Surprisingly, and in stark contrast to most of the computer vision literature, we find inconsistent gains from leveraging large-scale pre-training on Clevr-4. For instance, on the count taxonomy, pre-training gives substantially worse performance that training lightweight ResNet18 from scratch. On average across all splits, SimGCD with randomly initialized ResNet18 8 actually performs best. Generally, we find that the final category discovery model inherits biases built into the pre-training, and can struggle to overcome them even after finetuning. Our results highlight the importance of carefully selecting the initialization for a given GCD task, and point to the utility of Clevr-4 for doing so.
4.2 Limitations of existing category discovery methods
Next, we analyze SimGCD [19], the current state-of-the-art for the GCD task. We show that on Clevr-4 it is not always better than the GCD baseline [7] which it extends, and identify the source of this issue in the generation of the pscudo-labels for the discovered categories. In more detail, the
5

---

Table 4: Category discovery accuracy (ACC) Clevr-4. Compared to our reimplementation of the GCD baseline [7] and SimGCD ofate-of-the-an [19], our method provides substantial boosts on average across Clevr-4. Results averages across five random seeds. Also shown is the classification accuracy of a fully-supervised upper-bound on a disjoint test-set.
Texture
Shape
Color
Count
Average
Model
Backhone
Old
New
All
Old
New
All
Old
New
All
Old
New
All
Fully supervised
ResNet18
99.1
100.0
100.0
96.8
99.0
GCD
ResNet18
62.4
97.5
45.3
93.9
99.7
90.5
90.7
95.0
88.5
71,9
96.4
60.1
79.7
SimGCD
ResNet18
58.1
95.0
40.2
97.8
98.9
97.2
96.7
99.9
95.1
67.6
95.7
53.9
80.1
μGCD (Ours)
ResNet18
69.8
99.0
55.5
94,9
99.7
92.1
99.5
100.0
99.2
75.5
96.6
65.2
84.9
GCD baseline uses only one of the two losses used by SimGCD, performing contrastive learning on features, followed by simple clustering in the models' embedding space. To compare SimGCD and GCD, we start from a ResNet18 feature extractor, training it from scratch to avoid the potential biases identified in section 4.1. We show results in table 4, reporting results for 'All', 'Old" and 'New' class subsets. We follow standard practise when reporting on synthetic data and train all methods with five random seeds [34, 35] (error bars in the appendix B. 1), and sweep syper-paramette augmentations for both methods. We also train a model with full supervision and obtain 99% average performance on Clevr-4 (on independent test data), showing that the model has sufficient capacity.
and data
Overall, we make the three following observations regarding the performance of GCD and SimGCD on Clevr-4: (i) Both methods' performance on texture and count is substantially worse than shape and color. (ii) On the harder texture and coun! splits, the GCD baseline actually outperforms the SimGCD state-of-the-art. Given that SimGCD differs from GCD by adding classification head and corresponding loss, this indicates that jointly training classifier and feature-space losses can hurt performance. (iii) Upon closer inspection, we find that the main performance gap texture and count comes from accuracy on the 'New' categories; both methods cluster the 'Old' categories almost perfectly. This suggests that the 'New' class pseudo- labels from SimGCD are not strong enough; GCD, with no juuuudo-ouperiision for novel classes, achieves higher clustering performance.
4.3 Addressing limitations in current approaches
Given these findings, we seek to improve the quality of the pseudo-labels for 'New' categories. Specifically, we draw inspiration from the mean mean-teacher setup for semi- supervise learning [17], which has been adapted with minor changes in many self-suue frameworks [14, 18, 24]. Here, a 'student' network is supervised by class pseudo-labels generated by 'teacher'. The teacher is an identical architecture with parameters updated with the Exponential Moving Average (EMA) of the student. The intuition is that the slowly updated teacher is more robust to the noisy supervision from pseudo-labels, which in turn improves the quality of the pscudo-labels themselves. Also, rather than jointly optimizing both SimGCD losses, we first train the backbone only with the GCD baseline loss, before finetuning with the classification head and loss.
These changes. together with careful consideration of the data augmentations, give rise to our proposed μGCD (mean-GCD) algorithm, which we fully describe next in section 5. Here, we note the improvements that this algorithm brings in Clevr-4 on the bottom line of table 4. Overall, HGCD outperforms SimGCD on three of the four Clevr-4 taxonomies, and further outperforms SimGCD by nearly 5% on average across all splits. 5GCD underperforms SimGCD on the shape split of Clevr-4 and we analyse this failure case in the appendix B.2.
5 The GGC algorithm
In this section, we detail a simple but strong method for GCD, PGCD, already motivated in section and illustrated in fig. 2. In a first phase, the algorithm proceeds in the same way as the GCD baseline [7], learning the representation. Next, we append a classification head and fine-tune the model with 'mean teacher' setup [17], similarly to SimGCD but yielding more robust lseudo-labels.
Concretely, we construct models, fo. as the composition of a feature extractor, & and a classification head, y. & is first trained with the representation learning framework from [7] as described above, and the composed model gives f = g o with values in R. where is the total number of categories in the dataset. Next, we sample batch of images, B, and generate two random augmentations of

---

Unsupervised loss
Supervised loss
Regularizer loss
Weak Aug
Teacher Or 'GCD' Init.
Coefre Chooolar
Teacher predictions
One hot labels
Uniform distribution
EMA
Step
Strong Aug
Student 0s 'GCD' Init.
Cosine classter
Grad
Student predictions
Student predictions
Student prodictions
Figure 2: Our "GGCD' method. We begin with representation learning from the GCD baseline, followed by finetuning mean-teacher style setup. Here, a 'teacher' provides supervision for 'student' network, and maintains parameters as the exponential moving average (EMA) of the student.
every instance. We pass one view through the student network fag. and the other through the teacher network for. where 0g and Or are the network parameters of the student and teacher, respectively. We compute the cross-entropy loss between the (soft) teacher pscudo-labela and student predictions:
1 Σ
L (0s;;
(p) log(ps(z))), (x) softmax(fo, (z)): mEB
(1)
[
where P. (x) E [0, 11sh are the softmax outputs of the student and teacher networks, scaled with temperature T.. We further use labelled instances in the batch with supervised cross-entropy component as:
(0s;
Σ (u(x)), log(ps(>)))) iEBc
Bcl
where B& B is the labelled subset of the batch and y(x) 13 is the one-hot class label of the example æ. Finally, we add a mean-entropy maximization regularizer from [26] to encourage pseudo-labels for all categories:
L (0s)) = P log (p)),)),
Ps
Σ BEB
Pz(
B
The student is trained with respect to the following total loss, given hyper-parameten, A1 and Ag: L(0s; B) (1 121-x]1 (0s; B)) + (0s B&) (Os). The teacher parameters are updated wt)))Os where (() is time-varying momentum.
as the moving average 07 = w()
Augmentations. While often regarded as 'implementation detail', important component of our method is the careful consideration of augmentations used in the computation of L". Specifically, on the SSB, we pass different views of the same instance to the student and teacher networks. We generate a strong augmentation which is passed to the student network, and weak augmentation which passed the teacher, similarly to [49]. The intuition is that, while contrastive learning benefits from strong data augmentations [9, 14], we wish the teacher network's predictions to be as stable as possible. Meanwhile, on Clevr-4, misaligned data augmentations e.g., aggressive cropping for count, or color jitter for color - substantially degrade performance (see appendix B.5).
Architecture. We adopt 'cosine classifier' as g. which was introduced in [62] and leverages - ["-normalized weight vectors and feature representations. While it has been broadly adopted for many tasks [8, 9, 19, 25, 43], demonstrate why this component helps in section 7.1. We find that normalized vectors are important to avoid collapse of the predictions to the labelled categories.
6 Results on real data
Datasets. We compare ,GCD against prior work on the standard Semantic Shift Benchmark (SSB) suite [20]. The SSB comprises three fine-grained evaluations: CUB [64], Stanford Cars [65] and
7

---

Table 5: Category discovery accuracy (ACC) the Semantic Shift Benchmark [20]. We report results from prior work using DINO intialization [14], and reimplement GCD baselines and SimGCD with DINOv2 pre-training [60] (noted with *).
CUB
Stanford Cars
Aircraft
Average
Pre-training
All
Old
New
All
Old
New
All
Old
New
All
k-means [57]
DINO
34.3
38.9
32.1
12.8
10.6
13.8
16.0
14.4
16.8
21.1
RankStats+ [47]
DINO
33.3
51.6
24.2
28.3
61.8
12.1
26.9
36.4
22.2
29.5
UNO+ [43]
DINO
35.1
49.0
28.1
35.5
70.5
18.6
40.3
56.4
32.2
37.0
ORCA |8]
DINO
35.3
45.6
30.2
23.5
50.1
10.7
22.0
31.8
17.1
26.9
GCD [7]
DINO
51.3
56.6
48.7
39.0
57.6
29.9
45.0
41.1
46,9
45.1
XCon [6.3]
DINO
52.1
54.3
51.0
40.5
58.8
31.7
47.7
44.4
49.4
46.8
OpenCon [52]
DINO
54.7
63.8
54.7
49.1
78.6
32.7
MIB |51]
DINO
62.7
75.7
56.2
43.1
66.9
31.6
PromptCAL [50]
DINO
62.9
64.4
62.1
50.2
70.1
40.6
52.2
52.2
52.3
55.1
SimGCD [19]
DINO
60.3
65.6
57.7
53.8
71.9
45.0
54.2
59.1
51.8
56.1
pGCD (Ours)
DINO
65.7
68.0
64.6
56.5
50.9
53.8
55.4
53.0
58.7
k-means*
DINOv2
67.6
60.6
71.1
29.4
24.5
31.8
18.9
16.9
19.9
38,6
GCD
DINOv2
71.9
71.2
72.3
65.7
67.8
64.7
55.4
47.9
59.2
64.3
SimGCD*
DINOv2
71.5
78.1
68.3
71.5
81.9
66.6
63.9
69.9
60.9
69.0
pGCD (Ours)
DINOv2
74.0
75.9
73.1
76.1
91.0
68.9
66.3
68.7
65.1
72.1
FGVC-Aircraft [55]. Though the SSB datasets do not contain independent clusterings of the same images (as in Clevr-4) the evaluations do have well-defined taxonomies i.e. birds, and aircrafts. Furthermore, the SSB contains curated novel class splits which control for semantic distance with the labelled set. We find that coarse-grained GCD benchmarks do not specify clear taxonomies in the labelled set, and we include a long-tailed evaluation on Herbarium19 [66] in appendix C.3.
Model initialization and compared methods. The SSB contains fine-grained object-centric datasets, which have been shown to benefit from greater shape bias [67]. Prior GCD meth- ods [7, 52, 63] initialize with DINO [14] pre- training, which
Table 6: Ablations. We find that a proper intial- ization, momentum decay schedule, and augmen- tation strategy are critical to strong performance.
CUB
Old
New
show in table 2 had the
#GCD (Ours)
65.7
68.0
64.6
strongest shape bias among self-supervised mod-
(1) W/o GCD init.
61.7
66,2
59.6
els. However, the recent DINOv2 [60] demon- strates a substantially greater shape bias. As such, we train our model both with DINO and DINOv2 initialization, further re- re-implementing GCD baselines [7, 68] and SimGCD [19] with DINOv2 for comparison.
(2) W/u stronger student augmentation
58.1
72.5
(3) With wa :=
1.6
1.1
1.8
(4) With uq : 0.0
62.7
66.4
60.9
(5) With wi
64.1
65.1
63.6
(6) W/n casine classifier
54.9
64.2
50.3
(7) W/o ME-Max regularizer
42.0
41.8
42.1
Implementation details. We implement all models in PyToorch [69] single NVIDIA P40 M40. Most models are trained with an initial learning rate of 0. which is decayed with cosine annealed schedule [70]. For EMA schedule, we ramp it up throughout training with cosine function [18]: w(t) WT (1 ) { + 1)/2. Here t is the current epoch and T' is the total number of epochs. Differently, however, most self-supervised learning literature [18], found a much lower
initial decay to be beneficial; we ramp up the decay from Whose = 0.7 to WT = 0.999 during training.
Further implementation details can be found appendix E.
6.1 Discussion.
In table 5, we find that fGCD outperforms the existing ttate-of-the-an SimGCD [19], by over 2% on average across SSB evaluations when using DINO initialization. When using the stronger DINOv2 backbone, we find that the performance of the simple k-means baseline nearly doubles in accuracy, substantiating our choice of shape-biased initialization on this object-centric evaluation. The gap between the GCD baseline [7] and the SimGCD state-of-the-art [19] is also reduced from over 10% to under 5% on average. Nonetheless, our method outperforms SimGCD by over 3% on average, well as each dataset individually, setting new state-of thth-arft

---

Regularizdl Classifier weight
W/o max entropy loss
50
Fredicted Instances
Weight
230
25
36
Indices ('M 1001 New 101
200)
(Old: 1001 New 101
Figure 3: Left:: Norms weight vectors in GCD classifiers, with and without regularizations. Right:
Prediction histogram of classifier without weight norm and max-entropy regularization.
Ablations. We ablate main design choices in table 6. L(I) shows the importance of pre-training with the GCD baseline loss [7] (though we find in section 4 that jointly training this loss with the L(2) further demonstrates that stronger augmentation for the student network is critical, with a 7% drop in CUB performance without it. L(3)-(5) highlight of tine-varying decay outperforms
classifier, in SimGCD [19], is difficult).
the importance of carefully designed EMA schedule, constant decay values. This intuitive as early on training, with a randomly initialized classification head, wish for the teacher to be updated quickly. Later in training, slow teacher updates mitigate the effect of noisy pseudo-labes labels within any given batch. Furthermore, in L(6)-(7), we validate the importance of entropy regularization and cosine classifiers in category discovery. In section 7.1, provide evidence as to why these commonly used components [8, 19, 43] are necessary, and also discuss the design of the student augmentation.
Further Analysis
In this section we further examine the effects of architectural choices pGCD and other category discovery methods. Section 7.1 seeks to understand the performance gains yielded by cosine classifiers, and section 7.2 visualizes the feature spaces learned by different GCD methods.
7.1 Understanding cosine classifiers category discovery
Cosine classifiers with entropy regularization have been widely adopted in recognition settings with limited supervision [14, 25], including in category discovery [19, 43]. In fig. 3, we provide justifications for this by inspecting the norms of the learned vectors in the final classification layer.
Specifically, consider a classifier (without bias term) as g W Radk containing vectors of dimension, one for each output category. In fig. 3, we plot the magnitude of each of these vectors trained with different constraints on CUB [64] (one of the datasets the SSB [20]). Note that the classifier is constructed such that the first 100 vectors correspond to the 'Old' classes, and are trained with ground truth labels. In our full method, with normalized classifiers, the norm of vectors is enforced to be the unit norm (blue dashed line). If we remove this constraint (solid orange line), we can see that the norms of vectors which are not supervised by ground truth labels (indices 101-200) fall substantially. Then, if we further remove the entropy regularization term (solid green line), the magnitudes the 'Old' class vectors (indices 1-200) increases dramatically.
This becomes issue at inference time, with per-class logits computed as:
bm (W... |ww | Vmn E {1...k}
with the class prediction returned as arg max Im- In other words, we show that without appro- priate regularisation, our GCD models trivially reduce the weight norm of 'New' class vectors ((wml Vm > 100), leaving all images to be assigned to one of the 'Old' classes. The effects of this are visualized in the right panel of fig. 3, which plots the histogram of class predictions for an unregularized GCD classifier. We can see that exactly zero examples predicted to 'New" classes.

---

Object Count
GCD ACC: 70.1 / 95.8 / 65.5
SimGCD ACC: 65.2 93.5 / 51.8
PGCD (Ours) ACC: 78.0 / 9 68.1
2.0
'Old'
3.0
5.0
6.0
'New'
8.0
9.0
Figure 4: PCA [71] of features from the GCD baseline [7], SimGCD [19] and pGCD on the count split of Clevr-4. While the baseline learns eliptical clusters for each category, SimGCD and ,GCD project images onto mie-dimensional object in feature space, which be considered as a 'semantic axis' along which the category changes. Clustering accuracy is reported for All'v"Old//Neew classes.
We further highlight that this effect obfuscated by the evaluation process, which reports non-zero accuracies for "New" classes through the Hungarian assignment operation.
7.2 PCA Visualization.
Finally, we perform analysis on the count split of Clevr-4. Uniquely amongst the four taxonomies, the count categories have a clear order. In fig. 4, we plot the first two principal components [71] of
the normalized features of the GCD baseline [7], SimGCD [19] and μGCD. It is clear that all feature spaces learn a clear 'number sense' [72] with image features placed in order of increasing object count. Strikingly, this sense of numerosity is present even beyond the supervised categories (count greater than 5) as byproduct of simple recognition task. Furthermore, while the baseline learns elliptical clusters for each category, SimGCD and μGCD project all images onto one-dimensional object in feature space. This object can could be considered as a 'semantic axis": low-dimensional manifold in feature space, R E Rd, along which the category label changes.
8 Conclusion and final remarks
Limitations and broader impacts. In this paper we presented synthetic dataset for category discovery. While synthetic data allows precise manipulation of the images, and hence more controlled study of the GCD problem, findings on synthetic data do not always transfer directly to real-world images. For instance, the failure case of μGCD on the shape split of Clevr-4 (see section 4.3) occurs due to some classification vectors being unused (see section 7.1). This ceases to be issue on real-world data with hundreds of categories (see table 5). In appendix F, we fully discuss the difficulties in developing a dataset like Clevr-4 with real-world data. Furthermore, while GCD has many real-world applications, like any form of unsupervised partially-supervieed machine learning, it can be unreliable and must be applied with caution in practise.
Remarks on Clevr-4. We note that Clevr-4 can find broader applicability in related machine learning fields. As examples, the dataset can be used for disentanglement research (see appendix F) and as a simple probing set for biases in representation learning. For instance, we find in section 4. that most of the ImageNet trained models are biased towards shape rather than texture, which is in contrast to popular findings from Geirhos el al. [13]. Furthermore, larger models are often explicitly proposed as "all-purpose' features for "any task' [60]; here we find simple tasks (e.g., color or count recognition) where initialization with such moodels hurts performance compared to training from scratch. Note that practical problems - e.g., vehicle re-identification [73] or crowd counting [74] may require understanding of such aspects of the image.
Conclusion. In this paper we have proposed a new dataset, Clevr-4, and used it to investigate the problem of Generalized Category Discovery (GCD). This included probing the limitations of unsupervised representations for the task, as well as for identifying weaknesses in existing GCD methods. We further leveraged our findings, together with consistent trends in related literature, to propose a simple but performant algorithm, μGCD. We find that fGCD not only provides gains on Clevr-4, but further sets new state-of-the-art the standard Semantic Shift Benchmark.
10

---

Acknowledgments and Disclosure of Funding
S. Vaze supported by Facebook Scholarship. A. Vedaldi is supported by ERC-CoG UNION 101001212. A. Zisserman is supported by VisualAI EP/T028572/1.
References
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NeurlPS, 2012.
[2] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[3] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. convnet for the 2020s. In CVPR, 2022.
[4] Susan Carey. The Origin of Concepts. Oxford University Press, 2009.
[5] Gregory Murphy. The Big Book of Concepts. Boston Review, 2002.
[6] Dedre Gentner and Linsey Smith. Analogical reasoning. Encyclopedia of human behavior, 2012.
[7] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In JEEE Conference on Computer Vision and Pattern Recognition, 2022.
[8] Kaidi Cao, Maria Brbic, and Jure Leskovec. Open-world semi- semi-supervised learning. In International Conference on Learning Representations, 2022.
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NewrIPS, 2020.
[10] Xu Ji, João F Henriques, and Andrea Vedaldi. Invariant information clustering for unsupervised image classification and segmentation. In ICCV, 2019.
[11] Laina, Yuki Asano, and Andrea Vedaldi. Measuring the interpretability of unsupervised representa- tions via quantized reversed probing. In ICLR, 2022.
[12] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[13] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann. and Wieland Brendel. Imagenet-trainee cnns are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR, 2019.
[14] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[15] Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[16] Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. dndetr-modulated detection for end-to-end multi-moda understanding. ICCV, 2021.
[17] Antti Tarvainen and Harri Valpola. Mcan teachers better role models: Weight-averaged consistency targets improve cuperviidd deep lcarning results. In NewrIPS, 2017.
[18] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NewrIPS, 2020.
[19] Xin Wen, Bingchen Zhao, and Xiaojuan Qi. A simple parametric classification baseline for generalized category discovery. ICCV, 2023.
[20] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
11

---

[21] Ting Chen, Simon Korbblith. Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[22] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint (xXixv2003304299 2020.
[23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, ArXiv e-prints, 2018.
[24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[25] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In ECCV, 2022.
[26] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. seme-super learning visual features by non von-parametrically predicting view assignments with support samples. In ICCV, 2021.
[27] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustcring for unsupervised learning of visual features. In ECCV, 2018.
[28] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[29] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, 2016.
[30] Carl Dnersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning hy context prediction, ICCV, 2015.
[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision leamerrs. In CVPR, 2022.
[32] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsoureed dense image annotations. IJCV, 2017.
[33] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In CVPR, June 2021.
[34] Trina Higgins, Loic Mathhey Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
[35] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variation using few labels. ICLR, 2020.
[36] Brooks Paige, Jan Willem van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, Philip Torr, et al. Learning disentangled representations with semi-supervised deep generative models. NewrIPS, 2017.
[37] Scott Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy making.... NeurlPS, 2015.
[38] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In ICMI, 2018
[39] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report 200--0-001--00, California Institute of Technology. 2011.
[40] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
[41] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuff, Ira Ktena, Krishnamurthy Dvi- jotham, and Taylan Comgil. A fine-grained analysis on distribution shift. ICLR, 2022.
[42] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In ICCV, 2019.
12

---

[43] Enrico Fini, Enver Sangineto. Stéphane Lathuilière, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In ICCV, 2021.
[44] Zhun Zhong, Enrico Fini. Subhankar Roy. Zhiming Luo, Elisa Ricci, and Nicu Sebe. Neighborhood contrastive learning for novel class discovery. In ICCV, 2021.
[45] Bingchen Zhao and Kai Han. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation. In NewrlPS, 2021.
[46] Kai Han, Sylvestre-Alvise Rebufli, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Automat- ically discovering and learning new visual categories with ranking statistics. In ICLR, 2020.
[47] Kai Han, Sylvestre-Alvie Rehuffi, Sebastien Ehrhardt, Andrea Vedaldi, and Andrew Zisserman. Autonovel: Automatically discovering and learning novel visual categories. IEFE TPAMI, 2021.
[48] Chuang Niu, Hongming Shan, and Ge Wang. Spice: Semantic pseudo-labeling for image clustering. IEEE Transactions on Image Processing, 2022.
[49] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consisteney and confidence. In NewrlPS, 2020.
[50] Sheng Zhang, Salman Khan. Zhiqiang Shen, Muzammal Naseer, Guangyi Chen, and Fahad Khan. Prompt- cal: Contrastive affinity learning via auxiliary prompts for generalized novel category discovery. ArXiv e-prints, 2022.
[51] Florent Chiaroni, Jose Dolz, Ziko Imtiaz Masud, Amar Mitiche, and Ismail Ben Ayed. Mutual information- based generalized category discovery. ArXiv e-prints, 2022.
[52] Yiyou Sun and Yixuan Li. Opencon: Open-world contrastive learning. In Transactions on Machine Learning Research, 2022.
[53] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
[54] Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
[55] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft, arXiv preprint 1006:5530.. 2013.
[56] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
[57] James MacQueen. Some methods for classification and analysis of multivariate observations. In Proceed- ings of the Fifth Berkeley Symposiun on Mathematical Statistics and Probability, 1967.
[58] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-cfficient image transformers and distillation through attention. In ICML, 2021.
[59] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image beri pre-training with online tokenizer. ICLR, 2022.
[60] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby. Russell Howes, Po- Yao Huang. Hu Xu, Vasu Sharma, Shang-Wer Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[61] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola. Aaron Maschinot, Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004.11622 2020.
[62] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018.
[63] Yixin Fei, Zhongkai Zhao, Siwei Yang, and Bingchen Zhao. Xcon: Learning with experts for fine-grained category discovery. In BMVC, 2022.
[64] Catherine Wah. Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-22 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
13

---

[63] Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3d object representations for fine-grained categorization. 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13). 2013.
[66] Kiat Chuan Tan, Yulong Liu, Barhara Ambrose, Melissa Tulig, and Serge Belongie. The herbarium challenge 2019 dataset. In Workshop on Fine-Grained Visual Categorization, 2019.
[67] Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. In ICML, 2017.
[68] David Arthur and Sergei Vassilvitskii. k-means++: the advantages of careful seeding. ACM-SIAM symposium on Discrete algorithms, 2007.
[69] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
[70] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR, 2017.
[71] Hervé Abdi and Lynne Williams. Principal component analysis.
[72] Neehar Kondapaneni and Pietro Perona, A number sense as an emergent property the manipulating brain. ArXiv e-prints, 2020.
[73] Hongye Liu, Yonghong Tian, Yaowei Wang. Lu Pang, and Tiejun Huang. Deep relative distance learning: Tell the difference between similar vehicles. In CVPR, 2016.
[74] Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma. Single-image crowd counting via multi-column convolutional neural network. In CVPR, 2016.
[75] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sme. JMLR, 2008.
[76] Hugo Touvron, Matthicu Cord, Alaacidin El-Nouby, Jakob Verbeek, and Herve Jegou, Three things everyone should know about vision transformers. ArXiv e-prints, 2022.
[77] Hugo Touvron, Matthieu Cord, and Herve Jegou. Deit ii:: Revenge of the vit. ECCV, 2022.
[78] Terrance DeVries and Graham Taylor. Improved regularization convolutional neural networks with cutout. ArXiv e-prints, 2017.
[79] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
[80] Wenbin An, Feng Tian, Ping Chen, Siliang Tang, Qinghua Zheng, and QianYing Wang. Fine-grained category discovery under Searse-grained supervision with hierarchical weighted self-contrastive learning. EMNLP, 2022.
[81] Laurynas Karazija. Iro Laina. and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsuper- vised multi-object segmenttiio....... In Proceedings the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[82] Zhuowan Li, Xingrui Wang, Elias Stengel-Fskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan I. Yuille. Super-clevr: virtual benchmark to diagnose domain robustness in visual reasoning. In CVPR, 2023.
[83] Leonard Salewski, A Sophia Koepke, Hendrik PA Lensch, and Zeynep Akata, Clevr-x: A visual reasoning dataset for natural language explanations. In xxA/ Beyond explainable Artificial Intelligence, 2020.
[84] Sagar Vaze. Nicolas Carion, and Ishan Misra. Genecis: A benchmark for general conditional image similarity. In CVPR, 2023.
[85] Reuben Tan, Mariya I. Vasileva, Kate Saenko, and Bryan A. Plummer. Learning similarity conditions without explicit supervision. In ICCV, 2019.
[86] Ishan Nigam, Pavel Tokmakov, and Deva Ramanan. Towards latent attribute discovery from triplet similarities. In ICCV, 2019.
[87] Jiawei Yao, Enbei Liu, Maham Rashid, and Jubua Hu. Augdme: Data augmentation guided deep multiple clustering. ArXiv e-prints, 2023.
14

---

[88] Liangrui Ren, Guoxian Yu, Jun Wang, Lei Liu, Carlotta Domeniconi, and Xiangliang Zhang. A diversified attention model for interpretable multiple clusterings. TKDE, 2022.
[89] Ruchika Chavhan, Henry Gouk, Jan Stuehmer, Calum Heggan, Mehrdad Yaghoobi, and Timothy Hospedales. Amortised invariance learning for contrastive self-supervision. ICLR, 2023.
[90] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. Why do self-supervised models transfer? investigating the impact of invariance downstream lasks. ArXiv e-prints, 2021.
[91] Ravid Shwartz-Ziv, Randall Balestriero, and Yann LeCun. What do we maximize in self-supervised learning?, 2022.
[92] Randall Balestriero, Ishan Misra, and Yann LeCun. A data-augmentation is worth thousand samples: Analytical moments and sampling-free training. In NeurIPS, 2022.
[93] Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving self-supervised learning characterizing idealized representations. NeurIPS, 2022.
[94] Wenhin An, Feng Tian, Qinghua Zheng, Wei Ding, QianYing Wang, and Ping Chen. Generalized category discovery with decoupled prototypical network. AAAI, 2023.
15

---

Appendices
The appendices are summarized the contents table below. We highlight: appendices A. and A.2 for details on Clevr-4; appendix B.4 for further analysis of pre-trained representations Clevr-4; and appendix C.3 for a long-tail evaluation of pGCD on the Herbarium 19 dataset [66].
Clevr-4
16
A.1
Clevr-4 generation
16
A.2 Clevr-4 details
17
A.3 Clevr-4 examples
17
B Analysis of results
18
B.1 Clevr-4 error bars
18
B.2 shape failure case
19
B.3 Semi-supervised k-means with pre-trained backbones
20
B.4
Clustering with sub-spaces of pre-trained features
20
B.5 Design of data augmentation and mis-aligned augmentations
22
B.6 Effect of X
23
Additional Experiments
23
C.I Results with estimated number of classes
23
C.2 Results with varying proportion of labelled examples
24
C.3 Results on Herbarium19
24
Description baselines and cCCD algorithms
24
Further Implementation Details
26
F Connections to related work
28
F1 Clevr-4: connections to real-world and disentanglement datasets
28
F22
μGCD method.
28
A Clevr-4
A.1 Clevr-4 generatio
We build Clevr-4 using Blender [54], a free 3D rendering software with a Python API. Following the CLEVR dataset [12], our images are constituted of multiple rendered objects in a static scene. Each object is defined by three 'semantic' attributes (texture, shape and color), and is further defined by its size, pose and position in the scene. We consider the first three attributes 'semantic' they are categorical variables which can neatly define image 'classes'. Meanwhile, we designate the size, pose and position attributes as 'nuisance' factors which are not related to the image category.
CLEVR Limitations. CLEVR is first limited for the purposes of category discovery as has only two textures ('rubber" and 'metal') and three shapes (" ('cube', 'sphere" and 'cylinder'). For category discovery, we wish to have more categories, both to increase the difficulty of the task, and to ensure sufficient number of classes in the 'Old' and 'New' subsets. Furthermore, we wish to have the same number of categories each split; otherwise, in principal, an unsupervised algorithm may be able to distinguish the taxonomy simply from the number of categories present.
16

---

100.0
0.2
0.2
0.2
Stappe
100.0
0.2
0.6
Figure Normalized mutual information be- tween the four taxonomies in Clevr-4. All tax- onomies have roughly no mutual information be- tween them (they are statistically independent).
Calar
0.2
0.2
100.0
0.2
Couna
0.2
0.6
0.2
100.0
Tomero
Shape
Color
Cmnsl
Expanding the taxonomies. To increase the number of categories in each taxonomy, introduce new textures, colours and shapes to the dataset, resulting in 10 categories for each taxonomy. We create most of the 8 new textures by wrapping a black-and-whit JPEG around the surface of the object, each of which have their own design (e.g., 'chessboard" or "circles"). Given an 'alpha' for the opaqueness of this wrapping. these textures can be distinguished independently of the underlying color. We further leverage pre-fabricated meshes packaged with Blender to introduce 7 new shapes to the dataset, along with new colors for the objects. The new shapes and colors were selected be clearly distinguishable from each other. Full definitions of the taxonomies are given in appendix A.2.
Image sampling process. For given image, first independently sample object texture, shape and color. We then randomly sample how many objects should be in the image (i.e., object count) and place this many objects in the scene. Each object has its own randomly sampled size (which is taken to be one of three discrete values), position and relative pose. Thus, differently to CLEVR. all objects in the image have the same texture, shape and color. This allows these three attributes. together with count, to define independent taxonomies within the data.
A.2 Clevr-4 details
We describe the categories each of the four taxonomies in Clevr-4 below. All taxonomies have 10 categories, five of which are used in the labeled set and shown in bold. Image exemplars of all categories are given in figs. and
Texture: rubber, metal, checkered, emojis, wave, brick, star, circles.
zigzag, chessboard
Shape:
cube, sphere, monkey, cone, torus, star, teapot, diamond, gear, cylinder
gray, red, blue, green, brown, purple, cyan, yellow, pink, orange . Count: 1, 2, 3, 4, 5, 6, 7, 8, 9,
Color:
fig. 6 plots the frequency of all categories in the taxonomies, while fig. 5 shows the mutual information between the four taxonomies. We find that all taxonomies, except for shape, are roughly balanced, and the four taxonomies have approximately mutual information between them realizing our desire of them being statistically independent.
A.3 Clevr-4 examples
We give examples of each of the four taxonomies in Clevr-4 in figs. 7 and 8.
17

---

1837 A0oge
a. objocts
072
tomo
brown
worg
Figure 6: Category frequency plots for each laxonomy in Clevr-4. All taxonomies roughly
balanced, except for shape. shape shows minor imbalance due to greater difficulty in placing many objects of some shapes (e.g. 'star' and "monkey") ) in scenes.
Texture
Shape
brick
checkered
cube
chessboard
circles
cylinder
diamond
emojis
metal
gear
monkey
rubber
star
sphere
star
zigzag
teapot
Figure 7: Examples of each category from the texture and shape taxonomies of Clevr-4.
B
Analysis of results
B.1 Clevr-4 error bars
show results the GCD baseline [7], the current state-of-the-art SimGCD [19] and our method,
uGCD, in fig. 9, The results shown for five random seeds for each method, and plotted with the
18

---

Color
Count
blue
brown
gray
3
green
orange
pink
purple
7
yellow
9
Figure Examples of each category from the color and count taxonomies of Clevr-4.
standard matplotlib boxplot function, which identifies outliers in colored circles. We also plot the median performance of our method each taxonomy in dashed lines.
Broadly speaking, the takeaways are the same as the results from Table of the main paper. However, while the mean performance of our method is worse than SimGCD on the shape split, we can see here that the median performance of ,GCD is within bounds, or significantly better, than the compared methods on all taxonomies.
B.2 shape failure case
Overall, find our proposed ,GCD outperforms prior state-of-the-art methods three of the four Clevr-4 splits (as well as on the Semantic Shift Benchmark [20]). We further show in appendix B. that, when accounting for outliers in the five random seeds, method is also roughly equivalent to the SimGCD [19] state-of-the. art on the shape split of Clevr-4.
Nonetheless, we generally find that our method is less stable on the shape split of Clevr-4 than on other taxonomies and datasets. provide some intuitions for this by visualizing the representations and predictions of our method in fig. 10.
19

---

All Clesss
OM Clssrs
New Classss
Methods
Charrtng hourgg
GCD
SimGCD
μGCD
36-
100
Vreww
Color
Creat
Trxtens
Caler
Couer
Treem
Color
Coart
CLEVE Taxswvey
CLIVER Toweeery
CLENE Tacunony
Figure 9: Box plots of results on Clevr-4, We show results for the GCD baseline [7], the current state-of-the-art SimGCD [19] and method, μGCD. We plot results for five random seeds for the four taxonomies, with outliers shown as colored circles. We also plot the median performance of our method on each taxonomy in dashed lines. On all taxonomies, ,GCD is within bounds, or significantly better, than the compared methods.
Preliminaries: In fig. 10, we plot TSNE projections [75] of the feature spaces of versions of our model, as well as the histograms of the models' predictions on the shape split. Along with the models' image representations (colored scatter points), we also plot the class vectors of the cosine classifiers (colored stars). On the left, we show our trained model when we randomly initilize the cosine classifier, while on the right we initialize the class vectors in the classifier with k-means centroids. We derive these centroids by running standard k-means the image embeddings of the backbone, which is pre-trained with the GCD-style representation learning step (see appendix D).
In the plot on the left. we find that though the feature space is very well separated (there is little overlap between clusters of different categories), the performance of the classifier is still only around 90%. The histogram of model predictions demonstrates that this is due to no images being assigned the 'star' category this vector the classifier completely unused. Instead, too many instances are assigned to 'gear'. In the TSNE plot, we can see that the 'gear' class vector is between clusters for both 'gear' and 'star' images, while the 'star' vector is pushed far away from both. We suggest that this is due to the optimization falling into a local optimum early on in training, as a result the feature-space initialization already being so strong.
Observations:
On the right, we find we can largely alleviate this problem by initializing the classification head carefully with k-means centroids from the pre- tre-trained backbone. We see that the problem nearly perfectly solved, and the histogram of predictions reflects the true class distribution of the labels.
Takeaway: We find that when the initialization of the model's backbone from the GCD-style representation learning step, see appendix D - is already very strong, random initialization of the classification head in fGCD result in local optima in the model's optimization process. This be alleviated by initializing the classification head carefully with h-means centroids resulting in almost perfect performance but the issue can persist with some random seeds (see appendix B.
B.3 Semi-supervised k-means with pre-trained backbones
In fig. 11. we probe the effect of running comi-supervised k-means [7] on top of different pre-trained backbones. This is simple mechanism by which models can leverage the information from the 'Old' class labels. We find that while this improves clustering performance on some taxonomics, is insufficient overcome the biases learned during the models' pretraining, corroborating findings from table of the main paper.
B.4 Clustering with sub-spaces of pre-trained features
In table 2 and fig. 11, we demonstrate that all pre-trained models have clear bias towards one of the Clevr-4 taxonomies. Specifically, we find that clustering in pre-trained feature spaces preferentially aligns with single attribute (e.g shape or color).
20

---

w/o classitier init.
w/ classifier init. ACC:
ACC: 91 91.4/ 1
Figure 10: Analysis of the shape failure mode, showing TSNE plots [75] and prediction histograms for two models, trained without (left) and with (right) initialization the classification head with k-means centroids. Left: When the backbone initialization (from the GCD representation learning step [7]) is already very strong, the classification head gets stuck a local optimum, with one class vector unused. Right: We find
alleviate this by initializing the class vectors with k- k-means centroids, almost perfectly solving the problem, but the issue can persist with some random seeds.
Taxonomy
80-
Texture
Shape
Clustering ACC (AI)
Color
Count
40
Clustering Method
K-Means
SS-K-Means
Pre-tmining (Backbonc)
Figure 11: Effect of semi-supervised k-means on representative pre-trained backbones. We find semi-supervised k-means is insufficient to overcome the bias learned during pre-training.
Here, investigate whether these clusters have any sub-structures. To do this, we perform PCA analysis on features extracted with two backbones: DINOv2 [60] and MAE [31]. Intuitively, we wish to probe whether the omission of dominant features from the backbones (e.g the shape direction with DINOv2 features) allows k-means clustering to identify other taxonomies. Specifically, we: (i) extract features for all images using given backbone, X E NxD, (ii) identify the principal components of the features, sorted by their component scores, W E DDDD; (iii) re-project the features onto the components, omitting those with the highest scores, X (X μ)) W[: :] (iv) cluster the resulting features, X E R with k-means. Here, fa is the average of the features X, and the results are shown in figs. 12 and 13.
Overall, find that by removing the dominant features from the backbones, performance other taxonomies can be improved (at the expense of performance on the 'dominant' taxonomy). The effect particularly striking with MAE, where we almost seven-fold increase in shape performance after the the three most dominant principal components are removed.
This aligns with the reported performance characteristics of DINOv2 and MAE. The object-centric recognition datasets on which these models are evaluated benefit from shape-biased representations (see section 6). We find here that both MAE and DINOv2 encode shape information, but that more work is required extract this from MAE features. This reflected by the strong linear probe and RNN performance of DINOv2, while MAE requires full fine-tuning to achieve optimal performance.
21

---

Nen PCA Comprecrn Recsoved
Chorcdig
Tceas
Coler
Coao:
Clarrt Treeseny
Figure 12: Re-clustering DINOv2 [60] features after removing dominant principal components.
Nen PCA Compeecrs Rezevved
Choroks
Tudtau
Gea-4
Figure 13: Re-clustering MAE [31] features after removing dominant principal components.
Finally, we note that decoding the desired information from pre-trained features is not always trivial, and we demonstrate section that even in the (partially) supervised, fine-tuning setting in GCD, both of these backbones underperform randomly initialized ResNer18 on the count taxonomy.
B.5 Design of data augmentation and mis-aligned augmentations
In our SSB experiments, the teacherin is passed a weaker augmentation, comprising only RandomC rop and RandomHori zzontalF Flip. We find this stabilizes the pseudo-labels produced by the teacher. However, the self-supervised literature consistently finds that strong augmentations are beneficial for representation learning [9, 14, 21J. As such, we experiment with gradually increasing the strength of the augmentation passed to the student model in table 7.
axes: the strength of the base augmentation ('Strong Base Aug' column); and how aggressive the cropping augmentation is ('Aggressive Crop' column). To make the base augmentation stronger, we add Solarization and Gaussian blurring [21]. For cropping. we experiment with a light RandomResi zeCrop (cropping within range of 0.9 and 1.0) and more aggressive variant (within range of 0.3 and 1.0). Overall, find that aggressive cropping strategy, as well as a strong base augmentation, critical for strong performance. We generally found weaker variants overfit. Though they also have lower peak clustering accuracy, the accuracy falls sharply later in training without the regularization from strong augmentation.
Specifically, experiment along
Table 7: Design of student augmentation.
CUB
Aggressive Crop
Strong Base Aug
Old
New
X
38.6
54.6
30.6
/
41.6
58.8
33.0
X
52.7
69.4
44.7
65.7
68.0
64.6
22

---

Table 8: Effect of mis-aligned augmentations on the GCD Baseline.
Color
Count
Aligned Augmentation
84.5
65.2
Misaligned Augmentation
26.1
46.6
14-
14-
Canciggce
Claccing Spp hand
014
64-
04-
04-
02
02
02
Figure 14: Effect hyper-parameters d₁. We investigate the effect of A (which balances the supervised and un-supervised losses), training μGCD models on the texture split.
Mis-aligned augmentations on Clevr-4 A benefit of Clevr-4 is that each taxonomy has simple semantic axes. As such, we are able to conduct controlled experiments on the effect of targeting these axes with different data augmentations. Specifically, in table 8, demonstrate the effect of having 'misaligned' augmentations on two splits Clevr-4. We train the GCD baseline with ColorJitte Jirter on the color split and Cut for the count split. The augmentations destroy semantic information for the respective taxonomies, resulting in substantial degradation of performance. The 'aligned' augmentations are light cropping and flipping for color, and light rotation for count. The results highlight the importance data augmentation in injecting inductive biases into deep representations.
B.6
Effect of X1
In fig. 14, we investigate the effect of the hyper-parameter A₁ which controls the tradeoff between the supervised and unsupervised losses in μGCD. We find that with 0.1 <= \ <= 0.4, the 'All' clustering accuracy is robust, while at λ = 0 (only unsupervised loss) and A = (only supervised loss), performance degrades. We note that the Hungarian assignment in evaluation results in imperfect 'Old' performance even at \r close to (more weight on the supervised loss). As such, we also show an 'Upper Bound' ('cheating') ) clustering performance in gray, which allows re-use of clusters in the 'Old' and 'New" accuracy computation.
C Additional Experiments
C.1 Results with estimated number of classes
In the main paper, we followed standard practise in category discovery [7, 19, 43, 46, 50, 52) and assumed knowledge of the number of categories in the dataset, k. Here, we provide experiments when this assumption is removed. Specifically, we train our model using an estimated number of categories in the dataset, where the number of categories is predicted using off-the-shelf method from [7]. We use estimates of & = 231 for CUB and & = 230 for Stanford Cars, while these datasets have a ground truth number of A 200 and 196 classes respectively.
We compare against figures from SimGCD [19] as well as the GCD baseline [7]. As expected, we find our method performs worse on these datasets when an estimated number of categories is used, though we note that the performance of SimGCD [19] improves somewhat on CUB, and the gap between our methods is reduced on this dataset. Nonetheless, the proposed fGCD still performs marginally better on CUB, and further outperforms the SoTA by nearly 7% on Stanford Cars in this setting.
23

---

Table 9: Results on the SSB with estimated number of categories. We the method from [7] to estimate the number of categories as k = 231 for CUB, and h = 230 for Stanford Cars. We run our method with this many vectors in the classification head, comparing against baselines evaluated with the same estimates of k. Results from baselines are reported from [19].
CUB
Stanford Cars
Average
Pre-training
All
Old
New
All
Old
New
All
GCD [7]
DINO [14]
47.1
55.1
44.8
35.0
56.0
24.8
41.1
SimGCD [19]
DINO [14]
61.5
66.4
59.1
63.1
41.3
55.3
μGCD (Ours)
DINO [14]
62.0
60.3
62.8
56.3
66.8
51.1
59.2
Classes
New
76
100
Clusecto (.C. (LCI
72-
2OCT)
98
57.
65-
95
64-
56
52-
90
42-
44
36
40
0.1
0.3
0,5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0,7
0.9
Prop 'Old Labsllel
Prop. Labelled
Prep.of Labellad
Figure 15: Results when varying the proportion of "Old' category images reserved reserved for
De. We find our pGCD method substantially outperforms the GCD baseline [7] across all settings.
C.2 Results with varying proportion of labelled examples
In the main paper, we follow standard practise in the GCD setting [7, 19, 50, 52] and sample fixed proportion of images, p 0.5, from the labelled categories and use them in the labeled set, Dg. Here, we experiment with our method if this proportion changes, showing results in fig. 15. We find our proposed pGCD substantially outperforms the GCD baseline [7] across all tested values of p.
C.3 Results on Herbarium19
We evaluate our method on the Herbarium19 dataset [66]. We use the 'Old'/'New' class splits from [7] which are randomly sampled rather than being curated as they in the SSB. Nonetheless, the dataset is highly challenging, being long-tailed and containing 683 classes in total. 341 of these classes reserved 'Old', and the dataset contains total of 34K images. It further contains clear taxonomy (herbarium species), making it a suitable evaluation for GCD. We compare fGCD against prior work in table 10, again finding that we set a new state-of-the-art.
D Description of baselines and ,CCD algorithms
In this section provide step-by-step outlines of: the GCD baseline [7]: the SimGCD [19] baseline; and our method, ,GCD. Full motivation of the design decisions in μGCD can be found section 4.
Task definition and notation: Given dataset with labelled (Dz) and unlabelled (Du) subsets, model must classify all images Dy into one the categories in Dus and prior knowledge of k is assumed. During training, batches (B), are sampled with both labelled images (Bc E Dc) and unlabelled images (Bu Dz). The performance metric the clustering (classification) accuracy on Dz--
possible categories. De contains only subset of
GCD [7]. Train a backbone, Φ and perform classification by clustering in its feature space.
(1) Train & using an unsupervised InfoNCE loss [23] on all the data, as well as supervised contrastive loss [61] ] on the labeled data. Letting a; and a' represent two augmentations of the same image in batch B, the unsupervised and supervised losses are defined as:
24

---

Table 10: Results on Herbarium19 [66]. which a long-tailes recognition dataset.
Herbarium 19
Pre-training
All
Old
New
k-means [57]
DINO [14]
13.0
12.2
13.4
RankStats+ [46]
DINO [14]
27.9
55.8
12.8
UNO+ [43]
DINO [14]
28.3
53.7
12.8
GCD [7]
DINO [14]
35.4
51.0
27.0
ORCA [8]
DINO [14]
20.9
30.9
15.5
OpenCon [52]
DINO [14]
39.3
58.9
28.6
PromptCAL [50]
DINO [14]
37.0
52.0
28.9
MIB [51]
DINO [14]
42.3
56.1
34.8
SimGCD [19]
DINO [14]
43.3
57.9
35.3
pGCD (Ours)
DINO [14]
45.8
61.9
37.2
exp(z.,x()/t /T feat,i = log Inzi expl (xpp(z..t)//5
exp .t,) feat.s= = - EmEN(o) log En ex ////T
projection head, which is used during training and discarded afterwards; and 7 is temperature value. .Nii) represents the indices of images in the labeled subset of the batch, B2 E B, which belong to the same category as I₁ Given a weighting coefficient, X1, the total contrastive loss the model's 's features is given
where: 2; = ho,)
Leeat = (1- - A1) Σ
C'eaat + A1
CJeats
IEB
iEBg
(2) Perform classification by embedding all images with the trained backbone, 4, and apply semi- supervised k-means (SS-k-means) clustering on the entire dataset, Diu Dz. SS-k-means is identical to unsupervised k-means [57] but, at each iteration, instances from Dz are always assigned the 'correct' cluster using their labels, before being used in the centroid update. In this way, the cluster centroid updates for labelled classes are guided by the labels in Dc-
SimGCD [19]. Train a backbone representation, 4, and linear head, g. to classify images amongst the A classes in the dataset, yielding model = g Φ. Train the backbone jointly with the feature space loss from eq. (4), and with linear classification losses based on the output of g.
(1) Generate pseudo-labels for image, æ,, (x,) [0, 11th in order to train the classifier, fa. Infer pscudo-labels on images in a batch, B, and compute an additional supervised cross-entropy loss the labelled subset, Be.
Pass two views of an image to the same model. Each view generates soft pseudo-label for the other, for instance as:
(r();; = sglsoftmax(fo((=))
Here sg is the stop-grad operator and Fr is the pseudo-label temperature.
Compute model predictions as (s( softmax (()) and a standard pseudo- labelling loss [14, 18, 26] (i.e. soft cross-entropy loss) as:
C (0:
Σ 2.EB
(pr(a.)) (og(ps(xx))) - (ar()))) log(ps(*<)))
(6)
B
Temperatures chosen such that TT Ts to encourage confident pseudo-labels [14].
Optimize the model, fo. jointly with: the pseudo-label loss (eq. (6)) and f (see eq. (4)).
The model further trained with: the standard supervised cross-entropy loss on the labelled subset of the batch, Ca. (0: Be); and entropy regularization term, CG, (0):
25

---

(0; = [Bcl
Σ (y(x), log(ps(z))), Cao(0) - - log log(ps)): Ps = Eps(a)) iEBc
[B|
WEB
Here, y()) is ground-truth label and, given hyper-parameters λ1 and 22. the total loss is defined as:
[(0; B) = (1 A1) (0; B) + (C (0; (O; + Ai (0; + G (0; Bc)) + X2 (0).
GGCD (Ours). Train backbone representation, 4, and linear head, 9. to classify images amongst the k classes in the dataset, yielding a model for = o tb Train the backbone first with the feature space loss from eq. (4), and then with linear classification losses based the output of g.
(1) Train backbone + using Step (1) from the GCD baseline algorithm.
(2) Append a classifier, to the backbone and duplicate it to yield two models. One model (a teacher network, faz)) is used generate pseudo-labels for student network, feg. as Pr (2x) [0, 1 Infer pseudo-labele on all images in a batch, B3 and compute an additional supervised cross-entropy loss on the labelled subset, Bg. The student and teacher networks are trained follows:
Generate a strong augmentation of an image, 2; and a weak augmentation, 25 [49]. Pass the weak augmentation to the teacher generate a pseudo-label and construct loss:
(x,) = sg sglsoftmax(fe
(0s:BB
Σ (pr (zx), log(ps(x:))) (7) 28
B
Optimize the student's parameters, 0g,, with respect to: the pseudo-label loss from eq. (7); the supervised loss, Lags and the entropy regularization loss, C Formally, the 'student', fes. is optimized for: L(0s; B) = (1 Ci (0s; B) + Cas (0s; B2) + (ls))
Update the teacher network's ' parameters with the Exponential Moving Average (EMA) of the student network [17]. Specifically, update the 'teacher' parameters, Or, as:
0T w()0T + -1- +1-
where t is the current epoch and (t) is time-varying decay schedule.
At the end of training, the 'teacher', far, is used for evaluation. Remarks:
We first highlight the different ways in which the labels from Dc are used between the three methods. Specifically, the GCD baseline [7] only uses the labels in feature-space supervised contrastive loss. However, in addition to this, SimGCD [19] and pGCD also use the labels in standard cross-entropy loss in order train part of linear classifier, g.
We further note the high level similarity between SimGCD and ,GCD, in that both train parametric classifiers with pseudo-label loss. While SimGCD uses different views passed to the same model to generate pseudo-labels for each other (similarly to SWaV (9]), μGCD uses pseudo-labels from a 'teacher' network to train 'student' (similarly mean-teachers [17]).
This is in keeping with trends related fields, which find that there exists a small kernel of method- ologies - e,g. mean-teachers [17], cosine classifiers [62], entropy regularization [26] which are robust across many tasks [14, 22, 25], but that finding strong recipe for a specific problem critical. We find this to be true in supervised classification [58, 76, 77], self-supervised learning [14, 21], and semi-supervised learning [17, 26, 49]. Our use of mean-teacher. to provide classifier pseudo-labels, well careful choice of model initialization and data augmentation, yields performant μGCD algorithm for category discovery.
E Further Implementation Details
When re-implementing prior work, we aim to follow the hyper-parameters of the GCD baseline [7] and SimGCD [19], and use the same settings for our method. We occasionally find that tuned hyper-parameters are beneficial in some settings, which we detail below.
26

---

Learning rates. We swept learning rates at factors of 10 for all methods and architectures. When training models from scratch (ResNet18 on Clevr-4) or when finetuning a DINO/DINOv2 model [14, 60] on the SSB, we found learning rate of 0.1 to be optimal. When finetuning an MAF [31] or DINOv2 model on Clevr-4, we found better to lower the learning rate to 0.01. All learning rates are decayed from their initial value by factor of 10-3 throughout training with a cosine schedule.
Loss hyper-parameters. For the tradeoff between the unsupervised and supervised components of the losses, A is set to 0.35 for all methods. For the entropy regularization, we follow SimGCD and use 22 = 1.0 for FGVC-Aireraft and Stanford Cars, and X2 = 2.0 for all other datasets. We swept to find better settings for this term on Clevr-4, but did not find any setting to consistently improve results. We also train with 2 weight decay, set 10e 4 for all models.
Student and teacher temperatures. Following [14], we set the temperature of the student and teacher to Ts 0.1 and fr = 0.04 respectively, for both our method and SimGCD. This gives the teacher 'sharper' (more confident) predictions than the student. We further follow the teacher- temperature warmup schedule from [14], also used in SimGCD, where the teacher temperature is decreased from 0.07 to 0.04 in the first 30 epochs of training. On Herbarium19 [66] (which has many more categories than the other evaluations, see appendix C.3), we use teacher temperature of 2 x 10-3 (warmed up from 3.5 x 10-3 over 10 epochs).
Teacher Momentum Schedule. In pGCD, at each iteration, the teacher's parameters are linearly interpolated between the teacher's current parameters and the student's, with the interpolation (' ('decay' or 'momentum') ) changing over time following [18), as: (()) WT (1 Waase) ) (cos( 7) + 1)/2.
Here T is the total number of epochs and f the current epoch. We use wT = 0.999
and Whose = 0.7. We note for clarity that, though the momentum parameter is dictated by the epoch number, the teacher update happens at each gradient step.
Augmentations. On Clevr-4 we use an augmentation comprising of ShanddmHorizzntaalll. and RandomRotation.
On the SSB [20], we use RandomHori wwwelHorioonttlln aontalFii and RandomCrop. We use these augmentations for all methods, and for pGCD use these augmentations to pass views to the 'teacher'. An important part of our method on the SSB is to design strong aug- mentations to pass to the student. Our 'strong augmentation' adds aggressive RandomRes si zeCrop, as well as solarization and Gaussian blurring [21] (see appendix B.5 for details). On Clevr-4, due to the relatively simple nature of the images, strong augmentations can destroy the semantic image content; for instance color jitter and aggressive cropping degrade performance on color and count respectively. We find it helpful to pass Cutout [78] to the teacher on the color taxonomy, and texture benefits from the strong augmentation defined above.
Training time. Following the original implementations, we train all SimGCD [19] and GCD base- line [7] models for 200 epochs, which we find sufficient for the losses (and validation performance) to plateau. For our method, we randomly initialize classifier on model which has been trained with the GCD baseline loss, and further finetune for another 100 epochs. On our hardware (either an NVIDIA P40 or M40) we found training to take roughly 15 hours for SSB datasets, and around 4 hours for a Clevr-4 experiment.
Early stopping. We note that GCD is a transductive setting, or a clustering problem, where models are trained (in an unsupervised fashion) on the data used for evaluation, Dr. As such, an important criterion is which metric to use to select the best model. SimGCD [19] and the GCD baseline [7] use the performance choice for the baseline, we found it can lead to underestimated performance for SimGCD on some datasets. For SimGCD, we instead found it better to simply take the model at the end of training. For pGCD, we instead propose to choose the model with the minimum unsupervised loss on the unlabeled set.
a validation of images from the labeled categories. While this is reasonable
Other details:
When finetuning pre-trained transformer models - DINO [14], DINOv2 [60] or MAE [31] we finetune the last transformer block of the model. For Clevr-4, when training ResNet18, 8, we finetune the whole model. Finally, for the ,GCD failure case of shape, we suggest in appendix B.2 that μ&CD can get stuck in local optima if its initialization is already very strong. As such, in this case, we initialize the linear head with k-means centroids, reduce the learning rate and teacher temperature 0.01, and set Whase to 0.9.
27

---

F Connections to related work
E1 Clevr-4: connections to real-world and disentanglement datasets
Datasets with different granularities. When multiple taxonomies are defined in exisiting datasets, they are most often specified only at different granularites, for instance in CIFARI00 [53], FGVC- Aircraft [55] and iNaturalist [79]. While recognition at different granularites is related to our task and was explored in [80]- the constituent taxonomies are not statistically independent, as the Clevr-4 splits are. We note that, given the number of categories in each taxonomy, an unsupervised model could in principal solve the clustering problem the different granularities.
CUB-200-2011 [64]. Fei el al. [63] discuss the existence of alternate, but valid, clusterings of images from fine-grainet datasets like CUB [64] - e-g., based pose background. We note that the CUB 'Birds' dataset presents an opportunity for constructing an interesting dataset for category discovery. Each image in CUB labelled for presence (or absence) of each of 312 attributes, where these attributes come from different attribute types. Each attribute type (e.g., 'bill shape', 'breast color') provides different taxonomy with respect to which cluster the data. However, found these attribute annotations are too noisy to yield meaningful conclusions.
Disentanglement datasets. We suggest that Clevr-4 is also useful benchmark for disentanglement research [34, 35]. This research field aims to learn models such that the ground-truth data generating factors (i.e., attributes of an object) are encoded in different subspaces of the image representation. The current CLEVR dataset [12] cannot be used easily for this, as its images contain multiple objects, each with different attributes. Instead, in Clevr-4, all objects share the same attributes, allowing each image to be fully parameterized by the object shape, texture, color and count. Furthermore, compared synthetic datasets for disentanglement [38], Clevr-4 contains more categorical taxonomies, as well as more classes within those taxonomies.
Finally, we note that there exist other extensions of the CLEVR dataset [12], such as ClevrTex [81] Super-CLEVR [82] and CLEVR-X [83]. which also add new textures and/or categories to the original datasets. However these datasets cannot be used for category discovery (or disentanglement) research as, unlike in Clevr-4, they contain scenes with objects of differing attributes. As such, each image cannot be parameterized with respect to the object attributes in way which gives rise clear taxonomies.
Other related fields The GCD task and the Clevr-4 dataset are related to number of other machine learning sub-fields. Conditional Similarity research [84-86] aims to learn different embedding functions given different conditions. For instance, the GeneCIS benchmark [84] evaluates the ability of models to retrieve different images given a query and different conditioning text prompt. Meanwhile, the multiple clustering [87, 88] and self-supervised learning [89, 90] fields investigate the how different choices of data augmentation result in different clusterings of the data. The self-supervised field particularly aims to understand why these inductive biases result in different generalization properties [91-93].
We hope that Clevr-4 can be complementary to these works, and provide test-bed for controlled experimentation these research questions.
F.2 GCD method.
We note here that the idea of momentum encoders has been widely used in representation learning [18,
24, 50], semi-supervised learning [17, or to update class prototypes in category discovery [52, 94]. We use a mean-teacher model end-to-end, for the backbone representation and the classification head.
We highlight that, similar to rich vein of literature in related fields [14, 21, 26, 58, 76, 77], our goal is to find specific recipe for the GCD task.
28


Reviews:
[{"review_id": "TojbquR0Mj", "reviewer": "Reviewer_V8PZ", "summary": "This paper addresses generalized class discovery (GCD) from a unique perspective. The authors argues that current GCD benchmarks are unable to ascertain whether models are using the available labels to solve the GCD task, or simply solving an unsupervised clustering problem. In light of this, this paper introduces a new dataset where the data can be clustered according to different rules, i.e., object count, shape or texture. This paper also proposes a simple method that effectively solves such a problem.", "questions": "- The proposed is a simple reinvent of existing techniques that are common in semi- and self-supervised literature. As mentioned above, the question is how does such a simple method solves the proposed GCD problem when different clustering rules are acceptable? \n\nThank the authors for the rebuttal, which addressed most of my concerns. Please include the modifications as mentioned in the discussion in the final version.", "limitations": "The limitations have been carefully discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of this paper is sound.\n- The paper is well-written and easy to follow.\n- The contribution is convincing.", "weaknesses": "- The introduced dataset is focused on synthetic data. It could be more convincing to evaluate on real-world data.\n- The proposed method is simple yet powerful, but it's unclear which mechanism solves the GCD problem when multiple clustering rules are available.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's feedback on on our work. We hope to address their concerns as follows:\n\n**“The introduced dataset is focused on synthetic data. It could be more convincing to evaluate on real-world data”**:\n\nWe found it difficult to find an appropriate real dataset which contains sufficiently complete annotations to construct several taxonomies over the same images. The closest dataset we found was CUB-200-2011, which contains annotations for different attribute types (e.g ‘head color’, ‘bill shape’ etc). However, upon investigation, we found the annotations to be too noisy to build a proper benchmark.\n\nWe discuss this issue in the limitations (L331) and the difficulties of constructing real-world datasets of this form in Appendix 6.1. \n\nGiven the difficulties of finding an appropriate real dataset, we consider that the advantages of a synthetic dataset — which allows precise manipulation of the images and hence a more controlled study of the GCD problem — outweigh the disadvantages.\n\n**“...unclear which mechanism [in muGCD] solves the GCD problem when multiple clustering rules are available”**:\n\nIn Fig 3 (left), we find that the GCD baseline method can, to some extent, learn a representation which distinguishes images according to the desired taxonomy. Our intuition is to begin from this feature space, and train a linear head on top with strong pseudo-labels (generated with the mean-teacher approach) which is then specialized for the desired taxonomy. We thank the reviewer for raising this important point, and we will update the manuscript to extend our discussion in L216 - 234.\n\n"}, {"review_id": "F5dobYx7J9", "reviewer": "Reviewer_s1RT", "summary": "1. This paper tackles the problem of generalized category discovery (GCD) and identifies the drawbacks of existing methods that they are verified only with labels for a single clustering of the data. In such a case, the model may simply perform unsupervised clustering, not correctly using the available labels. \n2. A synthetic dataset, named “Clevr-4” is proposed which contains four independent taxonomies, i.e., shape, texture, color or count. \n3.The authors demonstrate the limitations of unsupervised clustering in the GCD setting and propose a new method, µGCD, based on mean teachers, which outperforms existing baselines on Clevr-4 and sets a new state-of-the-art on the Semantic Shift Benchmark (SSB).\n", "questions": "1. Additional ablation study on lambda_1 which is used to balance the supervised and unsupervised loss terms, should be added to verify the hyperparameter sensitivity.", "limitations": "The motivation of verifying GCD methods on taxonomies not merely on the semantic labels is reasonable. The experiments and analysis are comprehensive. Limitations of the method would be limited technical novelty. Codes are not provided. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation of this paper is clear and interesting. The authors propose that existing GCD methods may simply perform unsupervised clustering based on the natural grouping of the data. The issue is overlooked by existing methods and is indeed worth investigating. \n2. A new benchmark is proposed which provides four independent sets of labels for a common dataset, which is valuable to extensively verify GCD methods’ ability to extrapolate the taxonomies specified by the labeled data using different pre-trained feature extractors. ", "weaknesses": "1. Additional recent methods should be compared:\nPu N, Zhong Z, Sebe N. Dynamic Conceptional Contrastive Learning for Generalized Category Discovery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 7579-7588.\n2. Fig.2 is misleading. Why there are three 'student predictions'? Does the arrows between them mean that the three losses act sequentially, rather than all together?\n3. Since the proposed method adopts a two-stage training strategy, i.e., the proposed method could be regarded as an additional 100 epochs of training based on GCD, the training cost could become a concern. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments, and respond to their concerns as follows:\n\n**“Additional recent methods should be compared…[DCCL @ CVPR 2023]”**\n\nWe thank the reviewer for bringing this interesting paper to our attention, we were unaware of DCCL @ CVPR 2023 as CVPR 2023 was held after the NeurIPS submission deadline. We will include its results and discuss it in the related work. However, we note that its performance is lower than SimGCD and $\\mu$GCD on the Semantic Shift Benchmark (against which we compare), thus confirming that our claim of ‘SoTA’ on the SSB remains valid, even after this paper is taken into account.\n\n**“Additional ablation study on $\\lambda_1$…”**:\n\nWe note that we inherit the value of $\\lambda_1$ from prior work (the GCD baseline and SimGCD). However, we have now also ablated its effect on $\\mu$GCD in the Author Response PDF (Fig R1). Overall, we find that while performance degrades at extreme values of lambda – i.e near 0 (with only the unsupervised loss) or near 1 (with only the supervised loss) –  $\\mu$GCD is robust to $\\lambda_1$ changes in the range $0.1 <= \\lambda_1 <= 0.4$. \n\nWe thank the reviewer for raising this point and we will include this analysis in the updated manuscript.\n\n**“...the training cost [of $\\mu$GCD] could become a concern”**\n\nWe found the training time of all models to be feasible under an academic compute budget, with models being trained on a single NVIDIA M40 or P40 in roughly 4 hours on Clevr-4, and in roughly 15 hours on the SSB datasets (see Appendix L250). Furthermore, we found that the losses of all baseline methods had plateaued after 200 epochs, verifying that the improved performance of $\\mu$GCD is not simply due to longer training.\n\n**“Fig.2 is misleading. Why are there three 'student predictions'?”**:\n\nThe student model only produces one set of predictions, and we show these three times in the diagram to visualize the three losses (which are summed and optimized jointly). We agree that the figure may be confusing and will update the diagram, and accompanying text, to clarify.\n\n"}, {"review_id": "unEcsFJhuO", "reviewer": "Reviewer_dGiY", "summary": "This paper contributes one new dataset and one new method for category discovery. The dataset is designed in a way that each image can be clustered into 4 different clustering based on the attribute of the image (shape, count, texture, and color), this design can be used to reveal the difficulty of unsupervised clustering and the necessity of category discovery methods. The proposed method extends on the SimGCD method by adding an EMA teacher and strong-weak augmentations.", "questions": "1. I am interested in how would applying semi-supervised k-means algorithm on pretrained models in tab 2 performs, because using the semi-supervised k-means is IMO the easiest way to inject certain clustering criteria to the model.\n2. Since the proposed method is similar to SimGCD, then how about applying the strong augmentation to SimGCD or GCD and compare with that performance? It seems that the strong augmentation plays an important role for the performance as shown in line 2 of tab 6.\n3. I am also interested in the performance of tuning the augmentations used by SimGCD and GCD method on Clevr-4. As `misaligned augmentations degrade performance`, yet there are no experiment supporting this claim.", "limitations": "I think the methodology contribution of this paper is limited, the proposed method seems like a combination of FixMatch and SimGCD.\nThe proposed Clevr-4 might be of interest to the community, but as I discussed in the weakness and questions section, I think currently the discussion on Clevr-4 is not comprehensive enough.\n\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I really like the discussion of the limitation of unsupervised clustering method on self-supervised representations.\n2. The design of the Clevr-4 dataset is interesting, and could be a direction for future works.", "weaknesses": "1. I would like to note a few related fields, (1) deep multiple clustering [R1, R2] which deals with unsupervised clustering for different clustering criteria, and (2) conditional similarity [R3, R4]\n2. The proposed method seems to be a strighforward extension of the SimGCD method, I think it is not significant enough to be a claimed contribution.\n3. I think the discussion on the experiment of previous method on Clevr-4 is not comprehensive, as mentioned in line 262-269, augmentations seem to play an important role for category discovery, then I think the discussion on Clevr-4 should contain some experiments of using different augmentation for differe splits.\n\n\n[R1] AugDMC: Data Augmentation Guided Deep Multiple Clustering, https://arxiv.org/abs/2306.13023\n\n[R2] A Diversified Attention Model for Interpretable Multiple Clusterings, TKDE2022\n\n[R3] Learning Similarity Conditions Without Explicit Supervision, CVPR 2019\n\n[R4] Towards Latent Attribute Discovery From Triplet Similarities, ICCV 2019\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback, and hope to address their concerns as follows:\n\n**“How would applying semi-supervised $k$-means algorithm on pre-trained models in tab 2 perform”**:\n\nWe have re-evaluated a representative selection of the backbones from Tab 2 in the paper, in both unsupervised and semi-supervised k-means settings, for all four Clevr-4 taxonomies. The results are shown in the Author Response PDF (Fig R2). Overall, we find that though semi-supervised k-means can sometimes marginally improve performance on some taxonomies, it is *far from sufficient* to overcome the bias from the model’s pre-training. We suggest that this points to the utility of Clevr-4 as a test-bed for GCD, as well as as a probing set for biases in pre-trained representations.\n\n**“The discussion on Clevr-4 should contain some experiments of using different augmentation for different splits”**:\n\nIn the Author Response PDF (Tab R1), we have now included experiments of the GCD baseline trained on different taxonomies with different augmentations. Specifically, for the *count* split, we trained the model with ‘CutOut’, which blacks out a large portion of the image and hence hides a number of the objects. On the *color* split, we trained the model with ‘ColorJitter’, which disturbs the colors in the image. As such, these augmentations (though commonly used in the vision literature) are ‘misaligned’ with the respective taxonomies, and hence result in substantial performance degradation. We find that such failure modes, though quite intuitive, are difficult to isolate with existing GCD (or SSL) datasets, thus underlining the use case for Clevr-4. \n\nWe thank the reviewer for the suggestion, and we will update our manuscript to include this analysis and discussion.\n\n**“...how about applying the strong augmentation to SimGCD or GCD and compare with that performance”**:\n\nThis SimGCD experiment is almost exactly the same as the experiment from L(4) of the ablation in Tab 6. In this case, we train our model with $\\omega(t)$ always equal to 0, thereby using the weights from the last iteration as the ‘teacher’. Here, we find that performance degrades by 3%, pointing to the importance of the slowly updated mean-teacher used in $\\mu$GCD. We thank the reviewer for raising this point, and we will clarify it in the text.\n\n**“I would like to note a few related fields…[deep multiple clustering (DMC), conditional similarity (CS)]”**:\n\nWe thank the reviewer for pointing out these related problems, they are certainly relevant and we will discuss them fully in the updated manuscript. We will also conduct investigations to see if techniques from these fields can be brought into GCD. Similarly, we hope that our introduced Clevr-4 benchmark can also find use in DMC/CS research.\n\n**“The proposed method seems to be a straightforward extension of the SimGCD”**:\n\nWe thank the reviewer for the comment and we propose to clarify our methodological contributions as: “we present a novel algorithm for GCD, $\\mu$GCD, which extends the SimGCD method with a mean-teacher based approach”.\n"}, {"review_id": "f1byyQHoez", "reviewer": "Reviewer_aYTM", "summary": "This paper works on generalized category discovery, a setting that requires classifying unlabelled samples according to the taxonomy defined by the labeled set. This work identifies that a shortcut exists in previous benchmarks, and methods overlooking the classification metric defined by the labeled set could still perform well. It thus proposes the CLEVR-4 benchmark, in which the taxonomy/isolation of categories is separated as 'shape', 'texture' or 'color', or 'count'. Based on this benchmark, it evaluates the performance of previous methods in terms of pre-training/representation and pseudo-labeling and proposes $\\mu$GCD. The technique shows better performance in both the proposed benchmark and prior benchmarks.", "questions": "One concern on methodology is whether $\\mu$GCD could be listed as a major contribution. It is somehow technically incremental to me and is more like SimGCD Plus (momentum teacher & weak-strong aug) rather than a new method. I understand it suits the proposed benchmark better, and I feel okay with the overall contribution of this paper, the only concern is whether could it be listed as a side-by-side contribution with the benchmark.", "limitations": "have been discussed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "*Originality & significance.* The proposed benchmark is absolutely novel and original. It identifies a key issue in current GCD benchmarks: unsupervised classification algorithms are generally specialized for semantics, and the task could be easily suited without relying on the labeled set. The benchmark is in line with a recent trend of aligning machine outputs with human requirements: what is the definition needed by humans? I am happy to see this benchmark and the revisiting of previous works that it enables.\n\n*Quality.* The benchmark is accompanied by careful evaluations of different aspects of previous methods (representation & classification), and analytical figures (fig. 3/6/7) that help understand the source of improvement. Ablation study and error bars are also provided, which is good and makes the contribution sound.\n\n*Clarity* The presentation is natural and fluent, and I did not find issues in delivery.\n\n*Reproducibility.* Technical details are well provided to facilitate reproduction. Hope to see the dataset soon.", "weaknesses": "I have to say that my major concerns are anticipated and well discussed in the appendix. There is one small concern on methodology and I list it in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review of our work!\n\nWe first clarify that we will publicly release all code and the Clevr-4 dataset upon paper acceptance. Secondly, regarding the phrasing of our method (**“one concern is whether $\\mu$GCD could be listed as a major contribution”**), we propose to clarify our methodological contribution as: “we present a novel algorithm for GCD, $\\mu$GCD, which extends the SimGCD method with a mean-teacher based approach”. We are happy to answer any further concerns should they arise."}]

Author rebuttal:
**Global response**:

We sincerely thank all reviewers for the time they spent reviewing our manuscript, and for their thoughtful feedback. We are encouraged that the reviewers found: the ideas *‘unique’* and *'overlooked by previous methods'* (V8PZ, s1RT); our proposed dataset *‘interesting’*, *'valuable'* and *'absolutely novel and original'* (dGiY, S1RT, aYTM); and overall our analysis to be *'comprehensive'* and our contributions *'sound'* and *'convincing'* (s1RT, V8PZ, aYTM). 

We have provided detailed responses to individual reviewers below, and have provided additional experiments suggested by the reviewers in the Author Response PDF. We also clarify that we plan to publicly release all code and the Clevr-4 dataset.

Meta-review: The paper offers a novel approach to Generalized Class Discovery (GCD) by introducing a new dataset, Clevr-4, that challenges existing benchmarks. It proposes a simple yet effective method but leaves some questions about its mechanisms. The paper is well-executed, featuring thorough evaluations and clarity. However, concerns exist about the method's originality and a lack of comprehensive experimental discussion. Overall, the paper is a valuable contribution to the field, filling a gap in existing benchmarks.

The reviewers are all satisfied with the authors' response and recommend accept. AC reads the paper (roughly), the review, and the discussion, and recommends accept (poster).
Decision: Accept (poster)