SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given these findings, we seek to improve the quality of the pseudo-labels for 'New' categories. Specifically, we draw inspiration from the mean-teacher setup for semi-supervised learning [17], which has been adapted with minor changes in many self-supervised frameworks [14, 18, 24]. Here, a 'student' network is supervised by class pseudo-labels generated by a 'teacher'. The teacher is an identical architecture with parameters updated with the Exponential Moving Average (EMA) of the student. The intuition is that the slowly updated teacher is more robust to the noisy supervision from pseudo-labels, which in turn improves the quality of the pseudo-labels themselves. Also, rather than jointly optimizing both SimGCD losses, we first train the backbone only with the GCD baseline loss, before finetuning with the classification head and loss.
These changes, together with careful consideration of the data augmentations, give rise to our proposed  $\mu$ GCD (mean-GCD) algorithm, which we fully describe next in section 5. Here, we note the improvements that this algorithm brings in Clevr-4 on the bottom line of table 4. Overall,  $\mu$ GCD outperforms SimGCD on three of the four Clevr-4 taxonomies, and further outperforms SimGCD by nearly  $5\%$  on average across all splits.  $\mu$ GCD underperforms SimGCD on the shape split of Clevr-4 and we analyse this failure case in the appendix B.2.
5 The  $\mu$  GCD algorithm
In this section, we detail a simple but strong method for GCD,  $\mu$ GCD, already motivated in section 4 and illustrated in fig. 2. In a first phase, the algorithm proceeds in the same way as the GCD baseline [7], learning the representation. Next, we append a classification head and fine-tune the model with a 'mean teacher' setup [17], similarly to SimGCD but yielding more robust pseudo-labels.
Concretely, we construct models,  $f_{\theta}$ , as the composition of a feature extractor,  $\Phi$ , and a classification head,  $g$ .  $\Phi$  is first trained with the representation learning framework from [7] as described above, and the composed model gives  $f = g \circ \Phi$  with values in  $\mathbb{R}^k$ , where  $k$  is the total number of categories in the dataset. Next, we sample a batch of images,  $\mathcal{B}$ , and generate two random augmentations of
Figure 2: Our  $\mu$ GCD' method. We begin with representation learning from the GCD baseline, followed by finetuning in a mean-teacher style setup. Here, a 'teacher' provides supervision for a 'student' network, and maintains parameters as the exponential moving average (EMA) of the student.
every instance. We pass one view through the student network  $f_{\theta_S}$ , and the other through the teacher network  $f_{\theta_T}$ , where  $\theta_S$  and  $\theta_T$  are the network parameters of the student and teacher, respectively. We compute the cross-entropy loss between the (soft) teacher pseudo-labels and student predictions:
$$
\mathcal {L} ^ {u} \left(\theta_ {S}; \mathcal {B}\right) = - \frac {1}{| \mathcal {B} |} \sum_ {\boldsymbol {x} \in \mathcal {B}} \left\langle \boldsymbol {p} _ {T} (\boldsymbol {x}), \log \left(\boldsymbol {p} _ {S} (\boldsymbol {x})\right) \right\rangle , \quad \boldsymbol {p} _ {*} (\boldsymbol {x}) = \operatorname {s o f t m a x} \left(f _ {\theta_ {*}} (\boldsymbol {x}); \tau_ {*}\right), \tag {1}
$$
where  $\pmb{p}_{*}(\pmb{x}) \in [0,1]^{k}$  are the softmax outputs of the student and teacher networks, scaled with temperature  $\tau_{*}$ . We further use labelled instances in the batch with a supervised cross-entropy component as:
$$
\mathcal {L} ^ {s} \left(\theta_ {S}; \mathcal {B} _ {\mathcal {L}}\right) = - \frac {1}{| \mathcal {B} _ {\mathcal {L}} |} \sum_ {i \in \mathcal {B} _ {\mathcal {L}}} \langle \boldsymbol {y} (\boldsymbol {x}), \log \left(\boldsymbol {p} _ {S} (\boldsymbol {x})\right) \rangle , \tag {2}
$$
where  $\mathcal{B}_{\mathcal{L}} \in \mathcal{B}$  is the labelled subset of the batch and  $\pmb{y}(\pmb{x}) \in \{0,1\}^{k}$  is the one-hot class label of the example  $\pmb{x}$ . Finally, we add a mean-entropy maximization regularizer from [26] to encourage pseudo-labels for all categories:
$$
\mathcal {L} ^ {r} \left(\theta_ {S}\right) = - \langle \bar {\boldsymbol {p}} _ {S}, \log (\bar {\boldsymbol {p}} _ {S}) \rangle , \quad \bar {\boldsymbol {p}} _ {S} = \frac {1}{| \mathcal {B} |} \sum_ {\boldsymbol {x} \in \mathcal {B}} \boldsymbol {p} _ {S} (\boldsymbol {x}). \tag {3}
$$
The student is trained with respect to the following total loss, given hyper-parameters  $\lambda_{1}$  and  $\lambda_{2}$ :  $\mathcal{L}(\theta_S; \mathcal{B}) = (1 - \lambda_1)\mathcal{L}^u(\theta_S; \mathcal{B}) + \lambda_1\mathcal{L}^s(\theta_S; \mathcal{B}_{\mathcal{L}}) + \lambda_2\mathcal{L}^r(\theta_S)$ . The teacher parameters are updated as the moving average  $\theta_T = \omega(t)\theta_T + (1 - \omega(t))\theta_S$ , where  $\omega(t)$  is a time-varying momentum.
Augmentations. While often regarded as an 'implementation detail', an important component of our method is the careful consideration of augmentations used in the computation of  $\mathcal{L}^u$ . Specifically, on the SSB, we pass different views of the same instance to the student and teacher networks. We generate a strong augmentation which is passed to the student network, and a weak augmentation which is passed to the teacher, similarly to [49]. The intuition is that, while contrastive learning benefits from strong data augmentations [9, 14], we wish the teacher network's predictions to be as stable as possible. Meanwhile, on Clevr-4, misaligned data augmentations — e.g., aggressive cropping for count, or color jitter for color — substantially degrade performance (see appendix B.5).
Architecture. We adopt a 'cosine classifier' as  $g$ , which was introduced in [62] and leverages  $L^2$ -normalized weight vectors and feature representations. While it has been broadly adopted for many tasks [8, 9, 19, 25, 43], we demonstrate why this component helps in section 7.1. We find that normalized vectors are important to avoid collapse of the predictions to the labelled categories.
6 Results on real data
Datasets. We compare  $\mu$ GCD against prior work on the standard Semantic Shift Benchmark (SSB) suite [20]. The SSB comprises three fine-grained evaluations: CUB [64], Stanford Cars [65] and
Table 5: Category discovery accuracy (ACC) on the Semantic Shift Benchmark [20]. We report results from prior work using DINO initialization [14], and reimplement GCD baselines and SimGCD with DINOv2 pre-training [60] (noted with *).
Pre-training	CUB	Stanford Cars	Aircraft	Average
All	Old	New	All	Old	New	All	Old	New	All
k-means [57]	DINO	34.3	38.9	32.1	12.8	10.6	13.8	16.0	14.4	16.8	21.1
RankStats+ [47]	DINO	33.3	51.6	24.2	28.3	61.8	12.1	26.9	36.4	22.2	29.5
UNO+ [43]	DINO	35.1	49.0	28.1	35.5	70.5	18.6	40.3	56.4	32.2	37.0
ORCA [8]	DINO	35.3	45.6	30.2	23.5	50.1	10.7	22.0	31.8	17.1	26.9
GCD [7]	DINO	51.3	56.6	48.7	39.0	57.6	29.9	45.0	41.1	46.9	45.1
XCon [63]	DINO	52.1	54.3	51.0	40.5	58.8	31.7	47.7	44.4	49.4	46.8
OpenCon [52]	DINO	54.7	63.8	54.7	49.1	78.6	32.7	-	-	-	-
MIB [51]	DINO	62.7	75.7	56.2	43.1	66.9	31.6	-	-	-	-
PromptCAL [50]	DINO	62.9	64.4	62.1	50.2	70.1	40.6	52.2	52.2	52.3	55.1
SimGCD [19]	DINO	60.3	65.6	57.7	53.8	71.9	45.0	54.2	59.1	51.8	56.1
μGCD (Ours)	DINO	65.7	68.0	64.6	56.5	68.1	50.9	53.8	55.4	53.0	58.7
k-means*	DINOv2	67.6	60.6	71.1	29.4	24.5	31.8	18.9	16.9	19.9	38.6
GCD*	DINOv2	71.9	71.2	72.3	65.7	67.8	64.7	55.4	47.9	59.2	64.3
SimGCD*	DINOv2	71.5	78.1	68.3	71.5	81.9	66.6	63.9	69.9	60.9	69.0
μGCD (Ours)	DINOv2	74.0	75.9	73.1	76.1	91.0	68.9	66.3	68.7	65.1	72.1
FGVC-Aircraft [55]. Though the SSB datasets do not contain independent clusterings of the same images (as in Clevr-4) the evaluations do have well-defined taxonomies — i.e. birds, cars and aircrafts. Furthermore, the SSB contains curated novel class splits which control for semantic distance with the labelled set. We find that coarse-grained GCD benchmarks do not specify clear taxonomies in the labelled set, and we include a long-tailed evaluation on Herbarium19 [66] in appendix C.3.
Model initialization and compared methods. The SSB contains fine-grained, object-centric datasets, which have been shown to benefit from greater shape bias [67]. Prior GCD methods [7, 52, 63] initialize with DINO [14] pretraining, which we show in table 2 had the strongest shape bias among self-supervised models. However, the recent DINOv2 [60] demonstrates a substantially greater shape bias. As such, we train our model both with DINO and DINOv2 initialization, further re-implementing GCD baselines [7, 68] and SimGCD [19] with DINOv2 for comparison.
Table 6: Ablations. We find that a proper initialization, momentum decay schedule, and augmentation strategy are critical to strong performance.
CUB
All	Old	New
μGCD (Ours)	65.7	68.0	64.6
(1) W/o GCD init.	61.7	66.2	59.6
(2) W/o stronger student augmentation	58.1	72.5	50.9
(3) With ωt := 1	1.6	1.1	1.8
(4) With ωt := 0.0	62.7	66.4	60.9
(5) With ωt := 0.7	64.1	65.1	63.6
(6) W/o cosine classifier	54.9	64.2	50.3
(7) W/o ME-Max regularizer	42.0	41.8	42.1
Implementation details. We implement all models in PyTorch [69] on a single NVIDIA P40 or M40. Most models are trained with an initial learning rate of 0.1 which is decayed with a cosine annealed schedule [70]. For our EMA schedule, we ramp it up throughout training with a cosine function [18]:  $\omega(t) = \omega_T - (1 - \omega_{base})(\cos(\frac{\pi t}{T}) + 1)/2$ . Here  $t$  is the current epoch and  $T$  is the total number of epochs. Differently, however, to most self-supervised learning literature [18], we found a much lower initial decay to be beneficial; we ramp up the decay from  $\omega_{base} = 0.7$  to  $\omega_T = 0.999$  during training. Further implementation details can be found in appendix E.
6.1 Discussion.
In table 5, we find that  $\mu$ GCD outperforms the existing state-of-the-art, SimGCD [19], by over  $2\%$  on average across all SSB evaluations when using DINO initialization. When using the stronger DINOv2 backbone, we find that the performance of the simple  $k$ -means baseline nearly doubles in accuracy, substantiating our choice of shape-biased initialization on this object-centric evaluation. The gap between the GCD baseline [7] and the SimGCD state-of-the-art [19] is also reduced from over  $10\%$  to under  $5\%$  on average. Nonetheless, our method outperforms SimGCD by over  $3\%$  on average, as well as on each dataset individually, setting a new state-of-the-art.
Figure 3: Left: Norms of weight vectors in GCD classifiers, with and without regularizations. Right: Prediction histogram of a classifier without weight norm and max-entropy regularization.
Ablations. We ablate our main design choices in table 6. L(1) shows the importance of pre-training with the GCD baseline loss [7] (though we find in section 4 that jointly training this loss with the classifier, as in SimGCD [19], is difficult). L(2) further demonstrates that stronger augmentation for the student network is critical, with a  $7\%$  drop in CUB performance without it. L(3)-(5) highlight the importance of a carefully designed EMA schedule, our use of a time-varying decay outperforms constant decay values. This is intuitive as early on in training, with a randomly initialized classification head, we wish for the teacher to be updated quickly. Later on in training, slow teacher updates mitigate the effect of noisy pseudo-labels within any given batch. Furthermore, in L(6)-(7), we validate the importance of entropy regularization and cosine classifiers in category discovery. In section 7.1, we provide evidence as to why these commonly used components [8, 19, 43] are necessary, and also discuss the design of the student augmentation.
7 Further Analysis
In this section we further examine the effects of architectural choices in  $\mu$ GCD and other category discovery methods. Section 7.1 seeks to understand the performance gains yielded by cosine classifiers, and section 7.2 visualizes the feature spaces learned by different GCD methods.
7.1 Understanding cosine classifiers in category discovery
Cosine classifiers with entropy regularization have been widely adopted in recognition settings with limited supervision [14, 25], including in category discovery [19, 43]. In fig. 3, we provide justifications for this by inspecting the norms of the learned vectors in the final classification layer.
Specifically, consider a classifier (without a bias term) as  $g = \mathbf{W} \in \mathbb{R}^{d \times k}$ , containing  $k$  vectors of  $d$  dimension, one for each output category. In fig. 3, we plot the magnitude of each of these vectors trained with different constraints on CUB [64] (one of the datasets in the SSB [20]). Note that the classifier is constructed such that the first 100 vectors correspond to the 'Old' classes, and are trained with ground truth labels. In our full method, with normalized classifiers, the norm of all vectors is enforced to be the unit norm (blue dashed line). If we remove this constraint (solid orange line), we can see that the norms of vectors which are not supervised by ground truth labels (indices 101-200) fall substantially. Then, if we further remove the entropy regularization term (solid green line), the magnitudes of the 'Old' class vectors (indices 1-200) increase dramatically.
This becomes an issue at inference time, with per-class logits computed as:
$$
l _ {m} = \left\langle \mathbf {w} _ {m}, \Phi \right\rangle = \left| \mathbf {w} _ {m} \right| \left| \Phi \right| \cos (\alpha) \quad \forall m \in \{1 \dots k \}
$$
with the class prediction returned as  $\arg \max l_{m}$ . In other words, we show that without appropriate regularisation, our GCD models trivially reduce the weight norm of 'New' class vectors  $(|\mathbf{w}_m| \forall m > 100)$ , leaving all images to be assigned to one of the 'Old' classes. The effects of this are visualized in the right panel of fig. 3, which plots the histogram of class predictions for an unregularized GCD classifier. We can see that exactly zero examples are predicted to 'New' classes.
Figure 4: PCA [71] of features from the GCD baseline [7], SimGCD [19] and  $\mu$ GCD on the count split of Clevr-4. While the baseline learns elliptical clusters for each category, SimGCD and  $\mu$ GCD project images onto a one-dimensional object in feature space, which be considered as a 'semantic axis' along which the category changes. Clustering accuracy is reported for 'All'/'Old'/'New' classes.
We further highlight that this effect is obfuscated by the evaluation process, which reports non-zero accuracies for 'New' classes through the Hungarian assignment operation.
7.2 PCA Visualization.
Finally, we perform analysis on the count split of Clevr-4. Uniquely amongst the four taxonomies, the count categories have a clear order. In fig. 4, we plot the first two principal components [71] of the normalized features of the GCD baseline [7], SimGCD [19] and  $\mu$  GCD. It is clear that all feature spaces learn a clear 'number sense' [72] with image features placed in order of increasing object count. Strikingly, this sense of numerosity is present even beyond the supervised categories (count greater than 5) as a byproduct of a simple recognition task. Furthermore, while the baseline learns elliptical clusters for each category, SimGCD and  $\mu$  GCD project all images onto a one-dimensional object in feature space. This object can be considered as a 'semantic axis': a low-dimensional manifold in feature space,  $\mathbb{R} \in \mathbb{R}^d$ , along which the category label changes.
8 Conclusion and final remarks
Limitations and broader impacts. In this paper we presented a synthetic dataset for category discovery. While synthetic data allows precise manipulation of the images, and hence more controlled study of the GCD problem, findings on synthetic data do not always transfer directly to real-world images. For instance, the failure case of  $\mu$ GCD on the shape split of Clevr-4 (see section 4.3) occurs due to some classification vectors being unused (see section 7.1). This ceases to be an issue on real-world data with hundreds of categories (see table 5). In appendix F, we fully discuss the difficulties in developing a dataset like Clevr-4 with real-world data. Furthermore, while GCD has many real-world applications, like any form of unsupervised or partially-supervised machine learning, it can be unreliable and must be applied with caution in practise.
Remarks on Clevr-4. We note that Clevr-4 can find broader applicability in related machine learning fields. As examples, the dataset can be used for disentanglement research (see appendix F) and as a simple probing set for biases in representation learning. For instance, we find in section 4.1 that most of the ImageNet trained models are biased towards shape rather than texture, which is in contrast to popular findings from Geirhos et al. [13]. Furthermore, larger models are often explicitly proposed as 'all-purpose' features for 'any task' [60]; here we find simple tasks (e.g., color or count recognition) where initialization with such moods hurts performance compared to training from scratch. Note that practical problems — e.g., vehicle re-identification [73] or crowd counting [74] — may require understanding of such aspects of the image.
Conclusion. In this paper we have proposed a new dataset, Clevr-4, and used it to investigate the problem of Generalized Category Discovery (GCD). This included probing the limitations of unsupervised representations for the task, as well as for identifying weaknesses in existing GCD methods. We further leveraged our findings, together with consistent trends in related literature, to propose a simple but performant algorithm,  $\mu$ GCD. We find that  $\mu$ GCD not only provides gains on Clevr-4, but further sets a new state-of-the-art on the standard Semantic Shift Benchmark.
S. Vaze is supported by a Facebook Scholarship. A. Vedaldi is supported by ERC-CoG UNION 101001212. A. Zisserman is supported by VisualAI EP/T028572/1.
[88] Liangrui Ren, Guoxian Yu, Jun Wang, Lei Liu, Carlotta Domeniconi, and Xiangliang Zhang. A diversified attention model for interpretable multiple clusterings. TKDE, 2022.
[89] Ruchika Chavhan, Henry Gouk, Jan Stuehmer, Calum Heggan, Mehrdad Yaghoobi, and Timothy Hospedales. Amortised invariance learning for contrastive self-supervision. ICLR, 2023.
[90] Linus Ericsson, Henry Gouk, and Timothy M Hospedales. Why do self-supervised models transfer? investigating the impact of invariance on downstream tasks. ArXiv e-prints, 2021.
[91] Ravid Shwartz-Ziv, Randall Balestriero, and Yann LeCun. What do we maximize in self-supervised learning?, 2022.
[92] Randall Balestriero, Ishan Misra, and Yann LeCun. A data-augmentation is worth a thousand samples: Analytical moments and sampling-free training. In NeurIPS, 2022.
[93] Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving self-supervised learning by characterizing idealized representations. NeurIPS, 2022.
[94] Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, QianYing Wang, and Ping Chen. Generalized category discovery with decoupled prototypical network. AAAI, 2023.
The appendices are summarized in the contents table below. We highlight: appendices A.1 and A.2 for details on Clevr-4; appendix B.4 for further analysis of pre-trained representations on Clevr-4; and appendix C.3 for a long-tail evaluation of  $\mu$ GCD on the Herbarium19 dataset [66].
A Clevr-4 16
A.1 Clevr-4 generation 16
A.2 Clevr-4 details 17
A.3Clevr-4 examples 17
B Analysis of results 18
B.1 Clevr-4 error bars 18
B.2 shape failure case 19
B.3 Semi-supervised  $k$  -means with pre-trained backbones 20
B.4 Clustering with sub-spaces of pre-trained features 20
B.5 Design of data augmentation and mis-aligned augmentations 22
B.6 Effect of  $\lambda_{1}$  23
C Additional Experiments 23
C.1 Results with estimated number of classes 23
C.2 Results with varying proportion of labelled examples 24
C.3 Results on Herbarium19 24
D Description of baselines and  $\mu$ GCD algorithms 24
E Further Implementation Details 26
F Connections to related work 28
F.1 Clevr-4: connections to real-world and disentanglement datasets 28
F.2  $\mu$  GCD method. 28
A Clevr-4
A.1 Clevr-4 generation
We build Clevr-4 using Blender [54], a free 3D rendering software with a Python API. Following the CLEVR dataset [12], our images are constituted of multiple rendered objects in a static scene. Each object is defined by three 'semantic' attributes (texture, shape and color), and is further defined by its size, pose and position in the scene. We consider the first three attributes as 'semantic' as they are categorical variables which can neatly define image 'classes'. Meanwhile, we designate the size, pose and position attributes as 'nuisance' factors which are not related to the image category.
CLEVR Limitations. CLEVR is first limited – for the purposes of category discovery – as it has only two textures ('rubber' and 'metal') and three shapes ('cube', 'sphere' and 'cylinder'). For category discovery, we wish to have more categories, both to increase the difficulty of the task, and to ensure a sufficient number of classes in the 'Old' and 'New' subsets. Furthermore, we wish to have the same number of categories in each split; otherwise, in principal, an unsupervised algorithm may be able to distinguish the taxonomy simply from the number of categories present.
Figure 5: Normalized mutual information between the four taxonomies in Clevr-4. All taxonomies have roughly no mutual information between them (they are statistically independent).
Expanding the taxonomies. To increase the number of categories in each taxonomy, we introduce new textures, colours and shapes to the dataset, resulting in 10 categories for each taxonomy. We create most of the 8 new textures by wrapping a black-and-white JPEG around the surface of the object, each of which have their own design (e.g., 'chessboard' or 'circles'). Given an 'alpha' for the opaqueness of this wrapping, these textures can be distinguished independently of the underlying color. We further leverage pre-fabricated meshes packaged with Blender to introduce 7 new shapes to the dataset, along with 2 new colors for the objects. The new shapes and colors were selected to be clearly distinguishable from each other. Full definitions of the taxonomies are given in appendix A.2.
Image sampling process. For a given image, we first independently sample object texture, shape and color. We then randomly sample how many objects should be in the image (i.e., object count) and place this many objects in the scene. Each object has its own randomly sampled size (which is taken to be one of three discrete values), position and relative pose. Thus, differently to CLEVR, all objects in the image have the same texture, shape and color. This allows these three attributes, together with count, to define independent taxonomies within the data.
A.2 Clevr-4 details
We describe the categories in each of the four taxonomies in Clevr-4 below. All taxonomies have 10 categories, five of which are used in the labeled set and shown in bold. Image exemplars of all categories are given in figs. 7 and 8.
- Texture: rubber, metal, checkered, emojis, wave, brick, star, circles, zigzag, chessboard
- Shape: cube, sphere, monkey, cone, torus, star, teapot, diamond, gear, cylinder
Color: gray, red, blue, green, brown, purple, cyan, yellow, pink, orange
- Count: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
fig. 6 plots the frequency of all categories in the taxonomies, while fig. 5 shows the mutual information between the four taxonomies. We find that all taxonomies, except for shape, are roughly balanced, and the four taxonomies have approximately no mutual information between them – realizing our desire of them being statistically independent.
A.3 Clevr-4 examples
We give examples of each of the four taxonomies in Clevr-4 in figs. 7 and 8.
Figure 6: Category frequency plots for each taxonomy in Clevr-4. All taxonomies are roughly balanced, except for shape. shape shows minor imbalance due to greater difficulty in placing many objects of some shapes (e.g., 'star' and 'monkey') in scenes.
Figure 7: Examples of each category from the texture and shape taxonomies of Clevr-4.