SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remarks: We first highlight the different ways in which the labels from  $\mathcal{D}_{\mathcal{L}}$  are used between the three methods. Specifically, the GCD baseline [7] only uses the labels in a feature-space supervised contrastive loss. However, in addition to this, SimGCD [19] and  $\mu$ GCD also use the labels in a standard cross-entropy loss in order to train part of a linear classifier,  $g$ .
We further note the high level similarity between SimGCD and  $\mu$  GCD, in that both train parametric classifiers with a pseudo-label loss. While SimGCD uses different views passed to the same model to generate pseudo-labels for each other (similarly to SWaV [9]),  $\mu$  GCD uses pseudo-labels from a 'teacher' network to train a 'student' (similarly to mean-teachers [17]).
This is in keeping with trends in related fields, which find that there exists a small kernel of methodologies — e.g., mean-teachers [17], cosine classifiers [62], entropy regularization [26] — which are robust across many tasks [14, 22, 25], but that finding a strong recipe for a specific problem is critical. We find this to be true in supervised classification [58, 76, 77], self-supervised learning [14, 21], and semi-supervised learning [17, 26, 49]. Our use of mean-teachers to provide classifier pseudo-labels, as well as careful choice of model initialization and data augmentation, yields a performant  $\mu$ GCD algorithm for category discovery.
E Further Implementation Details
When re-implementing prior work, we aim to follow the hyper-parameters of the GCD baseline [7] and SimGCD [19], and use the same settings for our method. We occasionally find that tuned hyper-parameters are beneficial in some settings, which we detail below.
Learning rates. We swept learning rates at factors of 10 for all methods and architectures. When training models from scratch (ResNet18 on Clevr-4) or when finetuning a DINO/DINOv2 model [14, 60] on the SSB, we found a learning rate of 0.1 to be optimal. When finetuning an MAE [31] or DINOv2 model on Clevr-4, we found it better to lower the learning rate to 0.01. All learning rates are decayed from their initial value by a factor of  $10^{-3}$  throughout training with a cosine schedule.
Loss hyper-parameters. For the tradeoff between the unsupervised and supervised components of the losses,  $\lambda_{1}$  is set to 0.35 for all methods. For the entropy regularization, we follow SimGCD and use  $\lambda_{2} = 1.0$  for FGVC-Aircraft and Stanford Cars, and  $\lambda_{2} = 2.0$  for all other datasets. We swept to find better settings for this term on Clevr-4, but did not find any setting to consistently improve results. We also train with  $L^2$  weight decay, set to  $10e^{-4}$  for all models.
Student and teacher temperatures. Following [14], we set the temperature of the student and teacher to  $\tau_{S} = 0.1$  and  $\tau_{T} = 0.04$  respectively, for both our method and SimGCD. This gives the teacher 'sharper' (more confident) predictions than the student. We further follow the teacher-temperature warmup schedule from [14], also used in SimGCD, where the teacher temperature is decreased from 0.07 to 0.04 in the first 30 epochs of training. On Herbarium19 [66] (which has many more categories than the other evaluations, see appendix C.3), we use a teacher temperature of  $2\times 10^{-3}$  (warmed up from  $3.5\times 10^{-3}$  over 10 epochs).
Teacher Momentum Schedule. In  $\mu$ GCD, at each iteration, the teacher's parameters are linearly interpolated between the teacher's current parameters and the student's, with the interpolation ('decay' or 'momentum') changing over time following [18], as:  $\omega(t) = \omega_T - (1 - \omega_{base})(\cos(\frac{\pi t}{T}) + 1)/2$ .
Here  $T$  is the total number of epochs and  $t$  is the current epoch. We use  $\omega_{T} = 0.999 \approx 1$  and  $\omega_{base} = 0.7$ . We note for clarity that, though the momentum parameter is dictated by the epoch number, the teacher update happens at each gradient step.
Augmentations. On Clevr-4 we use an augmentation comprising of RandomHorizontalFlip and RandomRotation. On the SSB [20], we use RandomHorizontalFlip and RandomCrop. We use these augmentations for all methods, and for  $\mu$ GCD use these augmentations to pass views to the 'teacher'. An important part of our method on the SSB is to design strong augmentations to pass to the student. Our 'strong augmentation' adds aggressive RandomResizeCrop, as well as solarization and Gaussian blurring [21] (see appendix B.5 for details). On Clevr-4, due to the relatively simple nature of the images, strong augmentations can destroy the semantic image content; for instance color jitter and aggressive cropping degrade performance on color and count respectively. We find it helpful to pass Cutout [78] to the teacher on the color taxonomy, and texture benefits from the strong augmentation defined above.
Training time. Following the original implementations, we train all SimGCD [19] and GCD baseline [7] models for 200 epochs, which we find sufficient for the losses (and validation performance) to plateau. For our method, we randomly initialize a classifier on a model which has been trained with the GCD baseline loss, and further finetune for another 100 epochs. On our hardware (either an NVIDIA P40 or M40) we found training to take roughly 15 hours for SSB datasets, and around 4 hours for a Clevr-4 experiment.
Early stopping. We note that GCD is a transductive setting, or a clustering problem, where models are trained (in an unsupervised fashion) on the data used for evaluation,  $\mathcal{D}_{\mathcal{U}}$ . As such, an important criterion is which metric to use to select the best model. SimGCD [19] and the GCD baseline [7] use the performance on a validation set of images from the labeled categories. While this is a reasonable choice for the baseline, we found it can lead to underestimated performance for SimGCD on some datasets. For SimGCD, we instead found it better to simply take the model at the end of training. For  $\mu$ GCD, we instead propose to choose the model with the minimum unsupervised loss on the unlabeled set.
Other details: When finetuning pre-trained transformer models – DINO [14], DINOv2 [60] or MAE [31] – we finetune the last transformer block of the model. For Clevr-4, when training a ResNet18, we finetune the whole model. Finally, for the  $\mu$ GCD failure case of shape, we suggest in appendix B.2 that  $\mu$ GCD can get stuck in local optima if its initialization is already very strong. As such, in this case, we initialize the linear head with  $k$ -means centroids, reduce the learning rate and teacher temperature to 0.01, and set  $\omega_{base}$  to 0.9.
F Connections to related work
F.1 Clevr-4: connections to real-world and disentanglement datasets
Datasets with different granularities. When multiple taxonomies are defined in existing datasets, they are most often specified only at different granularites, for instance in CIFAR100 [53], FGVC-Aircraft [55] and iNaturalist [79]. While recognition at different granularites is related to our task – and was explored in [80] – the constituent taxonomies are not statistically independent, as the Clevr-4 splits are. We note that, given the number of categories in each taxonomy, an unsupervised model could in principal solve the clustering problem at the different granularities.
CUB-200-2011 [64]. Fei et al. [63] discuss the existence of alternate, but valid, clusterings of images from fine-grained datasets like CUB [64] - e.g., based on pose or background. We note that the CUB 'Birds' dataset presents an opportunity for constructing an interesting dataset for category discovery. Each image in CUB is labelled for presence (or absence) of each of 312 attributes, where these attributes come from different attribute types. Each attribute type (e.g., 'bill shape', 'breast color') provides a different taxonomy with respect to which to cluster the data. However, we found these attribute annotations are too noisy to yield meaningful conclusions.
Disentanglement datasets. We suggest that Clevr-4 is also a useful benchmark for disentanglement research [34, 35]. This research field aims to learn models such that the ground-truth data generating factors (i.e., attributes of an object) are encoded in different subspaces of the image representation. The current CLEVR dataset [12] cannot be used easily for this, as its images contain multiple objects, each with different attributes. Instead, in Clevr-4, all objects share the same attributes, allowing each image to be fully parameterized by the object shape, texture, color and count. Furthermore, compared to synthetic datasets for disentanglement [38], Clevr-4 contains more categorical taxonomies, as well as more classes within those taxonomies.
Finally, we note that there exist other extensions of the CLEVR dataset [12], such as ClevrTex [81], Super-CLEVR [82] and CLEVR-X [83], which also add new textures and/or categories to the original datasets. However these datasets cannot be used for category discovery (or disentanglement) research as, unlike in Clevr-4, they contain scenes with objects of differing attributes. As such, each image cannot be parameterized with respect to the object attributes in a way which gives rise to clear taxonomies.
Other related fields The GCD task and the Clevr-4 dataset are related to a number of other machine learning sub-fields. Conditional Similarity research [84-86] aims to learn different embedding functions given different conditions. For instance, the GeneCIS benchmark [84] evaluates the ability of models to retrieve different images given a query and different conditioning text prompt. Meanwhile, the multiple clustering [87, 88] and self-supervised learning [89, 90] fields investigate the how different choices of data augmentation result in different clusterings of the data. The self-supervised field particularly aims to understand why these inductive biases result in different generalization properties [91-93].
We hope that Clevr-4 can be complementary to these works, and provide a test-bed for controlled experimentation of these research questions.
F.2  $\mu$  GCD method.
We note here that the idea of momentum encoders has been widely used in representation learning [18, 24, 50], semi-supervised learning [17, 25], or to update class prototypes in category discovery [52, 94]. We use a mean-teacher model end-to-end, for the backbone representation and the classification head. We highlight that, similar to a rich vein of literature in related fields [14, 21, 26, 58, 76, 77], our goal is to find a specific recipe for the GCD task.