SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sagar Vaze Andrea Vedaldi Andrew Zisserman
Visual Geometry Group
www.robots.ox.ac.uk/~vgg/data/clevr4/
In this paper we tackle the problem of Generalized Category Discovery (GCD). Specifically, given a dataset with labelled and unlabelled images, the task is to cluster all images in the unlabelled subset, whether or not they belong to the labelled categories. Our first contribution is to recognize that most existing GCD benchmarks only contain labels for a single clustering of the data, making it difficult to ascertain whether models are using the available labels to solve the GCD task, or simply solving an unsupervised clustering problem. As such, we present a synthetic dataset, named 'Clevr-4', for category discovery. Clevr-4 contains four equally valid partitions of the data, i.e. based on object shape, texture, color or count. To solve the task, models are required to extrapolate the taxonomy specified by the labelled set, rather than simply latching onto a single natural grouping of the data. We use this dataset to demonstrate the limitations of unsupervised clustering in the GCD setting, showing that even very strong unsupervised models fail on Clevr-4. We further use Clevr-4 to examine the weaknesses of existing GCD algorithms, and propose a new method which addresses these shortcomings, leveraging consistent findings from the representation learning literature to do so. Our simple solution, which is based on 'mean teachers' and termed  $\mu$ GCD, substantially outperforms implemented baselines on Clevr-4. Finally, when we transfer these findings to real data on the challenging Semantic Shift Benchmark (SSB), we find that  $\mu$ GCD outperforms all prior work, setting a new state-of-the-art.
Developing algorithms which can classify images within complex visual taxonomies, i.e. image recognition, remains a fundamental task in machine learning [1-3]. However, most models require these taxonomies to be pre-defined and fully specified, and are unable to construct them automatically from data. The ability to build a taxonomy is not only desirable in many applications, but is also considered a core aspect of human cognition [4-6]. The task of constructing a taxonomy is epitomized by the Generalized Category Discovery (GCD) problem [7, 8]: given a dataset of images which is labelled only in part, the goal is to label all remaining images, using categories that occur in the labelled subset, or by identifying new ones. For instance, in a supermarket, given only labels for 'spaghetti' and 'penne' pasta products, a model must understand the concept of 'pasta shape' well enough to generalize to 'macaroni' and 'fusilli'. It must not cluster new images based on, for instance, the color of the packaging, even though the latter also yields a valid, but different, taxonomy.
GCD is related to self-supervised learning [9] and unsupervised clustering [10], which can discover some meaningful taxonomies automatically [11]. However, these cannot solve the GCD problem, which requires recovering any of the different and incompatible taxonomies that apply to the same data. Instead, the key to GCD is in extrapolating a taxonomy which is only partially known. In this paper, our objective is to better understand the GCD problem and improve algorithms' performance.
Figure 1: What is the key difference between Generalized Category Discovery (GCD) and tasks like self-supervised learning or unsupervised clustering? GCD's key challenge is extrapolating the desired clustering of data given only a subset of possible category labels. We present a synthetic dataset, Clevr-4, which contains four possible clusterings of the same images, and hence can be used to isolate the GCD task. Above, one can cluster the data based on object count, shape or texture.
To this end, in section 3, we first introduce the Clevr-4 dataset. Clevr-4 is a synthetic dataset where each image is fully parameterized by a set of four attributes, and where each attribute defines an equally valid grouping of the data (see fig. 1). Clevr-4 extends the original CLEVR dataset [12] by introducing new shapes, colors and textures, as well as allowing different object counts to be present in the image. Using these four attributes, the same set of images can be clustered according to four statistically independent taxonomies. This feature sets it apart from most existing GCD benchmarks, which only contain sufficient annotations to evaluate a single clustering of the data.
Clevr-4 allows us to probe large pre-trained models for biases, i.e., for their preference to emphasize a particular aspect of images, such as color or texture, which influences which taxonomy can be learned. For instance, contrary to findings from Geirhos et al. [13], we find almost every large model exhibits a strong shape bias. Specifically, in section 4, we find unsupervised clustering – even with very strong representations like DINO [14] and CLIP [15] – fails on many splits of Clevr-4, despite CLEVR being considered a ‘toy’ problem in other contexts [16]. As a result, we find that different pre-trained models yield different performance traits across Clevr-4 when used as initialization for category discovery. We further use Clevr-4 to characterize the weaknesses of existing category discovery methods; namely, the harms of jointly training feature-space and classifier losses, as well as insufficiently robust pseudo-labelling strategies for ‘New’ classes.
Next, in section 5, we leverage our findings on Clevr-4 to develop a simple but performant method for GCD. Since category discovery has substantial overlap with self-supervised learning, we identify elements of these methods that are also beneficial for GCD. In particular, mean-teacher based algorithms [17] have been very effective in representation learning [14, 18], and we show that they can boost GCD performance as well. Here, a 'teacher' model provides supervision through pseudolabels, and is maintained as the moving average of the model being trained (the 'student'). The slowly updated teacher is less affected by the noisy pseudo-labels which it produces, allowing clean pseudolabels to be produced for new categories. Our proposed method,  $\mu \mathrm{GCD}$  ('mean-GCD'), extends the existing state-of-the-art [19], substantially outperforming it on Clevr-4. Finally, in section 6, we compare  $\mu \mathrm{GCD}$  against prior work on real images, by evaluating on the challenging Semantic Shift Benchmark (SSB) [20]. We substantially improve state-of-the-art on this evaluation.
In summary, we make the following key contributions: (i) We propose a new benchmark dataset, Clevr-4, for GCD. Clevr-4 contains four independent taxonomies and can be used to better study the category discovery problem. (ii) We use Clevr-4 to garner insights on the biases of large pre-trained models as well as the weaknesses of existing category discovery methods. We demonstrate that even very strong unsupervised models fail on this 'toy' benchmark. (iii) We present a novel method for GCD,  $\mu$ GCD, inspired by the mean-teacher algorithm. (iv) We show  $\mu$ GCD outperforms baselines on Clevr-4, and further sets a new state-of-the-art on the challenging Semantic Shift Benchmark.
Representation learning. The common goal of self-, semi- and unsupervised learning is to learn representations with minimal labelled data. A popular technique is contrastive learning [21, 22], which encourages representations of different augmentations of the same training sample to be similar. Contrastive methods are typically either based on: InfoNCE [23] (e.g., MoCo [24] and SimCLR [21]); or online pseudo-labelling (e.g., SWaV [9] and DINO [14]). Almost all contrastive learning methods now adopt a variant of these techniques [25-28]. Another important component in many pseudo-labelling based methods is 'mean-teachers' [17] (or momentum encoders [24]), in which a 'teacher' network providing pseudo-labels is maintained as the moving average of a 'student' model. Other learning methods include cross-stitch [29], context-prediction [30], and reconstruction [31]. In this work, we use mean-teachers to build a strong recipe for GCD.
Attribute learning. We propose a new synthetic dataset which contains multiple taxonomies based on various attributes. Attribute learning has a long history in computer vision, including real-world datasets such as the Visual Genome [32], with millions of attribute annotations, and VAW, with 600 attributes types [33]. Furthermore, the disentanglement literature [34-36] often uses synthetic attribute datasets for investigation [37, 38]. We find it necessary to develop a new dataset, Clevr4, for category discovery as real-world datasets have either: noisy/incomplete attributes for each image [32, 39]; or contain sensitive information (e.g. contain faces) [40]. We find existing synthetic datasets unsuitable as they do not have enough categorical attributes which represent 'semantic' factors, with attributes often describing continuous 'nuisance' factors such as object location or camera pose [37, 38, 41].
Category Discovery. Novel Category Discovery (NCD) was initially formalized in [42]. It differs from GCD as the unlabelled images are known to be drawn from a disjoint set of categories to the labelled ones [43-47]. This is different from unsupervised clustering [10, 48], which clusters unlabelled data without reference to labels at all. It is also distinct from semi-supervised learning [17, 26, 49], where unlabelled images come from the same set of categories as the labelled data. GCD [7, 8] was recently proposed as a challenging task in which assumptions about the classes in the unlabelled data are largely removed: images in the unlabelled data may belong to the labelled classes or to new ones [19, 50-52]. We particularly highlight concurrent work in SimGCD [19], which reports the best current performance on standard GCD benchmarks. Our method differs from SimGCD by the adoption of a mean-teacher [17] to provide more stable pseudo-labels training, and by careful consideration of model initialization and data augmentations. [52] also adopt a momentum-encoder, though only for a set of class prototypes rather than in a mean-teacher setup.
3 Clevr-4: a synthetic dataset for generalized category discovery
Generalized Category Discovery (GCD). GCD [7] is the task of fully labelling a dataset which is only partially labelled, with labels available only for a subset of the categories. Formally, we are presented with a dataset  $\mathcal{D}$ , containing both labelled and unlabelled subsets,  $\mathcal{D}_{\mathcal{L}} = \{(x_{i},y_{i})\}_{i = 1}^{N_{l}} \in \mathcal{X} \times \mathcal{Y}_{\mathcal{L}}$  and  $\mathcal{D}_{\mathcal{U}} = \{(x_{i},y_{i})\}_{i = 1}^{N_{u}} \in \mathcal{X} \times \mathcal{Y}_{\mathcal{U}}$ . The model is then tasked with categorizing all instances in  $\mathcal{D}_{\mathcal{U}}$ . The unlabelled data contains instances of categories which do not appear in the labelled set, meaning that  $\mathcal{Y}_{\mathcal{L}} \subset \mathcal{Y}_{\mathcal{U}}$ . Here, we assume knowledge of the number of categories in the unlabelled set, i.e.  $k = |\mathcal{Y}_{\mathcal{U}}|$ . Though estimating  $k$  is an interesting problem, methods for tackling it are typically independent of the downstream recognition algorithm [7, 42]. GCD is related to Novel Category Discovery (NCD) [42], but the latter makes the not-so-realistic assumption that  $\mathcal{Y}_{\mathcal{L}} \cap \mathcal{Y}_{\mathcal{U}} = \emptyset$ .
The category discovery challenge. The key to category discovery (generalized or not) is to use the labelled subset of the data to extrapolate a taxonomy and discover novel categories in unlabelled images. This task of extrapolating a taxonomy sets category discovery apart from related problems. For instance, unsupervised clustering [10] aims to find the single most natural grouping of unlabelled images given only weak inductive biases (e.g., invariance to specific data augmentations), but permits limited control on which taxonomy is discovered. Meanwhile, semi-supervised learning [26] assumes supervision for all categories in the taxonomy, which therefore must be known, in full, a-priori.
A problem with many current benchmarks for category discovery is that there is no clear taxonomy underlying the object categories (e.g., CIFAR [53]) and, when there is, it is often ill-posed to understand it given only a few classes (e.g., ImageNet-100 [42]). Furthermore, in practise, there
are likely to be many taxonomies of interest. However, few datasets contain sufficiently complete annotations to evaluate multiple possible groupings of the same data. This makes it difficult to ascertain whether a model is extrapolating information from the labelled set (category discovery) or just finding its own most natural grouping of the unlabelled data (unsupervised clustering).
Clevr-4. In order to better study this problem, we introduce Clevr-4, a synthetic benchmark which contains four equally valid groupings of the data. Clevr-4 extends the CLEVR dataset [12], using Blender [54] to render images of multiple objects and place them in a static scene. This is well suited for category discovery,
Table 1: Clevr-4 statistics for the different splits of the dataset. Note that the same data must be classified along independent taxonomies in the different splits.
Texture	Color	Shape	Count
Examples	{metal, rubber}	{red, blue}	{torus, cube}	{1,2}
|Yc|	5	5	5	5
|Yu|	10	10	10	10
|Dc|	2.1K	2.3K	2.1K	2.1K
|Du|	6.3K	6.1K	6.4K	6.3K
|Dc| + |Du|	8.4K	8.4K	8.4K	8.4K
as each object attribute defines a different taxonomy for the data (e.g., it enables clustering images based on object shape, color etc.). The original dataset is limited as it contains only three shapes and two textures, reducing the difficulty of the respective clustering tasks. We introduce 2 new colors, 7 new shapes and 8 new textures to the dataset, placing between 1 and 10 objects in each scene.
Each image is therefore parameterized by object shape, texture, color and count. The value for each attribute is sampled uniformly and independently from the others, meaning the image label with respect to one taxonomy gives us no information about the label with respect to another. Note that this sets Clevr-4 apart from existing GCD benchmarks such as CIFAR-100 [53] and FGVC-Aircraft [55]. These datasets only contain taxonomies at different granularities, and as such the taxonomies are highly correlated with each other. Furthermore, the number of categories provides no information regarding the specified taxonomy, as all Clevr-4 taxonomies contain  $k = 10$  object categories.
Finally, we create GCD splits for each taxonomy in Clevr-4, following standard practise and reserving half the categories for the labelled set, and half for the unlabelled set. We further subsample  $50\%$  of the images from the labelled categories and add them to the unlabelled set. We synthesize  $8.4K$  images for GCD development (summarized in table 1), and further make a larger  $100K$  image dataset available. The full generation procedure is detailed in appendix A.1.
Performance metrics. We follow standard practise [7, 47] and report clustering accuracy for evaluation. Given predictions,  $\hat{y}_i$ , and ground truth labels,  $y_i$ , the clustering accuracy is  $ACC = \max_{\Pi \in S_k} \frac{1}{N_u} \sum_{i=1}^{N_u} \mathbb{1}\{\hat{y}_i = \Pi(y_i)\}$ , where  $S_k$  is the symmetry group of order  $k$ . Here  $N_u$  is the number of unlabelled images and the max operation is performed (with the Hungarian algorithm [56]) to find the optimal matching between the predicted cluster indices and ground truth labels. For GCD models, we also report  $ACC$  on subsets belonging to 'Old' ( $y_i \in \mathcal{V}_{\mathcal{L}}$ ) and 'New' ( $y_i \in \mathcal{V}_{\mathcal{U}} \setminus \mathcal{V}_{\mathcal{L}}$ ) classes. The most important metric is the 'All' accuracy (overall clustering performance), as the precise 'Old' and 'New' figures are subject to the assignments selected in the Hungarian matching.
4 Learnings from Clevr-4 for category discovery
In this section, we use Clevr-4 to gather insights into the category discovery problem. In section 4.1, we assess the limitations of large-scale pre-training for the problem, before using Clevr-4 to examine the weaknesses of existing category discovery methods in section 4.2. Next, in section 4.3, we use our findings to motivate a stronger method for GCD (which we describe in full in section 5), finding that this substantially outperforms implemented baselines on Clevr-4.
4.1 Limitations of large-scale pre-training for category discovery
Here, we show that pre-trained representations develop certain 'biases' which limit their performance when used directly or as initialization for category discovery.
Unsupervised clustering of pre-trained representations (table 2). We first demonstrate the limitations of unsupervised clustering of features as an approach for category discovery (reporting results with semi-supervised clustering in fig. 11). Specifically, we run  $k$ -means clustering [57] on top of features extracted with self- [9, 14, 21], weakly- [15], and fully-supervised [2, 3, 58] backbones, reporting performance on each of the four taxonomies in Clevr-4. The representations are trained on up to 400M images and are commonly used in the vision literature.
Table 2: Unsupervised clustering accuracy (ACC) of pre-trained models on Clevr-4. We find most models are strongly biased towards shape, while MAE [31] exhibits a color bias.
Pre-training Method	Pre-training Data	Backbone	Texture	Shape	Color	Count	Average
SWaV [9]	ImageNet-1K	ResNet50	13.1	65.5	12.1	18.9	27.4
MoCoV2 [22]	ImageNet-1K	ResNet50	13.0	77.5	12.3	18.8	30.4
Supervised [2]	ImageNet-1K	ResNet50	13.2	76.8	15.2	12.9	29.5
Supervised [3]	ImageNet-1K	ConvNeXT-B	13.4	83.5	12.1	13.1	30.5
DINO [14]	ImageNet-1K	ViT-B/16	16.0	86.2	11.5	13.0	31.7
MAE [31]	ImageNet-1K	ViT-B/16	15.1	13.5	64.7	13.9	26.8
iBOT [59]	ImageNet-1K	ViT-B/16	14.4	85.9	11.5	13.0	31.2
CLIP [15]	WIP-400M	ViT-B/16	12.4	78.7	12.3	17.9	30.3
DINOv2 [60]	LVD-142M	ViT-B/14	11.6	98.1	11.6	12.8	33.5
Supervised [58]	ImageNet-21K	ViT-B/16	11.8	96.2	11.7	13.0	33.2
Table 3: Effects of large-scale pre-training on category discovery accuracy (ACC) on Clevr-4. Contrary to dominant findings in the vision literature, we find that large-scale pre-training provides inconsistent gains on Clevr-4. For instance, on count, training a lightweight model (ResNet18) from scratch substantially outperforms initializing from DINOv2 (ViT-B/14) trained on 142M images.
Method	Backbone	Pre-training (Data)	Texture	Shape	Color	Count	Average	Average Rank
SimGCD	ResNet18	-	58.1	97.8	96.7	67.6	80.5	2.0
SimGCD	ViT-B/16	MAE [31] (ImageNet-1k)	54.1	99.7	99.9	53.0	76.7	2.0
SimGCD	ViT-B/14	DINOv2 [60] (LVD-142M)	76.5	99.9	87.4	51.3	78.8	2.0
We find that most models perform well on the shape taxonomy, with DINOv2 almost perfectly solving the task with  $98\%$  accuracy. However, none of the models perform well across the board. For instance, on some splits (e.g., color), strong models like DINOv2 perform comparably to random chance. This underscores the utility of Clevr-4 for delineating category discovery from standard representation learning. Logically, it is impossible for unsupervised clustering on any representation to perform well on all tasks. After all, only a single clustering of the data is produced, which cannot align with more than one taxonomy. We highlight that such limitations are not revealed by existing GCD benchmarks; on the CUB benchmark (see table 5), unsupervised clustering with DINOv2 achieves  $68\%$  ACC ( $\approx 140 \times$  random).
Pre-trained representations for category discovery (table 3). Many category discovery methods use self-supervised representation learning for initialization in order to leverage large-scale pretraining, in the hope of improving downstream performance. However, as shown above, these representations are biased. Here, we investigate the impact of these biases on a state-of-the-art method in generalized category discovery, SimGCD [19]. SimGCD contains two main loss components: (1) a contrastive loss on backbone features, using self-supervised InfoNCE [23] on all data, and supervised contrastive learning [61] on images with labels available; and (2) a contrastive loss to train a classification head, where different views of the same image provide pseudo-labels for each other. For comparison, we initialize SimGCD with a lightweight ResNet18 trained scratch; a ViT-B/16 pre-trained with masked auto-encoding [31]; and a ViT-B/14 with DINOv2 [60] initialization. For each initialization, we sweep learning rates and data augmentations.
Surprisingly, and in stark contrast to most of the computer vision literature, we find inconsistent gains from leveraging large-scale pre-training on Clevr-4. For instance, on the count taxonomy, pre-training gives substantially worse performance that training a lightweight ResNet18 from scratch. On average across all splits, SimGCD with a randomly initialized ResNet18 actually performs best. Generally, we find that the final category discovery model inherits biases built into the pre-training, and can struggle to overcome them even after finetuning. Our results highlight the importance of carefully selecting the initialization for a given GCD task, and point to the utility of Clevr-4 for doing so.
4.2 Limitations of existing category discovery methods
Next, we analyze SimGCD [19], the current state-of-the-art for the GCD task. We show that on Clevr-4 it is not always better than the GCD baseline [7] which it extends, and identify the source of this issue in the generation of the pseudo-labels for the discovered categories. In more detail, the
Table 4: Category discovery accuracy (ACC) on Clevr-4. Compared to our reimplementation of the GCD baseline [7] and SimGCD state-of-the-art [19], our method provides substantial boosts on average across Clevr-4. Results are averages across five random seeds. Also shown is the classification accuracy of a fully-supervised upper-bound on a disjoint test-set.
Model	Backbone	Texture	Shape	Color	Count	Average
All	Old	New	All	Old	New	All	Old	New	All	Old	New	All
Fully supervised	ResNet18	99.1	-	-	100.0	-	-	100.0	-	-	96.8	-	-	99.0
GCD	ResNet18	62.4	97.5	45.3	93.9	99.7	90.5	90.7	95.0	88.5	71.9	96.4	60.1	79.7
SimGCD	ResNet18	58.1	95.0	40.2	97.8	98.9	97.2	96.7	99.9	95.1	67.6	95.7	53.9	80.1
μGCD (Ours)	ResNet18	69.8	99.0	55.5	94.9	99.7	92.1	99.5	100.0	99.2	75.5	96.6	65.2	84.9
GCD baseline uses only one of the two losses used by SimGCD, performing contrastive learning on features, followed by simple clustering in the models' embedding space. To compare SimGCD and GCD, we start from a ResNet18 feature extractor, training it from scratch to avoid the potential biases identified in section 4.1. We show results in table 4, reporting results for 'All', 'Old' and 'New' class subsets. We follow standard practise when reporting on synthetic data and train all methods with five random seeds [34, 35] (error bars in the appendix B.1), and sweep hyper-parameters and data augmentations for both methods. We also train a model with full supervision and obtain  $99\%$  average performance on Clevr-4 (on independent test data), showing that the model has sufficient capacity.
Overall, we make the three following observations regarding the performance of GCD and SimGCD on Clevr-4: (i) Both methods' performance on texture and count is substantially worse than on shape and color. (ii) On the harder texture and count splits, the GCD baseline actually outperforms the SimGCD state-of-the-art. Given that SimGCD differs from GCD by adding a classification head and corresponding loss, this indicates that jointly training classifier and feature-space losses can hurt performance. (iii) Upon closer inspection, we find that the main performance gap on texture and count comes from accuracy on the 'New' categories; both methods cluster the 'Old' categories almost perfectly. This suggests that the 'New' class pseudo-labels from SimGCD are not strong enough; GCD, with no (pseudo-)supervision for novel classes, achieves higher clustering performance.
4.3 Addressing limitations in current approaches