SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison to supervised baseline. We compare HUME to a supervised linear classifier by utilizing ResNet-18 MOCOV2 pretrained representations [4] for both models. In particular, for a supervised classifier we add a linear layer on top of pretrained representations which is standard evaluation strategy of self-supervised methods. As a regularization representation space in HUME, we utilize BiT [7], CLIP ViT [6] and DINOv2 ViT [20]. The results are shown in Table 1. Remarkably, without using any supervision, on the STL-10 dataset HUME consistently achieves better performance than the supervised linear model in the transductive setting, and using CLIP and DINO in the inductive setting. Specifically, using the strongest DINO model, HUME outperforms the supervised linear model by  $5\%$  on the STL-10 dataset in terms of accuracy and by  $11\%$  in terms of ARI. On the CIFAR-10 dataset, HUME achieves performance comparable to the linear classifier. On the hardest CIFAR-100-20 there is still an expected gap between the performance of supervised and unsupervised methods. When comparing performance of HUME in inductive and transductive setting, the results show that utilizing more data in the transductive setting consistently outperforms the corresponding method in inductive setting by  $1 - 3\%$  in terms of accuracy. Finally, when comparing performance of different pretrained models, the results show that employing larger pretrained models results in better performance. For example, on the CIFAR-100-20 dataset HUME DINO achieves  $16\%$  relative improvement in accuracy over HUME BiT in both inductive and transductive settings. Thus,
these results strongly indicate that HUME framework can achieve even better performance by taking advantage of unceasing progress in the development of large pretrained models.
Table 1: Comparison of HUME to a supervised linear classifier in inductive (ind) and transductive (trans) settings using MOCOV2 self-supervised representations pretrained on the corresponding dataset and three different large pretrained models.
Method	STL-10	CIFAR-10	CIFAR-100-20
ACC	ARI	ACC	ARI	ACC	ARI
MOCO Linear	88.9	77.7	89.5	79.0	72.5	52.6
HUME BiT ind	87.5	76.2	85.9	73.8	47.8	33.5
HUME CLIP ind	90.2	80.2	88.2	77.1	48.5	34.1
HUME DINO ind	90.8	81.2	88.4	77.6	55.5	37.7
HUME BiT trans	90.3	80.5	86.6	75.0	48.8	34.5
HUME CLIP trans	92.2	84.1	88.9	78.3	50.1	34.8
HUME DINO trans	93.2	86.0	89.2	79.2	56.7	39.6
Comparison to unsupervised baselines. We next compare performance of HUME to the state-of-the-art deep clustering methods (Table 2). We use DINOv2 as the second representation space but the results for other models are available in Table 1. Results show that HUME consistently outperforms all baseline by a large margin. On the STL-10 and CIFAR-10 datasets, HUME achieves  $5\%$  improvement in accuracy over the best deep clustering baseline and  $11\%$  and  $10\%$  in ARI, respectively. On the CIFAR-100-20 dataset, HUME achieves remarkable improvement of  $19\%$  in accuracy and  $18\%$  in ARI. It is worth noting that considered baselines utilize nonlinear models (multilayer perceptrons) on top of the pretrained representations, while HUME employs solely a linear model. When compared to the K-means clustering baselines on top of concatenated features from DINO and MOCO, HUME achieves  $18\%$ ,  $9\%$  and  $8\%$  relative improvement in accuracy on the STL-10, CIFAR-10 and CIFAR-100-20 datasets, respectively. These results demonstrate that performance gains come from HUME's objective rather than from utilizing stronger representation spaces. Overall, our results show that the proposed framework effectively addresses the challenges of unsupervised learning and outperforms other baselines by a large margin.
Table 2: Comparison to unsupervised baselines. All methods use ResNet-18 MOCov2 self-supervised representations pretrained on the target dataset. We use DINOv2 as a large pretrained model.
Method	STL-10	CIFAR-10	CIFAR-100-20
ACC	ARI	ACC	ARI	ACC	ARI
MOCO + K-means	67.7	54.1	66.9	51.8	37.5	20.9
DINO + K-means	60.1	35.4	75.5	67.6	47.2	33.9
DINO + MOCO + K-means	77.1	70.0	81.4	77.1	51.3	36.1
SCAN	77.8	61.3	83.3	70.5	45.4	29.7
SPICE	86.2	73.2	84.5	70.9	46.8	32.1
HUME	90.8	81.2	88.4	77.6	55.5	37.7
Large-scale ImageNet-1000 benchmark. We next study HUME's performance on the ImageNet-1000 benchmark and compare it to the state-of-the-art deep clustering methods on this large-scale benchmark. All methods use the ResNet-50 backbone. SCAN is trained using MOCOv2 self-supervised representations and both TWIST and Self-classifier are trained from scratch as single-stage methods. HUME utilizes the same MOCOv2 self-supervised representations and DINOv2 as the second representation space. The results in Table 3 show that HUME
Table 3: Performance of HUME on the large-scale ImageNet-1000 dataset and comparison to unsupervised baselines. All methods use the ResNet50 backbone. HUME uses DINOv2 large pretrained model and ResNet-50 MOCOV2 self-supervised representation.
Method	ACC	ARI
SCAN	39.7	27.9
TWIST	40.6	30.0
Self-classifier	41.1	29.5
HUME	51.1	38.1
achieves  $24\%$  relative improvement in accuracy and  $27\%$  relative improvement in ARI over considered baselines, thus confirming the scalability of HUME to challenging fine-grained benchmarks.
(b)
Figure 3: (a) Different aggregation strategies on the CIFAR-10 dataset. We use MOCOV2 and different large pretrained models to instantiate HUME. (b) Accuracy of the reliable samples on the CIFAR-10 dataset. We use MOCOV2 and DINOv2 to instantiate HUME. The well-established setting for testing SSL methods on the CIFAR-10 dataset uses 4, 25 and 400 reliable samples per class (depicted with lines in red color).
Ablation study on aggregation strategy. HUME achieves a strikingly high correlation between its objective function and ground truth labeling (Figure 1). However, due to the internal stochasticity, the found labeling with the lowest generalization error does not need to always correspond to the highest accuracy. Thus, to stabilize the predictions we obtain final labeling by aggregating found labelings from independent runs. We simply use the majority vote of all tasks in all our experiments. Here, we investigate the effect of using different aggregation strategies on the performance. Figure 3 (a) shows the results on the CIFAR-10 dataset and the corresponding plots for the STL-10 and CIFAR-100-20 datasets are shown in Appendix F. Given the high correlation of our objective and human labeling, we aggregate top- $n$  of labelings w.r.t. generalization error and compute the corresponding point on the plot. Thus, the leftmost strategy (at  $x = 1$ ) corresponds to the one labeling that has the lowest generalization error, while the rightmost data point (at  $x = 100$ ) corresponds to aggregation over all generated labelings, i.e., the strategy we adopt in all experiments. By aggregation over multiple labelings the algorithm produces more stable predictions. Expectedly, given the high correlation between HUME's objective and human labeling, we can achieve even better performance by aggregating only top- $n$  predictions compared to our current strategy that considers all tasks; however, we do not optimize for this in our experiments. Finally, it can be seen that utilizing larger pretrained models eliminates the need of aggregation procedure, since they lead to stronger and robust performance.
Reliable samples for semi-supervised learning (SSL). We next aim to answer whether HUME can be used to reliably generate labeled examples for SSL methods. By generating a few reliable samples per class (pseudo-labels), an unsupervised learning problem can be transformed into an SSL problem [14]. Using these reliable samples as initial labels, any SSL method can be applied. HUME can produce such reliable samples in a simple way. Specifically, we say that a sample is reliable if  $(i)$  the majority of the found labelings assigns it to the same class, and  $(ii)$  the majority of the sample neighbors have the same label. We provide the detailed description of the algorithm in Appendix G. We evaluate the accuracy of the generated reliable samples on the CIFAR-10 dataset and show results in Figure 3 (b). In the standard SSL evaluation setting that uses 4 labeled examples per class [39, 42, 43], HUME produces samples with perfect accuracy. Moreover, even with 15 labeled examples per class, reliable samples generated by HUME still have perfect accuracy. Remarkably, in other frequently evaluated SSL settings on the CIFAR-10 dataset [39, 42, 43] with 25 and 400 samples per class, accuracy of reliable samples produced by HUME is near perfect (99.6% and 99.7% respectively). Results on the STL-10 and CIFAR-100-20 datasets and additional statistics of the reliable samples are provided in the Appendix E.
We next utilize the recent state-of-the-art SSL method FreeMatch [39] to compare the results of running FreeMatch with reliable samples produced by HUME to running FreeMatch with ground truth labeling. The results in Table 4 show that in the extremely low data regime FreeMatch with reliable samples produced by HUME outperforms FreeMatch with ground truth labels. Indeed, FreeMatch with ground truth labels utilizes samples which are sampled uniformly at random, while
HUME's reliable samples by definition are such samples whose most neighbours belong to the same class predicted by HUME. Consequently, FreeMatch which is based on adaptive thresholding for pseudo-labeling, benefits from utilizing labeled samples for which it can confidently set the threshold, especially in a low data regime. For instance, FreeMatch with reliable samples achieves  $10\%$  relative improvement in accuracy over FreeMatch with ground truth labeling on the CIFAR-10 dataset with one sample per class. Comparing FreeMatch with 4 reliable samples produced by HUME with an original HUME method, we observe improvement of  $8\%$  on the CIFAR-10 dataset, demonstrating that HUME's results can be further improved by applying SSL with HUME's reliable samples. Overall, our results strongly demonstrate that HUME is highly compatible with SSL methods and can be used to produce reliable labeling for SSL methods.
Table 4: Comparison of accuracy of FreeMatch trained using reliable samples produced by HUME and FreeMatch trained using ground truth labeling on the STL-10 and on the CIFAR-10 dataset. We apply FreeMatch using 4 and 100 samples per class on the STL-10 dataset and 1, 4, 25 and 400 samples per class on the CIFAR-10 dataset. Each experiment is run 3 times and the results are averaged.
Method	STL-10	CIFAR-10
4	100	1	4	25	400
FreeMatch w/ reliable samples	81.3 ± 4.3	91.3 ± 0.3	91.2 ± 5.3	95.1 ± 0.1	93.3 ± 2.5	94.3 ± 0.4
FreeMatch w/ ground truth labels	77.8 ± 1.1	94.0 ± 0.1	83.3 ± 9.6	94.8 ± 0.3	95.1 ± 0.0	95.7 ± 0.1
Self-supervised learning. Self-supervised methods [44, 45, 46, 47, 48, 49, 5, 19] aim to define a pretext task which leads to learning representations that are useful for downstream tasks. Recently, contrastive approaches [1, 2, 3, 4, 6, 20] have seen a significant interest in the community. These approaches learn representations by contrasting positive pairs against negative pairs. Another line of work relies on incorporating beneficial inductive biases such as image rotation [49], solving Jigsaw puzzles [46] or by introducing sequential information that comes from video [45]. Regardless of a learning approach, a linear probe, i.e., training a linear classifier on top of the frozen representations using the groundtruth labeling, is a frequently used evaluation protocol for self-supervised methods. In our work, we turn this evaluation protocol into an optimization objective with the goal to recover the human labeling in a completely unsupervised manner. In Appendix C, we show that stronger representations lead to better unsupervised performance mirroring the linear probe evaluation. Given that HUME framework is model-agnostic, it can constantly deliver better unsupervised performance by employing continuous advancements of self-supervised approaches. Furthermore, HUME can be used to evaluate performance of self-supervised methods in an unsupervised manner.
Transfer learning. Transfer learning is a machine learning paradigm which utilizes large scale pretraining of deep neural networks to transfer knowledge to low-resource downstream problems [50, 7]. This paradigm has been successfully applied in a wide range of applications including but not limited to few-shot learning [10, 9], domain adaptation [51, 52] and domain generalization [53, 54]. Recently, foundation models [55, 56, 11, 57, 12] trained on a vast amount of data achieved breakthroughs in different fields. The well-established pipeline of transfer learning is to fine-tune weights of a linear classifier on top of the frozen representations in a supervised manner [50]. In our framework, we leverage strong linear transferability of these representations to act as a regularizer in guiding the search process of human labeled tasks. Thus, it can be also seen as performing an unsupervised transfer learning procedure. While language-image foundation models such as CLIP [6] require human instruction set to solve a new task, HUME provides a solution to bypass this requirement.
Clustering. Clustering is a long studied machine learning problem [58]. Recently, deep clustering methods [59, 13, 60] have shown benefits over traditional approaches. The typical approach to clustering problem is to encourage samples to have the same class assignment as its neighbours in a representation space [13, 60, 61]. Other recent methods rely on self-labeling, i.e., gradually fitting the neural network to its own most confident predictions [14, 16, 62, 63]. Alternative approaches [64, 59] train an embedding space and class prototypes to further assign samples to the closest prototype in the embedding space. In contrast, our framework redefines the way to approach a clustering problem.
Without explicitly relying on notions of semantic similarity, we seek to find the most generalizable labeling regardless of the representation space in the space of all possible labelings.
Generalization. One of the conventional protocols to assess an ability of the model to generalize is to employ held-out data after model training. In addition, different measures of generalization [65, 66, 67, 68, 69, 70] have been developed to study and evaluate model generalization from different perspectives. Although, traditionally a generalization error is used as a metric, recent work [17] employs test-time agreement of two deep neural networks to find labelings on which the corresponding neural network can generalize well. Often, these labelings reflect inductive biases of the learning algorithm and can be used to analyze deep neural networks from a data-driven perspective.
Meta-optimization. The proposed optimization procedure requires optimization through the inner optimization process. This type of optimization problems also naturally arises in a gradient-based meta-optimization [27, 71, 72]. Other works [23, 25, 24] tackle the similar problem and study how to embed convex differentiable optimization problems as layers in deep neural networks. Although meta-optimization methods are, in general, computationally heavy and memory intensive, the proposed optimization procedure only requires propagating gradients through a single linear layer. This makes HUME a simple and efficient method to find human labelings. In addition, the inner optimization problem (Eq. 5) is convex allowing for the utilization of the specialized methods such as L-BFGS [73, 74], making our framework easily extendable to further boost the efficiency.
Semi-supervised learning. Semi-supervised learning (SSL) assumes that along with an abundance of unlabeled data, a small amount of labeled examples is given for each class. Consequently, SSL methods [39, 45, 37, 75, 76, 38] effectively employ available labeled data to increase the model's generalization performance on entire dataset. As shown in Figure 3 (b) and in Table 4, HUME can be used to construct reliable samples and convert the initial unsupervised problem to the SSL problem. Thus, the proposed framework can also be used to eliminate the need of even a few costly human annotations, bridging the gap between supervised and semi-supervised learning.
The instantiation of HUME employs advances from different fields of study to solve the proposed optimization problem and consequently, it inherits some limitations which we discuss below.
Meta-optimization. For simplicity, HUME leverages MAML [27] to solve the proposed optimization problem (Eq. 7). However, its non-convex nature prevents convergence to the labelings which attain global optima. Though, it can be seen that the aforementioned objective is a special case of well-studied bilevel optimization problems with a convex inner part. Thus, HUME can greatly benefit from utilization of specialized optimization methods [77, 78, 79, 80] to further boost the performance.
Number of classes and empirical label distribution. HUME assumes the number of classes and empirical label distribution is known a priori - a common assumption in existing unsupervised learning approaches [13, 14, 59]. Although, in this work, we also follow these assumptions, the proposed framework is compatible with methods which can estimate the required quantities [81, 82].
We introduced HUME, a simple framework for discovering human labelings that sheds a new light on solving the unsupervised learning problem. HUME is based on the observation that a linear model can separate classes defined by human labelings regardless of the representation space used to encode the data. We utilize this observation to search for linearly generalizable data labeling in representation spaces of two pretrained models. Our approach improves considerably over existing unsupervised baselines on all of the considered benchmarks. Additionally, it shows superior performance to a supervised baseline on the STL-10 dataset and competitive on the CIFAR-10 dataset. HUME could also be used to generate labelings for semi-supervised methods and to evaluate quality of self-supervised representations. Our model-agnostic framework will greatly benefit from new more powerful self-supervised and large pretrained models that will be developed in the future.
[77] Aurélien Ouattara and Anil Aswani. Duality approach to bilevel programs with a convex lower level. In American Control Conference, 2018.
[78] Jiri V Outrata. On the numerical solution of a class of Stackelberg problems. Zeitschrift für Operations Research, 34:255-277, 1990.
[79] Mihai Anitescu. On using the elastic mode in nonlinear programming approaches to mathematical programs with complementarity constraints. SIAM Journal on Optimization, 15(4): 1203-1236, 2005.
[80] Jane J Ye and DL Zhu. Optimality conditions for bilevel programming problems. Optimization, 33(1):9-27, 1995.
[81] Kai Han, Andrea Vedaldi, and Andrew Zisserman. Learning to discover novel visual categories via deep transfer clustering. In International Conference on Computer Vision, 2019.
[82] Yiqi Wang, Zhan Shi, Xifeng Guo, Xinwang Liu, En Zhu, and Jianping Yin. Deep embedding for determining the number of clusters. In AAAI Conference on Artificial Intelligence, 2018.
We utilize ResNet-18 backbone [1] as  $\phi_1(\cdot)$  pretrained on the train split of the corresponding dataset with MOCOv2 [2]. Features are obtained after avgpool with dimension equal to 512. For the ImageNet-1k dataset, we utilize ResNet-50 backbone as  $\phi_1(\cdot)$  pretrained with MOCOv2. Here, features are obtained after avgpool with dimension equal to 2048. We use the same backbone and pretraining strategy for baselines as well. To enforce orthogonality constraint on the weights of the task encoder we apply Pytorch parametrizations [3]. When precomputing representations we employ standard data preprocessing pipeline of the corresponding model and do not utilize any augmentations during HUME's training. In all experiments, we use the following hyperparameters: number of iterations  $T = 1000$ , Adam optimizer [4] with step size  $\alpha = 0.001$  and temperature of the sparsemax activation function  $\gamma = 0.1$ . We anneal temperature and step size by 10 after 100 and 200 iterations. We set regularization parameter  $\eta$  to value 10 in all experiments and we show ablation for this hyperparameter in Appendix B. To solve inner optimization problem we run gradient descent for 300 iterations with step size equal to 0.001. At each iteration we sample without replacement 10000 examples from the dataset to construct subset  $(X_{tr}, X_{te})$ ,  $|X_{tr}| = 9000$ ,  $|X_{te}| = 1000$ . Since STL-10 dataset has less overall number of samples, we use 5000  $(|X_{tr}| = 4500$ ,  $|X_{te}| = 500)$  in the inductive setting and 8000  $(|X_{tr}| = 7200$ ,  $|X_{te}| = 800)$  in the transductive setting. For the ImageNet-1000 dataset we use inner and outer step size equal to 0.1, number of inner steps equal to 100, sample 20000 examples with  $|X_{tr}| = 14000$ ,  $|X_{te}| = 6000$  on each iteration. We do not anneal temperature and step size for the ImageNet-1000 dataset and other hyperparameters remain the same. To reduce the gradient variance, we average the final optimization objective (Eq. 7) over 20 random subsets on each iteration. To stabilize training in early iterations, we clip gradient norm to 1 before updating task encoder's parameters. We use  $N_{\text{neigh}} = 500$  in Algorithm G1 to construct reliable samples for semi-supervised learning.
B Robustness to a regularization parameter
HUME incorporates entropy regularization of the empirical label distribution in the final optimization objective (Eq. 7) to avoid trivial solutions, i.e., assigning all samples to a single class. To investigate the effect of the corresponding hyperparameter  $\eta$ , we run HUME from 100 random initializations  $W_{1}$  for each  $\eta \in \{0,1,2,5,10\}$  on the CIFAR-10 dataset. Figure B1 shows results with different values of hyperparameter  $\eta$ . The results show that  $\eta$  is indeed a necessary component of HUME objective, i.e., setting  $\eta = 0$  leads to degenerate labelings since assigning all samples to a single class is trivially invariant to any pair of representation spaces. Furthermore, the results show that HUME is robust to different positive values of the parameter  $\eta$ . We set  $\eta = 10$  in all experiments.
Figure B1: Performance of HUME on the CIFAR-10 dataset with different values of the entropy regularization. We use MOCOV2 self-supervised representations pretrained on the CIFAR-10 dataset and BiT large pretrained model.
C Ablation study on different self-supervised methods
In all experiments, we use MOCOv2 [2] self-supervised representations. Here, we evaluate HUME's performance with different self-supervised learning method. In particular, we utilize SimCLR [5] and [6] to obtain representation space  $\phi_1(\cdot)$ . Table C1 shows the results in the inductive setting using DINO large pretrained model as the second representation space. The results show that HUME instantiated with MOCO consistently outperforms HUME instantiated with SimCLR on all of the datasets. This is expected result since MOCO representations are stronger also when assessed by a supervised linear probe. Alternatively, utilizing BYOL shows consistent improvements over MOCO representations. These results demonstrate that HUME can improve by employing stronger self-supervised representations. Interestingly, even with SimCLR representations HUME still outperforms unsupervised baselines pretrained with MOCOv2 in Table 2.
Table C1: Comparison of different self-supervised representations. We use DINOv2 large pretrained model. Stronger self-supervised representations lead to better performance.
Method	STL-10	CIFAR-10	CIFAR-100-20
ACC	ARI	ACC	ARI	ACC	ARI
SimCLR Linear	85.3	71.2	87.1	74.4	70.4	49.7
MOCO Linear	88.9	77.7	89.5	79.0	72.5	52.6
BYOL Linear	92.1	83.6	90.7	81.0	77.2	59.3
HUME SimCLR	86.9	74.3	85.2	71.8	51.8	33.9
HUME MOCO	90.8	81.2	88.4	77.6	55.5	37.7
HUME BYOL	91.5	82.7	89.8	79.7	56.0	40.3
D Correlation plots on the STL-10 and CIFAR-100-20 datasets.