SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Artyom Gadetsky
artem.gadetskii@epfl.ch
Maria Brbić
mbrbic@epfl.ch
We present HUME, a simple model-agnostic framework for inferring human labeling of a given dataset without any external supervision. The key insight behind our approach is that classes defined by many human labelings are linearly separable regardless of the representation space used to represent a dataset. HUME utilizes this insight to guide the search over all possible labelings of a dataset to discover an underlying human labeling. We show that the proposed optimization objective is strikingly well-correlated with the ground truth labeling of the dataset. In effect, we only train linear classifiers on top of pretrained representations that remain fixed during training, making our framework compatible with any large pretrained and self-supervised model. Despite its simplicity, HUME outperforms a supervised linear classifier on top of self-supervised representations on the STL-10 dataset by a large margin and achieves comparable performance on the CIFAR-10 dataset. Compared to the existing unsupervised baselines, HUME achieves state-of-the-art performance on four benchmark image classification datasets including the large-scale ImageNet-1000 dataset. Altogether, our work provides a fundamentally new view to tackle unsupervised learning by searching for consistent labelings between different representation spaces.
A key aspect of human intelligence is an ability to acquire knowledge and skills without external guidance or instruction. While recent self-supervised learning methods [1, 2, 3, 4, 5, 6] have shown remarkable ability to learn task-agnostic representations without any supervision, a common strategy is to add a linear classification layer on top of these pretrained representations to solve a task of interest. In such a scenario, neural networks achieve high performance on many downstream human labeled tasks. Such strategy has also been widely adopted in transfer learning [7, 8] and few-shot learning [9, 10], demonstrating that a strong feature extractor can effectively generalize to a new task with a minimal supervision. However, a fundamental missing piece in reaching human-level intelligence is that machines lack an ability to solve a new task without any external supervision and guidance.
Close to such ability are recent multi-modal methods [6, 11, 12] trained on aligned text-image corpora that show outstanding performance in the zero-shot learning setting without the need for fine-tuning. However, zero-shot learning methods still require human instruction set to solve a new task. In a fully unsupervised scenario, labels for a new task have been traditionally inferred by utilizing clustering methods [13, 14, 15, 16], designed to automatically identify and group samples that are semantically related. Compared to (weakly) supervised counterparts, the performance of clustering methods is still lagging behind.
In this work, we propose HUME, a simple model-agnostic framework for inferring human labeling of a given dataset without any supervision. The key insights underlying our ap
proach are that: (i) many human labeled tasks are linearly separable in a sufficiently strong representation space, and (ii) although deep neural networks can have their own inductive biases that do not necessarily reflect human perception and are vulnerable to fitting spurious features [17, 18], human labeled tasks are invariant to the underlying model and resulting representation space. We utilize these observations to develop the generalization-based optimization objective which is strikingly well correlated with human labeling (Figure 1).
The key idea behind this objective is to evaluate the generalization ability of linear models on top of representations generated from two pretrained models to assess the quality of any given labeling (Figure 2). Our framework is model-agnostic, i.e., compatible with any pretrained representations, and simple, i.e., it requires training only linear models.
Overall, HUME presents a new look on how to tackle unsupervised learning. In contrast to clustering methods [13, 14] which try to embed inductive biases reflecting semantic relatedness of samples into a learning algorithm, our approach addresses this setting from model generalization perspective. We instantiate HUME's framework using representations from different self-supervised methods (MOCO [4, 3], SimCLR [1, 2], BYOL [19]) pretrained on the target dataset and representations obtained using large pretrained models (BiT [7], DINO [20], CLIP [6]). Remarkably, despite being fully unsupervised, HUME outperforms a supervised linear classifier on the STL-10 dataset by  $5\%$  and has comparable performance to a linear classifier on the CIFAR-10 dataset. Additionally, it leads to the new state-of-the-art performance on standard clustering benchmarks including the CIFAR-100-20 and the large-scale ImageNet1000 datasets. Finally, our framework can construct a set of reliable labeled samples transforming the initial unsupervised learning problem to a semi-supervised learning.
Figure 1: Correlation plot between distance to ground truth human labeling and HUME's objective on the CIFAR-10 dataset. HUME generates different labelings of the data to discover underlying human labeling. For each labeling (data point on the plot), HUME evaluates generalization error of linear classifiers in different representation spaces as its objective function. HUME's objective is strikingly well correlated  $(\rho = 0.93, p = 2.6 \times 10^{-45}$  two-sided Pearson correlation coefficient) with a distance to human labeling. In particular, HUME achieves the lowest generalization error for tasks that almost perfectly correspond to human labeling, allowing HUME to recover human labeling without external supervision. Results on the STL-10 and CIFAR-100-20 datasets are provided in Appendix D.
2 HUME framework
In this section, we first introduce our problem setting and then present a general form of our framework for finding human labeled tasks without any supervision.
Problem setting. Let  $\mathcal{D} = \{x_i\}_{i=1}^N$  be a set of samples. We assume this dataset consists of  $K$  classes, where  $K$  is known a priori, and each example  $x_i$  can belong only to one particular class  $k \in \{0, \dots, K-1\}$ . We define a task  $\tau: \mathcal{D} \to \{0, \dots, K-1\}$  as a labeling function of this dataset. We refer to a task  $\tau$  as human labeled if it respects the true underlying labeling of the corresponding dataset  $\mathcal{D}$ .
2.1 Test error and invariance of human labeled tasks to representation space
Measuring the performance of a model on a held-out dataset is a conventional method to assess an ability of the model to generalize on the given task  $\tau$ . Specifically, for the dataset  $\mathcal{D}$ , we can construct two disjoint subsets  $(X_{tr}, X_{te})$  of the dataset  $\mathcal{D}$ . Let  $f: \mathcal{D} \to \Delta^{K-1}$  be a probabilistic classifier which transforms the input  $x \in \mathcal{D}$  to class probabilities, i.e.,  $\Delta^{K-1}$  is a  $(K-1)$ -dimensional simplex. After training  $f$  on  $X_{tr}$  with loss function  $\mathcal{L}$  and labeling  $\tau(X_{tr})$ , we can compute the test error on
Figure 2: Overview of the HUME framework. HUME utilizes pretrained representations and linear models on top of these representations to assess the quality of any given labeling. As a result, optimizing the proposed generalization-based objective leads to labelings which are strikingly well correlated with human labelings.
$X_{te}$  which can provide us with an unbiased estimate of the true error of the model on the task  $\tau$ :
$$
\mathcal {L} \left(f \left(X _ {t e}\right), \tau \left(X _ {t e}\right)\right) = \frac {1}{\left| X _ {t e} \right|} \sum_ {x \in X _ {t e}} \mathcal {L} (f (x), \tau (x)). \tag {1}
$$
In HUME, we utilize this score to measure the quality of any given task  $\tau$ . By utilizing this score, we aim at searching for a human labeled task over the set of all possible tasks on the dataset  $\mathcal{D}$ . However, the main challenge is that neural networks can have their own inductive biases and attain low test error on tasks that capture spurious correlations and do not reflect human labeling [17]. To distinguish between such tasks and human labeled tasks, the key insight behind our framework is that for many human labeled tasks classes defined by human labeling are linearly separable regardless of the representation space used to represent a dataset. In other words, human-labeled tasks are invariant to sufficiently strong representation spaces. We next formally define what we mean by a sufficiently strong representation space and invariance of a task to the pair of representations.
Definition 1. Let  $\phi(x): \mathcal{D} \to \mathbb{R}^d$  be a mapping from the sample space to a low-dimensional representation space. We say that a representation space is sufficiently strong with respect to  $\tau$  if a linear model  $f$  trained on top of  $\phi(\cdot)$  attains low test error in Eq. (1).
Definition 2. Let  $\phi_1(x): \mathcal{D} \to \mathbb{R}^{d_1}$  and  $\phi_2(x): \mathcal{D} \to \mathbb{R}^{d_2}$  be two mappings from the sample space to low-dimensional representations. We say that a task  $\tau$  is invariant to the pair of representations  $(\phi_1(x), \phi_2(x))$  if both representation spaces are linearly separable with respect to  $\tau$ , i.e., both linear models  $f_1$  and  $f_2$ , trained on top of  $\phi_1(\cdot)$  and  $\phi_2(\cdot)$  respectively, attain low test error in Eq. (1).
We employ the property of invariance to the given pair of fixed pretrained representations to seek for a human labeled task. Thus, we only train linear classifiers on top pretrained representations while the representations are always frozen during training. The simultaneous utilization of several representation spaces acts as a regularizer and prevents learning tasks that capture spurious correlations which can reflect inductive biases of the individual representation space.
Specifically, given a task  $\tau$ , we aim to fit a linear model  $f_{i}$  with weights parametrized by  $W_{i} \in \mathbb{R}^{K \times d_{i}}$  on  $X_{tr}$  in each representation space  $\phi_{i}(\cdot)$ ,  $i = 1,2$ . Let  $\hat{W}_{i}(\tau)$  be the solution for the corresponding  $\phi_{i}(\cdot)$ . We aim at optimizing the test error of both linear models with respect to  $\tau$ :
$$
\arg \min  _ {\tau} \mathcal {L} \left(\sigma \left(\hat {W} _ {1} (\tau) \phi_ {1} \left(X _ {t e}\right)\right), \tau \left(X _ {t e}\right)\right) + \mathcal {L} \left(\sigma \left(\hat {W} _ {2} (\tau) \phi_ {2} \left(X _ {t e}\right)\right), \tau \left(X _ {t e}\right)\right), \tag {2}
$$
where  $\sigma(\cdot)$  is a softmax activation function. We draw an attention of the reader to the fact that both  $\hat{W}_1(\tau)$  and  $\hat{W}_2(\tau)$  implicitly depend on  $\tau$  as solutions of the inner optimization problem on  $X_{tr}$  with labeling  $\tau(X_{tr})$ . We discuss how to efficiently solve this optimization problem and propagate gradients through the optimization process in the corresponding Section 2.3.
An unresolved modeling question is the choice of the representation spaces  $\phi_{1,2}(\cdot)$ . We utilize self-supervised pretraining on the target dataset  $\mathcal{D}$  to obtain robust and well clustered representations
for representation space  $\phi_1(\cdot)$ . Representation space  $\phi_2(\cdot)$  acts as a regularizer to guide the search process. Thus, we utilize features of a large pretrained model as the representation space  $\phi_2(\cdot)$ . This is an appealing modeling design choice from both efficiency and model performance perspective. In particular, by using a large pretrained model we do not need to train a model on the given dataset of interest. Despite the simplicity, the linear layer fine-tuning on top of the fixed representations of the deep pretrained models has shown its efficiency in solving many downstream problems [6, 20, 7, 10]. The approach to model  $\tau$  is discussed in the next section.
2.2 Task parametrization
The proposed objective in Eq. (2) requires solving difficult discrete optimization problems with respect to  $\tau$  which prevents us from using efficient gradient optimization techniques. Additionally, it requires designing three separate models  $(\phi_1(\cdot),\phi_2(\cdot),\tau (\cdot))$  which can be computationally expensive and memory-intensive in practice. To alleviate both shortcomings, we utilize  $\phi_1(\cdot)$  to simultaneously serve as a basis for the task encoder and as a space to which the task should be invariant to. Namely, we relax the outputs of  $\tau$  to predict class probabilities instead of discrete class assignments, and parametrize the task  $\tau_{W_1}(\cdot):\mathcal{D}\to \Delta^{K - 1}$  as follows:
$$
\tau_ {W _ {1}} (x) = \mathcal {A} \left(W _ {1} \hat {\phi} _ {1} (x)\right), W _ {1} W _ {1} ^ {T} = I _ {K}, \hat {\phi} _ {1} (x) = \frac {\phi_ {1} (x)}{\| \phi_ {1} (x) \| _ {2}}, \tag {3}
$$
where  $\phi_1(\cdot)$  denotes the self-supervised representations pretrained on the given dataset  $\mathcal{D}$  and these representations also remain fixed during the overall training procedure. We produce sparse labelings using sparsemax [21] activation function  $\mathcal{A}$  since each sample  $x_i \in \mathcal{D}$  needs to be restricted to belong to a particular class. The above parametrization may be viewed as learning prototypes for each class which is an attractive modeling choice for the representation space  $\hat{\phi}_1(\cdot)$  [22, 9]. Thus,  $W_1\hat{\phi}_1(\cdot)$  corresponds to the cosine similarities between class prototypes  $W_1$  and the encoding of the sample  $\hat{\phi}_1(\cdot)$ . Moreover, sparsemax activation function acts as a soft selection procedure of the closest class prototype. Eventually, it can be easily seen that any linear dependence  $W_1\hat{\phi}_1(x)$  give rise to the task which, by definition, is at least invariant to the corresponding representation space  $\hat{\phi}_1(x)$ .
Given the above specifications, our optimization objective in Eq. (2) is simplified as follows:
$$
\arg \min  _ {W _ {1}} \mathcal {L} \left(\sigma \left(\hat {W} _ {2} \left(W _ {1}\right) \phi_ {2} \left(X _ {t e}\right)\right), \tau_ {W _ {1}} \left(X _ {t e}\right)\right), \tag {4}
$$
where  $\hat{W}_2(W_1)$  denotes the weights of the linear model  $f_{2}$  trained on the  $(X_{tr},\tau_{W_1}(X_{tr}))$ , which implicitly depend on the parameters  $W_{1}$ . We use the cross-entropy loss function  $\mathcal{L}$ , which is a widely used loss function for classification problems. The resulting optimization problem is continuous with respect to  $W_{1}$ , which allows us to leverage efficient gradient optimization techniques. Although it involves propagating gradients through the inner optimization process, we discuss how to efficiently solve it in the subsequent section.
2.3 Test error optimization
At each iteration  $k$ , we randomly sample disjoint subsets  $(X_{tr}, X_{te}) \sim D$  to prevent overfitting to the particular train-test split. We label these splits using the current task  $\tau_{W_1^k}(\cdot)$  with parameters  $W_1^k$ . Before computing the test risk defined in Eq. (4), we need to solve the inner optimization problem on  $(X_{tr}, \tau_{W_1^k}(X_{tr}))$ , specifically:
$$
\hat {W} _ {2} \left(W _ {1} ^ {k}\right) = \arg \min  _ {W _ {2}} \mathcal {L} \left(\sigma \left(W _ {2} \phi_ {2} \left(X _ {t r}\right)\right), \tau_ {W _ {1} ^ {k}} \left(X _ {t r}\right)\right). \tag {5}
$$
It can be easily seen that the above optimization problem is the well-studied multiclass logistic regression, which is convex with respect to  $W_{2}$  and easy to solve. To update parameters  $W_{1}^{k}$ , we need to compute the total derivative of Eq. (4) with respect to  $W_{1}$  which includes the Jacobian  $\frac{\partial\hat{W}_2}{\partial W_1^k}$ . Different approaches [23, 24, 25, 26] can be utilized to compute the required Jacobian and propagate gradients through the above optimization process. For simplicity, we run gradient descent for the fixed number of iterations  $m$  to solve the inner optimization problem and obtain  $\hat{W}_2^m (W_1^k)$ . Afterwards, we employ MAML [27] to compute  $\frac{\partial\hat{W}_2^m}{\partial W_1^k}$  by unrolling the computation graph of the
inner optimization's gradient updates. The remaining terms of the total derivative are available out-of-the-box using preferred automatic differentiation (AD) toolbox. This results in the efficient procedure which can be effortlessly implemented in existing AD frameworks [28, 29].
Regularization. The task encoder  $\tau$  can synthesize degenerate tasks, i.e., assign all samples to a single class. Although such tasks are invariant to any representation space, they are irrelevant. To avoid such trivial solutions, we utilize entropy regularization to regularize the outputs of the task encoder averaged over the set  $X = X_{tr} \cup X_{te}$ , specifically
$$
\mathcal {R} (\bar {\tau}) = - \sum_ {k = 1} ^ {K} \bar {\tau} _ {k} \log \bar {\tau} _ {k}, \tag {6}
$$
where  $\overline{\tau} = \frac{1}{|X|}\sum_{x\in X}\tau_{\theta}(x)\in \Delta^{K - 1}$  is the empirical label distribution of the task  $\tau$ . This leads us to the final optimization objective, which is:
$$
\arg \min  _ {W _ {1}} \mathcal {L} \left(\sigma \left(\hat {W} _ {2} \left(W _ {1}\right) \phi_ {2} \left(X _ {t e}\right)\right), \tau_ {W _ {1}} \left(X _ {t e}\right)\right) - \eta \mathcal {R} (\bar {\tau} \left(W _ {1}\right)), \tag {7}
$$
where  $\eta$  is the regularization parameter. This regularization corresponds to entropy regularization which has been widely used in previous works [13, 30, 31]. The pseudocode of the algorithm is shown in Algorithm 1.
Algorithm 1 HUME: A simple framework for finding human labeled tasks
Input: Dataset  $\mathcal{D}$ , number of classes  $K$ , number of iterations  $T$ , representation spaces  $\phi_{1,2}(\cdot)$ , task encoder  $\tau_{W_1}(\cdot)$ , regularization parameter  $\eta$ , step size  $\alpha$
1: Randomly initialize  $K$  orthonormal prototypes:  $W_{1}^{1} = \mathrm{ortho\_rand}(K)$
2: for  $k = 1$  to  $T$  do
3: Sample disjoint train and test splits:  $(X_{tr}, X_{te}) \sim \mathcal{D}$
4: Generate task:  $\tau_{tr},\tau_{te}\gets \tau_{W_1^k}(X_{tr}),\tau_{W_1^k}(X_{te})$
5: Fit linear classifier on  $X_{tr}$ :  $\hat{W}_2(W_1^k) = \arg \min_{W_2} \mathcal{L}(\sigma(W_2\phi_2(X_{tr})), \tau_{tr})$
6: Evaluate task invariance on  $X_{te}$ :  $\mathcal{L}_k(W_1^k) = \mathcal{L}(\sigma (\hat{W}_2(W_1^k)\phi_2(X_{te})),\tau_{te}) - \eta \mathcal{R}(\overline{\tau})$
7: Update task parameters:  $W_1^{k+1} \gets W_1^k - \alpha \nabla_{W_1^k} \mathcal{L}_k(W_1^k)$
8: end for
Datasets and evaluation metrics. We evaluate the performance of HUME on three commonly used clustering benchmarks, including the STL-10 [32], CIFAR-10 and CIFAR-100-20 [33] datasets. The CIFAR-100-20 dataset consists of superclasses of the original CIFAR-100 classes. In addition, we also compare HUME to large-scale unsupervised baselines on the fine-grained ImageNet-1000 dataset [34]. We compare our method with the baselines using two conventional metrics, namely clustering accuracy (ACC) and adjusted rand index (ARI). Hungarian algorithm [35] is used to match the found labeling to the ground truth labeling for computing clustering accuracy. We interchangeably use terms generalization error and cross validation accuracy when presenting the results.
Instantiation of HUME. For the representation space  $\phi_1(\cdot)$ , we use MOCOV2 [4] pretrained on the train split of the corresponding dataset. We experiment with the SimCLR [1] as a self-supervised method in Appendix C. For the representation space  $\phi_2(\cdot)$ , we consider three different large pretrained models: (i) BiT-M-R50x1 [7] pretrained on ImageNet-21k [36], (ii) CLIP ViT-L/14 pretrained on WebImageText [6], and (iii) DINOv2 ViT-g/14 pretrained on LVD-142M [20].
Baselines. Since HUME trains linear classifiers on top of pretrained self-supervised representations, supervised linear probe on top of the same self-supervised pretrained representations is a natural baseline to evaluate how well the proposed framework can match the performance of a supervised model. Thus, we train a linear model using ground truth labelings on the train split and report the
results on the test split of the corresponding dataset. For the unsupervised evaluation, we consider two state-of-the-art deep clustering methods, namely SCAN [13] and SPICE [14]. Both methods can be seen as three stage methods. First stage employs self-supervised methods to obtain good representations. We consistently use the ResNet-18 backbone pretrained with MOCOv2 [4] for all baselines as well as HUME. During the second stage these methods perform clustering on top of the frozen representations and produce reliable pseudo-labels for the third stage. Finally, the third stage involves updating the entire network using generated pseudo-labels. Thus, pseudo-labels are produced using a clustering algorithm from the second step and third stage is compatible with applying any semi-supervised method (SSL) [37, 38] on the set of reliable samples. Instead of optimizing for performance of different SSL methods which is out-of-scope of this work, we compare clustering performance of different methods and report the accuracy of generated pseudo-labels which is a crucial component that enables SSL methods to be effectively applied. Additionally, we utilize the recent state-of-the-art SSL method FreeMatch [39] to study the performance of FreeMatch when applied to HUME's reliable samples. As additional unsupervised baselines, we include results of K-means clustering [15] on top of the corresponding representations and K-means clustering on top of concatenated embeddings from both representation spaces used by HUME. For stability, all K-means results are averaged over 100 runs for each experiment. On the ImageNet-1000 dataset, we compare HUME to the recent state-of-the-art deep clustering methods on this benchmark. Namely, in addition to SCAN, we also consider two single-stage methods, TWIST [40] and Self-classifier [41]. TWIST is trained from scratch by enforcing consistency between the class distributions produced by a siamese network given two augmented views of an image. Self-classifier is trained in the similar fashion as TWIST, but differs in a way of avoiding degenerate solutions, i.e., assigning all samples to a single class. HUME can be trained in inductive and transductive settings: inductive corresponds to training on the train split and evaluating on the held-out test split, while transductive corresponds to training on both train and test splits. Note that even in transductive setting evaluation is performed on the test split of the corresponding dataset to be comparable to the performance in the inductive setting. We report transductive and inductive performance of HUME when comparing it to the performance of the supervised classifier. For consistency with the prior work [13], we evaluate clustering baselines in the inductive setting.
Implementation details. For each experiment we independently run HUME with 100 different random seeds and obtain 100 different labelings. To compute the labeling agreement for the evaluation, we simply use the Hungarian algorithm to match all found labelings to the labeling with the highest cross-validation accuracy (HUME's objective). Finally, we aggregate obtained labelings using majority vote, i.e., the sample has class  $i$  if the majority of the found labelings predicts class  $i$ . We show experiments with different aggregation strategies in Section 3.2. We set the regularization parameter  $\eta$  to 10 in all experiments. We show robustness to this hyperparameter in Appendix B. We provide other implementation details in Appendix A. Code is publicly available at https://github.com/mlbio-epfl/hume.