SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peng Cui $^{15}$ , Dan Zhang $^{23}$ , Zhijie Deng $^{4*}$ , Yinpeng Dong $^{15}$ , Jun Zhu $^{15*}$ $^{1}$ Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Bosch Joint ML Center, Tsinghua University, Beijing, 100084 China  
 $^{2}$ Bosch Center for Artificial Intelligence  $^{3}$ University of Tübingen  
 $^{4}$ Qing Yuan Research Institute, Shanghai Jiao Tong University  $^{5}$ RealAI xpeng.cui@gmail.com, dan.zhang2@de.bosch.com, zhijied@sjtu.edu.cn dongyinpeng@tsinghua.edu.cn, dcszj@tsinghua.edu.cn
Large-scale pre-trained models have achieved remarkable success in many applications, but how to leverage them to improve the prediction reliability of downstream models is undesirably under-explored. Moreover, modern neural networks have been found to be poorly calibrated and make overconfident predictions regardless of inherent sample difficulty and data uncertainty. To address this issue, we propose to utilize large-scale pre-trained models to guide downstream model training with sample difficulty-aware entropy regularization. Pre-trained models that have been exposed to large-scale datasets and do not overfit the downstream training classes enable us to measure each training sample's difficulty via feature-space Gaussian modeling and relative Mahalanobis distance computation. Importantly, by adaptively penalizing overconfident prediction based on the sample difficulty, we simultaneously improve accuracy and uncertainty calibration across challenging benchmarks (e.g.,  $+0.55\%$  ACC and  $-3.7\%$  ECE on ImageNet1k using ResNet34), consistently surpassing competitive baselines for reliable prediction. The improved uncertainty estimate further improves selective classification (abstaining from erroneous predictions) and out-of-distribution detection.
Large-scale pre-training has witnessed pragmatic success in diverse scenarios and pre-trained models are becoming increasingly accessible [9, 4, 3, 14, 44]. The community has reached a consensus that by exploiting big data, pre-trained models can learn to encode rich data semantics that is promised to be generally beneficial for a broad spectrum of applications, e.g., warming up the learning on downstream tasks with limited data [45], improving domain generalization [24] or model robustness [19], and enabling zero-shot transfer [54, 43].
This paper investigates on a new application – leveraging pre-trained models to improve the calibration and the quality of uncertainty quantification of downstream models, both of which are crucial for reliable model deployment in the wild [20, 13, 28, 32]. Model training on the task dataset often encounters ambiguous or even distorted samples. These samples are difficult to learn from – directly enforcing the model to fit them (i.e., matching the label with  $100\%$  confidence) may cause undesirable memorization and overconfidence. This issue comes from loss formulation and data annotation, thus using pre-trained models for finetuning or knowledge distillation with the cross-entropy loss will not solve the problem. That said, it is necessary to subtly adapt the training objective for each sample according to its sample difficulty for better generalization and uncertainty quantification. We exploit
Figure 1: Visualization of training samples with low (high) sample difficulty that is scored by CLIP-ViT-B [44] using the derived relative Mahalanobis distance (RMD).
pre-trained models to measure the difficulty of each sample in the downstream training set and use this extra data annotation to modify the training loss. Although sample difficulty has been shown to be beneficial to improve training efficiency [33] and neural scaling laws [49], its effectiveness for ameliorating model reliability is largely under-explored.
Pre-trained models aid in scoring sample difficulty by shifting the problem from the raw data space to a task- and model-agnostic feature space, where simple distance measures suffice to represent similarities. Besides, large-scale multi-modal datasets and self-supervised learning principles enable the pre-trained models (see those in Table 1) to generate features that sufficiently preserve high-level concepts behind the data and avoid overfitting to specific data or classes. In light of this, we perform sample difficulty estimation in the feature space of pre-trained models and cast it as a density estimation problem since samples with typical discriminative features are easier to learn and typical features shall reappear. We advocate using Gaussian models to represent the training data distribution in the feature space of pre-trained models and derive the relative Mahalanobis distance (RMD) as a sample difficulty score. As shown in Fig. 1, there is a high-level agreement on sample difficulty between RMD and human cognition.
Equipped with the knowledge of sample difficulty, we further propose to use it for regularizing the prediction confidence. The standard cross entropy loss with a single ground truth label would encourage the model to predict all instances in Fig. 1-a) as the class Tusker with  $100\%$  confidence. However, such high confidence is definitely unjustified for the hard samples. Thus, we modify the training loss by adding an entropic regularizer with an instance-wise adaptive weighting in proportion to the sample difficulty. Profiting from the high-quality sample difficulty measure and the effective entropic regularizer, we develop a strong, scalable, and easy-to-implement approach to improving both the predictive accuracy and calibration of the model.
Our method successfully improves the model's performance and consistently outperforms competitive baselines on various image classification benchmarks, ranging from the i.i.d. setting to corruption robustness, selective classification, and out-of-distribution detection. Importantly, unlike previous works that compromise accuracy [22, 31] or suffer from expensive computational overhead [26], our method can improve predictive performance and uncertainty quantification concurrently in a computationally efficient way. The consistent gains across architectures demonstrate that our sample difficulty measure is a valuable characteristic of the dataset for training.
Uncertainty regularization. A surge of research has focused on uncertainty regularization to alleviate the overfitting and overconfidence of deep neural networks.  $L_{p}$  Norm [22] and entropy regularization (ER) [42] explicitly enforce a small norm of the logits or a high predictive entropy. Label smoothing (LS) [36] interpolates the ground-truth one-hot label vector with an all-one vector, thus penalizing  $100\%$  confident predictions. [35] showed that focal loss (FL) [30] is effectively an upper bound to ER. Correctness ranking loss (CRL) [34] regularizes the confidence based on the frequency of correct predictions during the training losses.
LS,  $L_{p}$  Norm, FL and ER do not adjust prediction confidence based on the sample difficulty. The frequency of correct predictions was interpreted as a sample difficulty measure [50]. However, CRL only concerns pair-wise ranking in the same batch. Compared to the prior art, we weight the overconfidence penalty according to the sample difficulty, which proves to be more effective at jointly improving accuracy and uncertainty quantification. Importantly, the sample difficulty is derived from a data distribution perspective, and remains constant over training.
Sample difficulty measurement. Prior works measure the sample difficulty by only considering the task-specific data distribution and training model, e.g., [50, 21, 40, 2, 1, 41, 49, 11, 33]. As deep neural networks are prone to overfitting, they often require careful selection of training epochs, checkpoints, data splits and ensembling strategies. Leveraging the measurement for data pruning, [49] showed existing work still underperforms on large datasets, e.g., ImageNet1k. The top solution used a self-supervised model and scored the sample difficulty via K-means clustering.
We explore pre-trained models that have been exposed to large-scale datasets and do not overfit the downstream training set. Their witness of diverse data serves as informative prior knowledge for ranking downstream training samples. Furthermore, the attained sample difficulty can be readily used for different downstream tasks and models. In this work, we focus on improving prediction reliability, which is an under-explored application of large-scale pre-trained models. The derived relative Mahalanobis distance (RMD) outperforms Mahalanobis distance (MD) and K-means clustering for sample difficulty quantification.
3 Sample Difficulty Quantification
Due to the ubiquity of data uncertainty in the dataset collected from the open world [23, 6], the sample difficulty quantification (i.e., characterizing the hardness and noisiness of samples) is pivotal for reliable learning of the model. For measuring training sample difficulty, we propose to model the data distribution in the feature space of a pre-trained model and derive a relative Mahalanobis distance (RMD). A small RMD implies that 1) the sample is typical and carries class-discriminative features (close to the class-specific mean mode but far away from the class-agnostic mean mode), and 2) there exist many similar samples (high-density area) in the training set. Such a sample represents an easy case to learn, i.e., small RMD  $\leftrightarrow$  low sample difficulty, see Fig. 1.
3.1 Large-scale pre-trained models
Large-scale image and image-text data have led to high-quality pre-trained vision models for downstream tasks. Instead of using them as the backbone network for downstream tasks, we propose a new use case, i.e., scoring the sample difficulty in the training set of the downstream task. There is no rigorously defined notion of sample difficulty. Intuitively, easy-to-learn samples shall reappear in the form of showing similar patterns. Repetitive patterns specific to each class are valuable cues for classification. Moreover, they contain neither confusing nor conflicting information. Single-label images containing multiple salient objects belonging to different classes or having wrong labels would be hard samples.
To quantify the difficulty of each sample, we propose to model the training data distribution in the feature space of large-scale pre-trained models. In the pixel space, data distribution modeling is prone to overfitting low-level features, e.g., an outlier sample with smoother local correlation can have a higher probability than an inlier sample [48]. On the other hand, pre-trained models are generally trained to ignore low-level information, e.g., semantic supervision from natural language or class labels. In the case of self-supervised learning, the proxy task and loss are also formulated to learn a holistic understanding of the input images beyond low-level image statistics, e.g., the masking strategy designed in MAE [14] prevents reconstruction via exploiting local correlation. Furthermore, as modern pre-trained models are trained on large-scale datasets with high sample diversities in many dimensions, they learn to preserve and structure richer semantic features of the training samples than models only exposed to the training set that is commonly used at a smaller scale. In the feature space of pre-trained models, we expect easy-to-learn samples will be closely crowded together. Hard-to-learn ones are far away from the population and even sparsely spread due to missing consistently repetitive patterns. From a data distribution perspective, the easy (hard)-to-learn samples should have high (low) probability values.
3.2 Measuring sample difficulty
While not limited to it, we choose the Gaussian distribution to model the training data distribution in the feature space of pre-trained models, and find it simple and effective. We further derive a relative Mahalanobis distance (RMD) from it as the sample difficulty score.
Class-conditional and agnostic Gaussian distributions. The downstream training set  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$  is a collection of image-label pairs, with  $x_i \in \mathbb{R}^d$  and  $y_i \in \{1, 2, \dots, K\}$  as the image and its label, respectively. We model the feature distribution of  $\{x_i\}$  with and without conditioning on the class information. Let  $G(\cdot)$  denote the penultimate layer output of the pre-trained model  $G$ . The class-conditional distribution is modelled by fitting a Gaussian model to the feature vectors  $\{G(x_i)\}$  belonging to the same class  $y_i = k$
$$
P (G (x) \mid y = k) = \mathcal {N} (G (x) \mid \mu_ {k}, \Sigma), \tag {1}
$$
$$
\mu_ {k} = \frac {1}{N _ {k}} \sum_ {i: y _ {i} = k} G \left(x _ {i}\right), \quad \Sigma = \frac {1}{N} \sum_ {k} \sum_ {i: y _ {i} = k} \left(G \left(x _ {i}\right) - \mu_ {k}\right) \left(G \left(x _ {i}\right) - \mu_ {k}\right) ^ {\top}, \tag {2}
$$
where the mean vector  $\mu_{k}$  is class-specific, the covariance matrix  $\Sigma$  is averaged over all classes to avoid under-fitting following [46, 27], and  $N_{k}$  is the number of training samples with the label  $y_{i} = k$ . The class-agnostic distribution is obtained by fitting to all feature vectors regardless of their classes
$$
P (G (x)) = \mathcal {N} (G (x) \mid \mu_ {\mathrm {a g n}}, \Sigma_ {\mathrm {a g n}}), \tag {3}
$$
$$
\mu_ {\mathrm {a g n}} = \frac {1}{N} \sum_ {i} ^ {N} G \left(x _ {i}\right), \quad \Sigma_ {\mathrm {a g n}} = \frac {1}{N} \sum_ {i} ^ {N} \left(G \left(x _ {i}\right) - \mu_ {\mathrm {a g n}}\right) \left(G \left(x _ {i}\right) - \mu_ {\mathrm {a g n}}\right) ^ {\top}. \tag {4}
$$
Relative Mahalanobis distance (RMD). For scoring sample difficulty, we propose to use the difference between the Mahalanobis distances respectively induced by the class-specific and class-agnostic Gaussian distribution in (1) and (3), and boiling down to
$$
\mathcal {R M} \left(x _ {i}, y _ {i}\right) = \mathcal {M} \left(x _ {i}, y _ {i}\right) - \mathcal {M} _ {\mathrm {a g n}} \left(x _ {i}\right), \tag {5}
$$
$$
\mathcal {M} \left(x _ {i}, y _ {i}\right) = - \left(G \left(x _ {i}\right) - \mu_ {y _ {i}}\right) ^ {\top} \Sigma^ {- 1} \left(G \left(x _ {i}\right) - \mu_ {y _ {i}}\right), \tag {6}
$$
$$
\mathcal {M} _ {\mathrm {a g n}} \left(x _ {i}\right) = - \left(G \left(x _ {i}\right) - \mu_ {\mathrm {a g n}}\right) ^ {\top} \Sigma_ {\mathrm {a g n}} ^ {- 1} \left(G \left(x _ {i}\right) - \mu_ {\mathrm {a g n}}\right). \tag {7}
$$
Previous work [46] utilized RMD for OOD detection at test time, yet we explore it to rank each sample and measure the sample difficulty in the training set.
A small class-conditional MD  $\mathcal{M}(x_i, y_i)$  indicates that the sample exhibits typical features of the sub-population (training samples from the same class). However, some features may not be unique to the sub-population, i.e., common features across classes, yielding a small class-agnostic MD  $\mathcal{M}_{\mathrm{agn}}(x_i)$ . Since discriminative features are more valuable for classification, an easier-to-learn sample should have small class-conditional MD but large class-agnostic MD. The derived RMD is thus an improvement over the class-conditional MD for measuring the sample difficulty, especially when we use pre-trained models that have no direct supervision for downstream classification. We refer to Appendix A for detailed comparisons.
How well RMD scores the sample difficulty. Qualitatively, Fig. 1a shows a high-level agreement between human visual perception and RMD-based sample difficulty, and see Appendix C for more visual examples. As shown, hard samples tend to be challenging to classify, as they miss the relevant information or contain ambiguous information. Furthermore, prior works [5, 6] also show that data uncertainty [8] should increase on poor-quality images. We manipulate the sample difficulty by corrupting the input image or alternating the label. Fig. 1b showed that the RMD score increases proportionally with the severity of corruption and label noise, which further shows the effectiveness of RMD in characterizing the hardness and noisiness of samples.
As there is no ground-truth annotation of sample difficulty, we construct the following proxy test for quantitative evaluation. Hard samples are more likely to be misclassified. We therefore use RMD to sort each ImageNet1k validation sample in the descending order of the sample difficulty and group them into subsets of equal size. Fig. 2 shows that an off-the-shelf ImageNet1k classifier (ResNet34 and standard training procedure) performed worst on the hardest data split, and its performance gradually improves as the data split difficulty reduces. This observation indicates an agreement between ResNet34 and RMD regarding what samples are hard and easy. Such an observation also holds for different architectures, see Fig. 7 in Appendix B.
Table 1: Pre-trained Models. We list two variants of CLIP [44] that use ResNet-50 [15] and ViT-Base [10] as the image encoder respectively.
Model	Pre-trained dataset
CLIP-ViT-B	400M image-caption data [44]
CLIP-R50
ViT-B	14M ImageNet21k (w. labels) [47]
MAE-ViT-B	1.3M ImageNet1k (w./o. labels) [7]
DINOv2	142M image data [38]
We hypothesize that self-supervised learning (MAE-ViT-B) plays a positive role compared to supervised learning (ViT-B), as it does not overfit the training classes and instead learn a holistic understanding of the input images beyond low-level image statistics aided by a well-designed loss (e.g., the masking and reconstructing strategy in MAE). [49] also reported that a self-supervised model on ImageNet1k (i.e., SwAV) outperforms supervised models in selecting easy samples to prune for improved training efficiency. Nevertheless, the best-performing model in Fig. 2 is CLIP. The error rate of ResNet34 on the hardest data split rated by CLIP is close to  $90\%$ . CLIP learned rich visual concepts from language supervision (cross-modality) and a large amount of data. Thus, it noticeably outperforms the others, which have only seen ImageNet-like data.
Fig. 2 also compares four large-scale pre-trained models listed in Table 1 for computing RMD. Interestingly, while MAE-ViT-B [14] is only trained on ImageNet1k (not seeing more data than the downstream classification task), it performs better than ViT-B [10] trained on ImageNet21k.
Figure 2: Error rate achieved by ResNet34 (trained on ImageNet1k) on the validation subsets, which respectively contain 500 samples ranked from the  $a$ th to  $b$ th hardest. Four different pre-trained models are used for computing RMDs and ranking.
4 Difficulty-aware Uncertainty Regularization
While we can clearly observe that a dataset often consists of samples with diverse difficulty levels (e.g., Fig. 1a), the standard training loss formulation is sample difficulty agnostic. In this section, we propose a sample difficulty-aware entropy regularization to improve the training loss, which will be shown to yield more reliable predictions.
Let  $f_{\theta}$  denote the classifier parameterized by a deep neural network, which outputs a conditional probability distribution  $p_{\theta}(y|x)$ . Typically, we train  $f_{\theta}$  by minimizing the cross-entropy loss on the training set
$$
\ell = E _ {\left(x _ {i}, y _ {i}\right)} \left\{- \log \left(f _ {\theta} \left(x _ {i}\right) [ y _ {i} ]\right) \right\}, \tag {8}
$$
where  $f_{\theta}(x_i)[y_i]$  refers to the predictive probability of the ground-truth label  $y_{i}$ .
It is well known that deep neural networks trained with cross-entropy loss tend to make overconfident predictions. Besides the overfitting reason, it is also because the ground-truth label is typically a one-hot vector which represents the highest confidence regardless of the sample difficulty. Although different regularization-based methods [36, 22, 35] have been proposed to address the issue, they all ignore the difference between easy and hard samples and may assign an inappropriate regularizer for some samples. In view of this, we propose to regularize the cross-entropy loss with a sample difficulty-aware entropic regularizer
$$
\ell = E \left\{- \log \left(f _ {\theta} \left(x _ {i}\right) [ y _ {i} ]\right) - \alpha s \left(x _ {i}, y _ {i}\right) \mathcal {H} \left[ f _ {\theta} \left(x _ {i}\right) \right] \right\}, \tag {9}
$$
where  $\alpha$  is a hyper-parameter used to control the global strength of entropy regularization and the value range is generally in  $(0, 0.5)$ .  $s(x_i, y_i) \in (0, 1)$  is a normalized sample-specific weighting derived from the RMD-based sample difficulty score (5)
$$
s \left(x _ {i}, y _ {i}\right) = \frac {\exp \left(\mathcal {R M} \left(x _ {i} , y _ {i}\right) / T\right)}{\max  _ {i} \left\{\exp \left(\mathcal {R M} \left(x _ {i} , y _ {i}\right) / T \right\} + c \right.}. \tag {10}
$$
As the entropic regularizer encourages uncertain predictions, the sample-adaptive weighting  $s(x_{i},y_{i})$  ranging from 0 to 1, is positively proportional to the sample difficulty score  $\mathcal{RM}(x_i,y_i)$ , where the parameter  $c$  equals to a small constant (e.g., 1e-3) ensuring the value range (0, 1). The parameter  $T$  is adjustable to control the relative importance among all training data, Fig. 10 in Appendix shows the distribution of  $s(\cdot)$  under different  $T$ .
Implementation details of sample difficulty score. Because the difficulty level of a sample may vary depending on how data augmentation is applied, we conduct five random runs to fit Gaussian distributions to eliminate the potential effect. Moreover, we check how the standard augmentation affects the score in Eq. 10, where the numerator can be computed based on each sample with randomly chosen data augmentation, and denominator can be obtained by taking the maximum from the dataset we created to fit the Gaussians in the first step. We do not find data augmentation changes the score too much. Therefore, we can calculate per-sample difficulty score once before training for the sake of saving computing overhead and then incorporate it into the model training using Eq. 9.
We thoroughly compare related uncertainty regularization techniques and sample difficulty measures on various image classification benchmarks. We also verify if our sample difficulty measure conveys important information about the dataset useful to different downstream models. Besides accuracy and uncertainty calibration, the quality of uncertainty estimates is further evaluated on two proxy tasks, i.e., selective classification and out-of-distribution detection.
Datasets. CIFAR-10/100 [25], and ImageNet1k [7] are used for multi-class classification training and evaluation. ImageNet-C [17] is used for evaluating calibration under distribution shifts, including blur, noise, and weather-related 16 types of corruptions, at five levels of severity for each type. For OOD detection, in addition to CIFAR-100 vs. CIFAR-10, we use iNaturalist [51] as the near OOD data of ImageNet1k.
Implementation details. We use standard data augmentation (i.e., random horizontal flipping and cropping) and SGD with a weight decay of 1e-4 and a momentum of 0.9 for classification training, and report averaged results from five random runs. The default image classifier architecture is ResNet34 [15]. For the baselines, we use the same hyper-parameter setting as recommended in [52]. For the hyper-parameters in our training loss (9), we set  $\alpha$  as 0.3 and 0.2 for CIFARs and ImageNet1k, respectively, where  $T$  equals 0.7 for all datasets. CLIP-ViT-B [44] is utilized as the default pre-trained model for scoring the sample difficulty.
Evaluation metrics. To measure the prediction quality, we report accuracy and expected calibration error (ECE) [37] (equal-mass binning and 15 bin intervals) of the Top-1 prediction. For selective classification and OOD detection, we use three metrics: (1) False positive rate at  $95\%$  true positive rate (FPR-95\%; (2) Area under the receiver operating characteristic curve (AUROC); (3) Area under the precision-recall curve (AUPR).
5.1 Classification accuracy and calibration
Reliable prediction concerns not only the correctness of each prediction but also if the prediction confidence matches the ground-truth accuracy, i.e., calibration of Top-1 prediction.