SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Renchunzi Xie $^{1}$  Hongxin Wei $^{2*}$  Lei Feng $^{1}$  Yuzhou Cao $^{1}$  Bo An $^{1}$
$^{1}$  School of Computer Science and Engineering, Nanyang Technological University, Singapore  
 $^{2}$  Department of Statistics and Data Science, Southern University of Science and Technology, China
{xier0002,yuzhou002}@e.ntu.edu.sg
weihx@sustech.edu.cn
lfengqaq@gmail.com
boan@ntu.edu.sg
Estimating the generalization performance is practically challenging on out-of-distribution (OOD) data without ground-truth labels. While previous methods emphasize the connection between distribution difference and OOD accuracy, we show that a large domain gap not necessarily leads to a low test accuracy. In this paper, we investigate this problem from the perspective of feature separability empirically and theoretically. Specifically, we propose a dataset-level score based upon feature dispersion to estimate the test accuracy under distribution shift. Our method is inspired by desirable properties of features in representation learning: high inter-class dispersion and high intra-class compactness. Our analysis shows that inter-class dispersion is strongly correlated with the model accuracy, while intra-class compactness does not reflect the generalization performance on OOD data. Extensive experiments demonstrate the superiority of our method in both prediction performance and computational efficiency.
Machine learning techniques deployed in the open world often struggle with distribution shifts, where the test data are not drawn from the training distribution. Such issues can substantially degrade the test accuracy, and the generalization performance of a trained model may vary significantly on different shifted datasets [Quinonero-Candela et al., 2008, Koh et al., 2021]. This gives rise to the importance of estimating out-of-distribution (OOD) errors for AI Safety [Deng and Zheng, 2021]. However, it is prohibitively expensive or unrealistic to collect large-scale labeled examples for each shifted testing distribution encountered in the wild. Subsequently, predicting OOD error becomes a challenging task without access to ground-truth labels.
In the literature, a popular direction in predicting OOD error is to utilize the model output on the shifted dataset [Jiang et al., 2021, Guillory et al., 2021, Garg et al., 2022], which heavily relies on the model calibration. Yet, machine learning models generally suffer from the overconfidence issue even for OOD inputs [Wei et al., 2022], leading to suboptimal performance in estimating OOD performance. Many prior works turned to measuring the distribution difference between training and OOD test set, due to the conventional wisdom that a higher distribution shift normally leads to lower OOD accuracy. AutoEval [Deng and Zheng, 2021] applies Fréchet Distance to calculate the distribution distance for model evaluation under distribution shift. A recent work [Yu et al., 2022]
introduces Projection Norm (ProjNorm) metric to measure the distribution discrepancy in network parameters. However, we find that the connection between distribution distance and generalization performance does not always hold, making these surrogate methods to be questionable. This motivates our method, which directly estimates OOD accuracy based on the feature properties of test instances.
In this work, we show that feature separability is strongly associated with test accuracy, even in the presence of distribution shifts. Theoretically, we demonstrate that the upper bound of Bayes error is negatively correlated with inter-class feature distances. To quantify the feature separability, we introduce a simple dataset-level statistic, Dispersion Score, which gauges the inter-class divergence from feature representations, i.e., outputs of feature extractor. Our method is motivated by the desirable properties of embeddings in representation learning [Bengio et al., 2013]. To achieve high accuracy, we generally desire embeddings where different classes are relatively far apart (i.e., high inter-class dispersion), and samples in each class form a compact cluster (i.e., high intra-class compactness). Surprisingly, our analysis shows that intra-class compactness does not reflect the generalization performance, while inter-class dispersion is strongly correlated with the model accuracy on OOD data.
Extensive experiments demonstrate the superiority of Dispersion Score over existing methods for estimating OOD error. First, our method dramatically outperforms existing training-free methods in evaluating model performance on OOD data. For example, our method leads to an increase of the  $R^2$  from 0.847 to 0.970 on TinyImageNet-C [Hendrycks and Dietterich, 2019] – a  $14.5\%$  of relative improvement. Compared to the recent ProjNorm method [Yu et al., 2022], our method not only achieves superior performance by a meaningful margin, but also maintains huge advantages in computational efficiency and sample efficiency. For example, using CIFAR-10C dataset as OOD data, Dispersion Score achieves an  $R^2$  of 0.972, outperforming that of ProjNorm (i.e., 0.947), while our approach only takes around  $3\%$  of the time consumed by ProjNorm.
Overall, using Dispersion Score achieves strong performance in OOD error estimation with high computational efficiency. Our method can be easily adopted in practice. It is straightforward to implement with deep learning models and does not require access to training data. Thus our method is compatible with modern settings where models are trained on billions of images.
We summarize our main contribution as follows:
1. We find that the correlation between distribution distance and generalization performance does not always hold, downgrading the reliability of existing distance-based methods.
2. Our study provides empirical evidence supporting a significant association between feature separability and test accuracy. Furthermore, we theoretically show that increasing the feature distance will result in a decrease in the upper bound of Bayes error.
3. We propose a simple dataset-level score that gauges the inter-class dispersion from feature representations, i.e., outputs of feature extractor. Our method does not rely on the information of training data and exhibits stronger flexibility in OOD test data.
4. We conduct extensive evaluations to show the superiority of the Dispersion Score in both prediction performance and computational efficiency. Our analysis shows that Dispersion Score is more robust to various data conditions in OOD data, such as limited sample size, class imbalance and partial label set. Besides, we show that intra-class compactness does not reflect the generalization performance under distribution shifts (SubSection 4.4).
2.1 Preliminaries: OOD performance estimation
Setup In this work, we consider multi-class classification task with  $k$  classes. We denote the input space as  $\mathcal{X}$  and the label space as  $\mathcal{Y} = \{1,\dots ,k\}$ . We assume there is a training dataset  $\mathcal{D} = \{\pmb {x}_i,y_i\}_{i = 1}^n$ , where the  $n$  data points are sampled i.i.d. from a joint data distribution  $\mathcal{P}_{\mathcal{X}\mathcal{Y}}$ . During training, we learn a neural network  $f:\mathcal{X}\to \mathbb{R}^{k}$  with trainable parameters  $\theta \in \mathbb{R}^p$  on  $\mathcal{D}$ .
In particular, the neural network  $f$  can be viewed as a combination of a feature extractor  $f_{g}$  and a classifier  $f_{\omega}$ , where  $g$  and  $\omega$  denote the parameters of the corresponding parts, respectively. The feature extractor  $f_{g}$  is a function that maps instances to features  $f_{g}: \mathcal{X} \to \mathcal{Z}$ , where  $\mathcal{Z}$  denotes the
feature space. We denote by  $z_{i}$  the learned feature of instance  $x_{i}$ :  $z_{i} = f_{g}(x_{i})$ . The classifier  $f_{\omega}$  is a function from the feature space  $\mathcal{Z}$  to  $\mathbb{R}^k$ , which outputs the final predictions. A trained model can be obtained by minimizing the following expected risk:
$$
\begin{array}{l} \mathcal {R} _ {\mathcal {L}} (f) = \mathbb {E} _ {(\boldsymbol {x}, y) \sim \mathcal {P} _ {\mathcal {X} \mathcal {Y}}} [ \mathcal {L} (f (\boldsymbol {x}; \theta), y) ] \\ = \mathbb {E} _ {(\boldsymbol {x}, y) \sim \mathcal {P} _ {\boldsymbol {x y}}} \left[ \mathcal {L} \left(f _ {\omega} (f _ {g} (\boldsymbol {x})), y\right) \right] \\ \end{array}
$$
Problem statement At test time, we generally expect that the test data are drawn from the same distribution as the training dataset. However, distribution shifts usually happen in reality and even simple shifts can lead to large drops in performance, which makes it unreliable in safety-critical applications. Thus, our goal is to estimate how a trained model might perform on the shifted data without labels, i.e., unlabeled out-of-distribution (OOD) data.
Assume that  $\tilde{\mathcal{D}} = \{\tilde{x}_i\}_{i=1}^m$  be the OOD test dataset and  $\{\tilde{y}_i\}_{i=1}^m$  be the corresponding unobserved labels. For a certain test instance  $\tilde{x}_i$ , we obtain the predicted labels of a trained model by  $\tilde{y}_i' = \arg \max f(\tilde{x}_i)$ . Then the ground-truth test error on OOD data can be formally defined as:
$$
\operatorname {E r r} (\tilde {\mathcal {D}}) = \frac {1}{m} \sum_ {i = 1} ^ {m} \mathbb {1} \left(\tilde {y} _ {i} ^ {\prime} \neq \tilde {y} _ {i}\right), \tag {1}
$$
To estimate the real OOD error, the key challenge is to formulate a score  $S(\tilde{\mathcal{D}})$  that is strongly correlated with the test error across diverse distribution shifts without utilization of corresponding test labels. With such scores, a simple linear regression model can be learned to estimate the test error on shifted datasets, following the commonly used scheme [Deng and Zheng, 2021, Yu et al., 2022].
While those output-based approaches suffer from the overconfidence issue [Hendrycks and Gimpel, 2016, Guillory et al., 2021], other methods primarily rely on the distribution distance between training data  $\mathcal{D}$  and test data  $\hat{\mathcal{D}}$  [Deng and Zheng, 2021, Tzeng et al., 2017], with the intuition that the distribution gap impacts classification accuracy. In the following, we motivate our method by analyzing the failure of those methods based on feature-level distribution distance.
2.2 The failure of distribution distance
In the literature, distribution discrepancy has been considered as a key metric to predict the generalization performance of the model on unseen datasets [Deng and Zheng, 2021, Tzeng et al., 2017, Gao and Kleywegt, 2022, Sinha et al., 2017, Yu et al., 2022]. AutoEval [Deng and Zheng, 2021] estimates model performance by quantifying the domain gap in the feature space:
$$
S (\mathcal {D}, \tilde {\mathcal {D}}) = d (\mathcal {D}, \tilde {\mathcal {D}}),
$$
where  $d(\cdot)$  denotes the distance function, such as Fréchet Distance [Dowson and Landau, 1982] or maximum mean discrepancy (MMD) [Gretton et al., 2006].
ProjNorm measures the distribution gap with the Euclidean distance in the parameter space:
$$
S (\mathcal {D}, \tilde {\mathcal {D}}) = \| \theta - \tilde {\theta} \| _ {2},
$$
where  $\theta$  and  $\tilde{\theta}$  denote the parameters fine-tuned on training data  $\mathcal{D}$  and OOD data  $\tilde{\mathcal{D}}$ , respectively.
The underlying assumption is inherited from the conventional wisdom in domain adaptation, where a representation function that minimizes domain difference leads to higher accuracy in the target domain [Tzeng et al., 2014, Ganin and Lempitsky, 2015, Tzeng et al., 2017]. Yet, the relationship between distribution distance and test accuracy remains controversial. For example, the distribution shift may not change the model prediction if the classifier's outputs are insensitive to changes in the shifted features. There naturally arises a question: given a fixed model, does a larger distribution distance always lead to a lower test accuracy?
To verify the correlation between distribution distance and model performance, we compare the model performance on different OOD test sets of CIFAR-10C, using the ResNet-50 model trained on CIFAR-10. For each OOD test set, we calculate the distribution distances in the feature space  $\mathcal{Z}$  via Fréchet distance [Dowson and Landau, 1982] and MMD [Gretton et al., 2006], respectively.
(a) Fréchet
(b) MMD
Figure 1: Distribution Gap Vs. OOD accuracy on CIFAR10-C via (a) Fréchet distance [Dowson and Landau, 1982] and (b) MMD [Gretton et al., 2006]. All colors indicate 14 types of corruption. The test accuracy varies significantly on shifted datasets with the same distribution distances.
Figure 1 presents the classification accuracy versus the distribution distance. The results show that, on different test datasets with similar distances to the train data in the feature space, the performance of the trained model varies significantly with both the two distance metrics. For example, calculated by Fréchet distance, the distribution gap varies only a small margin from 223.36 to 224.04, but the true accuracy experiences a large drop from  $61.06\%$  to  $46.99\%$ . Similar to MMD distance, when the distribution gap changes from 87.63 to 88.35, the OOD accuracy drops from  $66.53\%$  to  $47.73\%$ . The high variability in the model performance reveals that, the distribution difference is not a reliable surrogate for generalization performance under distribution shifts.
In this section, we first introduce the intuition of our method with a natural example. Next, we characterize and provide quantitative measure on the desirable properties of feature representations for predicting OOD error. We then present the advantages of our proposed method.
In representation learning, it is desirable to learn a separable feature space, where different classes are far away from each other and samples in each class form a compact cluster. Therefore, separability is viewed as an important characteristic to measure the feature quality. For example, one may train a nearest neighbor classifier over the learned representation using labeled data and regard its performance as the indicator. In this paper, we investigate how to utilize the characteristic of separability for estimating the model performance under distribution shifts, without access to ground-truth labels.
Intuitive example In Figure 2, we present a t-SNE visualization of the representation for training and test data. In particular, we compare the separability of the learned representation on shifted datasets with various severity. While the feature representation of training data exhibits well-separated clusters, those of the shifted datasets are more difficult to differentiate and the cluster gaps are well correlated with the corruption severity, as well as the ground-truth accuracy. It implies that, a model with high classification accuracy is usually along with a well-separated feature distribution, and vice versa. With the intuition, we proceed by theoretically showing how the feature separability affects the classification performance.
Theoretical explanation Recall that we obtain the model prediction by  $y_{i}^{\prime} = \arg \max f_{\omega}(\pmb{z}_{i})$ , where the intermediate feature  $\pmb{z}_{i} = f_{g}(\pmb{x}_{i})$ . Let  $P(y = i)$  denote the prior class probability of class  $i$ , and  $p(\pmb{x}|y = i)$  denote the class likelihood, i.e., the conditional probability density of  $x$  given that it belongs to class  $i$ . Then the Bayes error [Young and Calvert, 1974, Devijver and Kittler, 1982, Webb and Copsey, 1990, Duda et al., 2006] can be expressed as:
Figure 2: t-SNE visualization of feature representation on training and OOD test datasets (CIFAR-10C) with contrast corruption under different severity levels. The first left figure shows feature representation of the training dataset, while the rest of three figures illustrate those of the OOD datasets. From this figure, we observe that different clusters tend to be more separated as the corruption severity level gets smaller.
$$
E _ {\text {b a y e s}} = 1 - \sum_ {i = 1} ^ {k} \int_ {C _ {i}} P (y = i) p (\boldsymbol {x} \mid y = i) d \boldsymbol {x},
$$
where  $C_i$  is the region where class  $i$  has the highest posterior. The Bayes error rate provides a lower bound on the error rate that can be achieved by any pattern classifier acting on derived features. However, the Bayes error is not so readily obtainable as class priors and class-conditional likelihoods are normally unknown. Therefore, many studies have focused on deriving meaningful upper bounds for the Bayes error [Devijver and Kittler, 1982, Webb and Copsey, 1990].
For a binary classification problem where  $k = 2$ , we have
$$
E _ {\text {b a y e s}} \leq \frac {2 P (y = 1) P (y = 2)}{1 + P (y = 1) P (y = 2) \Delta}, \tag {2}
$$
where  $\Delta$  denotes the distance between features from the two classes [Devijver and Kittler, 1982, Tumer and Ghosh, 2003], such as Mahalanobis distance [Chandra et al., 1936] or Bhattacharyya distance [Fukunaga, 2013]. Based on the Bayes error for 2-class problems, the upper bound can be extended to the multi-class setting ( $k > 2$ ) by the following equation [Garber and Djouadi, 1988].
$$
E _ {\text {b a y e s}} ^ {k} \leq \min  _ {\alpha \in \{0, 1 \}} \left(\frac {1}{k - 2 \alpha} \sum_ {i = 1} ^ {k} (1 - P (y = i)) E _ {\text {b a y e s}; i} ^ {k - 1} + \frac {1 - \alpha}{k - 2 \alpha}\right), \tag {3}
$$
where  $\alpha$  is an optimization parameter. With Equations 2 and 3, we obtain an upper bound of Bayes error, which is negatively correlated with the inter-class feature distances. Specifically, increasing the feature distance will result in a decrease in the upper bound of Bayes error. In this manner, we provide a mathematical intuition for the phenomenon shown in Figure 2. However, Computing the upper bound of Bayes error directly, as indicated above, is challenging due to its computational complexity. To circumvent this issue, we propose a straightforward metric as a substitute, which does not require access to label information.
3.2 Dispersion score
In our previous analysis, we show a strong correlation between the test accuracy and feature separability under different distribution shifts. To quantify the separability in the feature space  $\mathcal{Z}$ , we introduce Dispersion score that measures the inter-class margin without annotation information.
First, we allocate OOD instances  $\{\tilde{x}\}_{i=1}^{m}$  into different clusters  $j$  based on the model predictions, i.e., their pseudo labels from the trained classifier  $f_{\omega}$ :  $j = \tilde{y}_i' = \arg \max f_{\omega}(z_i)$ .
With these clusters, we compute the Dispersion score by the average distances between each cluster centroid  $\tilde{\mu}_j = \frac{1}{m_j}\sum_{i=1}^{m_j}z_i\cdot\mathbb{1}\{\tilde{y}_i' = j\}$  and the center of all features  $\bar{\mu} = \frac{1}{m}\sum_{i=1}^{m}z_i$ , weighted by the sample size of each cluster  $m_j$ . Formally, the Dispersion score is defined as:
$$
S (\tilde {\mathcal {D}}) = \frac {1}{k - 1} \sum_ {j = 1} ^ {k} m _ {j} \cdot \varphi (\bar {\boldsymbol {\mu}}, \tilde {\boldsymbol {\mu}} _ {j})
$$
where  $k - 1$  is the degree of freedom and  $\varphi$  denotes the distance function. With the weight  $m_{j}$ , the induced score receives a stronger influence from those larger clusters, i.e., the majority classes. This enables our method to be more robust to the long-tailed issue, which naturally arises in the unlabeled OOD data. In subsection 4.3, we explicitly show the advantage of our method in the class-imbalanced case and analyze the importance of the weight in Appendix 4.5.
In particular, we use square Euclidean distances to calculate the distance in the feature space  $\mathcal{Z}$ . So it converts to:
$$
S (\tilde {\mathcal {D}}) = \log \frac {\sum_ {j = 1} ^ {k} m _ {j} \cdot \| \bar {\boldsymbol {\mu}} - \tilde {\boldsymbol {\mu}} _ {j} \| _ {2} ^ {2}}{k - 1} \tag {4}
$$
Following the common practice [Jiang et al., 2019], we adopt a log transform on the final score, which corresponds to multiplicative combination of class statistics.
We summarize our approach in Appendix A. Notably, the Dispersion score derived from the feature separability offers several compelling advantages:
1. Training data free. The calculation procedure of our proposed score does not rely on the information of training data. Thus, our method is compatible with modern settings where models are trained on billions of images.
2. Easy-to-use. The computation of the Dispersion score only does forward propagation for each test instance once and does not require extra hyperparameter, training a new model, or updating the model parameters. Therefore, our method is easy to implement in real-world tasks and computational efficient, as demonstrated in Tables 2 and 3.
3. Strong flexibility in OOD data. Previous state-of-the-art methods, like ProjNorm [Yu et al., 2022], usually requires a large amount of OOD data for predicting the prediction performance. Besides, the class distribution of unlabeled OOD datasets might be severely imbalanced, which makes it challenging to estimate the desired test accuracy on balanced data. In Subsection 4.3, we will show the Dispersion score derived from the feature separability exhibits stronger flexibility and generality in sample size, class distribution, and partial label set (see Subsection 4.3 and Appendix E).
In this subsection, we first compare the proposed score to existing training-free methods. Then, we provide an extensive comparison between our method and recent state-of-the-art method - ProjNorm. We then show the flexibility of our method under different settings of OOD test data. Additionally, we present an analysis of using ground-truth labels and K-means in Appendixes C and 4.6. Finally, we discuss the limitation of the proposed score for adversarial setting in Appendix F.
Train datasets. During training, we train models on the CIFAR-10, CIFAR-100 [Krizhevsky et al., 2009] and TinyImageNet [Le and Yang, 2015] datasets. Specifically, the train data of CIFAR-10 and CIFAR-100 contain 50,000 training images, which are allocated to 10 and 100 classes, respectively. The TinyImageNet dataset contains 100,000  $64 \times 64$  training images, with 200 classes.
Out-of-distribution (OOD) datasets. To evaluate the effectiveness of the proposed method on predicting OOD error at test time, we use CIFAR-10C and CIFAR-100C [Hendrycks and Dietterich, 2019], which span 19 types of corruption with 5 severity levels. For the testing of TinyImageNet, we use TinyImageNet-C [Hendrycks and Dietterich, 2019] that spans 15 types of corruption with 5 severity levels as OOD dataset. All the datasets with certain corruption and severity contain 10,000 images, which are evenly distributed in the classes.
Evaluation metrics. To measure the linear relationship between OOD error and designed scores, we use coefficients of determination  $(R^2)$  and Spearman correlation coefficients  $(\rho)$  as the evaluation metrics. For the comparison of computational efficiency, we calculate the average evaluation time  $(T)$  for each test set with certain corruption and severity.
Training details. During the training process, we train ResNet18, ResNet50 [He et al., 2016] and WRN-50-2 [Zagoruyko and Komodakis, 2016] on CIFAR-10, CIFAR-100 [Krizhevsky et al., 2009] and TinyImageNet [Le and Yang, 2015] with 20, 50 and 50 epochs, respectively. We use SGD with