Energy-based Automated Model Evaluation
Authors: Ru Peng
Heming Zou
Haobo Wang
Yawen Zeng
Zenan Huang
Junbo Zhao
Venue: ICLR 2024 poster (2024)
Keywords: Automated Model Evalutaion
Energy
Meta-distribution
Distribution shift
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We introduce a simple yet effective method (Meta-Distribution Energy) in predicting a model's generalization to previously unseen, unlabeled datasets with theoretical guarantees and achieves state-of-the-art performance.

Abstract: The conventional evaluation protocols on machine learning models rely heavily on a labeled, i.i.d-assumed testing dataset, which is not often present in real-world applications.
The Automated Model Evaluation (AutoEval) shows an alternative to this traditional workflow, by forming a proximal prediction pipeline of the testing performance without the presence of ground-truth labels.
Despite its recent successes, the AutoEval frameworks still suffer from an overconfidence issue, substantial storage and computational cost.
In that regard, we propose a novel measure --- Meta-Distribution Energy (MDE) that allows the AutoEval framework to be both more efficient and effective.
The core of the MDE is to establish a meta-distribution statistic, on the information (energy) associated with individual samples, then offer a smoother representation enabled by energy-based learning.
We further provide our theoretical insights by connecting the MDE with the classification loss.
We provide extensive experiments across modalities, datasets and different architectural backbones to validate MDE's validity, together with its superiority compared with prior approaches.
We also prove MDE's versatility by showing its seamless integration with large-scale models, and easy adaption to learning scenarios with noisy- or imbalanced- labels.

Published as a conference paper at ICLR 2024
ENERGY-BASED AUTOMATED MODEL EVALUATION
Ru Peng Heming Zou 1x Haobo Wang' Yawen Zeng² Zenan Huang Junbo Zhao
'Zhejiang University BByteDance
{rupeng, zouhem ng, Nanghaobo, ccurious ous, J. thaz 0z ju. edu.en yawenzengl 1 eegmmll.l..om coo
ABSTRACT
The conventional evaluation protocols on machine learning models rely heavily on labeled, i.id-assumed testing dataset, which is not often present in real-world applications. The Automated Model Evaluation (AutoEval) shows an alternative to this traditional workflow, by forming a proximal prediction pipeline of the test- ing performance without the presence of ground-truth labels. Despite its recent successes, the AutoEval frameworks still suffer from an overconfidence issue, substantial storage and computational cost. In that regard, we propose a novel measure - work to be both more efficient and effective. The core of the MDE is to establish mittadistribution statistic, on the information (energy) associated with individual samples, then offer smoother representation enabled by energy -ragyy learning. We further provide our theoretical insights by connecting the MDE with the clas- sification loss. We provide extensive experiments across modalities, datasets and different architectural backbones to validate MDE's validity, together with its su- periority compared with prior approaches. We also prove MDE's versatility by showing its seamless integration with large-scale models, and easy adaption to learning scenarios with noisy- or imbalanced- labels. Code and data are available: https://githute cwt/ www.o/////////////eeeeeeeeeeeeee http//eerrrrruuueeaeeeee
Energy (MDE) that allows the AutoEval frame-
INTRODUCTION
Model evaluation grows critical in research and practice along with the tremendous advances of ma- chine learning techniques. To do that, the standard evaluation is to evaluate model on a pre-split test set that is i)-fully labeled; ii)-drawn i.i.d. from the training set. However, this conventional way may fail in real-world scenarios. where there often encounter distribution shifts and the absence of ground-truth labels. In those environments with distribution shifts, the performance of a trained model may vary significantly Ciauoonero-Cander weak indicator of the model's 's generalization performance. Moreover, traditional cross-validation (Arlot Celisse 2010) and annotating samples are both laborious tasks, rendering it impractical to split label every test set in the wild. To address these challenges, predicting a model's 's performance on various dittof--iitributionn datasets without labeling, a.k.a Au- tomated Model Evaluation (AutoEval),, has emerged a promising solution and received some attention (Degg et al. 2021 Guillory et al. 2021 Garg et al. 2022).
al.) 20088 Koh et al. (021b), thereby limiting
in-distribution accuracy
The AutoEval works typically dedicated to the characteristics of the model's 's output on data. The past vanilla approaches are developed to utilize the model confidence on the shifted dataset (Guillory al. 2021 Garg al. 2022), and they have evidently suffered from the overconfidence problem. As result, some other metric branches are spawned, such as the agreement score of multiple mod- els' predictions (Chen al. 2021a| Jiang al., 202T). ). the statisties (e.g. distributional discrepancy) of network parameters YYu et al. [022]] Martin et al. 2021). Deng et al. (202T): Peng et al. (2023) introduce the accuracy of auxiliary self -upervuuee tasks as proxy to estimate the classification ac- curacy. The computational and/or storage expense is deemed as another problem in these AutoEval methods. For instance, Deng & Zheng (002 propose to measure the distributional differences between training and an out-of-distribution (OOD) testing set. Despite the feasibility of such an approach, it demands to access the training set every iterative loop of evaluation. While these prior approaches indeed prove the validity of AutoEval, most (if not all) of them involve extra heavy
"qqual contribution. Corresponning author.

---

Published as a conference paper at ICLR 2024
1230
0270
6290
0.27
Cartec
g3325
2875
1300
8.550
<923
E-22-
152
E.00
Press Sree
Kw
Press
(a) severity 1
(b) severity 2
(c) severity 3
(d) severity 4
(e) severity 5
Figure 1: Trrens between average energy and classification accuracy over different severity levels, we take CIFAR-IO-C fog sets as an example. The density (y-axis) is calculated as the proportion of classified correctly (blue unimodal) incorrectly (red unimodal) data within different energy ranges to the total samples. As the severity of the dataset strengthens, the accuracy degrades while the average energy increases accordingly (ie., the yellow dash line is moving to the right)
compute and/or external storage cost including the training set being stored/indexed, (retrained) model parameters, separate self-training objective- - which may cause unneglectable overhead to the system. To that regard, we pose the motivation of this work: can we establish a simpler, but more efficient and effective AutoEval framework, without resorting to much external resources?
To reach this goal is challenging. Most importantly, we hope to re-establish the AutoEval workflow by associating the inherent characteristic of the networks' output with its input more directly and transparently. Profoundly, we utilize energy, as introduced by [eCun et al. (2006) in the Energy- based Model (EBM), that we find aligned with our purpose. In this context, "energy" denotes the scalar value assigned to data point as fitted into the data manifold through the hypothesis class. In essence, the classifier can be viewed as an EBM (Zhao et al.) 2016 Grathwohl et al., 2019) with notable nature - "the correct classified data are given low energies, and vice versa" Based on this
finding, we empirically probe the relationship between energy and accuracy in Fig. D We observe a similar phenomenon with the previous AutoEval studies: as the dataset shift intensifies, the accuracy degrades while the average energy increases accordingly.
In line with the above observations, we propose a novel measure - Dieta-Distribution Energy (MDE) for accuracy prediction. Specifically, we present MDE as a meta-distribution (hence the name) statis- tic that is normalized based on characterizing the information (energy) of each sample individually. This indicator transforms the information quantity of overall samples into a statistic of the probabil- ity distribution, providing a softer representation of the dataset's distribution compared to the initial energy score. Also, we provide theoretical analysis for our method by connecting MDE to classifica- tion loss through Theorem 3.1 This theoretical justification indicates that under mild assumptions, the MDE measure consistently correlates with negative log-likelihood loss, thus reflecting the trends in model generalization. Hence, we posit a hypothesis as follows: the MDE- calculated from the test set alone- provides insights into the prediction of the model's testing accuracy.
For the measures derived in this way, we conduct rigorous empirical studies on different datasets -guided by the theory we pose above - that we prove the MDE on the test sets strongly corre- lates with its performance (Spearman's rank correlation p for vision >0.981 and for text 20.846). These results experimentally substantiate that our MDE's capability to predict the model's OOD test accuracy. Thus far, as a holistic AutoEval pipeline, we wish to emphasize that MDE outper- forms the prior training-free AutoEval approaches, and is more memory- and compute-efficient than the training-mun muutt methods. It is further capable to serve as a plug-and-pla evaluate off-the-shelf models including large-scale ones. Under varied modsl, data, and back- bone setups, MDE significantly surpasses its prior counterpart and sets a new SOTA record for test performance evaluation. Further, show that MDE remains effective even in strongly noisy and imnallnnbl scenarios. Finally, we visualize some in-depth analysis to demonstrate the inter- pretability of our method. In summary, we list our contributions follows: (i)-we propose simple but effective, plug-and-play AutoEval pipeline, which broadens the AutoEval technique towards production in the real world. (ii--MEE sets a new SOTA benchmark by significantly exceeding existing works and is backed by theoretical insights for its effectiveness.
module to elegantly
2 RELATED WORKS
Automated Model Evaluation is proposed to evaluate model performance on previously unseen, unlabeled datasets, hence also called unsupervised accuracy estimation. Recent methods mainly
2

---

Published as a conference paper at ICLR 2024
consider exploiting the properties of model output on unlabeled datasets for evaluation. Preliminary
research focuses on confidence score (Guillory et al. [202]] Garg et al. 2022 Lu et al. 2023c; Wang
jtt
al.
2023), such softmax probability. Subsequently, variety of directions have emerged along
with this research field consistently develops: disagreement of multiple models' predictions MMddani
et al. 2004 Donmez et al.) 2010; Platanios et al. 2016 2017, CCen al. (
al. 2021
Back et al. 2022), distribution discrepancy (Sun et al. 2021] Yu et al.
2022]Deng & Zheng 2021
norm and power law of network parameters (Unterthiner et 2020
Martin al. [02 |
Jain al.
2023), decomposition values of prediction matrix (Jaffe et al.
2015 Deng et al.
(023). bucketing
based on decision boundaries (Hu et al.) [023] Xie al. [023 Tu et al., 2023] Miao et al. 2023
conditional independence assumptions (Steinhardt & Liang] 2016). In addition Deng et al. (2021
2022 : Peng et al. (2023) add self-supervised tasks as a surrogate measure to estimate the classifier's
accuracy. Chen et al. (202 proposed an importance weighting approach guided by priori knowl- edge in accuracy estimation, akin to the re- re-weighting in Zhang et al. (2020). Chen et al. (2022)
propose SEES to estimate performance shift in both label and feature distributions. Mcanwhile, useful testbed was proposed to evaluate the model's 's generalization ability (Sun et al.) 2023). En- couragingly, the AutoEval concept has been extened to broader domains, e.g. database (Schelter iet al. 2020). structured data (Maggio et al., 2022), autonomous driving (Guan & Yuan 2023), text classification (Elsahar & Gallé 2019), feature engineering (Li et al. 2023), even the most closely watched LLM ( (Yue et al. (2023) and AIGC (Lu et (023a). Our approach differs from these above studies but aims to present a solid paradigm to address this evaluation task more perfectly.
Predicting ID Generalization Gap is to predict the performance gap between the paired training- test set, thereby facilitating an understanding of the model's generalization capability to in- distribution data. This field has explored long line of work from complexity measurement on training models and training data, representative studies involve Neyshabur et al. (2017); Dziugaite Involve/Neyshabur
& Roy (2017); Arora et al. (2018); (Zou et al. (2018); Jiang et al. (2018) 2019); Nagarajan & Kolter (2019a by: Corneanu et al. (2020): Zhang et al. 2021) )) For example, Jiian et al. (2018) introduces an
indicator of layer- wise margin distribution for the generalization prediction. Corneanu et al. (2020)
derives a set of persistent topology measures to estimate the generalization gap. Chuang et al. 2020)
gauges the generalization error via domain-invariant representations. [Baldock et al. (2021) propose a measure of example difficulty (i.e, prediction depth) in the context of deep model learning. The above works are developed for the same distribution between the training and test sets without ac-
cessing test data. In contrast, we focus on predicting model accuracy across various OOD datasets using the attributes of the test sample.
Energy-based Model is a non-normalized probabilistic model that captures dependencies between variables by associating scalar energy to each variable (LeCun et al. 2006). EBMs do not impose restrictions on the tractability of normalized constants, making them more flexible for parameteri- zation. As a result, researchers have started using EBMs to model more expressive families of prob- ability distributions (Ranzato et al.] (006)] 2007). However, the unknown normalization constant of EBMs makes training particularly difficult. Hence, Xie et al. (2016) first uses Langevin dynamics to effectively train a CNN classifier that can be regarded as an EBM Follow-up works investigate training EBMs through Markov chain Monte Carlo (MCMC) techniques (Du & Mordatch 2019) Song & Kingma 2021b. After (Xie al. 2016; Grathwohl et al. 2019) revealed that the classifier essentially acts as an energy model, energy-based applications have sprung up, such as GAN (Zhao eet al.] 2016), video (Xie al., 2019), point cloud (Xie et al. 2021), voxel (Xie al. 2018 tra- jectory (Xu et al., (022) and molecules (Liu et al. 2021). With the support of the discovery that "the correct classified data has higher energy, and vice versa". energy view has also been applied to OOD detection (Liu et al. 2020). But unlike Liu et al. (2020) using energy to detect OOD test samples that are different from training distributions, our work is towards predicting the model's 's ac- curacy on unlabeled OOD test sets. Inspired by these pioneering works, we formulate mergy-driven statistics as the accuracy surrogate to assess the feasibility of our method in AutoEval task.
[eenee,Xie&
3 ENERGY-BASED AUTOMATED MODEL EVALUATION
In this section, we propose an energy-based AutoEval framework pivoted on the meta-distribution energy. First, we formulate the AytoEval problem (Section B.I. We then describe the meta- distribution energy in detail. (Section 3.2) Also, connect the meta-distribution energy mathematical theorem on classification loss for a theoretical guarantee of our method (Section 3.3. A pseudo-code is provided in algorithm
3

---

Published as a conference paper at ICLR 2024
Algorithm Automated Model Evaluation via Meta-distribution Energy
C Input: Synthetic sets {D: unlabeled OOD set Du, classifier f. energy function ( f).
1: for = 1, 1,2, ..., C do
2: aCc; =: (x [y + arg maxjey Softmax (fz (x))]l
3: MDE;
E. log Softmax ((x; f)
4: end for
5: Fit a linear regressor (w, b) from the collection of {(acc, MDE,))
6: Regress the accuracy of on Du: acc, =: EgDD, w To MDE + b)
7: Mean absolute error: = |ace, - accal
Output: Correlation coefficients R², r, p and mean absolute error
3.1 PROBLEM STATEMENT
Notations. In this work, we consider a multi-class classification task with input space X C Rd and label space y {1,...; K]. We denote Ps and PT as the source and target distributions over x y, respectively. Let Ps and pr define the corresponding probability density functions. Given training dataset D6 i.i.d sampled from Ps. we train probabilistic classifier f RR + Ak.
where Ax is a unit simplex in K-1 dimensions. For a held-out test set D {(x] yt) M drawn from Ps. when accessed at data point (x*, y²), returns ý =: arg m softmax (f, (x*)) the
predicted lahel and fj (x=) ) as the associated logits of j-th class. Given the label y², the classification error (i.e., 0-1 loss) on that sample is computed as & (f (x=), errors on all points of DS, we can determine the in-distribution test accuracy of classifier f on Ps.
TI [v- * v). By calculating the
Automated Model Evaluation. However, under distribution shift (ps + pr). the in-distribution (source) test accuracy of DE fails to actually reflect the generalization performance of / on target Pr- To this end, this work aims evaluate how well f performs on the varied oui-i--distriuution (target) data without access to labels. Specifically, given trained f and an unlabeled OOD dataset DJT {- correlated the accuracy of f DJ Note that, the target distribution pr has the same K classes as the source distribution ps (known as the closed-set setting) in this work. And unlike domain adaptation, our goal is not to adapt the model to the target data.
N with N samples drawn i..dd from pr, expect develop quantity that strongly
3.2 META- DISTRIDITTIBUTIONN ENERGY FOR AUTOEVAL
In this part, we elaborate on the MDE measure and the AutoEval pipeline centered it.
Meta-Distribution Energy. The energy- based model (EBM) (LeCun et al. 2006 was introduced map each data point x to a scalar dubbed as energy via an energy function Z(x) R + R. It could transform the energy values into probability density p(x) through the Gibbs distribution:
e- -ZZ
- Z(Ta)/TT
ply x)) = Ty eZZ..// c Z(x)TT
where the denominator Iy ((aa/// is the partition function by marginalizing over y, and T is the positive temperature constant. Now the negative of the log partition function express the Gibbs free energy Zzx) at the data point x as:
-T- ((x) -T log / -(zaa///T
y'
In essence, the energy-based model has an inherent connection with the discriminative model. To interpret this, consider the above-mentioned discriminative classifier f : ERd - Ak. which
maps a data point æ E id to real number known as logits. These logits are used to parameterize categorical distribution using the Softmax function:
elv(x)TT
ply x) ΣK e5(())²²
(3)
4

---

Published as a conference paper at ICLR 2024
where fy (z) denotes the y-th term of fxx), i..., the logit corresponding to the y-th class. Combining Eq. 2 and Eq. we can derive the energy for a given imput data (z, y) as (x, y) = fz((z). Thus, given a neural classifier f(x) and an input point RD, we can express the energy function with respect to the denominator of the Softmax function:
7 -T. log Σ e
Assume an unlabeled dataset D [(=))]=11 with N samples, we define MDE as a meta- distribution statistic re-normalized on the energy density ((x; f) of every data point r:
1 N
1 N
eZ(nns))
MDE(#;
log Softmax Z(x;
log
(5)
En AZaa))
where Z (Zn; f) indicates the free energy of n-th data point Fns [ is the cardinality of DJT This dicator transforms global sample information into meta- meta- probabilistic distribution measure. Aiding by the information normalization, MDE offers a smoother dataset representation than initial energy score AvgEnergy (also proposed by us) which solely averages the energy function on the dataset.
AutoEval Pipeline. We give the procedure for using MDE to predict the OOD testing accuracy, but other measurements are also applicable. Given model f to be evaluated, we first compute the value pairs of its true accuracy and MDE on the synthetic test set. Then, the accuracy of the OOD test set write down the forms follows:
can be estimated by simple linear regression. Consequently,
acc =: ((a)) II & arg max Softmax (fy (x)) jEY
aèc =: ELED [w" (DEEE;; f) b].
|ace acce..
(8)
where ace and aèc are the ground-truth and estimated accuracy, respectively. [[]]] is an indicator function, (x, y) is input data and class label, is the absolute for accuracy estimation.
Remarks. According to our formulation, our MDE method poses three appealing properties: i)-a training-free approach with high efficiency by dispensing with extra overhead: ii)-a calibration with built-in temperature scaling to get rid of the overconfidence issue of only using model logits; iil)-a ce-nomallzedd distribbiinn statistic with a smoother dataset representation. These properties largely guarantee the efficiency and effectiveness of our MDE algorithm. More interestingly, since our method smoothly condenses all logits, our MDE metric demonstrates excellent robustness to label bias and noise; see Section 4.5 for details.
3.3 THEORETICAL ANALYSIS
From a theoretical side, we first set an assumption that the sample energy needs to be satisfied when the discriminative classifier (i.e., EBM) minimizes the loss function:
Corollary 3.1 For a sample (ar 4), incorrect answer y and positive margin m. Minimizing the loss function L will satisfy Z (x, f) Z (x. f)
if there exists at least point (21,2))
with 21 + m < 22 such that for all points 2/) with o + m 2 2/y, we have Liz,) (21 22) <
Ciz,,) (2 2 where [yy] contains the vector of energies for all values of y except and ÿ.
Theorem 3.1 Given well well-trained model with optimal loss Call , for each sample point (z,, y.).
the difference between its classification risk and MDE can be characterized as follows:
= MDE' Ciu (x:)/T - max f, (z.) (T) 10. Jey
if yi:
(9)
i $j41 yr.
where Y is the label space, MDE is the proposed mela-distribution energy indicator, Luul is the negative fog-likelihood loss, is the temperature constant approximate
We ascertain whether label yi corresponds to the maximum logits by comparing the term Eq.
and zero, thus assessing the model's accuracy. Thus, we theoretically establish the connection be- tween MDE and accuracy by mathematical theorem. Detailed theoretical analysis in Appendix C
5

---

Published as a conference paper at ICLR 2024
Table 1: Correlation comparison with existing methods on synthetic shifted datasets of CIFAR-IO, CIFAR- 100, and mageN,t,, MNLI. We report coefficient of determination (R²) and Spearson's s rank correlation (p) (higher is better). The training-mus must methods marked with "*" while the others are training-free methods. The highest score in each row is highlighted in bold.
Datasct
Network
ConfScore
p
Entrupy
R2
p
Frechet
R2
ATC
N2
AgreeScore* ProjNorm 12a
p
P
MDE
ResNet-20
0.991 0.953 0,990
0.958
0.984 0.930
0.962
0.890
0.990
0.955
0.974
0.954
0.992 0.964
CIFAR-10
RepVGG-AD
0.979 0.954 0.981
0.946
0.982
0.864 0.959
0.888
0.981
0.950
0.970
0.969
0.985
0.980
VGG-11
0.986 0.956 0.989 0.960 0.990 0.908
0.947
0.907
0.989
0.903
0.985
0.955
0.991
0.974
Average
0.985 0.954 0,987 0.955 0.985 0.901
0.956
0.895
0.987
0,936
0.976
0.959
0.989
0.973
ResNet-20
0.962
0.906 0,943 0,870
0.964
0.880
0.968
0.923
0.970
0,925
0.967
0.927
0.981
0.961
CIFAR-100
RepVGG-AO
0.985
0.938 0,977 0.926 0.955 0.864
0,982
0.963
0,983
0,953
0.973
0.933
0.992
0.978
VGG-11
0.979 0,950 0.972 0.937 0.986 0.889
0,991
0.958
0,980
0,953
0,966
0.881
0.991
0.960
Average
0.975 0.931 0,964 0,911 0.968
0.878 0.980
0,948
0.978
0,944
0.969
0,914 0.988
0.966
ResNet-50
0.932
0.711 0,937
0.755
0.957
0.818
0,986
0,910
0.971
0.895
0,944
0,930
0.994
0.971
TinyImageNet
DenseNet-161
0.964 0.821 0,925 0.704
0.948 0.813 0.989
0,943
0.983
0.866
0.957
0.930
0.994
0.983
Average
0.948
0.766 0,931 0.730 0.953 0.816 0.988
0.927 0.977
0,881
0,950
0,930
0.994 0.977
BERT
0.650
0.527
0.790
0,536
0.517
0.479
0.650
0.487
0.608
0.457
0.636
0.547
0.853
0.644
MNII
RoBERTa
0.734 0,470 0,741 0,516 0.587 0.494 0.643 0.430
0.825
0.682
0.790
0.531
0.846
0.716
Average
0.692 0.499 0.766 0.526
0.552 0.487
0.647
0.459
0.717
0,570
0.713
0.539
0.850 0.680
4 EXPERIMENTS
In this chapter, we assess the MDE algorithm across various data setups in both visual and text domains, which includes: correlation studies, accuracy prediction errors, the hyper-parameter sen- sitivity, as well as two stress tests: strong noise and class imbalance.
4.1 EXPERIMENTAL SETUP
In this work, we evaluate each method the image classification tasks CIFAR-10, CIFAR- 100 (Krizhevsky et al. 2009). TinylmageNet (Le & Yang) 2013), ImageNet-IK IK (Deng et al. 2009),
WILDS (Koh et al. (2021a) and the text inference task MNLI (Williams al. 2018). See Ap- pendix for details.
Training Details. Following the practice in Deng et al. 2023 we train models using public implementations] for CIFAR datasets. The models in ImagNet- IK are provided directly by timm library (Wightman al. 2019). Likewise, we use the commonly-used scripts] to train the models for Tiny MinyImageNet. Similarly, for the WILDS data setup, we align with the methodology proposed by (Garg et al.) 2022) for the selection and fine-tuming of models. For the MNLI setup, we use the same training settings as (Yu et al.) (022).
Compared Baselines. We consider eight methods as compared baselines: 1) Average Confi- dence (ConfScore) (Hendrycks & Gimpel| (016)). 2) Average Negative Entropy (Entropy) Guull
lory et al., (02T). 3) Frechet Distance (Frechet) (Deng Zheng] 202T), 4) Agreement Score (AgreeScore) (Jiang et al.) 2222]) 5) Average Thresholded Confidence (ATC) (Garg et al. 2022).
6) Confidence Optimal Transport (COT) (Lu et al. (023b). 7) Average Energy (AvgEnergy), 8) Pro- jection Norm (ProjNorm) (Yu et al. 2022). 9) Nuclear Norm (NuclearNorm)) (Deng et al. 2023).
The first six existing methods are developed using the model's output. The AvgEnergy we devised is based on initial energy score, highly tied to MDE. The final two currently the SOTA methods. For further details, see Appendix
To evaluate the performance of accuracy prediction, determination (R2), Pearson's 's correlation (r), and Spearman's 's rank correlation (p) (higher is better) to quantify the correlation between measures and accuracy. Also, we report the mean absolute error (MAE) results between predicted accuracy and ground-truth accuracy on the naturally shifted sets.
Evalutaion Metrics.
coefficients of

---

Published as a conference paper at ICLR 2024
Correlation comparison with SOTA Table 3: Mean absolute error (MAE) comparison and highly related methods on synthetic shifted with SOTA and highly related methods on natural datasets of different data setup.
Table 2:
shifted datasets of different data setup.
NuclearNorm AvgEnerg R²
MDE
Dataset
Natural Shifted Sets NuclearNorm AvgEnergy MDE
Dataset
Network
R²
CIFAR-10. 1
1.53
1.55
0.86
ResNet-20
0.996
0.959
0.989 0.955 0.992 0.964
CIFAR-10
CINIC-10
2.95
2.63
0.48
CIFAR-10
RepVGG-A0
0.989
0.936
0.990 0.959 0,985 0.980
4.78
VGG-11
0.997
0.993
0.991
Average
3,42
2.88
1.78
Average
0.994
0.935
0.991 0.957 0.989
Tey Net- V2-A
ResNet-20
0.986
0.955
0.977
0.956 0,981 0.961
Tiny ImageNet-V22 -2-B
2.36
1.92
1.11
RepVGG-AD
ImageNett
CIFAR-100
VGG-11
0.997
0.986 0.964 0.991
TinyImageNet
Tiny ImageNet-S
Taay Nott
1.90
1.24
0.47
Average
0.983
Ting ImageNet- Vid
9.16
8.49
6.08
Tiny ImageNei-Ade
6.01
5.66
3.59
ResNet-50
0,991
0.991
Average
3.84
3,23
2.15
TinyImageNet
DenseNet-161
0.993
0.968
0.983 0.961
0.994 0.983
QNLI
5.56
Average
0.992
0.969
0.987
0.964
0.994 0.977
KTE
6.49
5.39
3.96
0.650
01783
ScTaal
WNLI
7.21
8.69
7,50
6.48
6.06
4.79
MNLI
RoBERTa
0.685
0.832
0.650
0.846 0.716
ANLI
12.01
10.48
8.42
Average
0.668
0.496
0.808
0.595 0.850 0.680
Average
8:44
5.76
Table 4: Mean absolute error (MAE) comparison with existing methods natural shifted datasets of CIFAR-10, TinylmageNet, and MNLI. The training-mus' methods marked with "** while the others are training-free methods. The best result in each is highlighted in bold.
Dataset
Unseen Test Sets
ConfScore
Entropy
Frechet
ATC
AgreeScore*
ProjNorm*
MDE
CIFAR-10. CIFAR-10.2
9.61
3.72
5.55
4.87
3.37
2.65
0.86
CIFAR-10
7.12
8.95
6.70
5.90
3.78
4.59
1.01
CINIC-10
7.24
8.16
9.81
5,91
4.62
9.43
0.48
STL-10
10.45
15.25
11.80
15.92
11.77
12.98
4.78
Average
8.61
9.02
8.47
8.15
5.89
7.41
1.78
TinyImageNet-N V2-A
7.22
5.67
7.68
4.78
4.37
3.77
0.54
TinyImageNet- V2-B
8.80
10.61
11.65
5.57
6.36
5.04
1.11
TinyImageNet-V? V2-
10.67
8.04
14.58
9.38
5.69
3.56
0.88
TinylmageNet
'linelmaaeNeetS
11.44
9.54
8.32
13.17
6.35
9.80
0.47
TinyImageNer
10.18
8.02
11.28
14.81
7.10
9.50
2.41
Tiy LlnyImageNet- Vid
13.12
15.36
13.57
16.20
19.72
10.11
6.08
TinyImageNet Adv
14.85
14.93
10.27
15.66
10.98
12.94
3.59
Average
10.90
10.31
11.05
11.37
8.65
7.82
2.15
QNLI
16.10
17.31
15.57
10.54
14.33
15.88
5.56
RTE
12.32
18.18
16.39
14.46
10,92
9.43
3.96
WNLI
9.99
17.37
21.67
21.10
15.15
15.78
6.06
MNLI
SciTail
16.85
17.27
16.56
11.88
9.06
9.97
4.79
ANLI
25.14
22.19
14.69
20.85
12.34
17.93
8.42
Average
16.08
18.46
16.98
15.77
12.36
13.80
5.76
4.2
MAIN RESULTS: CORRELATION STUDY
We summarize the correlation results (R² and p) for all methods under different settings in Table 1 Encouragingly, our MDE surpasses every (even SOTA) baseline in a fair comparison across modalities, datasets, and backbones. We discuss these results from the following aspects:
and Fig. 2
In Table I and 6 MDE significantly outperforms common training-free methods. Specifically, the average πt MDE on CIFAR-I0 (0.973), CIFAR-100 (0.966), TinyImageNe (0.977), ImageNet (0.960) and MNLI (0.680) exceeds the ConfScore, Entropy, Frechet, and ATC by notable margin. These gains may benefit from the temperature scaling in MDE re-calibrating confidences. The MDE also superior to the training-must AgreeScore and ProjNorm. This advantaged scheme improves performance, reduces cost, and seamlessly meets the evaluation needs the popular LLM.
MDE vs. SOTA and Highly related methods. As shown in Table 2 MDE achieves better per- formance than the recent SOTA NuclearNorm in almost all setups, especially in the MNLI setup. This series of results substantiates that MDE is competitive technique with extensive applicability.

---

Published as a conference paper at ICLR 2024
Accuracy
Accuracy
MDE VS,, Accuracy
MDE V.S. Accuracy
pid 385
pe0,903
66
Accuracy
Accuracy
Accuracy B
Accuracy
40
X
16-
225 240
MDE
32.3255
MDE
12.8
23.5
MDE
(a)
DenseNet-12 121
(b) DenseNet-166 Accuracy
(c) DenseNet- 1st-169
(d) ResNet-50
MDE v.s. Accuracy
Accuracy
MDE v.s. Accuracy
King
Rina 838
iibara
pord
pod
Accuracy
5
Accuracy
Accurracy
Accuracy
%
8
22
20
2000
10-
MDE
MDF
MDF
MDE
(c) ResNet-101
(f) ResNet-152
(g) VGG-16
VGG-19
Figure 2: MDE's coefficients of determination (R²), Pearson's correlation (r) and Spearman's rank correlation (p) on synthetic shifted datasets of ImageNet setup.
Notably, MDE consistently outperforms the well-performing AvgEnergy which is highly tied us. It confirms the energy-based indicator can strongly correlate to accuracy. More importantly, MDE yields stronger correlation by smoother measure after re-normalizing the global sample energies.
Bigger and Textual datasets: ImageNet- MNLI. Further, we present scatter plots of MDE on ImageNet- 1k in Fig. 2 We emphasize that MDE remains robustly linearly related the performance of off-the-shel models, even in the extreme case of test accuracy below 20 (see subplots (a) and (g)). On the textual MNLI dataset, the average correlation obtained by our MDE is also effective (R==0.680, /=0855) These findings greatly bolster the deployment of our approach across diverse real-world scenarios. The complete set scatter plots can be found in Appendix H
4.3 MAIN RESULTS: ACCURACY PREDICTION ERROR
We show the mean absolute error (MAE) results for all methods in predicting accuracy real-world datasets Table 3 4 7 and [ For each natural shifted set, we report its MAE value as the average across all backbones. Among seven datasets,
conclude that method reduces the average MAE from 5.25 to 3.14 by about 40.0% % upon prior SOTA method (NuclearNorm), thus setting a new SOTA benchmark in accuracy prediction. Further, MDE shows strong performance regardless of the classification domain (e.g. MNLI) or the classification granularity (ranging from CIFAR-I0 to TinylmageNet). Interestingly, in certain extremely hard test sets (e.g. STL-10, Tinylmagenet-AGe ANLI), other methods fail with a relatively poor estimated eттor while we perform well yet. These results are not only excellent, and robust but also substantiated by the optimal correlation observed between MDE and accuracy. This reminds that the AutoEval technique heavily relies on the correlation degree between measure and accuracy.
4.4 ANALYSIS OF HYPERPARAMETER SENSITIVITY
As we adopt the MDE-based AutoEval framework, want to know the sensitivity of its perfor- mance to hyperparameters. So we study the impact of variations in temperature and random seed on performance. Here, report the results using VGG-11 the CIFAR-10 10 setup, which remains sistent in the subsequent experiments unless otherwise stated. All results of this section are placed in the appendix.
Scaled temperature constants. As an important factor in the MDE calculation, we study the tem- perature constant T. from 0.01 100. As figure 7 (a) shows, the performance declines when the
8

---

Published as a conference paper at ICLR 2024
MDE
NuclearNorm
NurlearNorm
6.1
15.0
Mean absolute error
1.4
12.5
4.77
2:46
7.5
5.0
1.6
1.51
2.5
0.56
C101 A1011
A
OMC
OMCA
CRICS
STLA
STLB
0.1
0.2
0.4
0.55
0.8
1.0
59.97
57.69
Imbalanced Ratio
Figure 3: Mcan absolute errors on two stress tests: (lefflystrongly noisy and (right)-class imbalance.
temperature increases and the best performance appears in T 1. The correlation coefficients and MAE of broader range of temperature can be found in Table
Different random seeds. To examine if the experimental results are robust the initial random state, we pick different random seeds for training (use as the default seed). As figure (b) shows, the performance of our framework robust randomness.
4.5 STRESS TESTS: STRONGLY NOISY AND CLASS IMBALANCED CASES
Strongly Noisy. the previous analysis, we tested our method on the naturally shifted test sets. Considering that real-world scenarios may be more complex, we test the robustness of MDE and NuclearNorm (SOTA) a "more realistic" test environment by applying new transformations on strictly screened to have overlap with
naturally shifted test sets. Note that new transformations various transformations in the synthetic set (i.e. CIFAR-1O-C), Specifically, we use Cutout Vries & Taylor 2017), Shear, Equalize and ColorTeeeeraturee (Cubuk et al. 2019) to generate CIFAR- 10.1-A/B, CIFAR- 1EAR-10.2240 1022-ABB, CINIC-10-A/B, STI.-10.1-A75 We note the following observations from the left of Fig. First, the greater the shifted intensity, the harder both methods are predict accuracy. The accuracy prediction results in the re-transformed test sets (-A/B) are worse than the untransformed state. Also, CINIC-10 10 and STL-10 with larger shifts, experience more performance decline compared to other datasets. Second, under the noised data undergoing new transformations, our method consistently achieves more superior results (MAE 5.92) than NuclearNorm.
Class Imbalance. Considering that real-world data is usually not balsnca like our work, some classes are under-sampled or over-sampled, resulting in label shift (ps(y) * pr(y)). To study the effect of class imbalance, we create long-tail imbalanced test sets from synthetic datasets (CIFAR-1O-C with the 2-th severity level). Specifically, we apply exponential decay (Cao et al.) 2019) to control the proportion of different classes. It is represented by the imbalance ratio (r) the ratio between sample sizes of the least frequent and most frequent classes that ranges from (0.1, 0.2, 0.4, 0.6, 0.8, 1.0}. As shown in the right of Fig. our method is robust under moderate imbalance (r 2 0.4) than NuclearNorm. Certainly, when there is severe class imbalance (r S 0.2), method is also seriously affected by label shift, but it still precede NuclearNorm. At this time, considering extra techniques such as label shift estimation (Lipton et al., 2018) may be potential idea for addressing this issue.
CONCLUSION
In this work, we introduce a novel measure, the Meta-Distribution Energy (MDE), to enhance the efficiency and effectiveness of the AutoEval framework. Our MDE addresses the challenges of overconfidence. high storage requirements, and computational costs by establishing the MDE - a meta-distribution statistic on the energy of individual samples, which is supported by theoreti- cal theorems. Through extensive experiments across various modalities, datasets, and architectural backbones, we demonstrate the superior performance and versatility MDE via micro-level results, hyper-parameter sensitivity, stress tests, and in-depth visualization analyses.

---

Published as a conference paper at ICLR 2024
ACKNOWLED
This work is majorly supported by the National Key Research and Development Program of China (No. 2022YFB330410 Fundamental Research Funds for the Central Universities (Project QiZhen @ @ZJU), and in part by the NSFC Grants (No. 62206247)
REFERENCES
Sylvain Arlot and Alain Celisse. A survey of cross-validation procedures for model selection. 2010.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning. pp- 254-263. PMLR, 2018.
Christina Back, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. ggreement-on-the-ee Pre- dicting the performance of neural networks under distribution shift. Advances in Neural Informa- fion Processing Systems, 35: 9274-19289, 2022.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10889, 2021.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642 2015.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aineelldisttibuton-wwwwwwe margin loss. Advances in neural information processing systems, 32, 2019.
Jiefeng Chen, Frederick Liu, Besim Avei, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting and estimating accuracy on unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems, 34:14980 1991449--4992,, 2021a.
Lingjiao Chen, Matei Zaharia, and James Zou. Estimating and explaining model performance when both covariates and labels shift. Advances in Neural Information Processing Systems, 35: 11467-11479, 2022.
Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International conference on machine learning, pp. 1617-1629. PMLR, 2021b.
Ching Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distri- bution shifts via domain-invarian representations. arXiv preprint arXiv:2007.035// 2020.
Ciprian A Corneanu, Sergio Escalera, and Aleix M Martinez. Computing the testing eтrоr with- out testing set. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2677-2685, 2020.
Ekin Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1 113-123, 2019.
Luke N Darlow, Elliot Crowley, Antreas Antoniou, and Amos Storkey, Cinic- 1 is not imagenet or cifar-10. arXiv preprint 100.0:181, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 255.255. leee, 2009.
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proc. CVPR, 2021.
10

---

Published as a conference paper at ICLR 2024
Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about clas- sifier accuracy under varying testing environments? In International Conference on Machine Learning. pp. 2579-2589, PMLR, 2021.
Weijian Deng, Stephen Gould, and Liang Zheng. On the strong correlation between model invari- ance and generalization. Advances in Neural Information Processing Systems, 35:28052-28067, 2022.
Weijian Deng, Yumin Suh, Stephen Gould, and Liang Zheng. Confidence and dispersity speak: Characterising prediction matrix for unsupervised accuracy estimation. arXiv preprint vrXiv:2302.01094 2023.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Terrance Veies and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint urXiv:1708.04552 2017.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13733-13742, 2021.
Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. Unsupervised supervised learn- ing i: Estimating classification and regression errors without labels. Journal of Machine Learning Research, 11(4), 2010.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
Gintare Karolina Dziugaite and Daniel Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint crXiv:1703.11000 2017.
Hady Elsahar and Matthias Gallé. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP- (JCNLP), pp. 2163-2173, 2019.
Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanic Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint mmXh::2201.04234. 2022.
Max Glockner, Vered Shwartz, and Yoav Goldberg. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 650-655, 2018.
Will Grathwohl, Kuan-Chieh Wang, Jörn- Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263. 2019.
Licong Guan and Xue Yuan. Instance segmentation model evaluation and rapid deployment for autonomous driving using domain differences. IEEE Transactions Intelligent Transportation Systems, 24(4):4050-400 2023.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Confer- on Computer Vision, pp. 1134-1144, 2021.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis- fies: Human anguage Technologies, Volume 2 (Short Papers), pp. 107-112, 2018.
11

---

Published as a conference paper at ICLR 2024
Ning Han, Yawen Zeng, Chuhao Shi, Guangyi Xiao, Hao Chen, and Jingjing Chen. Bic-net: Learn- ing efficient ppatio-tempone relation for text-video retrieval. ACM Transactions on Multimedia crmpuuting.ommmnicatti
and Applications, 20(3):1-21, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. arXiv preprint arXiv: ,
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and distii-dist examples in neural networks. arXiv preprint 1610.02 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distributio generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognilion, pp. 15262-15271, 2021b.
Qiang Hu, Yuejun Guo, Xiaofei Xie, Maxime Cordy, Mike Papadakis, Lei Ma, and Yves Le Traon. Aries: Efficient testing of deep neural networks via labeling-free accuracy estimation. In 2023 JEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1776-1787. IEEE, 2023.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4708, 2017.
Ariel Jaffe, Boaz Nadler, and Yuval Kluger. Estimating the accuracies of multiple classifiers without labeled data. In Artificial Intelligence and Statistics, pp. 407-415. PMLR, 2015.
Achin Jain, Gurumurthy Swaminathan, Paolo Favaro, Hao Yang, Avinash Ravichandran, Hrayr Harutyunyan, Alessandro Achille, Onkar Dabeer, Bernt Schiele, Ashwin Swaminathan, et al. A meta-learning approach to predicting performance and data requirements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3623-3632, 2023.
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. arXiv preprint urXiv: 13, 2018
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint 1rIiv: 2019.
Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and Zico Kolter. Assessing generalization of sgd via disagreement. arXiv preprint arXiv:2106. 137.11999, 2021.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of wild distribution shifts. In International Conference Machine Learning (ICML), 2021a.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021b.
12

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Liyao Li, Haobo Wang, Liangyu Zha, Qingyi Huang. Sai Wu, Gang Chen, and Junbo Zhao. Learning data-driven policy network for pre-training automated feature engineering. The Eleventh International Conference on Learning Representations, 2023.
Nankai Lin, Yingwen Fu, Xiaotian Lin, Dong Zhou, Aimin Yang, and Shengyi Jiang. Cl-xabsa: Contrastive learning for cross-lingual aspect-based sentiment analysis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:29 31:2935-2946, 2023. doi: 10. 109...93 3297964.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122-3130. PMLR, 2018.
Meng Liu, Keqiang Yan, Bora Oztekin, and Shuiwang Ji. Graphebm: Molecular graph generation with energy-based models. arXiv preprint arXiv:2 2021.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based cuitr--distribution detec- tion. Advances in neural information processing systems, 33:21 23:211644-21455, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 1xXiv:190..11690 2019.
Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. Limscore: Unveil- ing the power of large language models in text-to-image synthesis evaluation. arXiv preprint arXiv:2305. 11166. 2023a.
Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ketong Chen, Zhenlin Wang. Soheil Kolouri, Simon Stepputtis, Joseph Camphell, and Katia Sycara. Characterizing out-of-distributioo error via optimal transport. arXiv preprint arXiv:2305. 15640, 2023b.
Yuzhe Lu, Zhenlin Wang, Runtian Zhai, Soheil Kolouri, Joseph Campbell, and Katia Sycara. Predicting -ui-o--distr error with confidence optimal transport. arXiv preprint arXiv:2302.05018. 2023c.
Omid Madani, David Pennock, and Gary Flake Co-validation: labeled data to validate classification algorithms. Advances in neural information processing systems, 17, 2004.
Using model disagreement un-
Simona Maggio, Victor Bouvier, and Léo Dreyfus-Schmide Performance prediction under dataset shift. In 2022 26th International Conference Pattern Recognition (ICPR), pp. 2466-2474 IEFF, 2022.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' pp. 216-223, 2014.
Charles H Martin, Tongsu Peng, and Michael W Mahoney. Predicting trends in the quality of state- of of-the-art neural networks without access to training or testing data, Nature Communications, 12 (1):4122, 2021.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4248-448, 2019.
13

---

Published as a conference paper at ICLR 2024
Shuyu Miao, Lin Zheng, Jingjing Liu, and Hong Jin, K-means clustering based feature consis- tency aligmment for label-free model evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3298-3306, 2023.
Vaishnavh Nagarajan and J Zico Kolter. Deterministic pac-bayesian generalization bounds for deep networks via generalizing noise-resilience. arXiv preprint urXiv:1905.13333 2019a.
Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generaliza- tion in deep learning. Advances in Neural Information Processing Systems, 32, 2019b.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguisties, pp. 2340-2353, 2018.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. Advances neural information processing systems, 30, 2017.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversar- ial nli: A new benchmark for natural language understanding, arXiv preprint arXiv:1910. 14599, 2019.
Ru Peng. Yawen Zeng, and Jake Zhao. Distill the image nowhere: Inversion knowledge distil- lation for multimodal machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2379-2390, 2022.
Ru Peng, Qiuyang Duan, Haobo Wang, Jiachen Ma, Yanbo Jiang, Yongjun Tu, Xiu Jiang, and Junbo Zhao. Came: Contrastive automated model evaluation. arXiv preprini arXiv:2308. 1111, 2023.
Emmanouil Platanios, Hoifung Poon, Tom M Mitchell, and Eric Horvitz. Estimating accuracy from unlabeled data: A probabilistic logic approach. Advances in neural information processing systems, 30, 2017.
Emmanouil Antonios Platanios, Avinava Dubey, and Tom Mitchell. Estimating accuracy from unla- beled data: A bayesian approach. In International Conference on Machine Learning, pp. 1416 1425. PMLR, 2016.
Joaquin Cainoonero-Candele Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008.
Marc Aurelio Ranzato, Christopher Poultney, Sumit Chopra, and Yann Cun. Efficient learning of sparse representations with an energy- based model. Advances in neural information processing systems, 19, 2006.
Marc' Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, and Yann LeCun. A unified energy-based framework for unsupervised learning. Artificial Intelligence and Statistics, pp. 371-379.
PMLR, 2007.
Abhilasha Ravichander. Aakanksha Naik, Carolyn Rose, and Eduard Hovy. EQUATE: A benchmark evaluation framework for quantitative reasoning natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 349-361, 2019.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.0044 2018
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
Sehastian Schelter, Tammo Rukat, and Felix Bießmann. Learning validate the predictions of black box classifiers on unseen data. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pp. 1289-1299, 2020.
Vaishaal Shankar, Achal Dave, Rebecca Roclofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? In Proceedings of the IEEE/CVF Interna- fional Conference on Computer Vision, pp. 9661-9669, 2021.
14

---

Published as a conference paper at ICLR 2024
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv 2014.
Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288, 2021.
Jacob Steinhardt and Percy S Liang. Unsupervised risk estimation using only conditional indepen- dence structure. Advances in Neural Information Processing Systems, 29, 2016.
Xiaoxiao Sun, Yunzhong Hou, Hongdong Li, and Liang Zheng. Label-free model evaluation with semi-structured dataset representations. arXiv preprint 12200:211..00694, 2021.
Xiaoxiao Sun, Xingjian Leng, Zijian Wang, Yang Yang. Zi Huang, and Liang Zheng. Cifar- 10-warehouse: Broad and more realistic testbeds in model generalization analysis. ArXiv, abs/2310.04414 2023.
Weijie Tu, Weijian Deng, Tom Gedeon, and Liang Zheng. A prototypss representation for dataset-level applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2881-2892, 2023.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predict-
ing neural network accuracy from weights. arXiv preprint arXiv: 1448, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 553-355, 2018.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances Neural Information Processing Systems, 32, 2019.
Jiexin Wang, Jiahao Chen, and Bing Su. Toward auto-evaluation with confidence-based category relation -ware aware regression. In ICASSP 2023-2023 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Ross Wightman et al. Pytorch image models, 2019. hucgiagface/pycorod imagemmaeannnn models
URL https://github. com/
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad- cover-co challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers). pp. 1112-1122, 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020. URL https:// gi github. cow http:h/wwwngg..atata/aaaaooeoe/a
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2644. PMLR, 2016.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3d shape synthesis and analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8629-8638, 2018
Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Learning energy-based spatial-temporal gener- ative convnets for dynamic patterns. IEEE transactions on paftern analysis and machine intelli- gence, (112):516-534. 2019.
Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep energy- based learning unordered point sets for 3d generation, reconstruction and classifi- cation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14976-14985, 2021.
15

---

Published as a conference paper at ICLR 2024
Renchunzi Xie, Hongxin Wei, Yuzhou Cao, Lei Feng, and Bo An. On the importance of feature separability in predicting out-of-distribun error. arXiv preprint M.XIv:2303.11488 2023.
Yifei Xu, Jianwen Xie, Tianyang Zhao, Chris Baker, Yibiao Zhao, and Ying Nian Wu. Energy-based continuous inverse optimal control. IEEE transactions on neural networks and learning systems, 2022.
Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-of- distribution error with the projection norm. arXiv preprint 22X.v:220..05834, 2022.
Xiang Yue, Boshi Wang. Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint (xXix:3355003/// 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115. 2021.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweightedd adversarial training. arXiv preprint arXiv:2010.01736, 2020.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03//6. 2016.
Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous gen- eralization bounds at the imagenet scale: a pac-bayesian compression approach. arXiv preprint arXiv:1804.05862. 2018
Xiang Zhou, Yixin Nie, Hao Tan, and Mohit Bansal. datasets: Consequences, source, and suggestions. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing (EMNLP), pp. 8215-8228, 2020.
The curse of performance instability in analysis
16

---

Published as a conference paper at ICLR 2024
Table 5: Details of the datasets considered in our work.
Train (Source)
Valid (Source)
Evaluation (Target)
95 CIFAR- 10-C datasets, CIFAR-10.1,
CIFAR-10 (train)
CIFAR-10 (valid)
CIFAR-10.2, CI,,III CINIC-10
CIFAR-100 (train)
CIFAR-100 (valid)
95 CIFAR-100-- datasets
95 ImageNet-CC datasets, 3 ImageNet-v2 datasets, ImageNet-Sketch, ImageNet- Rendition, ImageNet-Aulversarial ImageNat-VilRobust VidRobust
ImageNet-1K (train)
ImageNet-1KK (valid)
75 TinylmageNet- C datasets, 3 TinyImageNet-v22 datasets,
TinylmageNer (train)
TinyImageNer (valid)
TinylmageNet-Skette Sketch. TinyImageNet-Reitn Renditttonn. Iew-imme----ddesaaarrreeeeeeee Tiny Beatrngeeee-ViiRRobbe
MNLI-M, MNLI-MM, SNLL, BREAK-NLI, HANS,
MNLI (train)
MNLI (valid)
SNLI-HRAD, 4 STRESS-TEST datasets, SICK, EQUATE, QNLI, RTE, WNLI, SciTail, ANLI
FMoW (2013-15, 2016-17) X
FMoW (2002-12) 12) (train) FMoW(2002-12) (valid)
(All, Africa, Americas, Oceania, Asia, and Europe)
RxRx1 (train) Camelyon17 (train)
RxRx (id-val) Camelyon17(id-val)
RxRxI (id-test, OOD-val, OOD-test Camelyon17 (id tis-test, OOl-val, OOD-test)
A DETAILS OF DATASET SETUP
In our work, we consider both natural and synthetic distribution shifts in empirical evaluation. A summary of the datasets we used is shown in Table 5] We elaborate on the setting of datasets and models as follows:
CIFAR-10. (i)-Model. We use ResNet-20 (He et al., 2016). RepVGG-A0 (Ding al. 202 p. and VGG-11 (Simonyan & Zisserman, 2014).. They are traiedd from scratch using the CIFAR- 10 train- ing set (Krizhevsky et al. 2009). (ii)-Synthetic Shift. We use CIFAR- 10-C benchmark (Hendrycks) & Dietterich, 2019) study the synthetic distribution shift. The CIFAR- 10-C datasets have con- trollable corruption with 95 sub-datasets, involving 19 types of corruption with 5 different intensity
levels applied to the CIFAR-10 validation set. (ii) -Ntural Shift. These include three test sets: 1) CIFAR-10.1 and CIFAR-I0.2 (Recht et al.) 2018) with reproduction shift, 2) CINIC-10 (Darlow al. 2018 that is a selection of downsampled 32x32 ImageNet images for CIFAR-10 class labels.
CIFAR-100. The datasets and models used are the same as the CIFAR- 10 setup, but here we only consider the case of synthetic shift, i.e. CIFAR- -10AR-100-CBB (Hendrycks & Dietterich] 2019).
ImageNet-1K. (1)Model. We use the image models provided by timm library ( (Wightman et al. 2019 They comprise three series of representative convolution neural networks: DenseNet
DDenseNe-122111//116992
(Huang et al.] 2017). ResNet (ResNet-50/101/155)
VGG (VGG-
16/19). These models are either trained or fine-tuned on ImageNet training set (Deng et al.) 2009).
(i)-Synthetic Shift. Similar to CIFAR 10-C, we employ the ImageNet-C (Hendrycks & Dietterich 2019) to investigate the synthetic shift. This dataset spans 19 types of corniption with 5 severity lev- els. (ii) Shift. We consider five natural shifts, which involve: 1) dataset reproduction shift for ImageNet- V2-A/B/C (Recht et al. 2019). 2) sketch shift for PrageNNe-S(kketee (Wang et al., 2019), 3) style shift for ImageNet-R(endiinee 4) adversarial shift for ImageNett-Adveersaiiaa) with 200 ImageNet classes (Hendrycks et al. 2021b
with 200 ImageNet classes (Hendrycks et al. 2021a),
5) temporal shift for VingeNet-Vii(Robuuse with 30 ImageNet classes (Shankar et al. 2021).
TinyImageNet. (i)-Model. We use classic classifiers: DenseNet-161 and ResNet-50. They pre-trained on ImageNet and fine-tuned on the TinyImageNet training set (Le & Yang. 2013). (i)--yythettic Shift. Following the practice of ImageNet-C, we adopt TinyImageNet-C which only applies 15 types of corruptions with 5 intensity levels to the TinylmageNet validation set. (iii) Naturrl Shift. We select the same naturally shifted dataset as in the ImageNet-Ie Setup, but only pick the parts that share the same classes TinylmageNet.
17

---

Published as a conference paper at ICLR 2024
MNLI (i)-Model. For the natural language inference task, utilize pre- pre-trained versions of BERT
(Devlin et al. 2018) and RoBERTa (Liu et al.) 2019) from HuggingFace library (Wolf the MNLI training set (Williams et al, (i)--Sntheticc Shift. In the MNLI setup, we combine these datasets to examine synthetic
et al.
2020). These transformer models are fine-tuned
2018
shntc:
MNLI-M, MNLI-MM, SNLI (Bowman al.] 2015), BREAK-NIJ (Glockner et al.) 2018).
HANS (McCoy et al. 2019). SNLI-HRAD (Gururangan et al. 20188) STRESS-TER (Naik et al.
2018). SICK (Marelli et 2014) and EQUATE (Ravichander et al. 2019). The STRESS-TES"
containing 4 sub-datasets, includes shifts such as length mismatch, spelling errors, word overlap, and antonyms. (more detailed descriptions of these datasets can be found in Zhou et al. (2020)
Nitur-atural Shift. We discuss two types of shifts: (1) domain shift in QNLI, RTE, WNLI (Wang
al. 2018). SeiTail (Khot et al., 2018); (2) adversarial shift in ANLI (Nie et al. 2019);
Camelyon17-WII.D (i)-Model. Following the setting of (Garg et al. 2022), we use ResNet-50 and DenseNet-121 They are pre-trained on ImageNet and fine-tuned on Camelyon training set.
(ii--yythetiic and Natural Shift. We used the official synthetic and natural shifted datasets provided in (Koh et al. 2021aa.
RxRxI-WILD. (i)Model. Following the setting of GGag al. 2022 we use ResNet-50 and DenseNet- 121. They are pretrained on ImageNet and fine-tuned on RxRxI's training set. (ii) Synthetth and Natural Shift. We used the official synthetic and natural shifted datasets provided
in Koh et al.] 2021a).
ResNet-50 and DenseNet- 12T. They are pretrained on ImageNet and fine-tuned on FMoW's training set. obtain 12 different synthetic and natural shifted
FMoW-WILD. (i)-Model. Following the setting of (Garg al. 2022)
(i)-Synthetic and Natural Shift. Similarly, datasets by considering images in different years and by considering five geographical regions as subpopulations (Africa, Americas, Oceania, Asia, and Europe) separately and together according to (Koh et al. 2021a).
B BASELINE METHODS
Below we briefly present the baselines compared our work, where we denote the classifier f, and the unlabeled dataset D, drawn from target distribution Py:
Average Confidence (ConfScore). The model's accuracy on target data is estimated as the expected value of the maximum softmax confidence (Hendrycks & Gimpel 2016);
CoufScore En-P,
max Softmax(f;xz)) jEy
(10)
Average Negative Entropy (Entropy). The target accuracy of a model is predicted by the expected value of the negative entropy (Guillory al. 2021h:
Entropy EzzD,
Ent (fftmax(f;()))
(11)
jEy
where (nt(p)= -p-log (p). Note that, the Difference of Confidence (DoC) (Guillory et al. 2021 equals to the ConfScore and Entropy indicators when there is label space shift between source and target distributions, i.e. closed-set setting.
Frechet Distance (Frechet). The model's accuracy target can be assessed by the Frechet Distance between the features of the training set D, and the target set (Deng & Zheng 2021):
Frechet FD (EEND. [(f(x)) EZ-D, [((())))
(12)
where FD (Do, D.) = Ile, - 1 2 Tr (, + En - 2 (E, )) and Σ are the mean feature vectors and the covariance matrices of dataset.
Agreement Score (AgreeScore). The model accuracy is estimated as the expected disagreement of two models (trained on the same training set but with different randomization) on target data (Jiang et al. 2021): :
[T] max Softmax(f{( + max Softmax(f) jey
AgreeScore = EEZD,
(13)
iEY
18

---

Published as a conference paper at ICLR 2024
where f and f² are two models that trained on the same training set but with different initial- izations.
Average Thresholded Confidence (ATC). This method learns threshold t on model confidence scores from source validation data D,, then predicts the target accuracy as the proportion unla- beled target data with score higher than the threshold (Garg et al. 2022
ATC EEwDa
I
Ent Softmax(f)(x)) JEY
(14)
EEWD
Ent Softmax(I,(x))) < jEy
<
I y arg max softmax (f, (=))
(15)
jEy
Average Energy (AvgEnergy). This measure is self-designed metric closely tied to our MDE, which predicts the model's 's accuracy by the expected value of the energy on target data:
K Sta/TT
AvgEnergy = En [2(z; BaND.
T log
(16)
Projection Norm (ProjNorm). This algorithm pseudo-labels labels the target samples using the classifier f. then uses these pseudo data (x, ÿ) to train new model f from the initialized network fo- The model's target accuracy is predicted by the parameters difference of two model (Yu et al. 2022):
7 =: arg max softmax (fy (x*))
(17)
jey'
ProjNorm
[0 07|12-
(18)
Nuclear Norm (NuclearNorm).
This approach uses the normalized value of the nuclear norm (i.e., the sum of singular values) of the prediction matrix to measure the classifier accuracy the target dataset (Deng et al. 2023):
|| (f)
NuclearNorm = EzwDa
(19)
/min (INV), K) |N|
where [|p]|. is the nuclear norm of and |N| is the cardinality of Du. K is the number of classes.
Confidence Optimal Transport (COT). This approach leverages the optimal transport framework to predict the error of model as the Wasserstein distance between the predicted target class proba- bilities and the true source label distribution (Lu et al. 2023b):
COT W (S (c),
(20)
where Woo is the Wasserstein distance with c(x, y) = || yllooo c(x, y) is cost function that tells us the cost of transporting from location x. to location y. P (c) is defined to be the pushforward of a covariate distribution Pr.
C DETAILED THEORETICAL ANALYSIS
we provide some more detailed discussions of this theorem here, including its basic assumptions and a complete proof of the theorem. We start with an assumption that the well-trained discriminative classifier (i.e., EBM) makes correct inferences of sample (2,, y1) with minimum energy.
Recalling Theorem 3.1
Assumption C.1 Vy E y and F yi. for sample (Ia, v.). the model will give the correct answer for x, if (z,, ye f) (I,, y: f).
To ensure the correct answer is robustly stable, we may opt to enforce that its energy is lower than the energies of incorrect answer y; by positive margin This modified assumption is as follows:
19

---

Published as a conference paper at ICLR 2024
Assumption C.2 For incorrect answer Ye. sample (I', y4) and positive margin m. the inference algorithm will give the correct answer for 24 if Z (æ ys:
(2 Ds: f)
Now, ready deduce the sufficient conditions for the minimum loss function. Let points (z1, 22) and (z1, 22) belong to the feasible region R, such that (21,22) E HP1 (that is, + m < 22) and (21, (2) HP2 (that is, 2 + m 24).
Corollary C.1 For sample (Is, V.) and positive margin m. Minimizing the loss function & will satisfy assumptions C.I or C.2 if there exists at least one point (21, 22) with 21 + m. < 22 such that for all points (2 22) with 21 + m 2 z2 we have Liz,) (21, 22) < Liz,1 (21,22) where [Zy] contains the vector of energies for all values of y except yi and yo-
Next, with the well-trained classifier f, proceed correlate the MDE and its classification accuracy in out-of-distribution data (x, pr. The temperature ''' is positive constant and defaults 1. To do this, first express the negative log-likelihood loss for f
efuun)/T
Call = E (n.y)~-P
log E ebs())/T)
K
-Sy(x)/T log Σ elitz)
= (a,y))>-r
(21)
Then, we represent the MDE computed by f on x ~ pr as follows:
MDE =
Ezepr
log Softmax
-T log Σ eb cbtu)/
1=1
log j-= cxs():)/
Ezapr
log
i= log Σラ=1 efs(ms))"
N
K
E Ezapr
log jml
cls(ws/TT + +log
Σ
Σ
ch(eS/T
(22)
Afterward, we represent the difference hetween the MDE indicator and the negative log-likelihood loss follows:
eb(es// )7)
A MDE Lall Expor
log
R(ox)~pr (Jy(z)/T)
(23)
For each sample point (r,, ny), the subtraction form can be rewritten as:
Σ CSCUN/ + fx.
A =
MDE'
C.u -log
T-
fy. (I.)/T max (zt) /T
(24)
JEY
00 < O,
if Vi.
(25)
if # Vi.
which is our deduced result. In this proof, assume ideal situation where T approaches 0, but usually in practical applications defaults to 1. Finally, by judging whether the term of Eq. 24 know whether the index corresponding to the maximum logits is the label
is less than 0, y. i.e., we can obtain the accuracy value the classifier f. Thus, we theoretically substantiated a correlation between MDE and accuracy by mathematical theorem.
20

---

Published as a conference paper at ICLR 2024
(a) original image
(b) brightness
(c) contrast
(d) elastic transform
(e) fog
(f) frost
gaussian blur
(h) impulse noise
(i) regppomrrrsion
saturate
(k) snow
(I) spatter
Figure 4: Visualized examples of synthetic sets in ImageNet-11 1 Setup
D SAMPLE VISUALIZATION OF SYNTHETIC SETS
In Fig. we provide some visualized examples of synthetic sets undergoing various transformations in ImageNet-Ik Setup.
E DISCUSSION: CLASS-LEVEL CORRELATION
From the previous results, we have observed strong linear correlation built at the dataset level between MDE and classification accuracy. A question naturally arises: can their correlation at the category level also be established? To this end, plot the T-SNE visualization of clustering the penultimate classifier features in Fig. 5] Whether on the ID or any OOD datasets, we found that the accuracy of each class showed consistent trend of "co-varying increments and decrements" with its MDE, i.e., characterized by a positive linear correlation. Also, the CINIC-10 and STL-10 clusters are poorer than the remaining datasets. effects because they have better classification accuracy, which corresponds to the lower accuracy of CINIC-10 and STL-10. Of course, that is because their samples are mainly transformed from the ImageNet dataset.
hard see that models can have better clustering
DISCUSSION: DICTA-DISTTBBUOON ENERGY FROM DIFFERENT LAYERS
Here, we wish to explore whether features from other layers can compute MDE scores that are as discriminative as the (default) classification head features. In Fig. 6 display the MDE scores
21

---

Published as a conference paper at ICLR 2024
Table 6: Correlation comparison with existing methods on synthetic shifted dataset of ImageNet. We report coefficient of determination (R²) and Spearson's 's rank correlation (p) (higher is better). The highest score in each row is highlighted in bold.
ConfScore
Entropy
Frechet
ATC
COT
NucleurNorm AvgEnergy
MDE
Dutasel
Network
P
Rº
R
K²
P
R²
P
N²
ResNei-152
0,980 0,949 0,979 0.946 0.945 0.879
0,980
0.899
0.968
0.943
0.979
0.961
0.980
0.955
0.982
0.967
DeuseNer-169
0.983 0.953 0,981 0.931 0.942 0,878 0.982
0.891 0.963 0.934
0.981
0.956
0.984
0.955
0.985
0.960
ImageNet
VGG-19
0.966 0,933 0.968
0.909 0.978 0.910
0.975
0.886
0.981 0.916 0.978
0.949
0.980
0.950
0.987 0.952
Average
0.976
0.955
0.892 0.971 0.979
0.955
0.981
0.985
0.960
Table 7: Mean absolute error (MAE) comparison with existing methods on natural shifted datasets of ImageNet. The best result in each row is highlighted bold.
Dataset
Unseen Test Sets
ConfScore
Entropy
Frechet
ATC
COT
NuclearNorm AvgEnergy
MDE
ImageNet-V2-A
8.76
9.72
6.36
6,88
7.04
4.04
3.58
2.21
ImageNet-V244
8.59
10.20
9.51
8.82
9.02
5.20
4.41
2.35
ImageNet-VV
14.92
10.27
9.12
8.74
8.85
5.66
5.01
4.27
ImageNet
ImageNet-V2.
-2-S
7.30
9.50
10.45
9.81
8.11
7.77
6.40
5.31
ImageNet-V2-RR
15.41
13.63
12.06
12.70
12.87
11.34
10.64
7.68
ImageNet- V2-Vid Vid
15.18
14.26
16.53
14.42
13.49
12.77
11.19
8.09
ImageNet-V22 V2-Avv
19.09
18.28
18.37
19.20
14.51
13.61
11.43
8.42
Average
12.75
12.27
11.77
11.51
10.56
8.63
7.52
Table 8: Mean absolute error (MAE) comparison with existing methods of natural shifted datasets on Wilds. The best result in each row is highlighted in bold.
Dataset
Shift
ConfScore
Entropy
Frechet
ATC
COT
NuclearNorm AvgEnergy
MDE
Camelyon17
Natural Shift
9.01
8.19
8.49
7.46
5,31
4.26
3,21
2.93
RaRx1
Natural Shift
6.63
6.32
5.49
5.45
4.45
3.67
1.62
FMoW
Natural Shift
10.52
9.61
7.47
6.13
5.26
4.49
2.90
2.24
calculated for these features from different layers (i.e. the output of each block) on the ID and OOD test sets. Without exception, the MDE scores calculated by shallow features all fall within the same numerical range, and their discriminability is far inferior to the MDE of the classification head feature. This justifies that the powerful representation ability of the classification head feature the foundation of why our MDE can work.
G DISCUSSION: ANALYZING CLASS DISTRIBUTION BY ENERGY BUCKETING
In this spot, aim to understand the distribution of sample categories based energy scores. In other words, what type of sample corresponds to what energy score? Specifically, in Fig we divide the samples into different buckets per the energy value, and then further analyze the proportion of each category in each block. We discuss these results from three aspects according to the ID and OOD data sets:
i) From Fig (a) and (d), we can see that within each (same) energy score range, the class dis- tribution in the ID data remains relatively balanced, while class distribution in the OOD data cxhibits an imbalanced trend.
ii) Comparing Fig g (a) and (d), we can observe that across (different) energy score segments, the proportion of the same category of OOD data fluctuates more drastically than ID data, such as dog at the top 10% energy scores and horse at 10% 20% energy ranges in (ig.
iii) The aforementioned two phenomena also held under different backbones, illustrated in other figures.
22

---

Published as a conference paper at ICLR 2024
Table MDE's coefficients of determination (R²), Pearson's 's correlation (r), Spearman's 's rank cor-
relation (p) and mean absolute emrors (MAE) scaled temperature constants.
0.01
0,988
0.989
0.991
0.991
0.990
0.988
0.987
0.986
0.984
0.983
0.983
0,989
0.988
0.987
0,981
0.974
0.968
0.964
0.961
0,960
0,959
0,959
y²
0.978
0.977
0.974
0.963
0.948
0.937
0.929
0.924
0.921
0.920
0.919
MAE
1.72
1.80
1.78
1.94
2.28
2.71
3.32
4.25
5.10
6.32
7.98
10
20
30
40
60
70
80
90
100
0,982
0.976
0.974
0.974
0.974
0.973
0.971
0.972
0,971
0.971
0.958
0.959
0.959 0.959 0.960
0.960
0.960
0.960
0.960
0.961
0,918
0.919
0,920
0.921
0,921
0.921
0.921
0.922
0.922
0.923
MAE
9.50
9.87
10.05 10.16
10.22
10.24 10.25
10.25 10.24 10.26
Table 10: MDE's coefficients of determination (R²), Pearson's correlation (r), Spearman's rank correlation (p) and mean absolute errors (MAE) different lincar regressor.
RobussL.nnerRRegresson LinearRegression
0.991
0.991
0.987
0,987
0,973
0,974
MAE
1.80
1.78
H COMPLETE SET OF CORRELATION SCATTER PLOTS
Here, we display the complete set of correlation scatter plots for all methoods/dataasets/modde archi- tectures, as follows in Fig. 10
I LIMITATION AND FUTURE WORK
We now briefly discuss the limitations of mest-distribution energy and future directions. Our method is grounded on an assumption that approximates the unknown test environments via data transfor- mations applied to the synthetic sets. So one limitation is that MDEE hinges on sufficient samples and shift types to make accurate predictions on the OOD test set. It would be practical to reduce the sample requirements of this method, ideally be one-sample version of MDE, Another is- sue is that MDE sometimes performs poorly on "extreme" shifts, as the energy of the data point doesn't 't reflect its information anymore, under these challenging scenarios such as adversarial tacks and class imbalance. This limitation may require new tailored techniques to be addressed, which suggests an interesting avenue for future work. Furthermore, we believe the concept of Au- toeval can also play role in many more AI fields, such text-video retrieval (Han et al.) (2023), machine translation (Peng et al. 2022), sentiment analysis (Lin et al. 2023), which also represents potential research direction in the future.
DIFFERENT LINEAR REGRESSORS.
For analysis of whether the regression performance of MDE is influenced by the choice of gression models, we selected both linear regressors and robust linear regressors for comparison. As indicated Table [0] the results suggest that the selection of different regression models has no significant impact on the accuracy prediction performance.
K AUTOEVAL DIFFERENCE FROM UNCERTAINTY ESTIMATION AND OOD DETECTION
AutoEval, uncertainty estimation and cittof-dditributtion (OOD) detection are significantly different tasks. First, the three tasks have different goals. Given labeled source data and unlabeled target
23

---

Published as a conference paper at ICLR 2024
data, uncertainty estimation is to estimate the confidence of model predictions to convey informa- tion about when a model's output should (or should not) be trusted, AutoEval directly predicts the accuracy of model output. Unlike OOD detection, which aims to identify outlier test samples that are different from training distributions, AutoRval is an unsupervised estimation for the model's accuracy across the entire test set. In this regard, AutoEval is a task that assesses the effectiveness and deployment worthiness of a model by directly predicting its accuracy in testing environment. Second, our work was not inspired by uncertainty estimation OOD detection techniques, but rather by the characteristics of the energy that fulfilled our desire to build an efficient and effective AutoEval framework.
L META-DISTRIBUTIE ENERGY V.S. SOFTMAX SCORE
Here, we demonstrate that the relationship between MDE and Softmax Score is not simple placement by comparing them from three aspects:
i) They are different in essence and mathematical form. Essentially, MDE is defined meta-distribution statistic of non-normalized energy at the dataset level, while Softmax Score represents the maximum value of the normalized logit vector for an individual sample. In terms of mathematical formulas, they have the following distinct expressions:
N
N
M(zn:))
MDE(2;
log Softmax 2(z = N
log
(26)
N
E cl(us)))
efx(x)
e (x)
max ply x) Σ, efi(x)
(27)
Exe(()
V
ii) Their usage in the AutoEval task is different. Softmax score typically reflects the classi- fication accuracy through measures such as the mean (ConfScore, Entropy), mean difference (DOC), the data proportion below a certain threshold (ATC). On the other hand, MDE pre- dicts accuracy by a regression model.
iii) MDE is more suitable than Softmax score for AutoEval task. To demonstrate this, decompose the softmax confidence by logarithmizing it as follows:
log max p(y x) = (x; f(x) fma (x))
y
(28)
Exx; finax (x)
Then, find fmmar (z) tends be lower and Ex f) tends be higher for OOD data, and vice versa. This shifting results in Softmax score that is no longer suitable for accuracy prediction, while MDE is not affected by this bothersome issue.
M COMPARISON IN TERMS OF EVALUATION TIME AND MEMORY USAGE BETWEEN MDE AND THE TRAINING-MUST METHOD
In this section, we want to clarify the advantage of MDE over the training-mus must approach in terms of time and memory consumption. However, due to the significantly different workflows of various methods (e.g., training the model from scratch, fine-tuning the model, calculating model features, statistically analyzing dataset distribution, computing the disagreement of ensemble prediction, etc.), is impossible to compare them directly and fairly. So, we simplify this problem to compare the time complexity and space complexity of different methods in rough granularity:
For time complexity:
AC ANE MDE (ours) Nucdaaroomm..ffeeeecchhoootttttn <AgreeScore (rriining-must).
For space complexity:
training- frree methods (AC, ANE, ATC, AvgEnergy, MDE(ours), NuclearNorm, Frechet) <training- must methods (ProjNorm, AgreeScore).
24

---

Published as a conference paper at ICLR 2024
INDU DI AM MOCC 1/A 13:5
PISTRC CUC NOL
AORTA HLL CND 1LAY
Poo 2ht WAL
POATREC DAL WEL 2E.0
MCC MICHH.11 11
one AoC MEC 11-13
DR BACO 1S. MA00210
20
20
25
0-
20-
20
-25-
-40
-40
-50
-00
-10
-50
-25
25
50
75
-20
-60
-2
20
60
-100
-75
-30
-5
50
75
(a) 10FAR10-test on ResNet-20
(b) CIFACRIO---- -0-tet on RGpVGG-AO
(c)
on VGG-11
AAl 1300
Iit A.D AER 12.05
D.E MOR 11.71
onS AN HOS 12410
PCC Sar MON 15-10
50
20
70-
-20
30-
-40-
-20-
40
-00-
-10
6
6
CIFAR-10.1 on 22
CIFAR-10.1 on RGpVGG-AO
(1) CIFAR-1O.1 on VGG-11
pestccoo 20.00 1th 1291
wit won tor yuue
Kag CaE ELAI PRIUER WAG. 2EL
MOS SILLNE ILAE PNC 10L MUN 11.15
PNC 11.0
60
75-
20
20
25
0-
-20
-20
-0-
-50-
-60
-60
15
72
3
25
50
15
-2
26
05
20
40
20
20
40
d2
CIFAR-10.2 on 2e
CIFAR- 10.2 on VGG-AA
CIFAR- 10.2 Vgg-
CAL MOL CLAH
20
-00
-20
-20
400
-40
-20
20
40
-00
-20
20
-40
-20
23
(i) CINIC-10 on ResNet-20
(k) CINIC-10 RepVGG-AO
()) CINIC-10 on VGG-11
25

---

Published as a conference paper at ICLR 2024
1X00
1299
COCON ICC STV ME KCMT
BAC MOCO MO 1105
WENIO LMOCTEA MAT
231 ME 1L.10
Wee LOCV 2020
50
-20
-70
-40
-50
-40
-50
-00
(m) STL-10 on ResNet-20
(n) STL-10 on RepVGG-A0
(o) STL-10 on VGG-11
Figure 5: T-SNE visualization of the classification features clustering CIFAR-10 setup. Different
colors correspond to different classes, and its accuracy and MDE are placed above.
26

---

Published as a conference paper at ICLR 2024
12.0
20.5
10.5
30
Layer
Layer
(a) CIFAR-100 on ResNet-20
(b) CIFAR-1O- 1EARR-O-ttn RepVGG-AO
(c) CIFAR-1O-t on VGG-11
11.
10.
11.0-
12.5
23
12.0
95
0.0
65
0.0
7.0
7.5
(d) CIFAR- 10.1 on ResNet-20 (e) CIFAR-10.1 on RepVGG-AA
(f) CIFAR -11.R VGG-11
11
11.5
11.0
9.3
(g) CIFAR-10.2 on ResNet-20
(h) CIFAR-IO4 on RepVGG-AO
CIFAR-10.2 VGG-11
15.
135
140
34.0
130
12.5
Layer
Layer
Layer
(i) CINIC-10 on ResNet-20
(k) CINIC-10 on RepVGG-AO
(I) CINIC-10 on VGG-11
12
12.5
12.0
11.
11.5
11.
12.
4.
6.5
Layor
Layer
(m) STL-10 on ResNet-20
(n) STL-10 on RepVGG-AO
(o) STL-10 on VGG-11
Figure 6: MDE scores calculated from the features of different layers in CIFAR-10 Setup.
27

---

Published as a conference paper at ICLR 2024
0.00
8.00
Commeattonn
31
1022
0.00
Temporoture
Tompensture (
Seed
Seed
(a) Temperature Constants
(b) Random Seeds
Figure 7: MDE's coefficients of determination (R²), Pearson's correlation (r), Spearman's 's rank cor- relation (p) and mean absolute errors (MAE) (a)-scaled temperature constants and (b)-different random seeds.
28

---

Published as a conference paper at ICLR 2024
14
17%
30%
30N
40C
50h
8PC
75%
M6%
99%
1806
15%
20h
39%
40%
5WL
40%
60%
90h
2005
30%
39%
30%
475
58%
60%
79%
996
10N
CIFAR-10 test ResNet-20
(b) CIFAR-10 test RepVGG-AO
CIFAR-I0 test VGG-11
3P%
6PC
MFC
59C
1856
1Ph
19C
0AC
PPL
RAC
5006
5PC
39%
6YC
05%
75%
99%
10M
(d) CIFAR-10.1 on ResNet-20
(e) CIFAR-10.1 on Rep
(1) CIFAR-10.1 on VGG-11
Eirryy 51/00
Eerry
Swwryy Stoee
(g) CIFAR-10.2 on ResNet-20
(h) CIFAR-10.2 on RepVGG-AO
(i) CIFAR-10.2 on VGG-11
argiane
ri
argare
Swwry Scon
Teergy SCOr
SWAgy 200e
CINIC- 10 on 20
(k) CINIC- on RepVGG-AO
(I) CINIC- on VGG-11
orpone
922
35%
3Ch
BEN
2004
15h
33%
45%
DUR
2006
50% 25%
476
18%
80%
125
100N
(m) STL-10 10 on ResNet-20
(n) STL- RGpVGG-AO
(0) STL-10 VGG-11
Figure 8: Bucketing of sample categories by energy scores in the CIFAR-10 Setup.
29

---

Published as a conference paper at ICLR 2024
ConfScore V.S. Accuracy
Entropy V.S. Accuracy
Frechet
Accuracy
50
Job4/8 393
Accuracs
60
1
20
-03
ConfScore
Entropy
Frechet
AcreeScare
ATC v.S. Accuracy
Pro Norm Accuracy
Brwpass
Accuraty
50
AgrenScare
(-1)ATTC
(-1) ProjNorm
NucicarNorm
Accuracy
AugEnergy VSS. Accuracy
MDE V.S. Accuracy
Ribbuss
80
959
aud 992
Accuracy
10
Pepcua
0.55 575
155 305 32 12.0 12
NuclearNorm
AugEnergy
MDE
Figure 9: Correlation comparison of all methods using ResNet-20 synthetic shifted datasets of CIFAR-10 setup. We report coefficients of determination (RP), Pearson's 's correlation (r) and Spearman's 's rank correlation (p) (higher is better).
30

---

Published as a conference paper at ICLR 2024
ConfScore V.S. Accuracy
Entropy Accuracy
Frechet Accuracy
Accuraty
B
Accuracy
20
Bradinn
1.01 ConfScore
Entropy
ScreeScare
ATC V.S. Accuracy
Pro Accuracy
Frep,255 250 pr00001
prd
Accuratyy
20
AgrenScare
(-1)ATTC
(-1) ProjNorm
NucicarNorm
Accuracy
AngEnerg V.S. Accuracy
MDE V.S. Accuracy
Rn-0u326
110 990
Rrobag)
Far
Accuracy
20
05
055
87
03
20.0
10.5
11.7
NNccearNorr
AugEnorgy
MID
Figure 10: Correlation comparison of all methods using RepVGG-A0 synthetic shifted datasets of CIFAR-10 setup. We report coefficients of determination (R²), Pearson's 's correlation (r) and Spearman's 's rank correlation (p) (higher is better).
31

---

Published as a conference paper at ICLR 2024
SonfScore Accuracy
Entropy V.S. Accuracy
Frechet Accuracy
60
Krwo,ss
50
Rrea250
Freaaza p-6
60
p-8 303
Accuraty
70
70
Mari
ConfScore
Entropy
(-1) Frechet
AcrenScore V.S. Accuracy
ATC v.S. Accuracy
Pro Norm vS.. Accuracy
50
110
Rlo2ago 210:413
Ri-nas
80
p=1.002
20
p=E.203
Accuracy
70
pen
Brfcc
80
333 633 0. 2. 0.55
-150 (-1]-ATC:
-0-28-057----- 122---5-----------
AgronScare
(-1) (ro-ProoNorm
NuclearNerm v.s. Accuracy
AugEnergy
Accuracy
MDE V.S. Accuracy
60
Natio Sud
240 933
Accuracy
70
30
05
0.7
0.5
3.0
10
30.0
335
33.5
11.5
120
12.5
NuclearNorm
AugEnorgy
Figure 11: Correlation comparison of all methods using VGG-11 on synthetic shifted datasets of
CIFAR-10 setup. We report coefficients of determination (IP2), Pearson's 's correlation (r) and Spear- man's 's rank correlation (p) (higher is better).
32

---

Published as a conference paper at ICLR 2024
Accuracy
Entropy V.S. Accuracy
Accuracy
Erw0a2
Fr-oaso
pirt0002
p-t
pet
Accuraty 6
Adi
1-1
10
20
Landa
Con/Score
Entropy
(-1) Frechet
AgronScore V.S. Accuracy
ATC V.S. Accuracy
Proj Norm vs. Accuracy
50
Ri-0333
50
Rio0a27 p=.:307
p=1
Accuracy
20
Pho
10
6:15 0.8 023000000000000 AgronScare
-2
-66
45 4.2
(-1)ATCC
(11 ProjNorm
MuclearNorm v.s. Accuracy
AugEnergy VS. Accuracy
MDE V.S. Accuracy
TwE997
Nafic Sud
50
too
Accuracy
Ro.
Accuracy
10
20
02
35
03
5.5
85 2.0 93 1-01--110L
10.5
35.5
23.0
11.5
32.0
KuclearNorm
AugEnergy
Figure 12: Correlation comparison of all methods using ResNet-20 on synthetic shifted datasets of CIFAR-100 setup. We report coefficients of determination (R²),, Pearson's 's correlation (r) and Spearman's rank correlation (p) (higher is better).
33

---

Published as a conference paper at ICLR 2024
ConfScore V.S. Accuracy
Entropy V.S. Accuracy
Frechet V.S. Accuracy
10-
pertiiis
penapy
Fed
46-
Accuracy
Accurao)
48-
20
42 11300330000 5.75
-22-20 1.2.-.10-000
-000 -430 -200
ContScore
Entropy
[1])Frechet
AgreeScore V.S, Accuracy
ATC V.S. Accuracy
ProjNorm vs. Accuracy
1-5572
JeE302
1=6206
dic
50
Accuracy
40
30
20
10
TAC.COOOOOR:-11100 SAA.PP/0E..00077700000000000 (-1)-ATC
AgreeScore
(-1) ProjNorm
Accuracy
AugEnergy vs. Accuracy
Accuracy
Pruncsh
E. 50%
Accuracy
16
30
1M00C Car
10
450
NuclearNorm
AugEnergy
MDE
Figure 13: Correlation comparison of all methods using Rep on synthetic shifted datasets of CIFAR-100 setup. We report coefficients of determination (R²), Pearson's 's correlation (r) and Spearman's 's rank correlation (p) (higher is better).
34

---

Published as a conference paper at ICLR 2024
ConfScore Accuracy
Entropy V.S. Accuracy
Prachet v.s. Accuracy
Erw0237
50
p-b.ar3
30
p-t 372
Accuracy
20
20
0 Cemte
ConfScore
Entropy
Frechrechht
AprenScore V.S. Accuracy
ATC V.S. Accuracy
Pro/Norm V.S. Accuracy
60
Mi-0u53
p=1.302
60-
Ri-bas
pet
red 20.
Accuracy
20
30
2.2
0000 2.0 0.2-2-00-----222 0.2-57750-000.
-28
AgronScare
(-1)ATTC
PrayPooooomm
Nucles Nurm v.S. Accuracy
Auggnyrry VS. Accuracy
MDE V.S. Accuracy
50
Accuracy
40
60
40-
10
03
24
65
06
077 33
13 32 AugEnergy
32
24
33
NuclearNorm
Figure 14: Correlation comparison of all methods using VGG-11 on synthetic shifted datasets of CIFAR-100 setup. We report coefficients of determination (lt2), Pearson's correlation (r) and Spearman's rank correlation (p) (higher is better).
35

---

Published as a conference paper at ICLR 2024
ConfScore V.S. Accuracy
Entropy V.S. Accuracy
Frechet V.S. Accuracy
Tea
Accuracy
16-
20
Ericesoee
1013 temprsson
52
66 65
-2. -35
101.0000000000000000000000000000 200-3000---00000 250-200-- (1)
Confscore
Entropy
Accuracy
ATC v.S. Accuracy
Norm vs. Accuracy
Ring
J=E254
Fg
50.
Accurony
48-
200%
10
665
6.7
0.92
585
-230
-1152 -11.5
AgreeScore
(-1)-ATC
(-1) ProjNor)
NuclearNorm v.S. Accuracy
AvgEnergy eeroyy Accuracy
Accuracy
p- 961
Fert
Accuracy
40
30
20
NudlerNorm
AugEnergy
MDE
Figure 15: Correlation comparison of all methods using ResNet-50 on synthetic shifted datasets of TinyImageNe setup. We report coefficients of determination (R²2), Pearson's correlation (r) and Spearman's 's rank correlation (p) (higher is better).
36

---

Published as a conference paper at ICLR 2024
CanfScore Accuracy
Entropt Accuracy
Accuracy
Fhet 221
Ret 704
Frw0a23
pet
Accuracy
10
Accurao
30
20
ConfScore
Entropy
(-1) Frechet
AgreeScore v.s. Accuracy
ATC V.S. Accuracy
Pro Norm vs. Accuracy
10
R--043
Rioosa p=t.2b?
1eb
1g
pet
nerive Bic
40
Accuracy
50
20
0-
250 11 21 AgrneScore
1.0 0.0 0.5-1 20-0-22- 10---------0---------- 20--0-2-16
11.55 11.2 31.99 12.0
(-1)ATTC
(-1) ProPoroNorm
NuclearNerm v.s. Accuracy
AugEnergy VS. Accuracy
MDE V.S. Accuracy
50
Accuracy
40.
9
20
10
01
022
03
24
05
0.5
00 8.5
75
05
90
34
3.5
5.5
33.0
15.2
KuclearNorm
AugEnergy
Figure 16: Correlation comparison of all methods using DenseNet-161 on synthetic shifted datasets of TinyImageNer setup. We report coefficients of determination (R²²), Pearson's 's correlation (r) and Spearman's rank correlation (p) (higher is better).
37

---

Published as a conference paper at ICLR 2024
ContScore V.S. Accuracy
Entropy V.S. Accuracy
Accuracy
r-0-773
280.882
82-
253
JAU
JAU
Accuracy
5300
246555
2
218224
2.65
0.88
1880
6.42
6.4
-0.0
-5
-22
-15
ConfScore
Entropy
frec-rechet
AgreeScare Accuracy
ATC Accuracy
ProjNarm Accuracy
Rt-0.157
Rt. 84-0467
R1-0.50 p=ii836
pet.528
MELLK
p=i.530
MILLI
MM11
Accuracy
118350
62-
62
-2.00 -035 -3.30-025 -3.70-022
-2
AgreeScore
(-1NATC
(-1)-Pro/Norm
NuclearNorm V.S. Accuracy
AvgEnergy V... Accuracy
MDE V.S. Accuracy
92
C450
783
MELM
5.85
Accuracy
Accuracy
Accuracy
42
S18554
STRESL
3102500
079555
10
50
52
3.2
NuclearNom
AvgEnergy
Figure 17: Correlation comparison of all methods using BERT synthetic shifted datasets of
MNLI setu. We report report coefficients of determination (R2), Pearson's 's correlation (r) and Spearman's rank correlation (p) (higher is better).
38

---

Published as a conference paper at ICLR 2024
Accuracy
Entropy V.S. Accuracy
V.S. Accuracy
160
mho
MS-R
Accuracy
BREACE
accuracy
FRASCEL1
06 1AU
20
72
318150
-200-0155-0 30-044-0-4000
-20
Conf5core
Entropy
Frechet
AgreeScore V.S. Accuraty
Accuracy
ProjNorm V.S. Accuracy
120.22
1=0.550
1e0.20
22
HIL-W
Accuracy
[RCKCL
EROACKL
80-
70-
1111300 0
3191200
0.06 AgreeSccrre
ATI
P-oy)rro/Nom
NuclearNorm v.S. Accuracy
AugEnergy v.S. Accuracy
V.S. Accuracy
150
100
152
190.87
1=0,000
200
Accuracy
Accuracy
FRESCK
Accuracy
PROACK
43
22
70-
1113300
70-
2111200
341 6.6 CA2
2.2
2.8
0.5
NudearNorm
AvgEnergy
Figure 18: Correlation comparison of all methods using RoBERTa on synthetic shifted datasets of MNLI setup. We report coefficients of determination (R²), Pearson's 's correlation (r) and Spearman's rank correlation (p) (higher better).
39


Reviews:
[{"review_id": "rgqlbug1KA", "reviewer": "Reviewer_xmk2", "summary": "This paper intends to solve the auto evaluation problem of a well-trained classification model on a test dataset with domain shift and without labels. The proposed method is based on an energy-based framework to estimate the Meta-Distribution Energy, which is used to train a regression model on synthesized dataset for prediction of classification accuracy on unlabelled test data.", "questions": "1. For Eq(5), is MDE defined on specific data $x_n$, where MDE(x; f) should be MDE($x_n$, f)? Or does it miss an expectation term over the dataset?\n2. How is the synthesized test data generated when training the regression model for accuracy prediction?\n3. If the proposed method indicates a significant drop in a new dataset, is there any way to correct this bias based on the proposed MDE?\n4. How is the correlation evaluated in Table 1 on a specific dataset? \n5. How to determine the best hyper-parameter of T on a new dataset? Is the best parameter related to the dataset or the classification model?\n6. Will the selection of different regression models affect the prediction accuracy?\n7. Is there any comparison on the evaluation of domain-shift data with real labels?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper introduces an energy-based automatic evaluation framework designed to enhance efficiency and mitigate overconfidence in existing methodologies. The proposed approach indicates better prediction on unseen test data over the other measurement methods on dataset in different modalities and with different classification models.", "weaknesses": "1. It is recommended to use a different symbol for the normalization term E(x) in Eq(2) to avoid confusion, like Z(x).\n2. In Eq(5) and Eq(6), the font of matchcal is usually used for a single letter.\n3. I cannot see a clear relationship between the proposed method and the energy-based model except that \"energy\" specifies the logits from the classification model. \n4. The performance of the proposed framework relies on the regression between MDE on the synthesized dataset and its accuracy. However, the type of domain shift is more complicated in the real world in most cases, thus very unpredictable. \n5. The experiment on adjusting temperature T should be conducted on a broader range, like from 0.01 to 100. The change from 1 to 10 is relatively small.\n6. A real dataset for evaluation needs to be included. The operation of shear, equalization, and color temperature adjustment is easy to synthesize, while the domain shift could come from more complex sources like [1][2].\n7. Some missing EBM references can be considered to be included into this paragraph to make it more complete for the readers. For example, [3] is the first EBM using CNN for energy function and trains it with Langevin dynamics. [3] is also the first one to point out that EBM and a classifier can be derived from each other. The EBM applications not only include video as you have mentioned in your paper, but also include point cloud [4], voxel [5], trajectory [6] and molecules [7].\n\n[1] Scanner invariant multiple sclerosis lesion segmentation from MRI. 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020.\n\n[2] Transfer learning for domain adaptation in MRI: Application in brain lesion segmentation. Medical Image Computing and Computer Assisted Intervention− MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III 20. Springer International Publishing, 2017.\n\n[3] A theory of generative convnet. ICML 2016.\n\n[4] Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for 3D Generation, Reconstruction and Classification.\n\n[5] Learning Descriptor Networks for 3D Shape Synthesis and Analysis. CVPR 2018.\n\n[6] Energy-Based Continuous Inverse Optimal Control.\n\n[7] Molecular Graph Generation with Energy-Based Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IBwBJCdVM5", "reviewer": "Reviewer_zk65", "summary": "- Authors propose a method for a problem called 'AutoEval'. The problem is described as evaluating the effectiveness of a model on data without ground truth labels.\n- The proposed method is very simple. It is based on energy models.\n- Authors perform experiments on datasets like CIFAR-10/100, TinyImagenet to validate their method.", "questions": "Please see the weakness section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The authors have performed a wide range of analysis experiments.", "weaknesses": "1. The problem addressed\n-  Authors motivate some new problem called Automated model evaluation. The definition is not very clear as they define it in many different ways.\n- I am not sure about the relevance of the problem. Or to phrase it better - I don't know much about this problem.\n \n2. Proposed method\n- The proposed method just use energy based model equation to create a simple function of the energy expressed in terms of logits. \n- One concern here is that it might be very similar to some method in Uncertanity estimation.\n- Can authors mathematically compare this method to some common methods in uncertainity estimation.\n\n3. Experimental validation\n- While the introduction had motivated the problem in a broad setting. Authors discussed OOD to motivate the problem. But I could not find the experiments on OOD.\n- The claims should match the experiments on which the method is validated. Maybe the authors can provide experiments on OOD.\n\n\n------\nauthors have addressed the questions I asked above. So I have updated to weak accept. Weak accept because experiments are still not on very large datasets", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0qWEHaZcal", "reviewer": "Reviewer_SNMZ", "summary": "This paper focuses on AutoEval, where the goal is to estimate classifier performance on unlabeled test sets. To this end, this work proposes to use energy score (MDE) as the statistic of each test set for inferring the corresponding classification accuracy. The experimental results on several benchmarks such as CIFAR-10 and CIFAR-100 show the proposed can achieve reasonable accuracy estimation.", "questions": "- Please clarify the claims on overconfidence issue, substantial storage, and computation cost \n- Please report the results on the ImageNet setup and the estimates of ImageNet-A/V2/S\n- Please clearly compare with existing works such as ATC, DoC, and Nuclear Norm on ImageNet. Moreover, [Characterizing Out-of-Distribution Error via Optimal Transport] should be included for comparison.\n- Other related works: Estimating and explaining model performance when both covariates and labels shift. In NeurIPS 2022", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "+ Algorithm 1 clearly shows the proposed method. The whole pipeline is well-introduced\n+ The experiment includes MNLI, a natural language inference task.", "weaknesses": "- *1. Overstated claims*: The paper asserts that \"the AutoEval frameworks still suffer from an overconfidence issue\" without providing clear, empirical examples. Additionally, the statement regarding \"substantial storage\" lacks a comparison with existing methods such as DoC and ATC, leaving the reader unconvinced of any real advantage of the proposed method. Furthermore, the claim that the proposed MDE method is superior in terms of \"computational cost\" is not substantiated, especially considering that unlike ATC, MDE requires the training of a linear regression model. To move forward, it would be essential to clarify these issues with specific data and comparative analysis.\n\n- *2. Limited contribution*: The novelty of your work is questioned due to the similarity with existing methods in the field. The use of Energy score as a replacement for Softmax score, without significant insights or enhancements, is seen as insufficient for constituting a substantial contribution. Prior works [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles] have already discussed the connection between OOD detection and AutoEval, which the current submission does not appear to extend beyond.\n\n- *3. Unconvincing theoretical analysis*: The theoretical grounding provided in Section 3.3 is deemed unclear and unconvincing. The methodology for ascertaining model accuracy needs to account for scenarios where an image is correctly classified with a low Softmax score. A more robust theoretical framework is necessary to support the claims made.\n\n- *4. Experimental results are not solid*: The absence of results for ImageNet-1K in Table 1, along with other test datasets like ImageNet-S/A/V2, raises concerns about the comprehensiveness of the experimental evaluation. Moreover, the omission of recent relevant works, such as \"Characterizing Out-of-Distribution Error via Optimal Transport,\" and the lack of comparison with methods like Nuclear Norm on ImageNet setup, call into question the thoroughness of the analysis.\n\n- *5. Limited Dataset Diversity*: The paper does not report on more natural shifts, which are included in benchmarks provided by methods like ATC. While BREEDs are mentioned in the supplementary material, results are not presented, and expectations for insights on datasets like i-WILDS are not met.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QEISwVnNwO", "reviewer": "Reviewer_Dsfd", "summary": "This paper proposes a novel measure called Meta-Distribution Energy (MDE) for automated model evaluation (AutoEval), which is a method for assessing the performance of machine learning models on unlabeled test data. The core idea of MDE is to convert the energy information of the model's output into a probability distribution statistic, which enables a smoother data representation. The paper also provides theoretical analysis connecting MDE with classification loss, proving its effectiveness. Experimental results demonstrate MDE's superior performance across various modalities, datasets, and model architectures, especially in noisy and class-imbalanced scenarios.", "questions": "Please refer to the weakness section mentioned above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The research topic of this paper -- AutoEval is valuable in the real world deployment of DNNs. I believe this area may shed some novel light on the unsupervised evaluation community.\n* This paper establishes a connection between MDE and classification loss through a mathematical theorem, providing theoretical justification for the effectiveness of the proposed method.\n* MDE demonstrates strong performance across a variety of modalities, datasets, and model architectures, and these detailed analyzes make it as a versatile solution for automated model evaluation.\n* MDE remains effective even in challenging scenarios such as strong noise and class imbalance, showcasing its robustness in practical applications.", "weaknesses": "* The paper does not provide source code, making re-implementation challenging. Please provide the code later to dispel this concern.\n* Interpretability can be further explored. While MDE provides strong correlation with classification accuracy, further research could focus on enhancing the interpretability of the method, making it easier for users to understand and trust the results.\n* Different methods seems to have different (wall-clock?) time required to come up with such evaluation -- perhaps some notes on that would be helpful as well.\n* Grammar should be thoroughly checked. For example, in the first paragraph, \"the information (energy) associated with individual samples, then offer a smoother representation enabled by energy-based learning.\" should be changed to \"xxxx, and then offers...\". Similarly, in the second paragraph, \"the correct classified data are given low energies, and vice versa\" should be changed to \"the correctly classified...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DExkMYtpYO", "reviewer": "Reviewer_2CiW", "summary": "This paper introduces a metric called Meta-Distribution Energy (MDE) for automated model evaluation (assessing model performance on unlabeled datasets). The method involves extracting energy scores from the model's output and extending them into a statistical representation at the dataset level. The paper demonstrates a strong linear relationship between MDE and classification accuracy on datasets with differing extents of distribution shifts. This allows for predicting model performance based on the MDE of an unlabeled test set, and the authors provide theoretical proof of this concept. In the experimental section, the paper presents the outstanding performance of MDE across different backbones, datasets, and modalities, even in scenarios with noise and class imbalance.", "questions": "Please try to address the questions I raised in the 'Weaknesses' chapter.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The task of AutoEval is crucial when deploying neural network models in real-world scenarios. It can help avoid the issue of not knowing the actual performance in practical applications and is also beneficial for model selection.\"\n2) The method MDE in the paper exhibits a relatively high level of novelty by introducing the statistical distribution of energy scores into the field of AutoEval for the first time. Furthermore, the paper extends the approach to improving energy scores, which can be applied to a wider range of tasks. Additionally, the authors provide detailed theoretical proof that strongly illustrates the effectiveness of the method.\n3) The paper conducts extensive experiments on different backbones, different datasets, and different modalities. The experimental results indicate a significant performance improvement compared to other methods in the field.\n4) The paper is of high writing quality, with a smooth and easy-to-grasp presentation of the key points.", "weaknesses": "1) It seems that the paper does not provide a more detailed explanation of the parameter T in MDE. Is T just an ordinary parameter, or does it have a more intuitive meaning? In Figure 3(a), only the trend of T from 1 to 10 is displayed. What is the impact of a wider range of T values on the results?\n2) In the paper, the authors categorized relevant methods into 'training-free' and 'training-must.' How much progress does MDE, as a 'training-free' method, make compared to 'training-must' methods in terms of evaluation time and memory usage? It would be ideal to conduct an experiment to illustrate this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)