Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shift
Authors: RENCHUNZI XIE
Ambroise Odonnat
Vasilii Feofanov
Ievgen Redko
Jianfeng Zhang
Bo An
Venue: Accepted by TMLR ()

Abstract: Estimating the test performance of a model, possibly under distribution shift, without having access to the ground-truth labels is a challenging, yet very important problem for the safe deployment of machine learning algorithms in the wild. Existing works mostly rely on information from either the outputs or the extracted features of neural networks to estimate a score that correlates with the ground-truth test accuracy. In this paper, we investigate -- both empirically and theoretically -- how the information provided by the gradients can be predictive of the ground-truth test accuracy even under distribution shifts. More specifically, we use the norm of classification-layer gradients, backpropagated from the cross-entropy loss after only one gradient step over test data. Our intuition is that these gradients should be of higher magnitude when the model generalizes poorly. We provide the theoretical insights behind our approach and the key ingredients that ensure its empirical success. Extensive experiments conducted with various architectures on diverse distribution shifts demonstrate that our method significantly outperforms current state-of-the-art approaches. The code is available at \url{https://github.com/Renchunzi-Xie/GdScore}.

Published in Transactions on Machine Learning Research (04/2025)
Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shift
Renchunzi Xie
renchunai.zie@ntuu.dee
College of Computing and Data. Science Nongang Technological University
Ambroisc Odonnat
mmbroisse.ooomnat@@maaaa
Huawei Nooh's 's Ark Lab, Inria
Poris, Pronce
Vasilii Feofanov
wwsitü.ffeooonore@mmaaccomm
Huawei Noah's Ark Lab Paris, Prance
Jevgen Redko
iengen.redhoogmai. com
Huawei Nooh's 's Ark Lab Paris, France
Jianfeng Zhang
reggen..redoo@@maii... com
Huawei Noah's Ark Lab Shenaben, China
Bo An
boan Dratr edu.
College of Compuling and Date Science Namyang Technological University
Reviewed on OpenReview: https: // openreview. net/ forum? id= FIWHRSuos
Abstract
Estimating the test performance of a model, possibly under distribution shift, without having access to the ground-truth labels is challenging, yet very important problem for the safe deployment of machine learning algorithms in the wild. Existing works mostly rely on information from either the outputs or the extracted features of neural networks to estimate a score that correlates with the ground-truth test accuracy. this paper, we investigate both empirically and theoretically how the information provided by the gradients can be predictive of the ground-truth test. accuracy we the norm of classification-layer gradients, backpropagated from the cross-entropy loss after only one gradient step over test data. Our intuition is that these gradients should be of higher magnitude when the model generalizes poorly. We provide the theoretical insights behind our approach and the key ingredients that ensure its empirical success. Extensive experiments conducted with various architectures on diverse distribution shifts demonstrate that our method significantly outperforms current state-of srate-of-the-art the-art approaches. The code is available at https://ee thub. com cww.com//////eeeeee htt///enchunii--e -ieii-eeiiiiie Xie/GdScore
under distribution shifts. More specifically,
1 Introduction
Deploying machine learning models in the real world is often subject to distribution shift between training and test data. Such a shift may significantly degrade the model's performance at test time (Quinonero-aandela
"Corresponding Authors. Univ. Rennes CNRS, IRISA

---

Published in Transactions on Machine Learning Research (04/2025)
et al., 2008; Geirhos et al., 2018; Koh et al., 2021) and lead to high risks related to AI satefy (Deng & Zheng, 2021). To alleviate this problem, common practice is to monitor the model performance regularly by collecting the ground truth of a subset of the current test dataset (Lu et al., 2023). However, this is usually time-consuming and expensive, highlighting the need for unsupervised methods to assess the test performance of models under distribution shift, commonly known as unsuperoised accuracy eslimation.
Limitations of current approaches. Current studies mainly focus on outputs or feature representation to derive a test error estimation score. Such score can represent the calibrated test error or the distribution discrepancy between training and test datasets (Hendrycks & Gimpel, 2016; Guillory et al., 2021; Garg et al., 2022; Deng & Zheng, 2021; Yu et al., 2022b; Lu et al., 2023). For instance, Hendrycks & Gimpel (2016) considered the average maximum softmax score of the test samples as the estimated error. Similarly, Garg et al. (2022) proposed to learn a confidence threshold from the training distribution. Deng & Zheng (2021) quantified the distribution difference between training and test datasets in the feature space, while Yu et al. (2022b) gauges the distribution gap at the parameter level. Although insightful, the current body of literature overlooks a potential tool for test accuracy estimation, namely the gradients that are known to correlate strongly with the generalization error of the deep neural networks (Li ct al., 2019; An et al., 2020).
Why do gradients matter? Recently, increasing attention has been paid to the intrinsic properties of gradients to design learning algorithms for meta-learning (Finn et al., 2017), domain generalization (Shi et al., 2021; Mansilla et al., 2021) or to improve optimization of DNNs Zhou et al. (2020); Zhao et al. (2022) by relying on them. In domain shift scenarios, Mansilla et al. (2021) proposed to clip the conflicting gradients and introduced a strategy to promote gradient agreement across multiple domains. Meanwhile, Zhao et al. (2022) designed a gradient-based regularization term to make the optimizer find flat minima. In the field of OOD detection, Huang et al. (2021) introduced a gradient- based function to detect ditribbiinnt (OOD) samples (relation to our method is compared in Appendix 7). Although the works mentioned above showed the potential of gradients to tackle the learning problems both in OOD and in-distribution (ID) settings, it is still unclear how unsupervised gradient- based scores can correlate with the test accuracy and be used to estimate it. This motivates us to ask:
Are gradients predictive of test accuracy under distribution shift?
In this paper, we shed new light on this open question: we surprisingly observe that there exists a strong linear relationship between gradient norm and test accuracy under distribution shift. Morcover, our theoretical analysis shows that the gradient norm conveys information on the generalization capacity of a well-calibrated model. In nutshell, this work provides direct evidence that gradient-based information correlates with generalization performance, which paves the way to better understanding of how neural networks generalize across unseen domains.
Our contributions. We hypothesize that the model requires a gradient step of large magnitude when it fails to generalize well on test data from unseen domains. To quantify the magnitude of gradients, we propose a simple yet efficient gradient- based statistic, GDSCORE, which employs the norm of the gradients backpropagated from standard cross-cntropy loss on the test samples. To avoid the need for ground-truth labels, we propose a pseudo-labeling strategy that benefits from both correct and incorrect predictions. We demonstrate that the norm of this one-step gradient of the classification layer strongly correlates with the generalization performance under diverse distribution shifts, acting a strong and lightweight proxy for the latter. The main contributions of our paper are summarized as follows:
1. We first provide several theoretical insights showing that correct pseudo-labeling and gradient norm have a direct impact the test error estimation. This is achieved by looking at the analytical expression of the gradient after one backpropagation step over the pre-trained model on test data under distribution shift and by upper-bounding the target out-of-distribution risk.
Based on these theoretical insights, we propose the GDSCORE, which gauges the magnitude of the classification-laee gradients and presents strong correlation with test accuracy. Our method does
2

---

Published in Transactions on Machine Learning Research (04/2025)
not require access to either test labels or training datasets and only needs one step of backpropagation which makes it particularly lightweight in terms of computational efficiency compared to other self-training methods.
3. We demonstrate the superiority of GDSCORE with large-scale empirical evaluation. We achieve new statc-of-the-ar results on 11 benchmarks across diverse distribution shifts compared to 8 competitors, while being faster than the previous best baseline.
Organization of the paper. The rest of the paper is organized as follows. Section 3 presents the necessary background on the problem at hand. In Section 4, we derive the theoretical insights that motivate the GDSCORE introduced afterward. Section 5 is devoted to extensive empirical evaluation of our method, while the ablation study is deferred to Section 6. Finally, Section & concludes our work.
2 Related Work
Unsupervised accuracy estimation. Unsupervised accuracy estimation is vital topic in practical applications due to frequent distribution shifts and the unavailability of ground-truth labels for test samples. To comprehensively undersland this field, we introduce two main existing settings related to this topic.
Some works aim to estimate the test accuracy or gauge the accuracy discrepancy between the training and the test set only via the training data (Corneanu et al., 2020; Jiang et al., 2019; Neyshabur et al., 2017; Unterthiner et al., 2020; Yak et al., 2019; Martin & Mahoney, 2020). For example, the mocdel-rrchiteetur--beeee
algorithm (Corneanu et al., 2020) derives plenty of persistent topology properties from the training data, which can identify when the model learns to generalize to unseen datasets. However, those algorithms are deployed under the assumption that the training and the test data are from the same distribution, which means they are vulnerable to distribution shifts.
2. Our work belongs to the second setting, which aims to estimate the classification accuracy of a specific lest dataset during evaluation using unlabeled test samples and/or labeled training datasets. The main research direction is to explore the negative relationship between the distribution discrepancy and model performance from the space of features (Deng & Zheng, 2021), parameters (Yu et al., 2022b) and labels (Lu et al., 2023). Another popular direction is to design an estimation score via the softmax outputs of the test samples (Guillory et al., 2021; Jiang et al., 2021; Guillory et al., 2021; Garg et al., 2022), which heavily relies on model calibration. Some works also learn from the field of unsupervised learning, such as agreement across multiple classifiers (Jiang et al., 2021; Madani et al., 2004; Platanios et al., 2016; 2017) and image rotation (Deng et al., 2021). In addition, the property of the test datasets presented during evaluation has also been studied recently (Xie et al., 2023). To the best of our knowledge, our work is the first to study the linear relationship between the gradients and model performance.
Gradients in generalization. The role of gradients in generalization has attracted increasing attention recently. To gauge the generalization performance of the hypothesis learned from the training data on unseen samples, known as the out-of-sample error (Hardt et al., 2016; London, 2017; Rivasplata et al., 2018), many studies try to provide a tight upper bound for generalization error from the view of gradient descent theoretically, indicating that gradients correlate with the discrepancy between the empirical loss and the population loss (Li et al., 2019; Chatterjee, 2020; Negrea et al., 2019; An al., 2020). However, those works assume that seen to unseen data are from the identical distribution, while unsupervised accuracy estimation discusses a more complex and realistic issue that they come from different distributions. Under distribution shift, gradients are also explored. For example, Mansilla et al. (2021) clips the conflicting gradicnts emerging in domain shift scenarios and promoles gradient agreement. across multiple domains via the gradient agreement strategy. Similarly, Zhao et al. (2022) designs a gradient- based regularization term to make the optimizer find the flat minima. In out-of-distribution (OOD) detoction which goal is to determine whether given sample in-distribution (ID) or out-of-distriuutin (Hendrycks & Gimpel, 2016; Hendrycks et al., 2018; Liu et al., 2020; Yang et al., 2021; Liang et al., 2017), (Huang et al., 2021) finds that ID data usually have higher
3

---

Published in Transactions on Machine Learning Research (04/2025)
gradient magnitude than OOD data from current source distribution to uniform distribution. However, despite their empirical success, the relationship between gradients and generalization is still unclear.
3 Background
Problem setup. We consider a K-class classification task with the imput space X' C RD and the label set y= (1....,,K) K). Our learning model is a neural network with trainable parameters E RP that maps from the input space to the label space fo : X' - RR We view the network as a combination a complex leature extractor f& and a linear classification layer fw, where g and w (w*) denote their corresponding parameters. Given a training example x,, the feedforward process can be expressed as:
fo(x,) = fw((g(x;)).
(1)
Let y = )== denote the one-hot encoded vector of label y; i.e., (/( = if and only if y = k, otherwise (k)) = Then, given a training dataset D {x,, Vi Hi that consists of n data points sampled i. from the source distribution p)(x,y) defined over X' x V, So is trained following the empirical cross-entropy loss minimization:
K
CD(fe)
(K) y;
log 50 (fg(x)),
(2)
where (1) denotes the output of the softmax for the class is approximating the posterior probability p(l= i.c., 10 (fg()) = xxp[w]fe(x)) (Ex Cxxc exp(wI (xx(w]]f(()))))
Unsupervised accuracy estimation. We now assume to have access to m test samples from the target distribution Diest = { ~ Pr(x), where Pr y) # Ps(x,8) y). For each test sample &,, we predict the label by V. = arg max fo (x;). We now want assess the performance of fe on a target distribution without using corresponding ground-truth labels )m by estimating as accurately as possible the following quantity:
Acc(Drest) (ustt)
10 7i) Σ
33)
where (-) denotes the indicator function. In practice, unsupervised accuracy estimation methods provide a proxy score (Dtta) that should exhibit a linear correlation with Acc(Drest). The performance of such methods is measured using the coefficient of determination R2 and the Spearman correlation coefficient p.
4 On the strong correlation between gradient norm and test accuracy
We start by deriving an analytical expression of the gradient obtaincd when fine-tuning a source pro-trained model on new lest data. We further use the intuition derived from it to propose our test accuracy estimation score and justify its effectiveness through more thorough theoretical analysis.
A motivational example. Below, we follow the setup considered by Denevi al. (2019); Balcan et al. (2019); Arnold et al. (2021) to develop our intuition behind the importance gradient norm unsupervisod accuracy estimation. Our main departure point for this analysis to consider fine-tuning: a popular approach to adapting a pre-trained model to different labeled datasets is to update either all or just fraction of its parameters using gradient descent on the new data. To this end, let us consider the following lincar regression example, where the test data from unseen domains are distributed as X (0, of', (Y|X ~N 1) parameterized by the optimal regressor O, E RR, while the data on which the model was trained is distributed as X~N((0, of), (IX=z)>N(00a) 1) with 0g E R. Consider the least-square loss over the test distribution:
1 Cr(c) Epy((ns) (y cx)²
2
4

---

Published in Transactions on Machine Learning Research (04/2025)
When do not observe the target labels, possible solution would be to analyze fine-tuning when using the source generator (Y]x=) 1) for pseudo-labeling. Then, we obtain that
V E (as cx)²1 = Py (ul cx)-
E. Pr) Pp lcr² ry
(c 0z)
((c 02) + (0c 0z))oy
This derivation, albeit simplistic, suggests that the gradient over the target data correlates modulo the variance of x - with 0c), capturing how far we are from the optimal parameters of the target model, and (0. - 0c), that can be seen as a measure of dissimilarity between the distributions the optimal source and larget parameters. Intuitively, both these terms are important for predicting the test accuracy performance suggesting that the gradient itself can be good proxy for the latter.
4.1 Proposed approach: GdScore
We now formally introduce our proposed score, termed GDSCORE, to estimate test accuracy in an unsupervised manner during evaluation. We start by recalling the backpropagation process of the pre-trained neural network fe from cross-entropy loss and then describe how leverage the gradient norm for the unsupervised accuracy estimation. The detailed algorithm can be found in Appendix B.
Feedforward. Similar to the feedforward in the pre-training process shown in Eq. 1, for any given test individual &, we have:
fo(x.) = (w(g((
(4)
As explained above, we do not observe the true labels of test data. We now detail our strategy for pseudo- labeling that allows obtain accurate and balanced proxies for test data labels based on accurate and potentially inaccurate model predictions.
Label generation strategy. Unconditionally generating pseudo-labels for test data under distribution shift exhibits an obvious drawback: we treat all the assigned pseudo-labels as correct predictions when calculating the loss, ignoring the fact that some examples are possibly mislabeled. Therefore, we propose the following confidence-based label-generation policy that outputs for every &; Dresti
arg max fo(x,), 1 U1,
max; sw (fg(x))) >T
V
5
otherwisc
where 7 denotes the threshold value, and U[1, k] denotes the discrete uniform distribution with outcomes (I,...,K) K). In a nutshell, we assign the predicted label to &i, when the prediction confidence is larger than threshold while using a randomly sampled label from the label space otherwise. The detailed empirical evidence justifying this choice is shown in Section 6, and we discuss the choice of proper threshold r in Appendix 6. From the theoretical point of view, our approach assumes that the classifier makes mistakes mostly on data with low prediction confidence, for which we deliberately assign noisy pseudo-labels. Feofanov et al. (2019) used a similar approach to derive an upper bound on the test error and proved its tightness in the case where the assumption is satisfied. We discuss this matter in more detail in Appendix E.
Backpropagation.
To estimate our score, calculate the gradients w.r.t. the weights of the classification layer w during the first epoch backpropagated over the standard cross- eross-entropy loss defined by:
L (fo) =
1
m
K
(k) U
(1) log S
(X,)
(6)
m

---

Published in Transactions on Machine Learning Research (04/2025)
where each unlabeled instance & is pseudo-labeled following Eq. 5. Then, the gradient of the classification layer w is evaluated as follows:
K
F = m
V
log
logs
(fg(2))
(7)
Note that our method requires neither gradients of the whole paramcter set of the pre-trained model nor iterative training. This makes it highly computationally efficient.
GdScore. Now, we can define GDSCORE using vector norm of gradients of the last layer. The score is expressed as follows:
S(Dtest) = Crom (fo)
(8)
where || lle denotes Lp-
4.2 Theoretical analysis
In this section, we provide theoretical insights into our method. We first clarify the connection between the true target cross-entropy error and the norm of the gradients. Then, we show that the gradient norm is upper-bounded by weighted sum of the norm of the inputs.
For the sake of simplicity, we assume the feature extractor fg is fixed and, by abuse of notation, we use x instead of fx(x). Reusing the notations introduced in Section 3, the true target cross-entropy error wriles
Notations.
[r(W) (Bpr(x)) Σ (k) log 2/10 (x),
where w =www) E Rx are the paramoters of the linear classification layer fw. For the case of notation, the gradient of LT w.r.t w denoted by VLr.
The following theorem, whose proof we defer Appendix H.1, makes the connection between the true risk and the -orm of the gradient explicit.
Theorem 4.1 (Connection between the true risk and the Ly-norm of the gradient). Let c E DxX and c' E Dxk be two linear classifiers. For any p.q 2 such that + = 1, we have that
[Lr(c') - Lr(c) we(c',))
(I)[IT(w)II,) -cll-
The left-hand side here is the difference in terms of the true risks obtained for the same distribution with two different classifiers. The right-hand side shows how this difference controlled by the maximum gradient norm over the two classifiers and a term capturing how far the two are apart.
In the context of the proposed approach, we want know the true risk of the source classifier w, on the test data and its change after one step of gradient descent. The following corollary applies Theorem 4.1 to characterize this exact case. The proof is deferred to Appendix H.2.
Corollary 4.2 (Connection after one gradient update). Let e be the classifier obtoined from w, after one gradient descent step, i.e,, c= w. - VLr with 17 2 Then, when w (ws, c), we have that
(r(ww)-Lr((e S max [W]T(W))),) (VLT(w))))
Note that in this case L1 (w,) can be seen term providing the true test risk after pseudo-labeling it with the source classifier. This shows the importance of pscudo-labeling as it acts departure point for obtaining a meaningful estimate of the right-hand side. When the latter is meaningful, the gradient norm on the right-hand side controls it together with a magnitude that tells us how far we went after one step of backpropagation.
In the next theorem, we provide an upper bound on the no-norm of the gradient as weighted sum of the norm of the inputs. The proof is deferred to Appendix H.3.

---

Published in Transactions on Machine Learning Research (04/2025)
Disperson Score Test Accuracy
ProjNorm Test Accuracy
GdScore Test Accuracy
Test Accuracy
60
Test Accuracy
Test Accuracy
40-
9.5
10.0
16.5 11.0
-7.4
-7.2
-7.0
-6.8
-4 -3
Dispersion Score
Negalive ProjNorm
Negative GdScore
True test accuracy Entity-13 with ResNet18. We compare the performance of GDSCORE with that of Dispersion Score and ProjNorm via scatter plots. Each point represents one dataset under certain corruption and certain severity, where different shapes represent different types of corruption, and darker color represents the higher severity level.
Figure 1: Test accuracy prediction
Theorem 4.3 (Upper-bounding the norm of the gradient). For any p 2 1 and A E DDX the Lp-norm of the gradient can be upper- bouuded as follows:
[VCT(W)|,, I (pr( -1xx)|
where a(w, x, =1 (1vg) (x), with by such that y y1xx)=
by a weighted combination of the norm of the inputs, where the weight a(w, x, E [0, 1] conveys how well the model predicts on x. In the case of perfect classification, the upper bound is tight and equals 0. In practice, as we do not have access the true risk, the gradients can be approximated by the proposed GDSCORE that requires to pseudo-label test data by Eq. As we said earlier, this implies that the model has to be well calibrated (see Appendix E), which is conventional to assume for self-training methods (Amini et al., 2023) including the approach of Yu et al. (2022b). Then, the network projects lest data into the low confidence regions, and the gradient for these examples will be large as we need to update w, significantly to fit. them.
Hence, the norm of the gradient upper-bountn
5 Experiments
5.1 Experimental setup
Pre-training datasets. For pre-training the neural network, we use CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009), TinyImageNet (Le & Yang, 2015); ImageNet (Deng et al., 2009), Offico-31 (Saenko et al., 2010), Office-Home (Venkateswara et al., 2017). CamelyonI7-WILDS (Koh et al., 2021), and BREEDS (Santurkar et al., 2020) which lever- ages class hicrarchy ImageNet (Deng et al., 2009) create 4 dalasets including Living-17, Nonliving- 26, Entity-13 and Entity-30. In particular, avoid timo-consuming training, directly utilize publicly available models pre-trained on Imagenet. For Office- 31 and Office-Home, we train a neural network on every domain.
600
ProjNorm Ours
Consumed (seconds)
500
400
300
200
100
CIFAR-10
CIFAR-100
TimylmageNet
offic-311
Office-Hame
Emtity-13
Entity-30
Datasets
Test datasets. In our comprehensive evaluation, we consider datasets with 3 types of distribution shifts: synthetic, natural, and novel subpopulation shift. To verify the effectiveness of our method under the synthetic shift, CIFAR-10C, CIFAR-100C,
Figure Runtime comparison of two self-training approaches with ResNet50.
7

---

Published in Transactions on Machine Learning Research (04/2025)
Table 1: Performance comparison on 11 benchmark datasets with 18 ResNet50, and WRN-50-2, where R² refers to coefficients of determination, and p refers to the absolute value of Spearman correlation coefficients (higher is better). The best results are highlighted in bold.
Rotation
CanfSoore R²
Entropy
AgresSeare
ATC
Próchet
Dispersion R²
ProjNorm xº
Qurs
Datnoet
Network
xº
xº
R2
R8
0.822
0.951
0.869
0.985
0.809
0.987
0.663
0.920
0.884
0.965
0.950
0.071
0.968
0.900
0.906
0.982
0.971
0.904
CIFAR 10
ResNet50
0.835
0.961
0.935
0,953
0.945
0.994
0.835
0.985
0.946
0.994
0,858
0.964
0.987
0.990
0.944
0.989
0.969
0.953
WRN
0,862
0.976
0,943
0.994
0,912
0.994
0,856
0,986
0.947
0.994
0,814
0,973
0,962
0.988
0.961
0.989
0.971
0,904
Average
0.840
0.963
0.916
0.001
0.530
0,502
0,785
0.367
0.926
0.5031
0.874
0.970
0.972
0.990
0.947
0.987
0.970
0.994
ResNet18
0,500
0.936
0.910
0.965
0.801
0,999
0,902
0.933
0.938
0.986
0.888
0.968
0,952
0.988
0.979
0.960
0.987
0.996
CIFAR 100
ResNet30
0.908
0.962
0.919
0.984
0.884
0.977
0.022
0.982
0.921
0.984
0.837
0.972
0.961
0.985
0.988
0.991
0.991
0.997
WRN-50-2
0.921
0.970
0.971
0.984
0.968
0.981
0.955
0.977
0.978
0.993
0.865
0.987
0.980
0.991
0.900
0.991
0.995
0.998
Average
0,896
0.056
0.006
0.927
0.340
0.965
0.961
0.062
0.988
0.967
0.991
0.907
ResNet18
0.786
0.946
0.670
0.960
0.502
0.842
0.501
0,853
0.761
0.945
0.826
0.920
0.906
0.986
0.970
0.981
0.971
0.994
TinyhnngeNet
ResNe150
0,736
0.957
0.070
0.869
0.651
0.892
0.560
0.853
0.751
0.945
0.826
0.971
0.977
0.986
0.979
0.987
0.980
0.995
WRNt-50-2
0,878
0.967
0.757
0.951
0,704
0,505
0.634
0.635
0.897
0.884
0,908
0.986
0.965
0:983
0.975 0,996
Average
0.505
0.959
0.727
0.920
0..50
0,800
0,300
0.878
0.693
0.921
0.817
0.976
0.970
0 0.957
0.072
0.981
0.976
0.995
18
0.979
0.991
0.963
0,500
0.974
0.983
0.802
0.974
0.940
0.971
0.975
0.993
0.98G
0.996
ImageNet
ResNe130
0.980
0.991
0.967
0,992
0.970
0.985
0.855
0.974
0.936
0.968
0,966
0.983
0.987
0.996
WRNN-50-2
0.983
0.001
0.963
0.99L
0.963
0.993
0.900
0.988
0.939
0.976
0.978
0.953
0.984
0.998
Average
0.961
0.969
0.999
0.970
0.967
0.555
0.979
0.030
0.972
0.960
0.993
0.966
0.997
18sNet18
0.753
0,942
0.470
0.828
0.322
0.714
0.000
0.085
0.843
0.942
0.143
0.257
0.618
0.714
0.009
0.428
0.675
0.829
0ffic-31
ResNe150
0.391
0.828
0.185
0.828
0.354
0.628
0,011
0.463
0.532
0.485
0.031
0.257
0.576
0.714
0.210
0.428
0.604
0.829
WRN-50-2
0.577
0.524
0,714
0,257
0.405
0.671
0.714
0,147
0.143
0.541
Average
0.567
0.790
0.493
0.790
0.367
0,276
0.006
0.211
0.593
0.790
0.071
0.057
0.622
0.714
0,162
0.333
01608
0.829
ResNe118
0.822
0.930
0.795
0.900
0.761
0.581
0.054
0.346
0.571
0.615
0.605
0.755
0.453
0.681
0.064
0.202
0.876
0.909
Offce-Home
ResNes50
0.851
0.944
0.709
0.895
0,712
0,853
0.025
0,216
0.187
0.731
0.807
0.885
0,383
0.727
0.109
0.475
0.629
0.944
WRNN-50-2
0.623
0.958
0,741
0.871
0.606
01846
0,132
0.405
0.383
0.643
0.589
0.706
0.456
0.713
0.172
0.531
0.800
0.916
Average
0.832
0.944
0.708
0.882
0.733
0.860
0.071
0.256
0.488
0.694
0.801
0.715
0.431
0.702
0.135
0.403
0.837
0.928
ResNei18
0.941
1.000
0.980
1.000
0,960
1.000
0,977
1.000
0.981
1.000
0.988
1.000
0,992
1.000
0.612
0.500
0.996
1.000
Comelyonl7-WWUu.
ResNot50
0.931
1.000
0.994
1.000
0.903
1.000
0.908
1.000
0.903
1
1.000
0.971
1.000
0.012
0.500
0.811
1,000
0.909
1.000
WRN-5002
0,918
1.000
0.061
1.000
1.000
0.965
1.000
0.942
1.000
0.994
1.000
0.001
0.500
0,789
0.300
0.997
1.000
Average
0.531
1.000
0.073
1.000
0,080
1.000
0.062
1.000
0.8F2
1.000
0.9854
1.000
0.334
0.660
0.737
0.096
0.998
1.000
ResNet18
0.927
0.961
0.795
0.940
0.794
0.505
0.543
0.919
0.823
0.945
0.950
0.981
0.937
0.908
0.962
0.981
0.969
0.991
Ebiity-13
ResNet50
0.932
0.976
0.728
0,908
0.901
0.783
0.950
0,764
0.502
0,944
0.074
0.960
WRN-50-2
0.930
0,983
0,930
0.977
0.019
0.973
0.871
0.005
0.936
0.980
0.906
0.958
0.815
0.905
0.950
0.977
0.968
0.995
Average
0.033
0.973
0.817 0.963
0.804
0.046
772
0.000
0.847
0.936
0.930
0.960
0.9448
0.977
0.830
0.022
0.966
0.994
BrsNet18
0.964
0.979
0.570
0.836
0.553
0.832
0.542
0.935
0.611
0.845
0.849
0.978
0.929
0.968
0.962
0.987
0.970
0.995
Eatity-30
ResNet30
0.961
0.980
0.878
0.960
0.838
0.956
0.914
0.975
0.924
0.973
0.835
0.956
0.783
0.914
0.937
0. 0.986
0.967
0.906
WRN-50-2
0.940
0.978
0.907
0.974
0.878
0.970
0.836
0.055
0.936
0.984
0.927
0.973
0.927
0.973
0.959
0.985
0.949
0.904
Average
0.955
0.918
0.781
0.920
0.756
0.919
0.820
0.901
0.871
0.909
0,880
0.902
0.9449
0.957
0.959
0.995
RosNot18
0.876
0.973
0.913
0.973
0.808
0.970
0.586
0.736
0.940
0.973
0.768
0.950
0.900
0.958
0.923
0.070
0.949
0.983
Living-17
ResNet30
0.906
0.956
0.580
0.967
0.853
0.961
0.633
0.802
0.938
0.976
0.771
0.926
0.851
0.929
0.963
0.924
0.901
0.975
WRN-50-2
0.900
0.957
0.928
0.980
0.921
0.977
0.632
0.793
0.966
0.984
0.931
0.967
0.931
0.996
0.915
0.970
0.910
0.976
Average
0,533
0.974
0.907
0.993
0.814
0.566
0.623
0.777
0.945
0.978
0.817
0.949
0.854
0.951
0.913
0.969
0.500
0.978
ResNet18
0.906
0.955
0.781
0.925
0.739
0.900
0.543
0.854
0.930
0.914
0.980
0.958
0.981
0.900
0.978
0.963
0.983
Noclivingg22
ResNet50
0.916
0.970
0.832
0.942
0.776
0.918
0.638
0.837
0.893
0.900
0.848
0.950
0.805
0.907
0.873
0.972
0.945
0.989
WRN-502
50-2
0.917
0.977
0.932
0.971
0.912
0.959
0.676
0.861
0.945
0.909
0.885
0.942
0.893
0.939
0.924
0.973
0.937
0.985
Average
0.913
0.967 0.848
0.946
0.803 0,5229
0.618
0.835
0.897
0.506
0.882
0.357
0.913
0.974
0.943
0.945
0.985
and ImageNet-C C (Hendrycks & Dietterich, 2019) that span 19 types of corruption across 5 severity levels, as well TinyImagen IinllmaeeNet-C (Hendrycks & Dietterich, 2019) with 15 types of corruption and 5 severity levels. For the natural shift, we use the domains excluded from training from Office-31, Office-Home, and Camelyon17- WILDS as the OOD datasets. For the novel subpopulation shift, we consider the BREEDS benchmarks, namely, Living-17, 17, Nonliving-26, Entity-13, and Entity-30, which are constructed from ImageNet-C.
Training details. To show the versatility of our approach across different architectures, we perform all our experiments on 18 ResNet50 (He et al., 2016) and WRN-50-2 (Zagoruyko & Komodakis, 2016) models. We train them for 20 epochs for CIFAR-10 (Krizhevsky & Hinton, 2009) and 50 epochs for the other datasots. In all cases, we use SGD with a learning rate of 10-3, cosine learning rate decay (Loshchilov & Hutter, 2016), a momentum of 0.9, and a batch size of 128. For all experiments, we use p= 0.3 to compute GDSCORE.
Evaluation metrics. We measure the performance of all competing methods using the coefficients of determination (R²) and Spearman correlation coefficients (p) calculated between the baseline scores and the
8

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
1.4
0.4
Pseudo
0.4
Random labels
True labels
Uniform lebels
02-
Entropy Smoothing Ours
Durs
0F38-10
CBSR 100
Thyinogeiig
office 1
OfficsHome
Enityy 3p
Average
GRAP-1 10
CaAR too
Thyinoppttt
office 2
offic Homee
Enity to
Liing- 17
Average
Dataset
Dataset
(a) Label generation stralegies
(b) Loss selection
Figure 3: Performance comparison (R²) 7 datasets with ResNet18 between (a) different label generation strategies and (b) different types of losses across 3 types of distribution shifts. Results confirm that our proposed method performs better on average across various datasets and types of shifts.
true test error. To compare the computational efficiency with two self-training methods, we calculate the average evaluation time needed for every test dataset.
Baselines. We compare our method GDSCORe with 8 baselines commonly considered in the unsupervised accuracy cstimation litcrature: Rotation Prediction (Rotation) (Deng et al., 2021), Averaged Confidence (ConfScore) (Hendrycks & Gimpel, 2016), Entropy (Guillory et. al., 2021), Agreement Score (AgreeScore) (Jiang et al., 2021), Averaged Threshold Confidence (ATC) (Garg et al., 2022), AutoEval (Fréchet) (Deng & Zheng, 2021), Dispersion Score (Dispersion) (Xic et al., 2023), and ProjNorm (Yu et al., 2022b). The first six methods are training-free free and the last method is an instance of self-training approaches. More details about the baselines can be found in Appendix C.
5.2 Main takeaways
GdScore correlates with test accuracy stronger than baselines across diverse distribution shifts. In Table 1, we present the OOD error estimation performance 11 benchmark datasets across 3 model architectures as measured by R2 and We observe that GDSCORE outperforms existing methods under diverse distribution shifts. Our method achieves an average R higher than 0.99 on CIFAR-100, while the average R² of the other baselines is always below. In addition, our method performs stably across different distribution shifts compared with the other cxisting algorithms. For example, Rotation performs well under the natural shift but experiences a dramatic performance drop under the synthetic shift, ranking from the second best to the eighth. However, our method achieves consistently high performance, ranking the best on average across the three types of distribution shifts.
Furthermore, we provide the visualization of estimation performance in Fig. 1, where we present the scatter plots for Dispersion Score, Norm and GDScORE on Entity-13 with ResNet18. We can see that GDSCore and test accuracy have a strong lincar relationship, while the other srate-of-tho-art methods struggle to have a linear correlation in cases when the test error is high. This phenomenon demonstrates the superiority of GDSCORe in unsupervised accuracy istimation. In the next paragraph, we also demonstrate the computational efficiency of our approach.
GdScore is more efficient self-training approach. In the list of baselines, both our method and Proj (Yu et al., 2022b) belong to self- self-training methods (Amini et al., 2023). The latter, however, requires costly itcrative training on the neural network during evaluation to obtain the complete set of finc-tuned parameters to calculate the distribution discrepancy in network parameters. Compared with ProjNorm, our method only trains the model for one epoch and collects the gradients of the linear classification layer to calculate the gradient norm, which is much more computationally efficient. Fig. presents the comparison of computational efficiency between the two methods on 7 datasets with ResNet50. From this figure, we can see that our method is up to 80% faster than ProjNorm average. This difference is striking the Office-31
9

---

Published in Transactions on Machine Learning Research (04/2025)
0.88
1.0
Block
Last layer
0.95
0.96
0.8
0.90
CIFAR-10
CIFARL100
0.94
Tinyimagelee
0.4-
0.80
0.92
CIFAR-100
TinylmageNet
0.2 TinytrageNee
0.75
4
8
norm
Epochs
(a) Norm types
(b) Layer selection
(c) Epoch selection
Figure 5: Sensitivity analysis on the effect of (a) norm types, (b) layer selection for gradients, and (c) epoch sclection. The first and the third experiments are conducted on CIFAR-10C, CIFAR-100C, and TinyImageNet- C, while the second experiment includes Tiny Imageeee,,,,C, Office-31, OfliceHHome, and WILDS-FMoV (Koh et al., 2021). All experiments are conducted with ResNet18.
dataset, where our method not only two orders magnitude faster than ProjNorm but also improves the ne score by factor of 2.
Robustness of our approach. GDSCORe achieves great performance across all datasets, archi- tectures, and types of shifts (Table 1). To highlight the robustness of our approach, we compare the dis- tributions of R² Figure 4, including as additional baseline the very recent Nuclear Norm (Deng et al., 2023) (more results in Appendix F). We show that GDSCORE is the best and most stable approach 10 datasets (except ImageNet).
1.0
0.8
0.6-
0.4-
0,2
6 Ablation study
Rotation
Confscore
Entropy
Agrite
ATC
Frechtt
Projnam
Dispersion
Nudear
GdScare
In this section, we will conduct comprehensive abla- tion studies to verify the efficiency of our proposed babel-genrration strategy, and choice of hyperparam- eters such as T and p as well as self- self-training settings to obtain GdScore such as epochs, blocks, and losses.
Methods
Figure 4: Robustness comparison for all estima-
tion baselines across diverse distribution shifts with ResNet18.
Random pseudo-labels boost the performance under natural shift. In Fig. 3(a), conduct ablation study to verify the effectiveness of our label generation strategy by comparing it with ground-truth labels, uniform labels (Huang et al., 2021), full random labels, and full pseudo-labeling. From the figure, we observe that while comparable with the ground truth under the synthetic drift and the subpopulation shift, the other strategics lead to a drastic drop in performance under the natural shift. This phenomenon is possibly caused by imprecise gradient calculation based on incorrect pseudo labels. Our labeling strategy performs better on average suggesting that random labels for low-confidence samples provide certain robustness of the score under natural shift.
Cross-entropy loss is robust to different shifts. To demonstrate the impact of different losses the test accuracy estimation performance, we compare the standard cross-entropy loss used in our method with the entropy loss for samples with low confidence (see definition in Appendix D). Moreover, we verify the effectiveness of the label smoothing, a simple yet effective tool for model calibration and performance improvement (Müller et al., 2019), by setting the smoothing rate as 0.4.
Fig. 3(b) illustrates this comparison revealing that standard cross-entropy loss is the most robust choice across different types of distribution shifts. We also note that the entropy loss enhances the estimation
10

---

Published in Transactions on Machine Learning Research (04/2025)
performance under the syuthetic and the novel subpopulation shifts, but struggles under the natural shift. On the contrary, the label smoothing regularization can increase the performance under the natural shift but decrease it under the synthetic and the novel subpopulation shifts.
Choosing smaller p for better estimation performance. To illustrate the effect of the choice of Ig-norm on the performance, we conduct a sensitivity analysis on three datasets with R8 summarized in Figure 5(a). We can see that there is an obvious decreasing trend as p becomes larger. Espocially, when p is smaller than 1, the estimation performance can fluctuate within satisfying range. This is probably because smaller p (i.e., 0 < p < 1) makes the Ly-norm more suitable for the sigh-dimensional space, while the Lo-norm with 2 1 is likely to ignore gradients that are close to 0 (Wang et al., 2016; Huang et al., 2021), so for all experiments, we used provides some theoretical insights into the Tep-norm of the gradient for O <<<<>
0.3. The remark below, whose proof we defer to Appendix H.4,
Remark 6.1 (Case 0 < < 1). Let c be the classifier obtained from ws after one gradient descent step, i.e., c c= W, 17- (WT) with 7 2 O. For any p E (0,1), we have
n [VLLT(w>)l)
|||c|p- - l|w/|bll-
Gradients from the last layer can provide sufficient information. In this part, we aim to understand whether backpropagation through other layers in the neural network can provide a better test accuracy estimate. For this, we separate the feature extractor fg into 3 blocks of layers with roughly equal size and calculate the GRDNORM scores for each of them. Additionally, we also try to gather the gradients over the whole network. Fig. 5(b) plots the obtained results on datasets, suggesting that the last layer provides sufficient information to predict the true test accuracy in an unsupervised way.
No gains after 1 epoch of backpropagation. Here, train the neural network for epochs, where E 21,2,5, 5, 10, 15, 20), and store the gradient vectors of the classification layer for each value of r. Fig. 5(c) suggests that the gradient norms after 1 step are sufficient to predict the model performance under distribution shifts. Further training gradually degrades the performance with the increasing r. The reason behind the phenomenon is that the gradients in the first epoch contain the most abundant information about the training datasct, while the neural network fine-tuned on the test dataset for several epochs is going forget previous training categories (Kemker et al., 2018).
Choice of proper threshold v In our experiments (see Section 5), we set the value of 7 0.5 across all dalasels and network architectures. This choice of is due to the intuition that if label contains softmax probability below 0.5, it means that this predicted label has over 50% chances of being wrong. It means that this label has a higher probability of being incorrect than to be correct. Thus, we tend to regard it as an incorrect prediction. To remonstrate the impact of threshold т on the final performance, we conduct an ablation study on CIFAR-10C and Office-31 with ResNet18 using varying values of r. We display in Table 2 the corresponding values of R22. We can observe that the final performance improves and achieves its best value for is 0.5, before decreasing slightly.
Table 2: Performance on CIFAR-10 and Office-31 with ResNet18 for varying value of r. The metric used in this table is the coefficient of determination R²a The best result is highlighted in bold.
Threshold
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
CIFAR-10C
0.963
0.963
0.964
0.965
0.971
0.972
0.967
0.962
0.963
0.959
Office-31
0.495
0.498
0.532
0.674
0.685
0.667
0.545
0.451
0.114
0.131
7 Connection to GradNorm
A current work, GradNorm (Huang et al., 2021), employs gradients to detect OOD samples whose labels belong to different label space from the training data. It. gauges the magnitude of gradients in the classification
11

---

Published in Transactions on Machine Learning Research (04/2025)
layer, backpropagated from KL divergence between the softmax probability and uniform distribution. Compared with GradNorm, our method bears three critical differences, in terms of the problem setting, methodology, and theoretical insights.
1) Problem setting: GradNorm focuses on OOD detection, which aims to determine whether a given sample is in-distribution (TD) or out-of-distributioo (Hendrycks & Gimpel, 2016; Hendrycks et al., 2018; Liu et al., 2020; Yang et al., 2021; Liang et al., 2017), while our method aims to estimate the test accuracy without ground-truth test labels. It requires GradNorm should be instance-level score for classification, but our method is a dataset-level score for linear regression. Furthermore, in OOD detection, the label spaces of OOD data and training data are disjoint, while in unsupervised test accuracy, the training and the OOD label spaces are shared. The two are also evaluated differently: AUROC score for OOD detection and correlation coefficients for error estimation. Those differences are summarized in Table 3.
Table Main differences between OOD detection and OOD error estimation.
Learning Problem
Goal
Scope
Metric
OOD detection
Predict ID/OOD Proxy to lest error
X;
AUROC
OOD error estimation
Drest
R2 and p
2) Methodology: GradNorm obtains the magnitude of gradients via a KT-divergence loss measuring the distribution distance from the training distribution to a uniform distribution, while our method obtains them using a standard cross entropy loss with the specifically designed label generation strategy. GradNorm assumes that OOD data should have a lower magnitude of gradients, but in our cases, test data are certified to have a higher magnitude of gradients. Table 4 presents the performance comparison of the two methods in unsupervised accuracy estimation on 7 datasets across 3 types of distribution shifts with ResNet18. It illustrates that GradNorm is inferior to our approach for unsupervised accuracy est suggesting that, the two problems cannot be tackled with the same tools.
3) Theoretical insights: GradNorm is certified to capture the joint information between features and outputs to detect OOD data from the oncoming dataset. However, the reason why OOD data have lower magnitude of gradients is still unclear. Our method provides clearer theoretical insights to demonstrate the relationship between gradients and test accuracy even under distribution shift, which further inspires future work to address generalization issues from the view of gradients
Table 4: Performance comparison between Huang et al. (2021) and our methods on datasets with ResNet18. The metric used in this table is the coefficient of determination R², The best results are highlighted in bold.
Method
CIFAR
CIFAR 100
TinylmageNet
Office-31
Office-
Entity-30
Living-17
(Huang et al., 2021)
0.951
0.978
0.894
0.596
0.848
0.964
0.942
Ours
0.972
0.983
0.971
0.675
0.876
0.970
0.949
8 Conclusion
In this paper, showcased the strong linear relationship between the magnitude of gradients and the model performance under distribution shifts. We proposed GDSCORE, simple yet efficient method to estimate the ground-truth test accuracy error by mcasuring the gradient magnitude of the last classification layer. Our method consistently achieves superior performance across various distribution shifts than previous works. Furthermore, it does not require the detailed architecture of the feature extractors and training datasets. Those properties guarantee that our method can be casily deployed the real world, and moet practical demands, such as large models and confidential information. We hope that our research sheds new light on the usefulness of gradient norms for unsupervised accuracy estimation.
12

---

Published in Transactions on Machine Learning Research (04/2025)
Acknowledgments
This research supported by the National Research Foundation Singapore and DSO National Laboratories under the AT Singapore Programme
( (ATSGAward No: ATSG2-GC-2023-000)
References
Massih-Reza Amini, Vasilii Feofanov, Loic Pauletto, Lies Hadjadj, Emilie Devijver, and Yury Maximov. Self-training: A survey. arXiv preprint arXiv:2202. 12040, 2023.
Haozhe An, Haoyi Xiong, Xuhong Li, Xingjian Li, Dejing Dou, and Zhanxing Zhu. Can we use gradient norm as a measure of generalization error for model selection in practice? 2020.
Sebastien Arnold, Shariq Iqbal, and Fei Sha. When maml can adapt fast and how to assist when it cannot.
In Proceedings of International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.
Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guaranlees for gradient-based meta-learning. In The International Conference on Machine Learning (ICML), pp.
2019.
Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization in gradient descent- based optimization. arXiv preprint 200: 10657, 2020.
Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Advances Neural Information Processing Systems, 34:14980 14992, 2021.
Ciprian Corneanu, Sergio Escalera, and Aleix M Martinez. Computing the testing error without a testing sct. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 2677 2677-2685, 2685, 2020.
Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-lcant stochastic gradient descent with biased regularization. In The International Conference on Machine Learning (ICML), pp. 1575, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. Tn 2009 TEEE Conference on Computer Vision and Pattern Recognation, (CVPR), pp- 248 255. leee, 2009.
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogrition (CVPR), pp. 15069-15078, 2021.
Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell about classifier accuracy under varying lesting environments? In International Conference on Machine Learning (ICML), pp. 2579-2555 2589, 2021.
Weijian Deng, Yumin Suh, Stephen Gould, and Liang Zheng. Confidence and dispersity speak: Characterizing prediction matrix for unsupervised accuracy estimation. In Andreas Kranse, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 7658-7674. PMLR, 23 29 Jul 2023. URL
mlr. hatsss//222/d/////// htmml.
Jiahua Dong, Zhen Fang, Anjin Liu, Gan Sun, and Tongliang Liu. Confident anchor-induced multi-source free domain adaptation. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, volume 34, pp. 2848-2860. Cur- ran Associates, Inc., 2021. URL htttps://proceeeeeee www.eeeddnnnnnnnnnn..eeeeeee newrips. httprrpp..c/ppppertteee///pppppp http///s//seaee////////... 15580aas2270333aabbbl7aaaaaa3393 Paper, pdf.
Vasilii Feofanov, Emilie Devijver, and Massih- Reza Amini. Transductive bounds for the multi-class majority vote classifier. Tn Proceedings of the AAAT Conference on Artificial Intelligence, pp. 3566-3573, 2019.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Chelsea Finn, Picter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint arXiv:2201.0433 2022.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained emns are biased towards texture; increasing shape bias improves accuracy and robustness. Xr preprint arXiv: 12231,, 2018.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IBEE/CVF International Conference on Computer Vision (ICCV), pp. 1134-1144, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1321-1330. JML...rr, 2017.
Moritz Hardt, Ben Recht, and Yoram Singer. Train laster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning (ICML), pp. 1225 1234. PMLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 770 778, 2016.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. arXie preprinl arXiv: 122612 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXic preprint arXiv:1610. 16t0.011...... 02136, 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arxic:1812, 04606, 2018.
Rui Huang, Andrew Geng, and Yixuan Li. On the importance of gradients for detecting distributional shifts in the wild. Advances in Newral Information Processing Systems, 34:677-689, 2021.
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In International Conference on Learning Representations, 2019.
Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and Zico Kolter. Assessing generalization of sgd via disagreement. arXiv preprint arXiv:2106.1377 2021.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting neural networks. In Proceedings of the Association for the Advancement of Artificial Intelligence conference on artificial intelligence (AAAT), 2018
Pang Wei Koh, Shiori Sagawa, Hcnrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664, PMI.R, 2021.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 281N, 7(7):3, 2015.
Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization error bounds of noisy gradient methods for non-convex learning. arXiv preprint arXiv:1902.00633 2019.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of dustri-disttibuutin image detection in neural notworks. Xiv preprint arXio:
1 706. 776602690, 2017.
14

---

Published in Transactions on Machine Learning Research (04/2025)
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection.
Advances in Newral Information Processing Systems, 14:21464-214755 2020.
Ben London. A pac- pac-bayesian analysis randomized learning with application to stochastic gradient descent. Advances Newral Information Processing Systems (NeurIPS), 30, 2017.
lyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 2016.
Yuzhe Lu, Yilong Qin, Runtian Zhai, Andrew Shen, Ketong Chen, Zhenlin Wang, Soheil Kolouri, Simon Stepputtis, Joseph Campbell, and Katia Sycara. Characterizing out-of-distribution error via optimal transport. arXiv preprint arxi:22005.155 15640, 2023.
Omid Madani, David Pennock, and Gary Flake. Co-validation: Using model disagreement on unlabeled data to validate classification algorithms. Advances in Neural Information Processing Systems (NewrIPS), 17, 2004.
Lucas Mansilla, Rodrigo Echeveste, Diego H Milone, and Enzo Ferrante. Domain generalization via gradient surgery. In Proceedings of the IEEE/CVE international conference compuler vision, pp- 2021.
Charles H Martin and Michael W Mahoney. Heavy-tailed universality predicts trends in test accuracies for very large pre-trained deep neural networks. In Proceedings of the 2020 SIAM International Conference on Dala Mining (SDM), pp. 505-513. SIAM, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information- thcoretic generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
Bchnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring gencralization in deep learning. Advances in Neurul Informalion Processing Systems (NeurIPS), 2017.
Emmanouil Platanios, Hoifung Poon, Tom M Mitchell, and Eric J Horvitz. Estimating accuracy from unlabeled data: A probabilistic logic approach. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Emmanouil Antonios Platanios, Avinava Dubey, and Tom Mitchell. Estimating accuracy from unlabeled data: A bayesian approach. In International Conference on Machine Learning (ICML), pp. 1416 1425. PMLR, 2016.
Joaquin Quinenero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Datasel shift in machine learning. Mit Press, 2008
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexcy Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 1112116-12128 2021.
Omar Rivasplata, Emilio Parrado- Herendderr John S Shawe- Taylor, Shiliang Sun, and Csaba Szepesvári. Pac-bayes bounds for stable algorithms with instance-dependent priors. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
Kate Sacnko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European Conference on Computer Vision (ECCV), pp. 213 226. Springer, 2010.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. Xix preprint ar Xi:22000 04859, 2020.
15

---

Published in Transactions on Machine Learning Research (04/2025)
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hanmun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprinl ar arXix:2104.0990 2021.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Tiang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 596 608. Cur- ran Associates, Inc., 2020. URL https://proceedis 2GfAAc99aabbee5555000888883333-4
neurips. cc/paper ftless/papae//222/11ee.
pdf.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Πly Tolstikhin. Prodicting neural network accuracy from weights. arXiv preprinl Xiv: 1 11448, 2020.
Hemanth Venkatcswara, Jose Euscbio, Shayok Chakraborty, and Sethuraman Panchanathan, Doep hashing network for unsupervised domain adaplation. In Procesdings of the IEEE Conference on Compuler Vision and Pattern Recognition (CVPR), pp.
2017.
Handing Wang, Yaochu Jin, and Xin Yao. Diversity assessment in many-objective optimization. IEEE transactions on cybernetics, 47(6):1510-1522, 2016.
Renchunzi Xie, Hongxin Wei, Yuzhou Cao, Lei Feng, and Bo An. On the importance of feature separability in predicting out-of-distribution error. arXiv preprint arXie:2303.1548 15488 2023.
Scott Yak, Javier Gonzalvo, and Hanna Mazzawi. Towards task and architectune-nddgeenentt generalization gap predictors. arXiv preprint Xiviv:1906. 101.1000001150 2019.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint crXi::2110.11394, 2021.
Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-of-distribution error with the projection norm. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25721-25746. PMLR, 17-23 Jul 2022a. URL https:/ pttp://///oceeeng.....m mlr. wwt..presa/veee hatpa/vvv2/yy2//// (222/y222i.2 2am22.mm.........mmmm...mmm..... html.
Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-of-distribution error with the projection norm. arXie preprint or arXiv:2202.0580 058344 2022b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Brilish Machine Vision Conference (BMVC), 2016.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference Machine Learning, pp. 26982 -1982-26992. PMLR, 2022.
Yingxue Zhou, Belhal Karimi, Jinxing Yu, Zhiqiang Xu, and Ping Li. Towards better generalization of adaptive gradient methods. Advances Neural Information Processing Systems, 33:810-821, 2020.
16

---

Published in Transactions on Machine Learning Research (04/2025)
A Appendix
B Pseudo-code of GdScore
Our proposed GDSCORe for unsupervised accuracy estimation can be calculated as shown in Algorithm
Algorithm 1: Unsupervised Accuracy Estimation via GDSCORE
Input:
Test dataset from unseen domains D {x,) a pre-trained model fe = fæ o Sw (feature extractor fg and classifier fw), a threshold value т. Output: The GDSCORE. for each test instance & do Obtain the maximum softmax probability via i max; 1 (fg(x,)) if F, then
Obtain pseudo labels via // arg max; (X;;;
else
Obtain random labels via V/ ~ U11,
end if
end for
Calculate the cross-entropy loss using assigned labels ia via Eq. 6. Calculate gradients of the weights in the classification layer via Eq. 7.
Calculate GDSCORE S(D) via Eq. 8.
C Baselines
(Deng al., 2021) By rotating images from both the training and the test sets with different angles, we can obtain new inputs and their corresponding labels yf which indicate by how many degrees they rotate. During pre-training, an additional classifier about rotation degrees should be learned. Then the Rotation Prediction (Rotation) metric can be calculated as:
Rotation.
(D =
1
Σ
(1 + v())),
1=1
rE/0", -900 180 270")
.90
where of denotes the predicted labels about rotation degrees.
ConfScore. (Hendrycks & Gimpel, 2016) This metric directly leverages the average maximum softmax probability as the estimation of the test error, which is expressed as:
Sey
(f
m
Entropy. (Guillory et al., 2021) This metric estimates the test error via the average entropy loss:
K
1
Se
(k) (x:( logs (fg(.)).
AgreeScore. (Jiang al., 2021) This method trains two independent neural networks simultancously during pre-training, and estimates the test error via the rate of disagreement across the two models:
(Duss)))
Σ
1 + 12..).
where Hi and V22, denote the predicted labels by the two models respectively.
17

---

Published in Transactions on Machine Learning Research (04/2025)
ATC. (Garg et al., 2022) It measures how many test samples have confidence larger than threshold that is learned from the source distribution. It can be expressed as:
m
(Drest)
(Z(:)) O (x.))) < <1),
m
where is the threshold value learned from the validation set of the training dataset.
Fréchet. (Deng & Zheng, 2021) This method utilizes Fréchet distance to measure the distribution gap between the training and the test datasets, which serves the test error estimation:
frr (Deett) Perain Mteadl + Tr(Etrain + Etest 2(2erain (teat))
where Ptrain and Ptest denote the mean feature vector of DD and Drest, respectively. Etrain and Etest refer to the covariance matrices of corresponding datasets.
Dispersion. (Xie et al., 2023) This paper estimates the test error by gauging the feature separability of the test datasct in the feature space:
K Sais (D wsst log k=1 my f
K
where μ denotes the center of the whole features, and μt denotes the mean of -class features.
Nuclcar Norm. (Deng et al., 2023) This paper estimates the test error by computing the nuclear norm of final softmax probabilities, which can be expressed as:
Snu (Dret) |||||
where Nuclear norm ||||| is defined the of singular values of P.
(Yu et al., 2022b) This method fine-tunes the pre-trained model on the test dataset with pseudo-labels, and measures the distribution discrepancy between the training and the test datasets in the parameter level:
ProjNorm.
(Ptes)))))) = ||ōres - 0|12.
where Oref denotes the parameters of the pre-trained model, while 0 denotes the parameters of the fine-tuned model.
Those algorithms mentioned in this paper can be summarized as Table 5.
Formulation of the entropy loss for low-confidence samples
In the ablation study, we explore the impact loss selection on the performance of unsupervised accuracy estimation. In particular, the detail about the entropy loss for samples with low confidence expressod follows: In particular, the entropy loss can be expressed as follows:
K
[(f(())) = 1
D log (1) ( (Sg(I (Z)on2T))
mh
km
m2
K
(1))
(fg (x(onST))) log s(t))) (f (x(OOST)))
m2
where the first term denotes the cross-entropy loss calculated for samples with confidence larger than the threshold value r, the second term denotes the entropy loss for samples with lower confidence than 7, and con means the sample confidence, mi and m2 denote the total number of samples with higher confidence and lower confidence than 7, respectively.
18

---

Published in Transactions on Machine Learning Research (04/2025)
Table 5: Method property summary including whether this method belongs to self-training or training-free approaches, and this method requires training data specific model architectures.
Method
Self-training
Training-free
Training-date data-fre Architetturerroniiccnn--ite
Rotation
ConfSeore
x
Entropy
X
Agreement
X
ATC
Fréchet
Dispersion
Nuclear
X
Norm
Ours
E Influence of the Calibration Error
Theoretically, the proposed pseudo-labeling strategy depends on how well the prediction probabilities are calibrated. However, in practice, we do not need to have a perfectly calibrated model as we employ a mixed strategy that assigns psoudo-labels to high-confidence examples and random labels to low-confidence ones (Sohn et al., 2020; Dong et al., 2021; Yu al., 2022a).
To demonstrate empirically, in Table 6, provide the expected calibration error (ECE, Guo al. (2017)) of ResNet18 depending on the difficulty of test data. For this, test first CIFAR-10 (ID), and then R- corrupted by brightness across diverse severity from 1 to 5. We can see that FCE is very low for ID data and remains relatively low across all levels of corruption severity, which shows that ResNet is quite well-calibrated on CIFAR-10.
Table 6: Expected Error Calibration (ECE) of ResNet18 on CIFAR-10 (ID) and CIFAR-10C corrupted by brightness across diverse severity from to 5.
Corruption Severity
ID
3
4
5
ECE
0.0067
0.0223 0.0230
0.0243
0.0255
0.0339
On the other hand, in the case of more complex distribution shift like Office-31, we can see that the calibration error has increased noticeably (Table 7). It is interesting to analyze this result together with Figure 3(a) of the main paper, where we compared the results between the usual pseudo-labeling strategy and the proposed one. Although our method has room for improvement compared to the oracle method, it is also significantly better than "pseudo-labels, indicating that the proposed label generation strategy is less sensitive to the calibration error.
Table 7: Expected Error Calibration (ECE) of ResNet18 on Office-31 data set.
Domain
DSLR (ID)
Amazon
Webcam
ECE
0.2183
0.2167
0.4408
F Comparison to Nuclear Norm
Here, we compare our method Nuclear Norm (Deng al., 2023) across 3 types of distribution shifts with ResNet18, ResNet50, and WRN-50- Results are shown in Table From this table, we observe our method outperforms Nuclear Norm under synthetic shift and natural shift.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Table 8: Performance comparison on 11 benchmark datasets with ResNet18, ResNet50 and WRN-50-2, where R² refers to coefficients of determination, and p refers to the absolute value of Spearman correlation coefficients (higher is better). The best results are highlighted in bold.
CIFAR 100
TinylmageNet
Office- Home
Camelyon17
Entity-15 13
Entity-30
Method
Network
R²
P
R²
p
R
p
R
p
R²
p
R²
p
ResNet18
0.989
0.995
0.983
0.994
0.692
0.783
0.858
1.000
0.978 0.991 0.980
0.993
ResNet50
0.979
0.994
0.965
0.994
0.731
0.895
0.849
1.000 0.989
0.996
0.978
0.994
Nuclear
WRN-50-2
0.962
0.988
0.956
0.992
0.766
0.874
0.983
1.000 0.989
0.995 0.985
0.996
Average
0.977
0.993
0.968
0.993
0.730
0.850
0.916
1.000
0.989
0.995 0.989 0.995
ResNet18
0.987
0.996
0.971
0.994
0.876
0.909
0.996
1.000
0.969
0.991
0.970
0.995
ResNet50
0.991
0.994 0.980
0.995
0.829
0.944 0.999 1.000 0.960
0.995
0.957
0.996
Ours
WRNt-50-2
0.995
0.998 0.975
0.996
0.809 0.916 0.997 1.000
0.968
0.995 0.949 0.994
Average
0.991
0.997 0.976
0.995 0.837 0.923 0.998 1.000 0.966
0.994
0.959 0.995
Table 9: Performance comparison between GDSCORE from different layers on CIFAR-10C with ResNet18.
The performance is measured by coelficients of determination (i.e., R²²).
ResNet18 Block 1 Block
Block 3 Last layer
All
CIFAR-10
0.880
0.972
0.984
0.986
0.983
G Why are the gradients of the entire network not predictive as that of the last layer?
We provide two intuitive explanations for this phenomenon, together with pointers to the existing body of work on our learning task, below:
Deep neural networks detect general features at lower (e.g., first) layers and specific features (parts, objects) at higher (e.g., last) layers (Chen et al., 2021). Consequently, higher layers would be more sensitive the changes distribution shift. than lower layers. So, reasonable that the last layer can capture more information about the distribution shift.
The gradients are getting weaker and weaker as moving back through the hidden layers (Raghu et al., 2021). Therefore, the information on the distribution shift contained the gradients may also decrease after backpropagation through layers.
This may explain why the gradients the last layer can achieve the best for this task. Moreover, the gap between different layers can be affected by network architecture. For example, recent work (Raghu al., 2021) shows that, compared to CNNs, VIIs have more uniform representations with greater similarity between lower and higher layers. The reason is that most information in VIT passes through skip connections. Therefore, it is natural that using different layers of ViT would not result in large gap in performance in method. This will not affect the effectivenoss of the proposed method using the last layer.
To certily it, we conduct an ablation study of layer selection with ResNet 18 and VIT in Table 9 and Table 10, respectively. Results shown below illustrate that the last layer performs satisfactorily for both ResNet models and transformer-based models. Furthermore, it worth noting that the estimation performance of ViT is more consistent even with deeper layers compared to R8sNet18.
H Proofs
H.1 Proof of Theorem 4.1
We start by proving the following lemma.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Table 10: Performance comparison between GDSCORE from different layers on CIFAR-1OC with ViT. The performance is measured by coefficients of determination (i.c., R²).
VIT
Block 1-18
Block 9-16
Block 17-24
Last layer
All
CIFAR-10
0.913
0.953
0.935
0.948
0.938
Lemma H.1. For any convez function :: + R and any p. such that + = 1, we have:
Va, b E dom(f);
(ʃ(a) - s(b)] v max (IV/()))))) - -1aabbly cE(a,b)
Proof. Using the fact that f is convex, have:
f(a) - f(b) s (Vf(a), b)
I (())))) b)
< Va(a), (a, - b,)l
$ f()) I//a bllg-
where we used Hölder's inequality for the last inequality. The same argument gives:
f(b) - f(a) < [v a
Using the absolute value, we can combine the two previous results and obtain the desired inequality.
The proof of Theorem 4.1 follows from applying Lemma H.l the convex function Lr.
H.2 Proof of Theorem 4.2
Proof. The proof follows from Theorem 4.1 by noting that || - cll, =
H.3 Proof of Theorem 4.3
We start by introducing some notations. We denote Lx.y the loss evaluated specific data-point v) ~ yx) We can then decompose the expected loss as Cr Ep((x.) fx.g. It follows by lincarity of the expectation that
vLr = (x Vxx.
Then, we prove the following lemmn that gives the formulation of the gradient of the cross-entropy. Lemma H.2. The gradient of the cross-entropy loss with respect to w (wa) wriles
(wx (-y 5 (x) k=1"
Proof. First, let's compute the partial derivative the softmax w.r.t we for any (1...., K}. We have:
0s (2)
(x)
wT x
EB e -
Ow
[x)]
[s(0)) (xx) (xx
1/ (x)
21

---

Published in Transactions on Machine Learning Research (04/2025)
Using the chain rule, the partial derivative of the loss w.r.t. We writes:
@Lxu(w) (Lx,((w)
(s(w,x)
Owk
0s(w,x) x)
dwk
(1) (x) [x)] )²} Bw
x
8w (x)
if (()) =1
Bw (x)
0,
otherwise
( x (1 (M)
/ (x))
dwn are the coordinates of Vx,,(w)) we obtain the desired formulation. As the (w)
We now proceed to the proof of Theorem 4.3.
Proof. Using the convexity of ||-||p and the Jensen inequality, we have that
[VLr(w)]lp = (x.)) VLxu(w)llp
S ())( fox.(w))ll
(Jensen inequality)
D K Σ
1/2
= (p)()
y (kxx)
x; -(0) (x))P
km
K
1/P
1/2
Σ (b) (x= (x)))
= -pp((x))
Σ
x
k=
1=1
1/p
((1- - ex)x 1/8
Epr(x.)
Σ
x
(Ry such that y(Ak)= =1)
where alw, x, y) (1- -0 (x)), with by such that (1())) 1. We used the fact that y is one-hot vector so it has only one nonzero entry.
H.4 Proof of Remark 6.1
Proof. Using the reverse Minkowski inequality, as < p 1, we have that
wellp = ||c + (Lr(w,)) 11p 2 ||c/| + [[CT(.)))l
Wrllpp - |||||p 2 1 V (w.,)llp
the same fashion, we have that We obtain the desired upper bound by combining those results.
|c||p = || - 77 Cr(ww)) 2 |lw + n I (W:)l ||c|| ||w,||p 2 n [[
22


Reviews:
[{"review_id": "fasSfMC3T3", "reviewer": "Reviewer_R4ZR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is generally easy to read and follow. The evaluation of the metric is very comprehensive, and shows that this simple measure is indeed a pretty decent proxy for test accuracy under certain kinds of distribution shifts. \n\nI have a few questions that I will be listing here. \n\n**Clarifying question re the approach**: The method just looks at the gradient of the outer layer of the pre-trained model using the soft-labelling described in sec. 4.1, and there is no update of weights using this gradient, right? I am asking this because throughout the intro and methodology section, that’s what the writing looks like. However, on page 9 “GdScore is a more efficient self-training approach” paragraph, the text says “our method only trains the model for one epoch….” If yes (assuming no updates are made), in Corollary 4.2 shouldn’t my main quantity of concern be $L_T(w_s)$ bc we want to get a proxy for $0/1$ error of the pre-trained model $w_s$? What is this telling me, then? If I were to take a step in that direction, the difference in loss would be small if my gradient norm at $w_s$ is controlled? Essentially saying that the model at $w_s$ was already good enough? Please clarify this a little.\n\n\n\n**Re the pseudo labelling**: In a very generic sense, you assume that if the source and train distribution are close enough, you’d probably get good pseudo-labels and random ones if they’re far apart (assuming the model’s confidence terrible, and uniform-ish). However, I could make adversarial distributions, for which the model is very confident with terrible predictions. Any comments on these kinds of distribution shifts?\n\n**Comparisons to other methods**: For all these methods, are the models they use also pre-trained on Imagenet, and then trained until same (similar) in distribution accuracy as your method? This feels like a very crucial detail that is missing.\n\n\n**Figure 5b)**: It is kinda odd that gradients of the entire network are not as predictive as that of the last layer. Why do you think that is the case?\n\n**Figure 5c)** is kind of an obvious result you’d expect (I think it also clarifies my Question 1? You’re just storing the gradients after 1 epoch, but please clarify). Any fine-tuning on the model won’t be predictive of the pre-trained model’s performance. But it is kind of surprising to see how fast, for CIFAR-100 for example, this predictive nature vanishes. Also, kind of surprising that it saturates for the other two datasets in the image and doesn't degrade any further. Some comment and discussion on this would be helpful. \n\nMinor: Please mention the GPUs you used, the compute time, all these details. Also, are the results averaged across different seeds? Please mention that too.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for unsupervised performance estimation under distribution shift. The authors show how gradient information can be a good predictive measure of test accuracy under such shifts. To avoid using ground truth test labels, they use a pseudo-labelling technique that basically uses the predicted label if the model is confident or a random one otherwise. The authors motivate their proposed gradient based measure with some simple theory. The evaluation is performed over a bunch of different distribution shifts for different pre-training datasets, and the authors compare their results to several baselines highlighting the goodness of their metric.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the questions in the section above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wJPlZVB6OG", "reviewer": "Reviewer_LHuS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "There are multiple strengths of this work. \n\nFirst of all, the draft works on both theoretical and empirical justification, which makes the draft look stronger than a pure theoretical work or pure empirical work.\n\nSecond, unlike existing works that mostly require the ground-truth label information in the test set to understand if the model is robust for the desired test sets, the proposed methods will not be constrained by lacking ground truth labels.\n\nFinally, The presented method, called GdScore, does show very high correlation with the test accuracy, and is clearly stronger than the baselines that it is compared to.\n\n\n\n\n\nWeakness :\n\nThis work uses the gradient from the classification layer, which imposes clear limitations to this work. For example, can this work be applied to foundation models which are not only favoring classification tasks? Can this work be favoring the models that are beyond the classification tasks?\n\nIn the continual learning domain, there are a lot of works that achieve the continual learning via adapting the layer normalization layers. That been said, two directions the authors can consider:\n\n-- For a large model where layer normalization, batch normalization are applied layer to layer, will the proposed methods still be significant? Assuming the model will be trained to do classification.\n\n-- Can the proposed methods be adapted to handle the normalization layers of the transformer?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies how gradients can be leveraged for accuracy estimation under the assumption of distribution shift. More specifically, the authors propose to use the norm of the classification-layer gradients and backpropaged from the cross-entropy loss for only one step, and use that information to estimate a score which reveals the confidence of the model in the new test dataset under the unknown distribution shift.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness part of the work in 'Strengths and weaknesses' section", "broader_impact_concerns": "No broader impact concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gAlDrVNYqC", "reviewer": "Reviewer_fmXC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- GdScore approach is elegant and intuitive, and results look very promising as well as more efficient\n- The self-labeling mechanism is an interesting method and can be reused to other problems\n- Paper is rigorous and all claims are justified theoretically and empirically\n- The paper provides meaningful qualitative and quantitative insights on the behavior of GdScore and other methods, as well as their sensitivity to different parameter choices\n\nWeaknesses:\n-  Assumption that classification errors mainly comes from the low confidence predictions is very strong, particularly in a distribution shift context. While it might hold for baseline datasets, I feel the paper does not discuss enough this assumption as well as the limitation it brings.\n- I would have wanted to see other measures of classification performance that would help assess the generalization capabilities of a given network, e.g. precision and recall in a binary classification case\n- The paper does not discuss enough the limitations of the metric presented - and therefore lacks nuance", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the context of assessing the performance of multi-class classification neural networks without access to ground-truth labels and under potential distribution shifts, the paper introduces GdScore, a novel statistic based on the norm of the gradients from the classification layer after one backpropagation step on test data. The authors show that this norm strongly correlates with test accuracy, offering a new and more efficient alternative to traditional methods.\n\nThe process works as follow:\n- Generate labels for the test data with a pseudo-labeling method based of prediction confidence\n- Using these labels, compute the gradient of the weights of the classification layer\n- Take the Lp norm of this gradient\n\nAfter a theoretical analysis and a formal introduction of the GdScore method, the paper presents empirical results on benchmark datasets with various types of distribution shifts in the test data. The testing shows that GdScore correlates well with test accuracy and outperforms the benchmark metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses section above. In short:\n- Further discuss the implications of the modeling assumptions made as well as their limitations / tradeoffs\n- Add further insights for alternative performance measures to get a more comprehensive view of how GdScore can help assess generalization\n- Paper and its organization can be confusing at times - e.g. related work at the complete end of the paper\n- Unclear what the ablation study is exactly - was not introduced previously\n- Add code for reproducibility", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers found the paper interesting and well written. The paper is complete as it provides both a theory and complementary, well conducted, experiments. The authors-reviewers discussion have been fruitful and the modifications asked by the reviewers were implemented and seem to have further raised the quality of the manuscript. 
Given all this, I recommend publication as is.
Decision: Accept (poster)