Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation
Authors: Weijian Deng
Yumin Suh
Stephen Gould
Liang Zheng
Venue: Submitted to ICLR 2023 (2023)
Keywords: Out-of-distribution generalization
Unsupervised Accuracy Estimation
Prediction DIversity
Distribution Shift

Abstract: This work focuses on estimating how well a model performs on out-of-distribution (OOD) datasets without using labels. Our intuition is that a well-performing model should give predictions with high confidence and high dispersity. While recent methods study the prediction confidence, this work newly finds dispersity is another informative cue. Confidence reflects whether the individual prediction is certain; dispersity indicates how the overall predictions are distributed across all categories. To achieve a more accurate estimation, we propose to jointly consider these two properties by using the nuclear norm of the prediction matrix. In our experiments, we extensively validate the effectiveness of nuclear norm for various models (e.g., ViT and ConvNeXt), different datasets (e.g., ImageNet and CUB-200), and diverse types of distribution shifts (e.g., style shift and reproduction shift). We show that the nuclear norm is more accurate and robust in predicting OOD accuracy than existing methods. Lastly, we study the limitation of the nuclear norm and discuss potential directions.

Under review as a conference paper at ICLR 2023
CONFIDENCE AND DISPERSITY SPEAK: CHARACTER- ISING PREDICTION MATRIX FOR UNSUPERVISED AC- CURACY ESTIMATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
This work focuses on estimating how well a model performs on out-of-distribution (OOD) datasets without using labels. While recent methods study the prediction confidence, this work newly reports prediction dispersity is another informative cue. Confidence reflects whether the individual prediction is certain; dispersity indicates how the overall predictions are distributed across all categories. Our key insight is that well performing model should give predictions with high confi- dence and high dispersity. Specifically, we need to consider the two properties so to make more accurate estimates. To this end, we use the nuclear norm which has been shown to characterize both properties. In our experiments, we exten- sively validate the effectiveness of nuclear norm for various models (e.g.. ViT and ConvNeXt), different datasets (e.g., ImageNet and CUB-200), and diverse types of distribution shifts (e.g., style shift and reproduction shift). We show that nuclear norm is more accurate and robust in predicting OOD accuracy than existing meth- ods. Furthermore, we validate the feasibility of other measurements (e.g., mutual information maximization) for characterizing dispersity and confidence. Lastly, we study the limitation of the nuclear norm and discuss potential directions.
INTRODUCTION
Model evaluation is critical in both machine learning research and practice. The standard evaluation protocol is to evaluate a model on held-out test set that is 1) fully labeled and 2) drawn from the same distribution as the training set. However, this way of evaluation is often infeasible for real- world deployment, where the test environments undergo distribution shifts and ground truths are not provided. In presence of a distribution shift, in-distribution accuracy may only be weak predictor of model performance (Deng & Zheng 2021 Garg et al., 2022). Moreover, annotating data itself is a laborious task, let alone is impractical label every new test distribution. Hence, way predict a classifier accuracy using unlabelled test data only has recently received much attention (Chuang
al. 2020 Deng & Zheng 2021 Guillory al. 2021 Garg al. 2022).
In the task of accuracy estimation, existing methods typically derive model-based distribution statis- ties of test sets (Deng Zheng, 2021] Guillory et al. [022 Deng et al., 2021: Garg et al. [022]] Back et al. 2022). Recent works develop methods based on prediction matrix on unlabeled data (Guillory al. 2021 Garg al. 2022). They focus on the overall confidence of the prediction ma- trix. Confidence refers to whether the model gives a confident prediction on an individual test data. can be measured by entropy or maximum softmax probability. Guillory et al. (202T) show that the average of maximum softmax scores on test set is useful for accuracy estimation. Garg et al. (022) predict accuracy as the fraction of test data with maximum softmax scores above a threshold.
In this work, we newly consider another property of prediction matrix: dispersity. It measures how spread out the predictions are across classes. When testing source-trained classifier on a target (out-of-distribution) dataset, target features may exhibit degenerate structures due to the distribution shift, where many target features are distributed a few clusters As a result, their corresponding class predictions would also be degenerate rather than diverse: the classifier predicts test features into specific classes and few into others. There are existing works that encourages the cluster sizes in the
target data to be balanced (Shi & Sha 2012 Liang et al. 2020
Yang al. 2021
Tang al. 2020

---

Under review as a conference paper at ICLR 2023
thereby increasing the prediction dispersity. In contrast, this work does not aim to improve cluster structures and instead studies the prediction dispersity to predict model performance on various test sets without ground truths.
To illustrate that dispersity is useful for accuracy estimation, we report our empirical observation in Fig. [ We compute the predicted dispersity score by measuring whether the frequency of predicted class is uniform. Specifically, we use entropy to quantify the frequency distribution, with higher scores indicating that the overall predictions are well-balanced. We show that the dispersity score exhibits a very strong correlation (Spearman's 's rank correlation > 0.950) with classifier perfor- mance when testing on various test sets. This implies that when the classifier does not generalize well on the test set, it tends to give degenerate predictions (i.e., low prediction dispersity), where the test samples are mainly assigned to some specific classes.
Based on the above observation, we propose to use nuclear norm, known to be effective in measur- ing both prediction dispersity and confidence (Cui et al., 2020 (02T), towards accurate estimation. Other measurements can also be used, such as mutual information maximizing (Bridle et al. [991 Krause et al. 2010 Shi & Sha 2012). Across various model architectures on a range of datasets, we show that the nuclear norm is more effective than state-of-the-art methods (e.g. ATC (Garg (ct al., 2022) and DoC (Guillory et al. 202I) in predicting OOD performance. Using uncontrollable and severe synthetic corruptions, strate that the nuclear norm still makes reasonably accurate estimations for test sets with moderate imbalances of classes. We additionally discuss potential solutions under strong label shifts.
show that nuclear norm is again superior. Finally, we demon-
2 RELATED WORK
Unsupervised accuracy estimation is proposed to evaluate model unlabeled datasets. Recent
methods typically consider the characteristics of unlabeled test sets (Deng & Zheng 2021
Guillory Chen et al.
ct al. 2021: Deng et al., 2021 Garg et al. 2022 Back et al. 2022 Yu et al. 2022
20216 a). For example, Deng & Zheng (021); Yu et al. (2022) : Chuang et al. 2020) consider the
distribution discrepancy for accuracy estimation. Chen et al. (2021b) achieve more accurate esti- mation by using specified slicing functions in the importance weighting. Chuang et al. (2020) learn domain-invariant classifiers on unlabeled test set to estimate the target accuracy. Guillory et al.) 22221) Garg et al. (2022) propose to predict accuracy based the softmax scores on unlabeled data. In addition, agreement score of multiple models' predictions on test data is investigated in (Madani et al. 2004| Platanios et al. 2016 2017, Donmez et al.) 2010 Chen et al.) 2021a). This work also focuses on estimating a model's OOD accuracy on various datasets and proposes to achieve robust estimations by considering the both prediction confidence and dispersity.
Predicting ID generalization gap. To predict the performance gap between certain pair of training-testing set, several works explore develop complexity measurements trained models and training data (Eilertsen et al., 20200] Unterthiner et al. 2020: Arora al. 2018 Corneanu et al.)
2020 Jiang et |01 9a. Neyshabur al.) 2017: Jiang et al. 2019b; Schiff et al. 2021 For
ample. Corneanu et al. (2020) predict the generalization gap by using persistent topology measures.
Jiang et al. (2)19)) develop measurement of layer-wise margin distributions for the generalization prediction. Neyshabur et al. 2017) use the product of norms of the weights across multiple layers. Baldock et al. (2021) introduce measure of example difficulty (i.e., prediction depth) study the learning of deep models. CChang et al. (2021) develop margin-based generalization bounds with optimal transport. The above works assume that the training and test sets are from the same distri- bution and they do not consider the characteristics of the test distribution. In comparison, we focus on predicting a model's accuracy on various OOD datasets.
Calibration aims to make the probability obtained by the model reflect the true correctness like- lihood (Guo et al., 2017 Minderer et al.. 202T). To achieve this, several methods have been de- veloped to improve the calibration of their predictive uncertainty, both during training (Karandikar ctt al. 1 Krishnan & Tickoo 2020) and after (Guo et al., 2017 Gupta et al., 2021 training. For perfectly calibrated model, the average confidence over a distribution corresponds to its accuracy over this distribution. However, calibration methods seldom exhibit desired calibration performance under distribution shifts (Ovadia et al.) 2019, Gong et al.) 2021). To estimate OOD accuracy, this work does not focus on calibrating confidence. Instead, we use the dispersity and confidence of prediction matrix to predict model performance on unlabeled data.
2

---

Under review as a conference paper at ICLR 2023
ResNee152 Aog
make
Top occuroo (R))
Sibe
997
0.8
038
0.8
308
032 1.0
0.52 Pretiction Digprriity
Prediction
crio Disporstty
Prediction Dispersity
Figure 1: Strong correlation between prediction dispersity and classifier accuracy. Each point corresponds to one test set of ImageNet-C. The straight lines are calculated by linear regression. We study four ImageNet models (ViT, DenseNet, Bei'T, and ResNet1 We compute the pre- dicted dispersity score by measuring how uniform the frequency of predicted class is. We observe that prediction dispersity exhibits a strong correlation (Spearman's 's rank correlation p > 0.950) with classification accuracy for various test datasets. This indicates that if a classier gives class predic- tions with high dispersity, likely achieves high accuracy, and not otherwise.
3 METHODOLOGG
3.1 PROBLEM DEFINITION
Notations. Consider classification task with input space X' C R and label space 3 (l....,AA k). Let ps and pr denote source and target distributions over X' x Y. respectively. Given a source training dataset DS train drawn from ps, train probabilistic predictor f IRd DR, where Ae denotes the k - 1 dimensional unit simplex. We assume held-out test set DJ
test = (() contains n, data i.i.d sampled from ps. When queried at source data (x* v²" of D Diest returns ÿ =: arg max Jn (xr") as the predicted label and ý =: maxjey fx(x)) as the associated soft- max confidence score. With label, we can easily compute the classification error on that data by E(f(x)), 3²) == Leondition (y" # 9). By calculating the errors on data of DS Disnss we evaluate the accuracy on the source (in-distribution) ps-
Unsupervised Accuracy Estimation. Due to distribution shift (ps * pr). the accuracy on in-distribution D is usually weak estimate of how well performs the target (out-of- distribution) pr- This work aims to assess the generalization of I on target (out-of-distriiutiin) Pr without access to labels. Concretely, given source-trained ʃ and unlabeled dataset DI {((=))))))))))))) ) with ne samples drawn i.i.d. from Pr, we aim to develop a quantity that strongly correlates with the accuracy of f on DJT Note that, the target distribution pr has the same K classes as the source distribution ps in this work (known the closed-set setting). Unlike domain adapta- tion, which aims to adapt the model to the target data, unsupervised accuracy estimation focuses on predicting model accuracy on various unlabeled test sets.
3.2 PREDICTION CONFIDENCE AND DISPERSITY
Let P E I x of i-th target data. The values of P are in the interval [0, 1]. Based on the predicted class of each softmax vector, we divide P into & class groups (k is the number of classes). Then, we analyze the following two properties of P.
denote the prediction matrix of f on DJT and its each row Pt.: is the softmax vector
Confidence. It measures whether softmax vector (each row of P) is certain. Common ways to measure the confidence include entropy and maximum softmax score. the overall confidence of P is high, then it implies that the classifier f' is certain on the given test set. Prediction confidence has been reported be useful in predicting classifier performance on various test sets (Guillory et al., 2021 Garg al., 2022). For example, the overall confidence of P measured by the average of maximum softmax score is predictive of classifier accuracy (Guillory et al. 2021). Other measures such entropy (Guillory et al. 2021 Garg al. 2022 also give similar observation.
Dispersity. It is another property of P that measures whether the predicted classes are diverse and well-distributed. High dispersity means that predictions on test samples are well-distributed among & classes. When testing source-tras cuure--train classifier f on a target dataset DT the target features may exhibit degenerate structures due to distribution shift. A commonly seen pattern that many
3

---

Under review as a conference paper at ICLR 2023
target features are distributed in few clusters. This likely leads to degenerate predictions: the clas- sifier tends to predict test features into some particular classes (and neglects other classes). Recent methods (Tagg et al. 2020 Liang et al.] 2020 Yang et al. (022) report that regularizing prediction dispersity by encouraging cluster size to be balanced is beneficial when training domain adaptive models. Here, we study whether prediction dispersity is useful for the problem of accuracy estima- tion, instead of adapting models to the target domain.
To verify the usefulness of dispersity in accuracy prediction, we conduct preliminary correlation study using ImageNet-C in Fig. Here, the prediction dispersity score is simply computed by measuring whether the number of softmax vectors each class is similar: we first calculate the histogram of the sizes of predicted class, and then use entropy to measure the degree of balance. With four ImageNet models, we observe that prediction dispersity has a consistently strong correlation (rank correlation p > 0.950) with model accuracy on various test sets (ImageNet-C). This shows that when classifier does not generalize well on test data, it tends to give degenerate predictions (low prediction dispersity), where the test samples are mainly assigned to some specific categories.
3.3 CHARACTERIZING DISPERSITY AND CONFIDENCE WITH NUCLEAR NORM
Based on the above observation, we aim to quantify dispersity and confidence of prediction matrix P to estimate classification accuracy. For this purpose, we resort to the nuclear norm which is known to be effective in measuring both prediction dispersity and confidence (Cui et al. 2020 2021).
Nuclear norm || Pll is defined as the sum of singular values of P. It is the tightest convex envelop of rank function within the unit ball (Fazel) 2002). A larger nuclear norm implics more classes are predicted and involved, indicating higher prediction dispersity. In addition, the nuclear-norm is an upperbound of the Frobenius-norm that || reflects prediction confidence Cui et al. 2020 In Section A of the appendix, briefly introduce how nuclear norm reflects the predic- tion confidence and dispersity. Since test sets can contain any numbers of data points, we normal- ize the nuclear norm of prediction matrix by its upper bound derived from matrix size and obtaín
||P|| / /iiin(n,, k) -7, We mainly use Pl| to measure the confidence and dispersity of prediction matrix P in this work. In the experiment, we also show that another measure mutual
||P|I
information maximization (Bridle et al. [991] Krause et al. 2010 Shi & Sha 2012 Yang et al.
2022 is also feasible for the task of accuracy estimation.
4
EXPERIMENT
4.1 EXPERIMENTAL SETUPS
ImageNet-1K. (i) Model. We use 6 representative neural networks provided by (Wightman 2019 First, we include three vision transformers: ViT-Base-PI6 (ViT) (Dosovitskiy et al. 2020). BET-
Base-P16 (BEiT) (Liu et al. 2022). and Swin-Small-P11 (Swin) (Liu et al. 2021). Second, we include three convolution neural networks: DenseNet-121 (DenseNet), R 152-BiT-M (Res152-BiT)) (Kolesnikov et al. 2020), ConvNeXt-Base (Liu et al. 2022). They either trained or fine-tuned on ImageNet training set (Deng et al.) 2009).
(ii) Synthetic Shift. We use ImageNet-C benchmark (Hendrycks & Dietterich 2019) to study the synthetic distribution shift. ImtgeNet-C is controllable in terms of both type and intensity of cor- ruption. It contains 95 datasets that are generated by applying 19 types of corruptions (e.g., blur and contrast) to ImageNet validation set. Each type has 5 intensity levels.
(ii) Real-world Shift. We consider four natural dataset shifts, including 1) dataset reproduction shift in ImageNet-V2-A/PE V2-A/B/C (Recht et al.) 2019). 2) sketch shift in AmageNet-S(kketch) (Wang et al.) 2019, 3) style shift in ImageNet-RRendittion
(Hendrycks et al. | [22T)), and 4) bias-controlled dataset shift in ObjectNet (Barbu et al. 2019). Note that, ImageNet-R R and ObjectNet only shares common 113 and 200 classes with ImageNet, respectively. Following (Hendrycks et al. 2021), we sub-select the model logits for the common classes of both test sets.
CIFAR-10 (i) Model. We use ResNet-20 (He et al., 2016). RepVGG-AO (Ding et al. 2021 and VGG-11 (Simonyan & Zisserman 2014). They are trained on CIFAR-I0 training set. (ii) Synthetic Shift. Similar to ImageNet-C. C, we use CIFAR-I0-E -O-C (Hendrycks & Dietterich 2019) to study the synthetic shift. It contains 19 types of comuptions where there are 5 intensity levels for
4

---

Under review as a conference paper at ICLR 2023
each type. (iii) Real-world Shift. We include three test sets: 1) CIFAR-10.1 with reproduction shift (Recht et al. 2018). 2) CIFAR-10.2 10.2 with reproduction shift (Recht et al. 2018). and 3) CINIC-10 test set that is sampled from a different database ImageNet.
CUB-200. We also consider fine-grained categorization with large intra-class variations and small inter-class variations (Wei al., 2021). We build up setup based on CUB-200-2011 (Wah al.) 2011) that contains 200 birds categories. (i) Model. We use 3 classifiers: ResNet-50, ResNet-101, and PMG (Du al. 2020). They are pretrained on ImageNett and finetuned on CUB-200-2011 I train- ing set. We use the publicly available codes provided by (Du et al. (020). (ii) Synthetic Shift. Fol- lowing the protocol in ImageNet-C, we create test set. (iii) Real-world Shift. We use
applying 19 types of corruptions with
5 intensity levels to CUB-200-220
with style shift (Wang et al. 2020). It contains bird paintings with various rendition (e.g., watercol- ors, oil paintings, pencil drawings, stamps, and cartoons) collected from web.
4.2 COMPARED METHODS AND EVALUATION METRICS
We use four existing measures for comparison. They are all developed based on the softmax output of classifier. 1) Average Confidence (AC) (Hendrycks & Gimpell 2017). The average of maximum softmax scores on the target dataset; 2) Average Negative Entropy (ANE) (Guillory et al. ( The average of negative entropy scores on the target dataset; 3) Average Thresholded Confidence (ATC) (Garg et al. 2022). This method first identifies threshold on source validation set. Then, ATC is defined as the expected number of target images that obtain a softmax confidence score than the threshold; 4) Difference of Confidence ( (DOC) (Guillory al.] [02T). It is defined as the source validation accuracy minus the difference of AC on the target dataset and source validation set. The difference of AC is regarded as a surrogate of distribution shift.
Evaluation Procedure. Given a trained classifier, we test it on 95 synthesized test sets under each setup. For each test set, we calculate the ground-truth accuracy and the estimated OOD quantity. Then, we evaluate the correlation strength between the estimated OOD quantity and accuracy. We also show scatter plots and mark real- world datasets to compare different approaches.
Evaluation Metrics. To measure the quality of estimations, we use Pearson Correlation coeffi- cient (r) (Benesty et al., 2009) and Spearman's Rank Correlation coefficient (p) (Kendall 1948) to quantify the linearity and monotonicity. They range from [-1, 1]. A value closer to (or indicates strong positive (or negative) correlation, and 0 implies no correlation [eenesty et al. 2009). To precisely show the correlation, we use prob axis scaling that maps the range of both accuracy and estimated OOD quantity from [0, 1] to [-o0, +oo], following Taori et al. (2020): Miller et al. 2021). We also report the coefficient of determination (IR²) (Nagelkerke et al.
1991 of the linear
fit between estimated OOD quantity and accuracy following (Yu et al. 2022). )). The coefficient R²
ranges from 0 to 1. An R² of indicates that regression predictions perfectly fit OOD accuracy.
4.3
MAIN RESULTS
Nuclear norm is an effective indicator to OOD accuracy. In Table we report the correlation results of nuclear norm under three setups: ImageNet- CIFAR-10, and CUB-200. We consis- tently observe a very strong correlation (122 > 0.945 and p > 0.960) between the nuclear norm and ODD accuracy under the three setups. The strong correlation still exists when using different model architectures under each setup. For example, the average coefficients of determination 1t achieved by nuclear norm are 0.979, 0.990, and 0.989 on ImageNet- 1k. CIAFR-10, and CUB-200, respec- tively. It demonstrates that nuclear norm well captures the distribution shift and makes excellent OOD accuracy estimations for different classifiers.
Nuclear norm is generally more robust and accurate than existing methods. Compared with existing methods, nuclear norm achieves the strongest correlation with classifier performance across all the three setups by considering prediction dispersity as well as the confidence. With different models on ImageNet, nuclear norm achieves an average It2 of 0.979, while the second best method (ATC) only obtains 0.924. Moreover, nuclear norm outperforms ATC by 0.262 and in average R under CUB-201 setup. Moreover, the prediction performance of nuclear norm is overall more robust than other methods. Specifically, we observe that the competing methods are less effective in predicting the accuracy of centain classifiers such as Swin under the ImageNet setup and ResNet-110
5

---

Under review as a conference paper at ICLR 2023
Table 1: Method comparison under ImageNet, CIFAR-10, and CUB-200 setups. We compare nuclear norm with four existing methods. To quantity the effectiveness in assessing OOD gener- alization, we report coefficients of determination (R²) and Spearman's 's rank correlation (p). The highest score in each row is highlighted in bold. We show that nuclear norm exhibits the highest correlation strength (R² and p) with OOD accuracy across three setups.
AC
ANE
ATC
DoC
Nuclear Norm
Setup
Model
R²²
P
R²
p
R²
p
R²2
If²
p
VIT
0,970
0.990
0.964
0.988
0.978
0.990
0.961
0.990
0.991
0.995
BeiT
0.977
0.994
0.964
0.989
0.985
0.995
0.979
0.994
0.988
0.996
Swin
0.794
0.929
0.732
0.909
0.815
0.935
0.791
0.929
0,949
0.961
ImageNet
DenseNet
0.938
0.984
0.929
0.979
0.961
0.989
0.937
0.984
0.995
0.997
Res152-BiT
0.891
0,981
0.877
0.979
0.916
0.982
0.908
0.981
0.981
0.991
ConvNeXt
0.894
0.971
0.866
0.960
0.888
0.967
0.899
0.971
0.967
0.982
Average
0.911
0.975
0.889
0.968
0.924
0.976
0.911
0.975
0.979
0.989
ResNet-20
0.916
0.991
0.916
0.991
0.934
0.992
0.937
0.991
0.989
0.995
RepVGG-AO
0.811
0.982
0.806
0.981
0.841
0.985
0.824
0.982
0.992
0.996
CIFAR-10
VGG-11
0.973
0.994
0.973
0.995
0.984
0.996
0.964
0.994
0.988
0.996
Average
0,900
0.989
0.900
0.988
0.920
0.991
0.908
0.989
0.990
0.995
ResNet-50
0.836
0,942
0.839
0.939
0.855
0.957
0.818
0.942
0.989
0.997
ResNet-101
0.303
0.734
0.319
0,739
0.351
0.775
0.308
0.734
0.987
0.998
CUB-200
PMG
0.892
0.979
0.893
0.977
0.977
0.991
0.903
0.979
0.990
0.998
Average
0.677
0.885
0.684
0.885
0.727
0.908
0.677
0.885
0.989
0.997
Top-1 accuracy (96)
(w)) 883
RF-0 267
Img-2288-
7w0 $31
glw 285
mg--2-3
R dsc
fw0 945
Img. Vilc
72
65
Ing V2
Img
In6
50
50
Img
33
Ing
Clgesthirt
Imge
ObjectNot
35
OnjectNet
mg 5
Imp-5
23
20
077075 on 0.5
5.6
8.
0.03
0.6
0.62
0.94
0.66
0.65
0.4
0.45
Nuclear Norm
Average Negative Entropy
Average Confidence
R ass
Imp-2--8
R²
V2--2-5
brightnees
Top-1 accuracy (9)
80
50
P-0.957
detocus, blur transform
es
Ing V2
65
Ing V2 A
Tog
gaussian blur
52
Ing
geuss roise
Objecthie
Osjecthatt
Imuu noize
Img
mootin blur
Seturate
Snon7
0.55
0.1
0.15
0.6
0.55
0.5
0.6
9.7 0.8
speckie naise
Difference of Confidence
Average Thresholded Confidence
200m, DiL
Figure 2: Correlation study under the ImageNet setup. We plot the actual accuracy of ConvNeXt and five measures including nuclear norm and four competing methods. Different shapes in each sub-figure represents different test sets. The straight lines are calculated by linear regression fit on synthetic datasets of ImageNet-C. We list the 19 types of corruptions in ImageNet-C using different shapes and colors in the bottom right figure. We also mark the 6 real-world datasets in each sub- figure with arrows. We observe nuclear norm exhibits stronger correlation with accuracy. Moreover, with nuclear norm, real-world test sets are closely around the linearly fit line.
under the CUB- 200 setup. For these difficult cases, nuclear norm remains useful with R² > 0.945, which further validates its effectiveness.
Nuclear norm can estimate accuracy of real-world datasets. To further validate the effectiveness of nuclear norm, we show its accuracy prediction real-world datasets as the scatter plots under

---

Under review as a conference paper at ICLR 2023
nil tht
net
CRAN16
592
pucace
DIAROAL
CNC
OVE
13 50 C.
002 025 1-K
204
65 82 Ga
064
0.6
640
Confidence
Average Thresholded Confidence
Difforence Confidence
Figure 3: Correlation study under the CIFAR-10 setup. We plot the actual accuracy of ResNet- 20 and the estimated OOD quantity. We show the results of nuclear norm, AC and ATC. The lines calculated by linear regression fit CIFAR-C. We mark the 3 real-world test sets in each sub- figure. We show that AC and ATC fail to estimate generalization on datasets with lower ground-truth accuracy. In comparison, nuclear norm more robust and accurate.
and and
For R3A
plaxee
Norm
Average Confidence
Average Thresholded Conficence
Difference of Confidence
Figure 4: Correlation study under the CUB-200 setup. We plot the actual accuracy of ResNet-50 and the estimated OOD quantity. We compare nuclear norm with AC and ATC. The straight lines are calculated by the linear regression on CUB-200-C. We mark the real-world test set CUB-P in each sub-figure. We show that AT and ATC cannot give accurate estimates for some datasets. In contrast, nuclear norm is more robust and accurate in predicting generalization: all test sets are closely around the linear line, yielding higher correlation strength.
the three setups (Fig. 2 Fig. and Fig. 4 respectively). We observe that nuclear norm can produce reasonably accurate estimation on real-world test sets. Under the ImageNet setup (Fig. ( the six test (e.g., Vma/A/ettV2//A/B/CC and ImageNet-R) demonstrates that nuclear norm well captures these real-world shifts and thus estimates OOD performance very well. Under CIFAR-10 and CUB-200 setups, we have similar observations.
very close the linear regression line.
Although existing methods (e.g., ATC) are effective on most real-world datasets, nuclear norm still shows its advantage over them. Other methods fail capture the shifts of ImageNet-S and ObjectNet under the ImageNet setup: they are far away from linear lines. In comparison, nuclear norm captures them well and both datasets are very close to linear lines. Furthermore, the scatter plots under the CIFAR-10 10 (Fig. 3) and CUB-200 (Fig. 4 show that the competing methods often give accuracy numbers lower than the ground truth when the test set is difficult, while nuclear norm still effective.
4.4 DISCUSSION AND ANALYSIS
(I) Beyond controllable synthetic shifts. The synthetic datasets (e.g., ImageNet-C) are algorithmii- cally generated in controllable manner. Here, we investigate whether measure is robust in pre- dicting OOD accuracy on random synthetic datasets. To this end, we randomly synthesize datasets for the CIFAR-10 10 setup. Specifically, we use 10 new corruptions of ImageNet-C (Mintun et al., 2021 that are perceptually dissimilar to ImageNet-C. The dissimilar corruptions include warps, blurs, color distortions, noise additions, and obscuring effects. When synthesizing each test set, we randomly choose corruptions and make corruption strength random. By doing so, create 200 random synthetic datasets denoted CIFAR-C-Rand.
In Fig. 5 we report the correlation results using ResNet-20 under the CIFAR- 10 setup. We also show the linear regression lines that are fit on datasets of CIFAR-10-C. We report results of four methods including nuclear norm, AC, ATC, and DoC. We have two observations. First, for each method, CIFAR-C-Rand datasets (marked with "+") are generally distributed around the linear lines. This indicates that all methods can make reasonable accuracy estimations MEFAR-C-Rand Second,
7

---

Under review as a conference paper at ICLR 2023
1 01A0-11-2trt
SPAR-ES Chaan
STAR-IE-C
accurac
Prea77
F 7/0903
R Pus
Nuclear
Averaga Confidence
Negazie Entropy
Thresholded Confidincce
Figure 5: Correlation study on randomly synthesized datasets under the CIFAR-10 setup. We report results with ResNet-20. Randomly synthesized datasets (CIFAR-IO-C-Rand) are marked with orange "4", and the solid lines are fit with robust linear regression on controllable CIFAR- 10-C. Overall, CIFAR-IO-C-Rant datasets are distributed around the linear line for every method. Looking more closely low-accuracy region (bottom left in each subfigure), nuclear norm gives more accurate estimations than other methods, indicating its effectiveness.
for the low-accuracy region (bottom left in each subfigure), nuclear norm gives more accurate and robust predictions than other methods (e.g., ATC and DoC).
(II) Other measures to consider prediction con-
Method
ImageNet
CIFAR
CUB
fidence and dispersity. Here, we discuss the us- age of other measures. We study mutual infor- mation maximizing (MI) which is commonly used in discriminative clustering (Bridle al.] 1991 Krause et al., 2010). Recent methods use it as regularization make model predictions confi- dent and diverse (Liang et al. 2020 Yang et al. Tang et al., 2020). Given prediction matrix R K IM is defined as H ( Ei P..))
ANE
0.968
0.988
0.885
MI
0.982
0.994
0.995
Nuelear Norm
0.989
0.995
0.997
Table 2: Correlation results using mutual information maximizing (MI). We report the average correlation strength (Spearman's 's rank correlation p) under each setup. We observe MI and nuclear norm have similar correlation strength. Compared with average negative en- tropy (ANE), MI exhibits stronger correlation with accuracy across three setups.
2021
PEL
n, E (P..... Its first term encourages the pre- dictions to be globally balanced. The second term is standard entropy that makes the prediction confi- dent. In Table 2 we report the correlation results using MI. We observe that MI and nuclear norm achieve similar average correlation strength. Compared with average negative entropy (ANE), MI exhibits stronger correlation across three setups. For example, MI yields 0.110 higher & than ANE on CUB. This further validates that prediction dispersity is informative for accuracy estimation.
III. Effect of test set size. As illustrated in Section 3.3 nuclear norm without scaling is related to the size of the prediction matrix. Since test sets can contain any numbers of data points, we normalize nuclear norm by its upper bound. Here, we change the size of each dataset of ImageNet-
C by randomly selecting 20-90% of all test samples. As shown Fig. 6 scaled nuclear norm is well correlated with accuracy under different test set sizes.
Rav152-8IT
Densebeut
of Bamii
Keet 011 pod
Ford
HwO 30%
592
0:55
Norm
Figure 6: Analysis of the influence test set size on nuclear norm. We conduct correlation study on randomly sub-sampled ImageNet-C. Specifically, vary the size of each dataset by randomly selecting 20-90% of test samples. We test three classifiers and observe the correlation strength remains very high (R2 0.960 and p > 0.970).
(IV) Discussion on label shift (class imbalance). In our work, we consider the common covari- ate shift (Sugiyama & Kawanabe 2012) where (x) # pr(x) and ps(v)x) ()))) (i.e., the class label of the input data is independent of distribution). Nuclear norm measures the predic- tion dispersity and thus implicitly assumes that the test set does not contain strong label shift (i.e.,
8

---

Under review as a conference paper at ICLR 2023
m=01
m=0 m=0.a
m-0
F1-Macro (%)
M
75
m=0
75
m-0.4
62
m= 1.0
15
m- 1.0
R -0-0.881
R- 950
46
RF-0.990 900
16
p=o 380
36
0.86
Nuclear Norm
Average Negative Entropy
Average Confidence
m-0
m m=g
F1-Macro (%)
m-
m=0.8 m=1.0
m=10
m=
59
p.0.922
28
p=0 962
48
R²
p=0 082
35
1.8 9.85 2:83 0.00 0.59
0.81
0.7
0.75
2.43 0.50
0.55
Mutual Information Maximization
Difference of Confidence
Average Thresholded Confidence
Figure 7: Comparison of various methods on imbalanced test sets. Using ViT under ImageNet setup, we study the robustness of existing methods several imbalance ratio m when test sets are long-tailed. A smaller m indicates higher imbalance intensity. The linear lines are fit on standard test sets (m = 1). We observe that both mutual information maximization (MI) and nuclear norm are less effective than other methods under strong-imbalanced datasets (m < 0.4). Furthermore, we
show that MI and nuclear norm are robust under muld-immallanee test sets (m 2 0.4).
class imbalance). As for the label shift (Grgg et al. 2020), the assumption about the distribution is ps(y) # pr (y) and ps(a)ly) (y) = pr pr(ary (i.e., the class-conditional distribution does not change).
Here, we discuss the robustness of nuclear norm to label shift. We first note that real-world test sets such as ImageNet-R, ObjectNet, and CUB-200-P are already imbalanced. We show that nuclear norm robustly captures them: they are very close to the linear lines (as shown in Fig. and Fig, 4 To further study the effect of label shift, we create long-tailed imbalance test sets. We use an exponential decay to make the proportion of each class different between source and target datasets following Cao et al. 2019 We use imbalance ratio m to denote the ratio between sample sizes of the least frequent and most frequent class. We test several imbalanced ratios: {0.1, 0.2, 0.4, 0.6, 0.8). We conduct the experiment on ImageNet-C and use 19 types of corruption datasets with the second intensity level. As shown in Fig. 7 observe that both nuclear norm and MI are influenced by label shift when the imbalance is strong (m < 0.4). For example, when the test set is of extreme class imbalance (m 0.1), the prediction of nuclear norm is not accurate. We also observe that under the strong imbalance (m < 0.4), exiting methods (e.g., ATC) is more stable than nuclear norm and MI. We further note that nuclear norm and MI are robust to the mild imbalance (m 0.4).
The above analysis indicates that nuclear norm is robust in the presence of moderate label shift, To deal with strong label shift, using extra techniques such as label shift estimation (Lipton al. 2018 Tian et al. 2020) and prior knowledge (Chen et al.) 2021b Sun et al. 2022) would be helpful. It would be interesting to further study this idea in future work.
CONCLUSION
This work studies OOD accuracy estimation where the goal to predict classifier accuracy on unlabeled test sets. While existing methods study the confidence of prediction matrix unlabelled data, this work newly considers the prediction dispersity. measures whether the overall predictions well-balanced across classes. We report that prediction dispersity is useful property which correlates strongly with classifier accuracy on various test sets. Driven by this new observation, we consider both prediction confidence and dispersity to achieve more accurate estimation. To this end, we use the nuclear norm of prediction matrix to characterize both properties. Across three setups, consistently observe that nuclear norm is more effective and robust in assessing classifier OOD performance than existing methods. We further conduct experiment on imbalanced test sets and show that nuclear norm still effective under moderate class imbalances.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. arXiv preprint
2018.
Christina Back, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on line-ine: Predict- ing the performance of neural networks under distribution shift. arXiv preprint arXiv:2206.13003 2022.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34: 34:10876-10889, 2021.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlle dataset for pushing the limits of object recognition models. In Advances in neural information processing systems, 2019.
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pp. 1-4. Springer, 2009.
John Bridle, Anthony Heading. and David MacKay. Unsupervised classifiers, mutual information and' phanton targets. Advances in neural information processing systems, 4, 1991.
Kaidi Cao. Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aaaaa margin loss. Advances in neural information processing systems, 32, 2019.
Jiefeng Chen, Frederick Liu, Besim Avei, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems, 34, 2021a.
Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Mandoline: Model evaluation under distribution shift. In International Conference on Machine Learning. pp. 1617-1629, 2021b.
Patryk Chrabaszez, lya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819, 2017.
Ching-Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under dis- tribution shifts via domain-invariant representations. In International Conference on Machine Learning, 2020.
Ching-Yao Chuang, Youssef Mroueh, Kristjan Greenewald, Antonio Torralba, and Stefanie Jegelka. Measuring generalization with optimal transport. In Advances in Neural Information Processing Systems, volume 34, pp. 8294-8306. 2021.
Ciprian A Corneanu, Sergio Escalera, and Aleix M Martinez. Computing the testing error without testing In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2677-2685, 2020.
Shuhao Cui, Shuhui Wang, Junbao Zhuo Liang Li, Qingming Huang, and Qi Tian. Towards dis- criminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
3941-3950, 2020.
Shuhao Cui, Shuhui Wang, Junbao Zhuo Liang Li, Qingming Huang, and Qi Tian. Fast batch nuclear-norm maximization and minimization for robust domain adaptation. arXiv preprint arXiv:2107.06154, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. 2009.
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15069-15078, 2021.
10

---

Under review as a conference paper at ICLR 2023
Weijian Deng, Stephen Gould, and Liang Zheng. What does rotation prediction tell us about clas- sifier accuracy under varying testing environments? In International conference on machine learning, 2021.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Com- puler Vision and Patlern Recognition, pp. 13733-13742, 2021.
Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. Unsupervised supervised learn- ing i: Estimating classification and regression errors without labels. Journal of Machine Learning Research, 11(4), 2010.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Ruoyi Du Dongliang Chang, Ayan Kumar Bhunia, Jiyang Xie, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Fine-grie grne-grained visual classification via progressive multi-granularity training of jigsaw patches. In European Conference Computer Vision, pp. 153-168, 2020.
Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the classifier: dissecting the weight space of neural networks. arXiv preprint arXiv:2002.06688, 2020.
Maryam Fazel. Matrix rank minimization with applications. PhD thesis, PhD thesis, Stanford University, 2002.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishman, and Zachary Lipton. A unified view of label shift estimation. In Advances Neural Information Processing Systems, pp. 3290-3300, 2020.
Saurabh Garg. Sivaraman Balakrishnan, Zachary Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict nut-of-distribution performance. In International Conference on Learning Representations, 2022.
Yunye Gong, Xiao Lin, Yi Yao, Thomas G Dietterich, Ajay Divakaran, and Melinda Gervasio. Confidence calibration for domain generalization under covariate shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp.
2021.
Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence unseen distributions. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 1134-114 1144, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proc. ICML., pp. 1321-1330, 2017.
Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. Calibration of neural networks using splines. In International Conference on Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribatis examples in neural networks. In International Conference on Learning Representations, 2017.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of duttof-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8349, 2021.

---

Under review as a conference paper at ICLR 2023
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In International Conference on Learning Represen- tations, 2019a.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where find them. In International Conference Learning Representations, 2019b.
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C Mozer, and Becca Roelofs. Soft calibration objectives for neural networks. In Ad- vances in Neural Information Processing Systems, pp. 29768-29777 2021.
Maurice George Kendall. Rank correlation methods. 1948.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European confer- ence on computer vision, pp. 491-507, 2020.
Andreas Krause, Pietro Perona, and Ryan Gomes.
Discriminative clustering by regularized infor- mation maximization. Advances neural information processing systems, 23, 2010.
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus un- certainty optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 18237-18248, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028 6039. PMLR, 2020.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122-3130, 2018.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Omid Madani, David Pennock, and Gary Flake. Co-validation: Using model disagreement un- labeled data to validate classification algorithms. In Advances in neural information processing systems, pp. 873-880, 2004.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: the strong correlation between out-of-distribution and in-distribution generalization. International Conference on Machine Learming. pp. 7721-7735, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems, pp. 15682-15694, 2021.
Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and cor- ruptions in natural corruption robustness. In Advances in Neural Information Processing Systems, 2021.
Nico JD Nagelkerke et al. A note on a general definition of the coefficient of determination. Biometrika, 78(3):691-692, 1991.
12

---

Under review as a conference paper at ICLR 2023
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. In Advances in neural information processing systems, pp. 5947-5956, 2017.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Process- ing Systems, 2019.
Emmanouil Platanios, Hoifung Poon, Tom M Mitchell, and Eric Horvitz. Estimating accuracy from unlabeled data: A probabilistic logic approach. In Advances in Neural Information Processing Systems, pp. 4361 -3700, 2017.
Emmanouil Antonios Platanios, Avinava Dubey, and Tom Mitchell. Estimating accuracy from unla- beled data: A bayesian approach. In International Conference Machine Learning, pp. 1416 1425, 2016.
Benjamin Recht, Maryam Fazel, and Pablo Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3):71-501,, 2010.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 10 classifiers generalize to eifar-10? arXiv preprint 1rXiv:1806.0000 00451, 2018
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400.
PMLR, 2019.
Yair Schiff, Brian Quanz, Payel Das, and Yu Chen. Predicting deep neural network generaliza- tion with perturbation response curves. In Advances in Neural Information Processing Systems, 2021.
Yuan Shi and Fei Sha. Information learning of discriminative clusters for unsupervised domain adaptation. arXiv preprint arXiv:1206.6438, 2012.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv14099 1556, 2014.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stutionary environments: Introduction to covariate shift adaptation. MIT press, 2012.
Tao Sun, Cheng Lu, and Haibin Ling. Prior knowledge guided unsupervised domain adaptation. In European Conference on Computer Vision, 2022.
Hui Tang, Ke Chen, and Kui Jia. Unsupervised domain adaptation via structurally regularized deep clustering- In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8725-8733 2020.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini. Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 13:18583-18599 2020.
Junjiao Tian, Yen-Cheng Liu, Nathaniel Glaser, Yen-Chang Hsu, and Zsolt Kira. Posterior re- calibration for imbalanced datasets. Advances in Neural Information Processing Systems, 33: 8101-8113, 81, 2020.
Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and llya Tolstikhin. Predict- ing neural network accuracy from weights. In International Conference on Learning Representa- tions, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518. 2019.
13

---

Under review as a conference paper at ICLR 2023
Sinan Wang, Xinyang Chen, Yunbo Wang, Mingsheng Long, and Jianmin Wang. Progressive adver- sarial networks for fine-graing ane-grained domain adaptation. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9213-9222. 2020.
Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang. and Serge Belongie. Fine-grained image analysis with deep learning: A survey. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2021.
Ross Wightman. Pytorch image models.
pytorch- imagce modelsi 2019.
Shiqi Yang, Joost de Weijer, Luis Herranz, Shangling Jui, et Exploiting the intrinsic neighbor- hood structure for source-free domain adaptation. In Advances in Neural Information Processing Systems, pp. 19393-294055, 2021.
Shiqi Yang, Yaxing Wang, Kai Wang. Shangling Jui, and Joost van de Weijer. Attracting and dispers- ing: A simple approach for source-free domain adaptation. In Advances in Neural Information Processing Systems, 2022.
Yaodong Yu, Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting out-of- distribution error with the projection norm. In Advances in Neural Information Processing Sys- tems, 2022.
14

---

Under review as a conference paper at ICLR 2023
A NUCLEAR NORM
Let P E IR denote the prediction matrix of f on Di nuclear norm ||]|| is the sum of singular values of P. Nuclear norm the tightest convex envelop of rank function within the unit ball (Fazel, 2002). A larger nuclear norm implies more classes predicted and involved, indicating higher pre- diction dispersity. In addition, nuclear norm ||| and Frobenius norm |P||e =V VTaaee( PT P) can bound each other (Recht et al. 2010: Fazel 2002). More specifically. they have the following relationship: 1/v Pll. & || S PIl. S vdl Pl[r, where d = mnin(n, k). In our work, because P is consists of softmax vectors, its Frobenius norm is bound by || S /n4-
Frobenius norm PI f reflects prediction confidence Cui et al. 2020). Based on the above rela- tionship, larger nuclear norm ||| implies larger Frobenius norm || F. indicating higher prediction confidence. Therefore, nuclear norm Pl| can be use to characterize both confidence
and dispersity of P. Moreover, nuclear norm || related the shape of P, so normalized
it by its upper bound vd and obtain P| = ||||| /d. In our work, we use ||||| to measure the confidence and dispersity of prediction matrix.
B EXPERIMENTAL SETUP
B.1 MODELS
ImageNet. Models are provided by PyTorch Image Models (timm- 1.5) Wightman 2019) They are either trained or fine-tuned on the ImageNet- 1k training set Deng et al. (2099).
http://wwwwb..oooccec///o////o// cifar-ma mooels. CIFAR-C C--nd is generated with the 10 new corruptions of ImageNet-C (Mintun al. (02T) that are perceptually dissimilar to ImageNet-C. We apply random corruptions based on http://hwwh..tooocccco///rrrr//a
CIFAR-10. We train models using the implementations from
the codes from
CUB-200. We train CIFAR models using the implementations from https:/gittub..omm/ com//RIS- MaA-----rrreesiee--ttr--rrraalie mentations from https://giiabbboomm/eeedy///ssss
CUB-200-C is generated based on the imple-
B.2 DATASETS
The datasets
standard benchmarks, which publicly available. We have double-checked their license. We list their open-source as follows.
CIFAR-10 Krizhevsky et al. (2009) (https://www.cs www.:cs..toooooeed.. kriz/eifar.html);
CIFAR-10-4 COFAR-10-C Hendryckk & Dietterich (2019)
CIFAR-10.1 Recht al. 2018)
http:///ghhuubboooooooottttaaaae -1011)
CINIC ChhiCChhabaszez et al.
2017)
ImageNet-Validation
Deng et al.
al. (2019) (ttps:/wwwwimmgge-nctrrr)..
2009)
ImageNet-V2-A/B/C Recht et Recht
ImageNet-Corruppion Hendrycks Dietterich (2019)
ImageNet-Skktcc Wang et al. (2019)
ImageNet- Hendrycks et al. (202T)
ObjectNet Barbu et al. (2019) (httpss//objectnet.dee)
CUB-200-2011 Wah et al.
2011 httpp://wwww.visooo.cccccc.....e
CUB-
Paintings Wang al. (2020)
(tttps//githbb.co////////aa
B.3 COMPUTATION RESOURCES
We run all experiment on one 3090Ti with PyTorch (1.11.0tcul13) 13). CPU is AMD Ryzen 9 5900X 12-Core Processor.
15

---

Under review as a conference paper at ICLR 2023
B.4 EXPERIMENTAL DETAIL
(I) Effect of temperature. We empirically find that using a small temperature for softmax is helpful for all methods. Therefore, use temperature of 0.4 for all methods in the experiment. We show the effect of temperature in term of correlation strength (Rt² and p) in Fig. 8 We have two observations. First, using small temperature (e.g. 0.4) helps for all methods including nuclear norm, ATC and DoC. The correlation results are stable when temperature ranges from 0.2 to 0.45. Second, when using various temperature values, nuclear norm consistently achieve stronger correlation.
AC
ANE
DoC
ATC
Nucloar Norm
DoC
Nuclear
1.0
1.0
0.9
0.0
0.2
0.5
0.5 10
Temperature
Temperature
Figure 8: Effeet of temperature for all methods. We report the correlation results (both R and p) using various temperature of softmax. We show that a small temperature (0.2 to 0.45) helps for all methods. Moreover, when using different temperature values, nuclear norm consistently exhibits stronger correlation than other methods.
0=03
3
0-0 0.2
F1-Macro (%)
Ma0.8
[R-0.005
p-0.918
p.0.956
p.0.562
0.75
88
8.00
0.83
0:34
0.75
5.8
0.85
0.83
6:94
0.43
0.38
0:08
0.77
Nuclear Norm
Rectified Nuclear Norm
Average Thresholded Confidence
Figure 9: Effect rectified nuclear norm. Under imbalanced test sets, we relax the regularization of nuclear norm "tail" classes (rectified nuclear norm). We conduct correlation study imbal- anced ImageNet-CC C using VIT. We observe that rectified nuclear norm can improve the nuclear norm under imbalanced test sets.
(III) Comparison with ALine-D (Back et al. 2022) and ProjNorm (Yu et al., (2022). For a fair
Correlation ProjNorm ALine-D Nuclear Norm
comparison, follow the
setting as (Bekk
et al. 2022h and report the results using ResNet18
P 0.980
0.995
0.997
on CIFAR-10-4 1O-C. As shown in Table 3 ob-
RP
0.973
0.974
0.990
serve that nuclear norm gains stronger correlation strength than the two methods. It achieves 0.997 and 0.990 in rank correlation (p) and coefficients of determination (R²), respectively. Furthermore, we would like to mention that ALine-D (Bakk et al.) 2022 requires set of models for accuracy estima-
Table 3: Method comparison under CIFAR-10 setup. We report the average cor- relation strength (Spearman's rank correlation and coefficients of determination R²2).
tion. ProjNorm (Yu et al.) 2022 requires fine-tuning pre-trained network each OOD test set with pseudo-labels. In contrast, nuclear Norm more efficient: computed on a classifier's prediction matrix on each unlabeled test set.
(II) Rectified nuclear norm for imbalanced test sets. We tried to relax the regularization of nuclear norm under imbalanced test sets. Nuclear Norm encourages the predictions to be well-distributed across all classes. For imbalanced test sets, we can relax this regularization on the tail classes. That is, we mainly consider the prediction dispersity of head classes.
16

---

Under review as a conference paper at ICLR 2023
To achieve this, explored one intuitive way to rectify the nuclear norm: modify the nor- malization (i.e., upper bound) of the nuclear norm. Specifically, we revise the normalization from /min(n, k) * ny to v /ininnn, Ameaa) Mig where knea is the number of major classes regularised by nuclear norm. We conducted experiment under ImageNet setup (k=1000) and empirically set Ahead based on the imbalanced intensity I' (the ratio between the number of last 10 "tail" classes
and the number of top 10 "head" classes): khead k (1 T'm 80, To estimate imbalanced intensity, we use BBSE Lipton et al. 2018 to estimate the class distribution.
In Figure we show that our attempt (rectified nuclear norm) can improve nuclear norm. We would like to view the above experiment as a starting point that inspires more research on the rectification of nuclear norm for strong imbalanced test sets.
(IV) Additional observations. First, ObjectNet of ImageNet setup is built in a bias-controlled manner (with controls for rotation, background, and viewpoint). We observe that its images are of- ten confidently misclassified, which makes predictions with the high nuclear norm. We believe this why ObjectNet always off the linear line. Second, for all accuracy estimation methods, they can well capture the model performance is high (top-right region of each figure). However, when model accuracy is low (bottom-left), cxisting methods cannot make reasonable estimations, espe- cially under CIFAR-10 and CUB-200. In contrast, nuclear norm can well handle the low-accuracy region by additionally considering the prediction dispersity. To improve the accuracy estimation, it would be helpful to further consider the characteristics of predictions when the model performs
poorly. Third, in Figures 2 3
and 4 we observe that the real-world test sets (e.g., ImageNet-R,
M
CINIC, and CUB-P) scatter around the linear lines fit synthetic datasets. This indicates that both real-world and synthetic datasets follow a similar linear trend. This gives an interesting hint: we can use synthetic datasets to simulate and capture the distributions of real-world test sets.
B.5
MORE CORRELATION RESULTS
B.5.1 IMAGENET SETUP
Top-1 accuracy (86)
r=0.900
A3u0
Img.2
Img-V24B-
R 964
r= 282
in
R1m0 $70
n-o, 9--555
Ing V2 C
pud.
Imp-V7-A
Imp-V7-A
Img-
Impe
mmg R
Img-
52
Cljecthut
ObjectNot
ObjectNet
5
mg
img-5
25
20
20
0.6
0.95
Nuclear Norm
Average Negative Entropy
Average Confidence
Top-1 1op-1accer accuracy 196)
*0
p=0.955
R"w0 2811
Imp/
Ing V2
60
R'=0 075 p-0.990
Impa
mn
brightness
defocus blur
elastic_ (Fensform
ast
45
Imp-
Ing A
65
Img-V/44
Gaussiar noise
frest
blur
58
DijectNett
50
Chyectivt
Impulte naise
mmp-
35
iny-5
motionn buur
Saturate
noise
26
SROW?
0.55
0.7
5.75
0.8
0.35
0.5
0.6
0.7
0.8
Speckic noise 200m pilr
Difference of Confidence
Average Thresholded Confidence
Figure 10: Correlation study under the ImageNet setup. We plot the actual accuracy of ViT and five measures including nuclear norm and four competing methods. Different shapes in each sub-figure represents different test sets. The straight lines are calculated by linear regression fit on synthetic datasets of ImageNet-C. We list the 19 types of corruptions in ImageNet-C using different shapes and colors in the bottom right figure. We also mark the 6 real-world datasets in each sub- figure with arrows. Compared with other methods, nuclear norm exhibits stronger correlation with accuracy. Moreover, with nuclear norm, real-world test sets are closely around the linearly fit line.
17

---

Under review as a conference paper at ICLR 2023
A7-00 (wO, 996 pro
Img-22-822
Top-1 accuracy (95)
Ing
/wo 22.562 pro.
fw0.565
pmo.
Img-27A
ImpeV/-A
Imp-
Img
Img
ObjectNet
OojectNet
ObjectNer
Img-5
mg
28
Nuclear Norm
Average Negative Entropy
Average Confidence
Top-1 accuracy 1%6)
R-w0 961
Img/2-C
Img-V V2.0
R7w0.978
brfahtness
ast
blur
wlastis Frarsform
45
Ing V
Imp-V20
frist
img
mg
Geussian rolse
60
DbjectNet
50
impuie noise
[peg
m
35
pive ereett
buu
shor noise
20
Spatter
5.05
2:75
6.8
noise
Difference of Confidence
Average Thresholded Confidence
200m, aru
Figure 11: Correlation study under the ImageNet setup. We plot the actual accuracy of BeiT and five measures including nuclear norm and four competing methods.
Top-1 accuracy (%)
#T pw2.991
Img V2
Ing V2
R
Img-V2c
P 544 pwo 991
mmg V2 Img-4/-4
p=0.9/1
68
Img-V2-A
Img V2
Imge-V-A
55
Imp-1
42
Impé
ObjectNet
42
Oeject
ObjectNet
23
Img-
Img-5
0.55
0.65
0.75
0.91
0.00
0.65
0.45
0.9
0.45
Nuclear Norm
Average Negative Entropy
Average Confidence
Top-1 accuracy 196)
A3a0 r-o
3n.3
Ing V2.C
Img-
R²u0.516 916 z- 25/
Img
contrast
p=o:
clastic, trandform
ing
Ing-
Inge
gauss Tokse
frost
ObjectNe
ObjectNat
glans D0L7
33
pog compressioo
mmp3
imge
ginelate
shol noise
Spatter
Difference Confidence
Average Thresholded Confidence
200m, BTur
Figure 12: Correlation study under the Image Net setup. We plot the actual accuracy of Res/52- BiT and five measures including nuclear norm and four competing methods.
18

---

Under review as a conference paper at ICLR 2023
B.5.2 CIFAR-10 AND CUB-200 SETUPS
Top-1 accuracv (9)
01
#Fleg rwO, 596
/wo 896
CIFARTD
Row0 rwo 500
CFAR10.2
pw2.026
CINIC
peo
pw0.952
73
CIAA110.1
CIPAR10.1
CIFARID
43
CINIC
Nuclear
Average Negative Entropy
Average Confidence
Top-1 accuracy (96)
91
9.24 982
CHFRROO22
R7- D61
OFAR1O.2
contrast
blur
claztic transform
25
CFANIO.1
75
CIFAR10.1
frist
50
GMC
CINIC
SAuAsaa Talke
Impule fie, noise
43
modtion blur
compression
27
Genelate
shot noise
Spatler fer noise
0.49
0.50
191
0.92
6:34
5.59
6.7
0.76
0.92
Difference of Confidence
Average Thresholded Confidence
200mL DEr
Figure 13: Correlation study under the CIFAR-10 setup. We plot the actual accuracy of Rep VGG-A0 and five measures including nuclear norm and four competing methods. The straight lines are calculated by linear regression fit synthetic datasets of CIFAR-IO-C. We list the 19 types of corruptions in CIFAR- 10-C using different shapes and colors in the bottom right figure. We also mark the 3 real-world datasets in each sub-figure with arrows. Compared with other methods, nu- clear norm exhibits stronger correlation with accuracy.
Too-1 accuracv (96)
00
rwo. 595
p-2
Awo 545 pmo 0.977
7w0 344
pm0.977
62
200
60
Nuclear Norm
Average Negative Entropy
Average Confidence
Tap-1 accuracy (%)
p- 33.979
A7 0.903
R 9.997
brlahtness
elastic transform
Contraat
blur
60
frost
42
CUB-200-P
geuesian rolse
20
20
mmuut notse
buur
pine
shor noise
Spatter
0.80
0.5
022 0.35 05 0.65
notse
Difference of Confidence
Average Thresholded Confidence
200m, b(r
Figure 14: Correlation study under the CUB-200 setup. We plot the actual accuracy of PMG and five measures including nuclear norm and four competing methods. The straight lines are calculated by linear regression fit synthetic datasets of CUB-200-C using different shapes and colors in the bottom right figure. We also mark the real- world CUB-200-P in each sub-figure with arrows. Compared with other methods, nuclear norm exhibits stronger correlation with accuracy.
We list the 19 types of corruptions in
19


Reviews:
[{"review_id": "camFnK8u7gh", "reviewer": "Reviewer_JXCX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of the paper argue that confidence and dispersity are two important factors regarding the performance on the out-of-distribution datasets. Along this line, the authors propose to use the (normalized) nuclear norm of the prediction matrix as a metric for the generalization capabilities of the models. Various experiments on image datasets show there exists a strong correlation between the accuracy on the out-of-distribution data and the proposed metric.\n", "summary_of_the_review": "Overall my review for this paper tends to be negative due to the major concerns I stated in the previous comments. If the good empirical performance shown is mostly due to the artifacts of the relatively label-balanced datasets then I feel maybe it is better to introduce more new datasets that have highly unbalanced classes to test the robustness of the models instead of making use of the artifact for a little improved numbers.", "strength_and_weaknesses": "Strength: it is an interesting thought to consider the distribution of the predicted labels and use them as a metric to evaluate the generalization capability for the out-of-distribution data.\n\nWeakness: I do have some serious concerns about the underlying assumptions used in the paper. To me the dispersity assumption may be too strong for a lot of real applications. For those image data sets used in the paper (imgnet, cifar10, CUB-200), those assumptions seem perfect since the class labels on the shifted test data are distributed relatively evenly. However for real world applications, the labels may be distributed very differently. For example some classes are rarely observed, and some classes are observed very frequently. The power law and long tail effects are prevalent. In such a situation, the dispersity assumption used in the paper is no longer valid, and thus the proposed metric may not work. This makes me wonder if the good performance shown in the paper is only an artifact of the “label-balanced” data sets.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7dJn6z8fTE8", "reviewer": "Reviewer_eGpU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies accuracy estimation from unlabeled data, in the presence of distribution shifts. They show that \"dispersity\" that captures the marginal distribution of the predicted labels correlates well with the accuracy. The paper assumes that the true label distribution is uniform and proposes a measure of dispersity that measures how much predicted label distribution deviates from uniform. This measure builds off of previous works on nuclear norm. Intuitively, this won't work when there is label shift and the paper shows that nuclear norm based estimation does indeed fail under large label shifts but is able to handle moderate label shifts. ", "summary_of_the_review": "This paper presents an interesting and intuitive idea of using dispersity in model predictions for estimating accuracy on unlabeled data from a shifted distribution. The results are compelling and I think this would be an interesting contribution to the ICLR community and inspire follow-up work. The paper could do a better job with exploring more interesting takeaways and insights based on the strong correlation between dispersity and performance. The paper could also benefit from exploration of other metrics (not based on nuclear norm) that also try to incorporate the marginal of the predictions to determine OOD performance. ", "strength_and_weaknesses": "Strengths: \n(i) The paper studies a very important problem of unsupervised accuracy estimation under distribution shifts. \n(ii) The paper considers an interesting idea and is well motivated. The paper is overall well-written and easy to follow as well. \n(iii) The paper performs very systematic and thorough experiments on a large number of datasets. \n\nWeaknesses:\n(i) The paper can do a better job at synthesizing the results of the experiments, especially because they could reveal interesting insights about deep networks or distribution shifts, beyond just providing a method that improves accuracy estimation. For e.g. why is ObjectNet always off the line---is this because of the label shift?\n(ii) The paper is missing some comparisons to recent related work: Baek et al. 2022 (agreement-on-the-line) and projection norm (Yu et al. 2022). Baek et al. is a recent paper, but projection norm was older. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6OHkJ-k1EPd", "reviewer": "Reviewer_UZsk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new heuristic to estimate the accuracy of a given model for OOD test observations without labels. It uses the nuclear norm (i.e., the normalized sum of singular values) of the prediction matrix (i.e., a matrix of softmax outputs for the test data) of a given model. The measure is motivated by results from previous work, which show that the nuclear norm integrates the confidence and dispersity of the model predictions. While the confidence was shown to be predictive for the OOD performance by previous work, the paper demonstrates that the dispersity is also a predictive feature. Empirically, the paper shows a strong correlation between the computed nuclear norm and the OOD classification accuracy on multiple datasets with synthetic and realistic corruptions. The proposed heuristic shows a slightly but consistently stronger association with OOD accuracy compared to existing measures across multiple datasets, models, and types of shifts. However, the nuclear norm performs worse on test datasets with imbalanced classes and seems to overestimate the OOD accuracy on randomly synthesized datasets", "summary_of_the_review": "For the goal of unsupervised accuracy estimation, the paper proposes a simple heuristic that shows a strong association with OOD accuracy across many models, datasets, and types of shifts. It is a solid paper with a thorough empirical evaluation, but the utility for imbalanced datasets is not sufficiently clear. It could be stated more clearly that the idea is heavily based on existing work, and the connection to the more general setup of domain adaptation could be clearer.", "strength_and_weaknesses": "**Strengths**\n\nThe paper is quite clear and easy to read. The goal is stated clearly. The empirical results are in line with the claims from the abstract and introduction.\n\nIt's helpful that the paper provides an initial motivation by showing the relationship between dispersity and OOD accuracy.\n\nThe association between nuclear norm and OOD accuracy is very strong and consistent across datasets. The used visualizations show the trend very clearly.\n\nThe paper is transparent about the limitations of the proposed heuristic and points out directions for future work.\n\n\n**Weaknesses**\n\nThe presented setting for unsupervised accuracy estimation seems to be a special case of the unsupervised domain adaptation setup with only one training domain. It is not clear why the problem is not presented from the more general perspective of domain adaptation. To better assess generalization, the evaluation could consider more evaluation metrics, not only accuracy.\n\nThe results suggest that the nuclear norm performs best on datasets with balanced classes. However, on imbalanced test datasets, the predictive value of the nuclear norm starkly decreases, whereas at least one of the competing methods seems to be much more robust. In light of these results, it would be helpful to have more thorough ablations with respect to the performance on imbalanced datasets. For instance: What if both the training and test data were highly imbalanced? How well do the other heuristics (AC, ANE, DoC, MI) perform under imbalance and/or label shift? Can you provide an experiment that demonstrates that the nuclear norm can be adapted to deal with label shift?\n\nThe nuclear norm seems to consistently overestimate the OOD accuracy for randomly synthesized datasets (CIFAR-10-$\\overline{\\text{C}}$-Rand). Is there a particular reason for this? It would be helpful if you would also provide the performance metrics for each subplot in Figure 5 and Figure 7.\n\nFrom the abstract, it sounds like the nuclear norm and its connection to confidence and dispersity would be a core contribution of the paper. Only at the end of the introduction was it indicated, and in Section 3.3 explained, that the nuclear norm has been used for similar applications (domain adaptation) and that the connection to confidence and dispersity was already established by previous work. I think it would be better to state the contribution more clearly/conservatively in the abstract.\n\n\n**Questions and comments**\n\nIn Table 1, the performance improvements compared to existing methods seem relatively small in many cases. Is there a way to compute standard deviations or other uncertainty estimates to quantify the significance of these results?\n\nAt the beginning of Section 3.2, the prediction matrix is defined on the real numbers. Shouldn't it be defined on the interval (0, 1) instead?\n\nIn Section 3.2 it is suggested that classifiers that do not generalize well tend to give degenerate predictions by assigning test samples to some specific categories. Are there any insights into *which* categories typically correspond to degenerate predictions? Or is it quasi-random and depends more on the seed than on the actual categories?\n\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject