Effective Robustness against Natural Distribution Shifts for Models with Different Training Data
Authors: Zhouxing Shi
Nicholas Carlini
Ananth Balashankar
Ludwig Schmidt
Cho-Jui Hsieh
Alex Beutel
Yao Qin
Venue: NeurIPS 2023 poster (2023)
Keywords: Effective robustness
natural distribution shifts
out-of-distribution robustness

Abstract: ``Effective robustness'' measures the extra out-of-distribution (OOD) robustness beyond what can be predicted from the in-distribution (ID) performance. Existing effective robustness evaluations typically use a single test set such as ImageNet to evaluate the ID accuracy. This becomes problematic when evaluating models trained on different data distributions, e.g., comparing models trained on ImageNet vs. zero-shot language-image pre-trained models trained on LAION. In this paper, we propose a new evaluation metric to evaluate and compare the effective robustness of models trained on different data. To do this, we control for the accuracy on multiple ID test sets that cover the training distributions for all the evaluated models. Our new evaluation metric provides a better estimate of effective robustness when there are models with different training data. It may also explain the surprising effective robustness gains of zero-shot CLIP-like models exhibited in prior works that used ImageNet as the only ID test set, while the gains diminish under our new evaluation. Additional artifacts including interactive visualizations are provided at https://shizhouxing.github.io/effective-robustness.

(OOD) robustness Effective Robustness against Natural Distribution Shifts for Models with Different Training Data
Zhouxing Shi" UCLA zshi@cs. ucla. edu
Nicholas Carlini Google Research ncar inarlinion iearliniggoogle.... com
Ananth Balashankar
Google Research
anatthsshnnkrr@ggglee.oom
Ludwig Schmidt
Cho-Jui Hsieh
University of Washington achmidt@cs. washington edu
Google, UCLA chohsieh@cs ucla. edu
Alex Beutel' OpenAl alexb@openai.coom com
Yao Qin
UCSB, Google Research yaoqin@ucsb.ed edu
Abstract
"Effective robustness" measures the extra beyond what can be predicted from the in-distribution (ID) performance. Existing effective robustness evaluations typically use a single test set such ImageNet evaluate the accuracy. This becomes problematic when evaluating models trained on different data distributions, e.g., comparing models trained on ImageNet vs. zero-shot language-image pre-trained models trained LAION. In this paper, propose
evaluation metric to evaluate and compare the effective robustness models trained on different data. To do this, we control for the accuracy on multiple ID test sets that cover the training distributions for all the evaluated models. Our new evaluation metric provides a better estimate of effective robustness when there models with different training data. It may also explain the surprising effective robustness gains of zero-shot CLIP-like models exhibited in prior works that used ImageNet as the only ID test set, while the gains diminish under new evaluation. Additional artifacts including interactive visualizations are provided at httts://shizhouree srrriinnngggthtll github io/
Introduction
Robustness against distribution shifts is important for machine learning models to work reliably across various environments. For natural distribution shifts on image classification datasets, Taori et al. (2020) proposed the notion of effective robustness control for in-distribution (ID) accuracy when evaluating duitt--diittiiutionn (OOD) accuracy. Following a long line work that has found strong correlation between ID and OOD accuracy many test sets (Recht et al., 2019; Yadav & Bottou, 2019), effective robustness allows researchers to assess whether an apparently improved OOD accuracy is result of effectively improved robustness or is simply expected outcome of enhanced ID accuracy.
Unfortunately, the current definition of effective robustness has subtle limitation: it requires fixed ID test set, which typically ImageNet (Deng al., 2009) when using ImageNet-like OOD test sets
Work done while at Google.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in Taori et al. (2020) or CIFAR-10 (Krizhevsky et al., 2009) when using CIFAR-liks OOD test sets in Miller et al. (2021). It is acceptable when models are trained predominately on only one dataset. However, the emergence of many large-scale models trained on significantly different datasets makes necessary to evaluate and compare models trained on different data distributions, under which becomes unclear which ID test set should be used.
In particular, models from Contrastive Language-Image Pre-training, such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) have recently exhibited unprecedented effective robustness gains during zero-shot inference (Radford al., 2021; Fang et al., 2022; Nguyen al., 2022). However these previous works simply take ImageNet as the single ID test set, even though the models are not trained on ImageNet. We demonstrate that the results of evaluating effective robustness using single ID test set can vary drastically depending on the selection of the ID test set. Therefore, this imprecise treatment on the ID test set existing works could end up exaggerating the effective robustness of zero-shot CLIP models compared to models that are exactly trained on ImageNet.
In this paper, we propose more precisely evaluate and compare the effective robustness of models trained on different datasets. Instead of controlling for a single ID accuracy that may bias towards models from particular training distribution, we propose to use multiple ID test sets that cover the training distributions of all the models. In particular, previous works performed single-dimensional linear regression on set of baseline models to predict OOD accuracy from single ID accuracy (Taori et al., 2020). And they then evaluate the actual OOD accuracy of the models beyond the expected value that can be predicted from the fitting line, as the effective robustness. We expand on this definition by allowing for multiple ID test sets, and perform multi- mulli-dimensooal linear regression to fit a plane to predict OOD accuracy from the accuracy on multiple II) test sets.
In summary, we make the following contributions:
We reveal a limitation in the existing effective robustness evaluation when used to compare models trained on different data distributions.
We then propose a new effective robustness evaluation which uses multiple ID test sets to more precisely compare the effective robustness of models trained different data.
We show that the OOD accuracy of various models including zero-shot CLIP models can usually be better predicted from accuracies on multiple ID test sets compared to using only one ID test set.
Our results provide new understandings on the effective robustness gains of CLIP-like models observed in prior works only using ImageNet as the ID test set, while the gains diminish under our new evaluation.
Background of Effective Robustness
Under natural distribution shifts, the OOD accuracy of a model often correlated with the ID accuracy. After applying a logit transformation on the accuracy, a linear trend between the trans- formed ID accuracy and OOD accuracy holds across many datasets (e.g., a distribution shift from ImageNet (Deng et al., 2009) to ImageNetV2 (Recht et al., 2019), or from CIFAR-10 10 (Krizhevsky et al., 2009) to CIFAR-10.2 (Hendrycks & Dietterich, 2018)) and models with various architectures and training methods (Taori et al., 2020; Miller et al., 2021). This phenomenon implies that most models showing higher OOD accuracies naturally resulted from better ID performance.
To eliminate the confounding effect of ID accuracy on OOD performance, Taori et al. (2020) proposed effective robustness that measures the OOD performance beyond the expected OOD accuracy given the ID accuracy. where the expected OOD accuracy is predicted according to the fitted linear trend of baseline models. Since they only use single ID test set, we refer to this version of effective robustness as single-II effective robustness.
Suppose there are n baseline models f1, f2. . fn- A baseline function F(x) is constructed to predict the OOD accuracy of each baseline model, accood(fi) (1 < i I n), given the single ID accuracy of the model x x= acoaa(a) The baseline function is instantiated as:
B(x) = expit(w logit(x) + +b)
where w and b are parameters, log = In(r5z) is the logit transformation, and expit(x) the inverse of lo) Since log)t(((z)) logit(z) + b, the baseline function is essentially linear

---

Imagefie models
Imagellet model
ImageNeet-22 accuracy (css., %)
VFCC models
ImageNett-2 accuracy (CSS., %))
YFCC models
Imaçelett mode's
Linear imagehet models
YFCC
90
10
ImageNet accuracy (cMS., 96]
YFCC accuracy (css., %))
(a) ImageNet- accuracy against ImageNet accuracy.
(b) ImageNet- V2 accuracy against YFCC accuracy.
Figure 1: Class-subsampled ("ess." for short) ImageNet-V2 V2 accuracy against ImageNet accuracy and YFCC accuracy, respectively, for 36 ImageNet models and 13 YFCC models that are also used in Table 3a. A linear fit is generated for ImageNet models and YFCC-15M models, respectively. Accuracies and linear fits are under the logit scale. Class-subsampling is used to only include classes that appear in all the involved test sets (see Section 5.1).
function after applying logit transformation on the accuracies, and it can be determined by solving a linear regression. Then the single-ID effective robustness model f evaluated
p(f) = aCCond (f) - (faccia(f)),
which subtracts the predicted OOD accuracy based on the ID accuracy acca(ff), from the actual OOD accuracy af)
3 Limitation of the Single ID Test Set
The existing effective robustness evaluation in Section 2 fixes single ID test set for all the models, which is reflective of the ID performance only if all the models are trained on the same dataset that matches the ID test set. However, large-scale pre-trained models emerge, it becomes necessary to compare models trained on different datasets, in order to know if the latest pre- pre-training techniques can yield effective robustness gains. In this section, we use the comparison between zero-shot CLIP models and standard ImageNet models as an example to show the limitation using a single test set: when only one ID test set is used, using different ID test sets leads to contradictory conclusions.
Following Fang al. (2022), we compare models trained on ImageNet (Deng al., 2009) and YFCC-15M (Thomee et al., 2016), respectively. On ImageNet, we include standard classifiers, and we also train CLIP models using templates filled with an ImageNet class name as the caption in format of "A photo of (class name]", We also train CLIP models on YFCC-15M, a dataset with image-text text pairs. And we use ImageNet- V2 (Recht al., 2019) as the OOD test set. We consider two different ID test sets. One ID test set is simply ImageNet. The other ID test set is constructed from YFCC-15M since we have CLIP models trained on YFCC. We refer to this test test set",, and we refer to the accuracy on this test set as "YFCC accuracy". We discuss its details in Section 5.1 and Appendix B.2. Both ID test sets we consider here match the training distribution of some of the models (ImageNet models and YFCC models respectively) but not all the models.
"YFCC
We then plot the ImageNet-V. V2 accuracy of the models against their ImageNet accuracy and YFCC accuracy, respectively. There is strong linear trend between the scaled ID accuracy and OOD accuracy for ImageNet models and YFCC models, respectively, and we plot fitting lines for these two sets of models, respectively. When the ID test set is ImageNet, Fig. 1a shows that the fitting line for YFCC models is generally above the fitting line for ImageNet models (except for the regime with extremely low accuracies), which appears to suggest that YFCC models have effective robustmess gains over ImageNet models, as also suggested in Fang al. (2022). However, Fig. 1b which uses YFCC as the ID test set, the fitting line of ImageNet models are now mostly above YFCC models, which instead appears to suggest that ImageNet models have greater effective robustness than YFCC models. We observe that when there mismatch in the training data and the ID test data, the

---

models appear to have greater effective robustness (YFCC models in Figure la and ImageNet models in Figure 1b), as their performance on the ID test data and the OOD performance predicted from the single ID accuracy tend to be lower. This makes it difficult to compare models trained on different data and leads to imprecise conclusions on effective robustness if only one II) test set is used.
Multi-ID Effective Robustness
Considering the limitations of using single ID test set, we propose new way for effective robustness evaluation using multiple ID test sets that cover the training data distributions of all the involved models. We name it multi-ID effective robustness. Specifically, for each training distribution, we propose to prepare an ID test set that matches the training distribution, respectively. In particular, we focus on comparing models trained two different datasets time in this paper, and we thereby use two ID test sets, where each of them corresponds to one of the training datasets.
While we refer to them as ID test sets, each of them is only the exact ID test set for some of the considered models that are trained on the distribution matching the test set, and is not exactly an ID test set for all the considered models. However, we assume that the training distributions of all the models are still relatively close compared to the OOD test distributions (c.g., images normally collected from social medias in ImageNet (Deng et al., 2009), YFCC (Thomee et al.. 2016), and LAION (Schuhmann et al., 2021) are relatively close compared to the OOD images in ImageNet- Sketch (Wang al., 2019) that consists of sketch images specifically). In this way, both ID test sets are relatively ID for all the models compared to the OOD test sets, and it can be meaningful to control for the performance
these ID test sets when comparing the OOD performance.
We still use acCoed (-) to denote the OOD accuracy, and we use acc1 (-) and acc2(-) to denote the accuracy on the two ID test sets, respectively. In contrast to the previous baseline function 3(x) in Eq. (I), propose new baseline function ((x,, y) that predicts the OOD accuracy based on the accuracies x and on the two ID test sets, respectively.
All the models in Figure are trained on either Im- ageNet or YFCC. Thus, to compare their effective robustness under our new evaluation, we use two ID
Imagehiet Models
YFCC Models
test sets for ImageNet and YFCC at the same time, in contrast Figure la and lb which use one ID test set
(%% (550) 908 ZA-anpedoutt
thh mraem
separately at each time and results on the two differ-
20
ent ID test sets lead to contradictory conclusions. As shown in Figure 2, we plot the OOD accuracy against the two ID accuracies on both two ID test sets in 3D space. We observe that the data points approximately lie plane when plotted on the logit scale. This motivates us to instantiate B(x, y) as:
10
racc act (css, %)
4 4-g ib
Imageltet acc. (cop., %))
(xx v) = expit(w., (z) + wy logit(y) (y) + b).
(3)
where Wzz Wy: b are parameters. f(x, y), which is the plane in Figure 2, is also a linear function w.r.t. xr and y under the logit scale, and thus it a reasonable ex- tension from ((x)) by using multi-dimeesional lin- function the logit scale. We determine the pa- rameters by solving an ordinary least squares regres- sion to fit the accuracies. Metrics for linear regression such as the coefficient of determination, a.k.a. R² can be used to evaluate the fitting quality of the baseline function. A high R² value indicates that the OOD accuracy is accurately predicted by the baseline function from the ID accuracies, and thus the evaluated models have similar effective robustness. And our multi-ID effective robustness for model f is defined as
Class-subsampled ("ess." for short) ImageNet-V2 accuracy against both ImageNet accuracy and YFCC accuracy for ImageNet models and YFCC models used in Figure I which shows the projections when only one of ImageNet accuracy and YFCC accuracy is used.
Figure 2:
p( acC (f) Bacc (f), acC2
Compared to the existing definition for effective robustness in Eq. (2), the major difference is the inclusion of two ID accuracies acc; (f) and acc2 (f) in the baseline function, compared to using a single ID accuracy accia() f))

---

Generalizing to more than two training datasets. Although we focus on handling two training datasets a time, our method may be generalized to more than two datasets in principle, by defining a baseline function based on multiple ID) accuracies acc] (.).... acc,, (.). However, it could be costly as it would require training more models to fit a high-quality baseline function. We leave it for future works to reduce the cost when dealing with larger number of datasets.
5 Experiments
5.1 Settings
Models. In order to fit a baseline function, we need large amount of models with diverse accuracies. To this end, we follow Taori et al. (2020) to train models with various proportions of data by subsampling from the entire training set (namely dataset subsampling), which effectively produces models with diverse accuracies. Moreover, we also combine examples from two datasets with different sampling ratios and train models on these combined datasets. This produces models with training distributions varying between the two training datasets and it is supposed to yield different combinations of the two ID accuracies. We use models trained on each single dataset as well as the combined datasets in the same fitting process, so that the baseline functions do not bias towards models trained on certain data. Our experiments include the following models:
Standard classifiers on CIFAR-10 and ImageNet. We train standard classifiers on CIFAR-10 and ImageNet (Deng et al., 2009). We use ResNet-18, 18, ResNet-50, and ResNet-101 (He et al., 2016). Additionally, we train models by combining CIFAR 10 and ImageNet various ratios, where we upsample CIFAR-10 images from 32 x 32 to 224 x 224. Furthermore, we include ViT-S/16, ViT-B/16, ViT-L/16 models (Dosovitskiy et al., 2021) pre-trained on the whole ImageNet.
CLIP models. On YFCC-15M (Thomee et al., 2016) and LAION-15M (Schuhmann et al., 2021) which consist of image-text pairs, we train CL.P models using ResNet-50 and 101. We also train models by combining ImageNet and YFCC-15M and LAION-15M, respectively. We discard models with ImageNet accuracy below 5%. Additionally, in Section 5.4, we also have downloaded ViT-based models from Mu et al. (2022); Ilharco et al. (2021) and CLIP models fine-tuned on ImageNet, which are only used for evaluation but not fitting the baseline functions. We provide additional details in Appendix B.I.
We use "(Name_ of dataset) models" to denote models trained only on the dataset, e.g., "CIFAR- -00 models". And we use "[Name_ of dataset A | of dataset B) models" represent models models".
trained on a combination of two datasets, c.g., (CIFAR-1O+Imageen
ID test sets. We focus on image classification. Labeled image classification datasets such as ImageNet can be directly used for evaluating ID accuracy. For datasets that consist of image-text pairs for language-image pre-training without original labels, including YFCC and LAION, we automatically generate classification labels by matching captions with ImageNet classes, which has been similarly performed in Fang et al. (2022) for training classifiers using caption data, and we then hold out a balanced test set from the original dataset. More details are reported in Appendix B.2. Although it is possible to obtain higher- qughit- test set by human labelling, we will show that using the automatically labelled test sets can already produce reasonable results.
OOD test sets. To compare the effectiveness robustness of models trained on CIFAR-10 10 and ImageNet, we use 3 CIFAR-like OOD test sets with natural distribution shifts, including CIFAR- 10.1 (Recht et al., 2019), CIFAR- 10.2 (Lu et al., 2020), and CINIC- (Darlow et al., 2018). We use 4 ImageNet-like OOD test sets to compare models trained on ImageNet with models trained on YFCC and LAION: ImageNet-V2 (Recht et al., 2019), ImageNet-R (Hendrycks et al., 2021a), ImageNet-Sketch (Wang et al., 2019), and ObjectNet (Barbu et al., 2019). We do not use ImageNet- (Hendrycks et al., 2021b) which involves adversarial filtering and has a different behavior in effective robustness evaluation (Taori et al., 2020; Fang et al., 2022).
Class subsampling and mapping. Considering that different test sets may not have the same classes, we follow prior works (Taori et al., 2020; Fang et al., 2022) to use class subsampling to
We reuse the term "class subsampling" from prior works (Taori et al., 2020; Fang et al., 2022), although it is not random sampling.
5

---

CIFAR-10 Mades
Imagehett Models
Dmaggeteetttttttt Models
Integafted
Imtegatee
90
538 CI--AAI
80
(% Ccee ope w-soyuibeuu
70
20
50
25
45 1a 2o ImageNet acc. (mapped, %6)
CIFAR-10 ecc. (%)
59 1P
44 imageaa imageliet ecc. (ons, %)
+ + 9---6----- VPcC aco. (cse., %6
(a) Using CIFAR-10.2 as the OOD test set. The ImageNet accuracy is mapped to CIFAR-10 10 classes (see Section 1.
(b) Using ImageNet-R R as the OOD test set.
Figure Visualization of the multi-ID effective robustness. The colored plane stands for the baseline function. Figure and Figure 5 (in Appendix A.1) show various projected 2D) views. See our website (https: h/pp::////hzzhuxing github. io/effective serfffcciive-ouuuteeeseeee robustness) for an interactive visualization.
retain classes appearing in all the test sets. We also follow Miller al. (2021) map a subset of ImageNet classes to CIFAR-10 classes when comparing CIFAR-10 models and ImageNet models,.
5.2
Evaluation on CIFA R-like OOD Test Sets
Imageket
Imagetict
CIFAR-10.2 accuracy (%)
CIPAR-J0+Imageeee
CIFAR-10.2 accuracy (%)
CIFAR-10 accuracy (%)
ImageNett accuracy (mapped, %%)
(a) CIFAR-10.2 accuracy against CIFAR-IO accuracy. ImageNet models have higher CIFAR-10.2 accuracy compared to CIFAR-10 models when controlling for CIFAR-I IEAAR-Oaaa accuracy only.
(b) CIFAR-10.2 accuracy against ImageNet accuracy. ImageNet models have lower CIFAR-10.2 accuracy compared to CIFAR-10 models when controlling for ImageNet accuracy only.
Figure 4: Projected views of Figure 3a. Figure 4a and Figure 4b correspond to single-ID evaluations using different ID test and yield contradictory conclusions on the effective robustness. Our multi-ID evaluation provides a more holistic view where all these models are approximately on same plane and thus have similar effective robustness.
We first experiment with models trained using CIFAR-IP 10 and ImageNet on CIFAR-like OOD test sets. We show the fitting quality in Table la and the effective robustness of various models in Table lb. Compared to the single-ID evaluation, our multi-ID evaluation achieves better fitting quality and predicts the OOD accuracy from the accuracies more precisely (higher R² and lower MAE), and thus provides a more precise understanding on the effective robustness. Specifically, while both single-ID effective robustness and multi-ID effective robustness have relatively high fitting quality on CIFAR-like test sets, using multi-ID effective robustness further improves the fitting quality. In terms of the effective robustness, under the single-ID evaluation, ImageNet models achieve 3.91+2.200 (%) and 2.77+1. 1.25 (%) effective robustness on CIFAR-10.2 and CINIC- respectively. The positive

---

Table 1: Results on CIFAR-like OOD test sets. 148 models are included, including CIFAR-I0 10 models, ImageNet models, and 10+ImageNet models (CIFAR+IN for short). The multi-ID evaluation achieves better fitting quality where the effective robustness values of CIFAR-10 models and ImageNet models also become closer to O.
(a) Fitting quality evaluated by R² and mean absolute error (MAE).
R2 (t)
MAE (%, J)
Test set
Single-ID
Multi-ID
Single-ID
Multi-ID
CIFAR-10. CIFAR-10.
0.996
0.997
1.07
0.93
0.981
0.996
2.22
0.95
CINIC-10
0.978
0,990
2.41
1.49
(b) Effective robustness values (%). different training data, respectively.
We report the mean and standard deviation for three groups of models with
CIFAR-I0 21 models
ImageNet 89 models
CIFAR+IN 38 models
Test sct
Evaluation
Single-ID
-1.6810.92
1,05+1.27
0.02=1.10
CIFAR-10.1
Multi-ID
-1.43+0092
0.10:11.12
0.19=1101
Single-ID
-1.6510.700
3.91112.00
10.6411.779
CIFAR-10.2
Multi-ID
-0.76110.7
0.56:11.27
0.03:11.29
Single-ID
-0.96:11.43
2.77:1125
+0.10:2.81
CINIC-10
Multi-ID
10.08+1552
-0.52++0.98
0.63+22.10
Single-ID
-1.43110.53
2.5811133
10.24111.58
Average
Multi-ID
-0.76.10.63
0.04
0.28
effective robustness values seems to suggest an advantage of ImageNet models compared to CIFAR- 10 models, which is consistent with the findings in Miller et al. (2021). However, under the multi-ID evaluation, the advantage of ImageNet models diminishes, and the effective robustness values of both CIFAR-10 10 models and ImageNet models are much closer to O. Therefore, the apparent advantage reported by prior works be explained the effect of training data on the single-ID evaluation, and our multi-ID evaluation resolves this confounder to provide a more precise understanding.
In Figure 3a, visualize the multi-ID effective robustness CIFAR-10.2 where the accuracies of all the models approximately lie on plane (the baseline function) on the logit scale, and thus these models have similar effective robustness the OOD accuracy of all the models be approximately predicted using a simple plane. We also show projected views of Figure 3a in Figure 4, where Figure 4a and Figure 4b correspond to the single-ID evaluation taking different ID test sets with contradictory conclusions. In contrast, our new evaluation provides a more holistic view.
5.3 Evaluation on ImageNet-like OOD Test Sets
We then conduct evaluation on ImageNet-like OOD test sets, and we compare ImageNet models with models trained on YFCC and LAION, respectively. We show the fitting quality in Table 2 and the effective robustness in Tables 3a and 3b. Consistent with results in Section 5.2, our multi-ID evaluation improves the fitting quality over the single-ID evaluation to better predict and understand the OOD accuracies from ID accuracies. On effective robustness, single-ID evaluation leads to perception of an effective robustness gain when there is mismatch between the training data and the single ID test set. Our multi-ID evaluation enjoys holistic view of all the training distributions and suggests that all the models evaluated here have similar effective robustness.
Specifically, the improvement of fitting quality is particularly significant for models involving LAION on ImageNet-R (Ai² improved from 0.216 to 0.982 and MAE reduced from 9.23% to 1.32%) and ImageNet-Sketchl (R² improved from 0.281 to 0.937 and MAE reduced from 7.90% to 2. 10%). On the effective robustness values, under the single-ID evaluation, YFCC and LAION models have positive effective robustness values (2.59+2.43 (%) on average for YFCC models and 5.96 (%) on average for LAION models), which is consistent with the findings Fang et al. (2022); Nguyen et al. (2022). In contrast, under our multi-ID evaluation, the average effective robustness becomes 0.77=0.85 (%) for YFCC models, and -0.00+00552 (%) for LAION models, much closer to While single-ID evaluation used by prior works (Fang al., 2022; Nguyen al., 2022) suggests effective
7

---

Table 2: Fitting quality of single-ID ID and multi-ID effective robustness. respectively, on ImageNet- like OOD test sets. For ImageNet v.s. YFCC, models involved include ImageNet, YFCC, and ImageNer+ YFCC models. For ImageNet v.s. LAION, models involved include ImageNet, LAION, and ImageNet=L.AION models. Multi-ID evaluation improves the fitting quality.
R (f)
MAE (%. L)
ImageNet v.s.
Test set
Single-ID
Multi-ID
Single-ID
Multi-ID
ImageNet-V2
0.990
0,999
1.44
0.54
YFCC (103 models)
ImageNet-R
0.879
0.965
2.55
1.30
Image Sketch
0.928
1.56
ObjectNet
0.903
0,936
2.60
1.98
ImtgeNNet V2
0.992
0.999
1.33
0.51
LAION (107 models)
ImageNet-R
0.216
0.982
9.23
1.32
ImageNet- Setcehh
0.281
0.937
7.90
2.10
ObjectNet
0.849
0.906
2.88
2.38
Table 3: Single-ID and multi-ID effective robustness (%) of the models on variants of ImageNet OOD test sets. The effective robustness of all the models becomes close to under the multi-ID evaluation.
(a) Models involving ImageNet and YFCC.
(b) Models involving ImageNet and LAION.
Models
Test sel
Single-ID
Multi-D
Models
Test set
Single-ID
Multi-ID
ImageNet-V2
-1.23=00.6
-0.19:+0.50
ImageNet-V22
-1.21=00.56
0.05:20.5
ImageNet
ImageNet-R
-2.80+1.34
-044++183
InsageNot (37 models)
ImageNNt-R
-9.5-22.99
10.441190
(36 models)
SkacchhttSketch
ObjectNet
-0.99-4423
-1.25=1.90
0.74=4.14
0,14=2,57
ImageNet- Sketch
ObjectNet
-7.3343440
-1,90:24.48
-0.72-3.03
1,1444.18
Average
11.721.200
0.07=1.68
Average
-3055221
-0.02-153
ImageNet-V2 V2
1,69+1844
-016:5037
InagoNet-V2 V2
1.2211.73
-0:33=0.57
YFCC (13 models)
ImageNet-Sketch
ImageNet-R
3.4443.25
1.90+2.25
0.89=1.299
LAION (14 models)
ImageNet.Sketch
ImageNet-R
8.71+7.15
9.4818.84
10.6511005
-1.10=1.98
ObjectNet
3.32+22555
1.27-0.85
ObjectNet
4.24
1.77+1120
Avcrage
2.391243
0.77:0.85
Average
5.9624.96
-0.00-0.52
ImageNet-V2 V2
0.70:=1.75
0.13=00799
InageNet-VV
0.65=1.43
0.07:0M
ImageNet+ (54 models)
ImageNet-Sketel
ImageNet.R
0.69+11.9
1.31442.75
0,21=1.52
0.16-1.76
ImageNet+LAAOX (56 models)
ImageNett
ImageNet-R
6.0128.43
5.99+-7.39
0.56:11
1.04+2.46
ObjeatNet
0.21++1.8
-0.60+ 1.01
ObjjccNNe
0.63-2.200
-1.01=1444
Avcrage
0.732196
-0.03:EL06
Average
3.3224.63
0.16:10.65
robustness gains of YFCC models compared to ImageNet models (Figure 5a), all the models have similar effective robustness under our multi-ID evaluation. Additionally, we provide an ablation study on using models with mixed training data in Appendix A.3 and additional interactive visualization on our website at https: / //sizhhooiiiigg github. io/ eeoeeffetiieeee -eefeeciii-roousteees
5.4 Evaluation on Additional Models
We also evaluate additional models that are not used in fitting the baseline functions. We download models pre-trained by existing works, including OpenCLIP (Ilharco et al., 2021) and SLIP (Mu al., 2022). OpenCLIP provides CLIP models trained on YFCC and LAION, and SLIP provides YFCC models trained using CLIP and also a combination of self-supervised learning (Chen et al., 2020a,b) and CLIP (SimCLR+CLIP namely SLIP). And we also fine-tune CLIP models on ImageNet for models we pre-train on YFCC and LAION. We use both vanilla fine-tuning and also WiSE- FT (Wortsman et al., 2022b) which aims improve the robustness after tnn-ttnnng, using a weight- space ensembling of the pre-trained model and the fine-tuned model. Details are in Appendix B.I.
In Table 4, we show results involving YFCC and LAION, respectively. Since these models used in the fitting, we do not use R², but we use MAE to measure the fitting quality. Our multi-II) evaluation generally reduces MAE compared to the single-ID evaluation, and thus the multi-ID evaluation can still more accurately predict the OOD accuracy from the ID accuracies for these models that are not used in the fitting. The effective robustness values of the models also generally become closer to O, especially for the zero-shot CLIP models. The results further validate that zero-shot CLIP models, although may achieve high OOD performance if pre-trained with large-seat scale data (Radford et al., 2021), generally do not improve effective robustness if we control for all the ID accuracies. Among the models evaluated here, SLIP models on YFCC and WiSE-FT models from LAION achieve relatively higher average effective robustness compared to other models, under our multi- -ID evaluation, although the gains are not consistently significant on all the test sets and become much smaller than those reflected in the single-ID evaluation. However, we are not certain
8

---

Table Fitting quality and effective robustness for downloaded and fine-tuned models. The models are not used in the fitting and directly evaluated. Note that MAE and effective robustness are different, where MAE takes absolute values but not effective robustness. For CLIP by Mu et (2022) and
SLIP, only models pre-trained on YFCC are available.
Models with pre training on YFCC
Models with pre preiiiinnng on LAION
Model
MAE (S. L)
Effective Robustness (Sr
MAE(S; J)
Effective Robustness (S)
Single-ID Multi-ID
Single-ID
Multi-ID
Single-ID Mukii-D
Single-ID
Multi-ID
ImaggeNetN-A -2
3.95
0.45
3.95
-0.3:10.44
4.70
0.38
4.70:10.01
0.38100.11
ImageNetN-R R
8.98
3.12
898+1.24
3.12+1127
39.80
7.49
39,80+0.03
7.49+0.09
OpenCLIP
ImageNetN- Sketch
5.32
2.49
5,32+1.07
2.49-0811
38.92
1.31
38.92=0.000
-1.31=0023
ObjectNet
5.68
1.04
5.68=0081
-0.22:+:104
6.94
1.35
6.94+0.09
-1.35=00.44
Average
5.98
1.77
5.98
1.26:00.9
22.59
2,63
22.3910.02
1.3010.007
ImageNetN-V2
1.14
0.85
0.1621.23
0.77=0.70
0.82
0.91
-0.12:0.89
0.31
ImageNetN-R
2.90
1.82
-2 1.96
-1.82:10.92
4.63
2.38
-4.4542.89
2.27:12.20
Vanilla FT
ImageNetN-
2.26
3,16
1,20=1186
3.16+1.19
4.78
7.50
3,47+4.67
7.50++3.29
ObjectNet
3.46
3.11
-3.4222.36
-3.11:11.43
4.07
2.27
-4.07=2.10
12.26=1195
Average
2.44
2,23
0.9911176
+0,25100.6
3.57
3:27
11229:2.19
2.011138
ImageNetN-V2
197
1.05
19970076
1005:0.59
1.72
0.81
L.72:0.45
0.81
ImageNetN-R
3.64
1.76
3.6410.61
1.76:0.60
13.08
7.40
13.0811885
7.40
FT
ImageNetNN Sketch
5.47
4.41
5.47+1.220
4.419
16.84
10.21
16.84+1..88
10.21
ObjectNet
2.12
1.30
2.12+11.8
-0.68:+..34
1.65
1.65
1.5241.64
0.31+1677
Average
3.30
2.13
0.99
1,63 0.80
8.32
5,02
8.29 1.04
4.68 10.55
Image N2
4.95
0.83
0.95
-0.3::0.455
CLIP by
ImageNetN R
6.54
1.86
6.541:2.66
-1.86.11.15
Mu al. (2022)
ImageNetN Sketch
1.49
4.16
0,58+1168
4.16-40.44
ObjectNet
9.32
1.49
9,32+1,45
1.49=0044
Average
5.57
2.09
1,35=1,49
-1.34::0.28
ImageNetNNV V2
5.25
0.64
0.25
-0.1210..44
ImageNatN. R
14.47
6.13
14.47455988
5.75+44.77
SLIP
ImageNetN-Skeee Sketch
7.71
2,90
7.71
2.119
ObjectNet
14.79
5.02
14.79-2.77
Average
10.56
3.67
10.56=3.11
3.19:2.05
on whether SLIP and WiSE-FT can alter the underlying training distribution, because SLIP uses SimCLR that introduces different training images, and WiSE-FT esseentially edits the weights of the models by weight-space ensembling. Thus, we do not draw a definite conclusion for the effective robustness of SLIP and WiSE-FT models and leave further validation for future work.
Related Work
For natural distribution shifts, the linear correlations between the ID and OOD performance ("accuracy-on-the-ling (Miller al., 2021) in the single-ID effective robustness) have earlier been observed in dataset reproduction works (Recht et al., 2018, 2019; Yadav & Bottou, 2019; Miller al., 2020). Taori al. (2020) evaluated many ImageNet models on several ImageNet-like OOD test sets, and given the widely held linear correlations, they proposed to evaluate effective robustness by controlling for ID accuracy. Miller et al. (2021) further validated accuracy-on-the -ot--oe-onttnenene tine with a broader scope. Nevertheless, such as some corruption shifts (Hendrycks & Dietterich, 2018) and shifts in the wild (Miller et al., 2021), and sometimes ID accuracy and OOD accuracy can inversely correlate (Miller et al., 2021; Teney et al., 2022). Baek et al. (2022) also observed linear correlation between ID agreement and OOD agreement for pair of neural networks (namely gegreement--o-thh-llee accuracy-on-ther
may not hold on some distribution shifts,
for testing whether holds by agreement-on-the-linn which does not require labeled data. We focus on distribution shifts where at least secuurcy-on-the-ling holds for models from each of the training datasets, and we further propose "accuracy-on too-thh-ppp thee-ppane" using multiple ID test sets.
Recently, CLIP-like models with language image pre-training which has been studied earlier in works such as Sariyildiz et al. (2020); Zhang et al. (2022); Desai & Johnson (2021), were shown to achieve exceptional effective robustness in the single-ID evaluation (Radford et al., 2021; Jia et al., 2021) 1). Fang et al. (2022) analyzed the cause of the effective robustness gain of CLIP and concluded that the pre-training data determined the robustness. Nguyen et al. (2022) experimented on more pre- pre-training data, and they observed difference in the single-ID effective robustness of models trained on different data. While Fang et al. (2022); Nguyen et al. (2022) both suggested that the pre-training data could determine the effective robustness gains, our evaluation suggests that zero-shot CLIP models do not have effective robustness gains. Besides, Kumar et al. (2021); Andreassen et al. (2022); Wortsman

---

et al. (2022b) studied the robustness of theettuned CLIP models. Moreover, Devillers et al. (2021); Santurkar et al. (2022) studied the transfer performance of CLIP models, which out of our scope on the robustness against natural distribution shifts.
7 Conclusion
To conclude, we propose a new and more precise effective robustness evaluation for models with different training data. In our evaluation, the OOD accuracy can generally be better predicted from multiple ID accuracies compared to previous effective robustness evaluation with a single ID test. We find that zero-shot CLIP models pre-trained on language- -image data do not have better effeetive robustness compared to standard image classifiers, and we provide a new understanding of the apparently significant effective robustness gains observed by prior works.
8 Limitations and Future Work
There remain several limitations that may be addressed in future works:
Our method requires fully knowing the training distributions of all the evaluated models, which is not directly applicable for large-scale pre-trained models with private training data. And this also requires the training methods not significantly alter the training distribution beyond basic data augmentation, while some methods such as SLIP may alter training distributions more significantly, and it is unclear how we can precisely define the training distribution for model with post-training processing, such as WiSE-FT (Wortsman et al., 2022b) and Model Soups (Wortsman et al., 2022a) with weight-space ensembling. Future work may study how these techniques may impact the ID performance evaluation (Section 5.4).
We assume that the two ID test sets have relatively close distributions compare to the OOD test sets. We have not considered how the difference between the multiple ID test sets may affect the evaluation, and how the effective robustness should be compared models trained highly different distributions.
We use fixed OOD test to evaluate the OOD performance, following previous works (Radford et al., 2021; Fang et al., 2022: Nguyen et al., 2022; Schuhmann et al., 2021). When models are pre-trained on large-scale data, it becomes unclear if these "OOD" test sets are still OOD, or these test sets could be less OOD for the pre-trained models compared to standard classifiers. There may also be some distribution overlap between these test sets and the pre- pre-training datasets, even though Radford et al. (2021) mentioned that the likelihood of direct data overlapping is low.
We focus on distribution shifts where least "accuracy-on-the -ote---iiteeee from existing works is known to hold for models trained on the same data (Taori et al.. 2020; Miller et al., 2021), yet there are counterexamples where "accuracy-on-ther -o-tt---i--- does not hold (Section 6) and requires further study, We may set threshold on the fitting quality and only adopt our method when the fitting quality is sufficiently good. And there are also other OOD test sets (Singla & Feizi, 2021; Rusak et al., 2021: Singla et al., 2022; Idrissi et al., 2022; Li et al., 2023; Moayeri et al., 2022; Vasudevan et al., 2022) that have not been studied in the effective robustness works yet.
While we mostly focus on comparing CLiP-like models with standard image classifiers, due to the notable OOD robustness CLIP-like models studied in prior works (Radford et al., 2021: Fang et al., 2022; Nguyen et al., 2022), this work may further be extended to cover other types of models (Goyal et al., 2022; Singh et al., 2022) as well as other modalities such as distribution shifts on language data (Miller et al., 2020; Awadalla et al., 2022).
Our multi-ID evaluation is intended for the scenario with models trained on different data. For models trained on a single dataset, while there is often correlation between the rankings derived from the single-ID evaluation and the multi-ID evaluation, respectively, the rankings are not necessarily consistent (see Appendix A.2), and thus our multi-ID evaluation is not intended to replace the single-ID evaluation this case. We suggest using single-ID and multi-ID evaluation comprchensively.
10

---

Acknowledgments & Funding Disclosure
We thank Alex Fang and Jindong Gu for helpful discussions and the reviewers for constructive feedback. This work was supported in part by NSF 2008173, 2048280, 2325121, 2331966, ONR N00014-2-11-330::00000
References
Andreassen, A. J., Bahri, Y., Neyshabur, B., and Roelofs, The evolution of out-of-distribution robustness throughout fine-tuning. Transactions on Machine Learning Research, 2022.
Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., and Schmidt, L. Exploring the landscape of distributional robustness for question answering models. In Findings of the Association for Computational Linguistics; EMNLP 2022, pp. 5971-5987, 2022.
Back, C., Jiang, Y., Raghunathan, A., and Kolter, Z. Agreement-on-tee thenneininne Predicting the perfor- mance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 11119927299999 2022.
Barbu, A., Mayo, D., Alverio, J., Luo. W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the limits object recognition models. Advances in neural information processing systems, 32, 2019.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33: 22243-22255, 2020b.
Darlow, L. N., Crowley, E. J., Antoniou, A., and Storkey, A. J. Cinic-10 is not imagenet or cifar- 10. arXiv preprint arXiv:1810.03500 2018.
Deng. J., Dong, W., Socher, R., Li, Li, K., and Li, F. Imagenet: A large-scale hierarchical image database. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 2009.
Desai, K. and Johnson, J. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11162- 11173, 2021.
Devillers, B., Choksi, B., Bielawski, R., and VanRullen, R. Does language help generalization in vision models? In Proceedings of the 25th Conference on Computational Natural Language Learning, pp. 171-182, 2021.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S.. Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
Fang. A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave. A., and Schmidt. Data deter- mines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pp. 6216-6234, PMLR, 2022.
Goyal, P., Duval, Q.. Seessel, L,, Caron, M., Singh, M., Misra, I., Sagun, L., Joulin, A., and Bojanowski, P. Vision models more robust and fair when pretrained uncurated images without supervision. arXiv preprint (tt::2202.08366. 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. dois 10.1109/CVPR.2016.999

---

Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19262-15277 2021b.
Idrissi, B. Y., Bouchacourt, D., Balestriero, R., Evtimov, I,, Hazirbas, C., Ballas, N., Vincent, P., Drozdzal, M., Lopez-Paz, D., and Ibrahim, M. Imagenet-x: Understanding model mistakes with factor of variation annotations. In The Eleventh International Conference Learning Representations, 2022.
Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidr, L. Openclip, 2021. URL https://doi. htwp//ooooorgg//// org/10. 10. 5281/zenoda 5143773.
Jia, C., Yang, Y., Xia, Y. Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y-HH,, Li, Z., and Duerig, T. Scaling visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Kendall, M. Rank correlation methods. 1948.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical Report TR-2009, 2009.
Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, Fine-tuning can distort pre- trained features and underperform aut-of-distribution.. In International Conference on Learning Representations, 2021.
Li, Z., Evtimov, I., Gordo, A., Hazirbas, C., Hassner, T., Ferrer, C. C.,, Xu, C., and Ibrahim, M. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20071-20083 2023.
Lu, S., Nott, B., Olson, A., Todeschini, A., Vahabi, H., Carmon, Y., and Schmidt, L. Harder or different? closer look distribution shift in dataset reproduction. In ICML Workshop on Uncertainty and Robusiness in Deep Learning, 2020.
Miller, J., Krauth, K., Recht, B., and Schmidt, L. The effect natural distribution shift on question answering models. In International Conference on Machine Learning, pp. 6905-6916 16. PMLR, 2020.
Miller, J. P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, I. Accuracy the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning, pp. 7721-7735.
PMLR, 2021.
Moayeri, M.. Singla, S., and Feizi, S. Hard imagenet: Segmentations for objects with strong spurious cues. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Mu, N., Kirillov, A., Wagner, D., and Xie, S. Slip: Self-supervision meets language-image pre- training. In European Conference on Computer Vision, pp. 529-544. Springer, 2022.
Nguyen, T.. Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems, 14:21455-2146 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry. G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763, PMLR, 2021.
12

---

Recht, B., Roclofs, R., Schmidt, L., and Shankar, V. Do cifar-10 10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.004: 00451, 2018.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet?
In International Conference on Machine Learning, pp. 5389-5400, PMLR, 2019.
Rusak, E., Schneider, S., Pachitariu, G., Eck, L.., Gehler, P. V., Bringmann, O., Brendel, W., and Bethge, M. your data distribution shifts, use self-learning. Transactions on Machine Learning Research, 2021.
Santurkar, S., Dubois, Y., Taori, R., Liang, P., and Hashimoto, T. Is a caption worth a thousand images? a controlled study for representation learning, arXiv preprint arXiv:2207.07633 2022.
Sariyildiz, M. B., Perez, J., and Larlus, D. Learning visual representations with caption annotations. In European Conference on Computer Vision, pp. 153-170. Springer, 2020.
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarezyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Singh, M., Gustafson, L., Adcock, A., de Freitas Reis, V., Gedik, B., Kosaraju, R. P., Mahajan, D., Girshick, R., Dollár, P., and Van Der Maaten, L. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 804-814, 2022.
Singla, S. and Feizi, S. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations, 2021.
Singla, S., Moayeri, M., and Feizi, S. Core risk minimization using salient imagenet. arXiv preprint arXiv:2203. 15566, 2022.
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18599 2020.
Teney, D., Oh, S. J., and Abbasnejad, E. Id and ood performance are sometimes inversely correlated real-world datasets. arXiv preprint XXiv:2209.006/33 2022.
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-77 73, 2016.
Vasudevan, V., Caine, B., Gontijo Lopes, R., Fridovich-Keil S., and Roelofs, R. When does dough become a bagel? analyzing the remaining mistakes on imagenet. Advances in Neural Information Processing Systems, 35:6720 6734, 2022.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Wortsman, M., Ilharco, G., Gadre, Y., Roelofs, R., Gontijo-Lopes, R., Morcos. A. S., Namkoong. H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine- tuned models improves accuracy without increasing inference time, In International Conference on Machine Learning, pp. 133665233999 PMLR, 2022a.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Korbblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7959-7971, 2022b.
Yadav, C. and Bottou, L. Cold case: The lost mnist digits. Advances in neural information processing systems, 32, 2019.
Zhang. Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pp. 2-25. PMLR, 2022.
13

---

A Additional Results
A.1 Projected Views
In Figure 5, we show projected views of Figure 3b.
Imagehiet
Imageket YFCC
Imageteet- acouraiy (cSs. %)
YFCC EmogchettYYCC
90
Imagelet--R accuracy (cos..
Imagchet +VFCC NVCCC
a
03
ImageNet accuracy (css., %6)
YFCC accuracy (cSs., %%)
(a) ImageNct-R accuracy against ImageNet accu- racy. YFCC models have higher ImageNct-R accu- racy compared to ImageNet models when control- ling for ImageNet accuracy only.
(b) ImageNet-R R accuracy against YFCC accuracy. YFCC models have similar ImageNet- R accuracy compared to ImageNet models when controlling for YFCC accuracy only.
Figure 5: Projected views of Figure 3b. Figure 5a and Figure 5b correspond to single-ID evaluation using diffrent ID test sets, Figure 5a suggests effective robustness gains of YFCC models but the gains diminish in Figure 5b. Our multi-ID evaluation shows holistic view where all the models have similar effective robustness.
A.2 Agreement between Single-ID and Multi-ID Evaluation
We conduct an experiment to check the correlation between the single-ID evaluation and our new multi-ID evaluation, in terms of the relative ranking between different models trained on the same data. We use Kendall's rank correlation test (Kendall, 1948) and report the statistic computed by scipy. stats. kendall taa in Tables effective robustness and multi-ID effective robustness are positively correlated for CIFAR- 10 and ImageNet models. There is also a weaker positive correlation for YFCC models. For LAION models, there is sometimes negative correlation. Overall, while there is often a positive correlation between the rankings provided by the single-ID evaluation and the multi-I evaluation, respectively, necessarily consistent all the datasets. Thus, when comparing models trained the same data, our multi-ID evaluation not intended to replace the single-ID evaluation. Our multi-ID evaluation is mainly for comparing models trained different data, and may be used as supplementary evaluation all the models are trained on a single dataset.
7. Results show that the rankings on the single-ID
not
Table 5: statistics in the Kendall's rank correlation test for evaluating the correlation between the rankings provided by the single-ID evaluation and the multi-ID evaluation, respectively, for models trained on the same data. We consider CIFAR-10 models and ImageNet models, respecitively, CIFAR-like OOD test sets. For ImageNet models, ImageNet instead of CIFAR-10 10 is used as the ID test set in the single-ID ID evaluation.
Test
CIFAR-1O.1
CIFAR-10.2
CINIC-10
CIFAR-I0 models
0.9619
0.6667
0.8095
ImageNet models
0.1664
0.1522
0.4907
A.3 Models with Mixed Training Data in the Fitting
As mentioned in Section 5.1, we train models with mixed data to obtain models with diverse accuracies. Tables 8 and 9, we show that if do not include models with mixed training data in the fitting, the MAE for these models can become higher, although the difference is not large.
14

---

Table statistics (similar Table 5) for ImageNet models and YFCC models on ImageNet- like OOD test sets. For YFCC models, YFCC instead of ImageNet is used as the ID test set in the single-ID evaluation.
Test set
ImageNet-V2
ImageNet-R
ImageNet-Skati Sketch
ObjectNet
ImageNee models YFCC models
0,5142
0.4412 0.8461
0.8031 0.7179
0.8063
0.0256
0.2564
Table statistics (similar to Table 5) for ImageNet models and LAION models on ImageNet-lig like OOD test sets. For LAION models, LAION instead of ImageNet is used as the ID test set in the single-ID evaluation.
Test set
ImageNet-V2 V2
ImageNet-R
ImageNet- Sketchh
ObjectNet
ImageNet models LAION models
0.3123 -0.4945
0.4384 0.6483
0.6216 -0.2747
0.9729
0.6483
In this work, we train models with data mixed various ratios and consider models with diverse combinations of accuracies, to obtain more convincing conclusions the effective robustness.
Table MAE (%) for ImageNet+ YFCC models when they are excluded and included in the fitting. respectively, where comparing the effective robustness of ImageNet models and YFCC models.
MmageNet+YFCC models in the fitting Included
Test set
Excluded
ImageNet-V2
0.71
0.64
ImageNet-R
1.30
1.21
ImageNet-SSe Sketch
0.98
0.94
ObjectNet
1.73
0.95
B Experimental Details
B.1 Details of Models
We Vision under the Apache License Version 2.0 to train standard classifiers CIFAR-10 and IeageNet. We follow the configurations provided in TF-Vision for vanilla ResNet training on ImageNet and train ResNet-18, ResNet-50 and ResNet-101 models. We reuse the configurations to train models on CIFAR-10, where we only change the dataset, number of classes, and image size, without tuning typerparameters for the training. And we load checkpoints of ViT-S/16, ViT-B/16, and ViT-L/16 models pre-trained on ImageNet, provided by TF-Vision.
For training CLIP models, mostly follow hyperparameters provided in Fang et al. (2022) and the implementation in Open-CLI (Ilharco et al., 2021). While Fang et al. (2022) used a batch size of 1024, 2048 for more parallelism. We use YFCC- 15M in Radford al. (2021), which is subset of YFCC-100M (Thomee et al., 2016). And we use LAION-15M which we uniformly sample from LAION-400M (Schuhmann al., 2021). For fine-tuning CLIP models, fine-tune for 50,000 steps, using learning rates 3 x 10-5 and x 10-4 respectively. For WiSE-FT, we take a = 0.5 which is the coefficient for weight-space ensembling. For OpenCLIP models*. trained on LAION-400M. For SLIP 5, we use all the CLIP and SLIP models trained on YFCC-15M.
ViT-B/32 models
For data subsampling, we uniformly sample proportion of training examples from the entire dataset, at ratios of (5%, 10%, 20%, 30%, 40%, 50%), respectively. For combining two training datasets at various ratios, given coefficient À (0 < À < 1), uniformly sample proportion of data from the two datasets at ratios of and (1 A). respectively, and then we combine the two subsets. When combining ImageNet and CIFAR-10, we take À (0.001, 0.01, 0.1, 0.5, 0.9, 0.99, 0.995); when combining ImageNet with YFCC and LAION, respectively, we take À E (0.01, 0,2.5.25,, 0.25,
15

---

Table MAE (%) for ImageNet+LAION models when they are excluded and included in the fitting. respectively, where comparing the effective robustness of ImageNet models and LAION models.
ImageNet+LAION models the fitting Included
Test set
Excluded
ImageNet- V2
0.54
0,51
ImageNet-R
1.56
1.19
ImageNet-Skee Sketch
2.62
2.03
ObjectNet 2.91
1.47
Models are trained using x 4 or 4x8 TPU v2 Pods, and they are evaluated using NVIDIA V100 GPUs, on the cloud.
B.2 Details of ID Test Sets
We construct an ID test set from YFCC-15M and LAION-15M, respectively, and we automatically generate classification labels by matching with ImageNet class names, which has also been similarly performed in Fang et al. (2022) for training classifiers using caption data. On YFCC-15M which contains metadata, metadata such as tags, we simply use the entire text for matching. We adopt label only if a unique ImageNet class can be determined by matching for the involved image. We then construct balanced and labelled test set by keeping 50 examples for cach class that has at least 100 examples in the labelled images. The test examples are then held from the training data. For the YFCC test set, there are 22550 examples for 451 classes; and for the LAION test set, there are 20400 examples 408 classes.
tags for matching. On LAION-15M which does provide
B.3 Licenses of the Datasets
We have used the following datasets:
CIFAR-10 (Krizhevsky et al., 2009). License is not clearly known.
YFCC' under various Creative Commons licenses.
LAION" under the Creative Common CC-BY 4.0 license for the metadata, and the images are under the copyright of the original authors.
CIFAR-10.1 under the MIT license.
CIFAR-10.2%. License is not clearly known.
CINIC-10'0 under the MIT license.
ImageNet-V2" under the MIT license.
ImageNet-R under the MIT license.
ImageNet-Sketch" under the MIT license.
ObjectNer with license provided on the wehpage.
wat::000---oor----eeeeeee core-dataset
16


Reviews:
[{"review_id": "V59da31PEL", "reviewer": "Reviewer_mbLQ", "summary": "This paper extends the work \"Measuring Robustness to Natural Distribution Shifts in Image Classification\" to more than one ID test set scenario. This paper proposes a robustness evaluation metric that uses more ID test sets covering the evaluated models' training distributions.", "questions": "See above", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem of Effective Robustness is important.\n\n2. The paper's findings are interesting: vision-language model evaluation needs future explored given the unique training setting.", "weaknesses": "1 **The motivation is not sound**. Intuitively, the effective robustness evaluation is ID training dataset driven. If we want to compare different model architectures, comparing the domain shift OOD performance of different models trained on the same training dataset is meaningful. In that case, using a single ID test set is enough. \n\nFor models trained on large-scale image-language pair datasets (CLIP pretrained on LAION), it is unclear how to define a domain-shift OOD evaluation: how to find a \"domain shift\" OOD dataset of LAION? If the model is pertained on LAION and then finetuned on ImageNet, how to guarantee ImageNet-v2 is not overlapped with LAION? The accuracy of ImageNet-v2 itself is meaningful. Comparing different vision-language models that both trained on LAION is also meaningful. However, it is hard to understand why we must compare two models trained on different training sets. Which expected conclusion are you trying to get? The conflicting findings and conclusion on L95~L108 do not make sense if the previous motivation is unclear.\n\nUsually, if we want to see the OOD performance of vision-language models, we show a list of performances on different test sets. \n\n2 **The Figure-1 and conclusion are confusing** In Figure1, how many models are you using as ImageNet models and YFCC models? Why do you want to compare these two groups of models? What is your expected possibility of conclusion? \nAlso, in Figure1(a), the ImageNet accuracy overlap between the two groups is limited, so it is hard to directly compare the two fitted lines.\n\n3 **Limited Novelty** If we skip the motivation problem and focus on the proposed method, it is straightforward to extend equation 1 to equation 3 if we want to consider two ID test sets.\n\n4 **Generalization to more than two ID tests** The paper only conducted experiments on two ID test sets, but claimed the effectiveness on multiple ID test sets. It is not straightforward to extend to more datasets. For instance, how to build the sampling classes in Section 5.2? How to guarantee the sampled overlap class is good enough to evaluate", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their time and attention!\n\n## Why do we compare the effective robustness of models trained on different datasets?\nAt a general level, state-of-the-art models are increasingly being trained on large swaths of private data sources, but despite these differences in training distributions their performance is regularly compared, often in leaderboards composed of challenging benchmarks.  Because the field is heading in this direction, we believe it is critical to understand what conclusions can and can’t be drawn from such cross-model and training-data-agnostic evaluations.\n\n\nWith respect to robustness, this issue has already arisen in prior work.  As mentioned in Line 29-36, previously the well-known CLIP paper [30] has claimed unprecedented effective robustness gains of CLIP models, and [12, 29] later studied the reasons behind the effective robustness gains of CLIP models. These works all compared the effective robustness of ImageNet models and CLIP models pre-trained on other data (under the single-ID evaluation), and [12, 29] concluded that training data affect the effective robustness. \n\nGiven these previous works, the motivation here is that we aim to further understand if it is truly enough to compare the effective robustness of models trained on different data using the previous single-ID evaluation, and if changing the training data can truly improve effective robustness. We have found the limitations of the previous evaluation and proposed a new evaluation. This work is not raising the problem of comparing the effective robustness of models trained on different data for the first time, but it is providing a new understanding on the existing problem that has been studied by multiple existing works [30, 12, 29]. We conclude that the evaluation needs to be aware of the training data, and changing the training dataset does not improve the effective robustness. Therefore, we believe the motivation of this work is well supported given the previous works [30, 12, 29].\n\n## Which datasets are really OOD?\nWhile we agree with the reviewer that there is a risk of overlap between different datasets, we believe this a common challenge in defining “out-of-distribution” and that the datasets we chose both follow best practice and still shed meaningful conclusions.\n\nWe have used several OOD test sets in this work, such as ImageNet-R, ImageNet-Sketch, etc. It is also consistent with many previous works studying LAION models, including the [29], the LAION-5B paper (Schuhmann et al., 2022), Gadre et al., 2023, etc.  Further, previous works have already found that the likelihood of data overlapping is low (e.g., see Section 6 in Schuhmann et al., 2022 or Section 5 in CLIP [30]). \n\nSchuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., ... & Jitsev, J. (2022). Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35, 25278-25294. \n\nGadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., ... & Schmidt, L. (2023). DataComp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108. \n\n## Figure 1\nThese models are also used in Table 4 which has mentioned that there are 36 ImageNet models and 13 YFCC models. We will also add this information in Figure 1. \n\n\"Why do you want to compare these two groups of models?\" It follows [12] which also compared ImageNet models and YFCC models. YFCC is a public dataset with image-text pairs and thus it enables CLIP training which is previously believed to improve effective robustness under the single-ID evaluation. \n\n\"What is your expected possibility of conclusion?\" When the ID test set is changed from ImageNet into YFCC, there can be two possibilities. \n- Possibility 1: YFCC models do not have better single-ID effective robustness compared to ImageNet models. It implies that the single-ID evaluation is problematic when there are models with different training data, as suggested in our paper. \n- Possibility 2: YFCC models still have better single-ID effective robustness compared to ImageNet models. Then it cannot directly reject the single-ID evaluation. \nAs shown in Figure 1, the first possibility actually happens. \n\nRegarding the ImageNet accuracy overlap, it is similar to that in [12] which also compared the fitting lines (with ImageNet as the ID test set only). There is still an overlap at the mid-accuracy regime. \n\n## Is the work novel?\nThe core insight of our work is that apparent differences in effective robustness can be driven by differences in training sets and the evaluation needs to be aware of training data.  We believe that this is a novel insight, missed by many previous works on robustness [30, 12, 29], and leads to new conclusions and understanding of how to measure and improve robustness.  Therefore, the novelty lies not in how we empirically verify the insight, but in the insight itself.\n\n## Generalization to more than two ID test sets\nIn principle, a straightforward generalization is possible -- by directly generalizing Section 5.2, one can take the intersection of classes from more than two datasets. To check whether the evaluation is good enough, one can check if the fitting quality is satisfactory. In practice, this can be costly, as training much more models is needed to achieve a convincing fitting quality, when the evaluation has more than two datasets. It is not perfect, and we have already mentioned it as a limitation in the paper (Line 149-151, Line 316-317). Nevertheless, by comparing two datasets at a time in this work, we can already obtain conclusions that the considered models trained on any of the considered datasets have similar effective robustness, and a 3-way evaluation is not quite needed here.\n"}, {"review_id": "PjI7Q2Ngp0", "reviewer": "Reviewer_Kcyx", "summary": "The authors identify that the definition of effective robustness is ill-equipped for comparing models trained on a variety of distributions; in particular, it only uses a single data distribution to predict OOD accuracy $acc_{OOD}$. The authors propose a more precise effective robustness measurement for models trained on two different datasets ($data_1^{train}$ and $data_2^{train}$), which is based on using regression to optimize the parameters $b,w_1, w_2$ in the following equation: $acc_{OOD}(model_i) = b + w_1 acc_{data_1^{test}}(model_i) + w_2 acc_{data_2^{test}}(model_i),$ where accuracies are given in logit space. Thus, if we wish to compare the effective robustnesses of models $A$ and $B$ that were trained on $data_1^{train}$ and $data_2^{train}$, respectively, we can use this new approach to predicting $acc_{OOD}$ values without being forced to use just one of the two data distributions ($data_1$ or $data_2$) as the distribution we use to predict $acc_{OOD}$. The benefit of this approach, across a wide variety of contexts/models, is that it fits the data better and gives lower effective robustness measurements for models that appeared effectively robust under the old definition because they were trained on a dataset from a distribution different from the one used to predict $acc_{OOD}$.", "questions": "1. Many of the results are intuitive/interesting, but I'm not sure how they are aligned with the notion of predicting OOD performance from an ID dataset (i.e., \"effective robustness\"). My understanding of the proposed definition of multi-ID effective robustness is that it involves predicting OOD performance using an ID *and* an OOD performance, rather than using two ID performances as the proposed name (\"multi-ID\" ER) suggests. I think the submission (and my score) could be significantly improved if the authors can clarify why multi-ID ER can be interpreted as an ER measurement without issues.\n\n2. While the original ER definition is limited in ways (including a way identified by this submission), it also has clear practical utility in that I can understand whether a model outperforms on OOD data relative to models trained on the same ID data. The practical utility of this submission's proposed alternative ER definition is unclear to me. For instance, given a model trained on $data_1$, is it possible to get a different understanding of its multi-ID ER simply by changing the second dataset used in the ER definition (e.g., from YFCC to LAION)? If so, how many dataset pairs ($data_1$, $data_i$) do I need to check my model on before I understand its expected OOD performance (or its ER)? \n\n3. Line 264: Why would WiSE-FT alter the underlying training distribution in a way that makes it unsuitable for multi-ID ER? It seems like its train distribution would be similar to that of the models that are trained on dataset mixtures.\n\n4. Why not report the R2 in some tables (e.g., Table 5)?\n\n5. Table 1 should be labeled as a Figure.\n\n6. It might be nice to see the plane equations on the plane plots.", "limitations": "Potential limitations are well addressed.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality:\n\n1. The submission's effective robustness (ER) definition is a simple and novel extension of the original definition. \n\nQuality:\n\n2. The first three sections of the paper are strong. In particular, Section 3 is a very convincing and interesting illustration of a problem with our current definition of effective robustness.\n\n3. The submission's approach (described in Section 4) provides interesting results that may be helpful for understanding when a model is effectively robust. The models and datasets used are comprehensive.\n\nClarity:\n\n4. Largely, the paper is well written and organized. \n\nSignificance\n\n5. This work offers an important insight about evaluation protocol in an active area of research.", "weaknesses": "My main concern is that, despite superficial similarity in their definitions, the interpretations of single- and multi-ID ER are drastically different. Accordingly, I am concerned about whether the new ER definition (multi-ID ER) will have practical utility for researchers interested in identifying models with ER. I expand on these concerns below and in the \"Questions\" section.\n\n1. My (and seemingly the authors') understanding of the purpose of effective robustness is that it identifies models that surpass the OOD accuracy we expect of them *based on their ID accuracy* (i.e., we want to know if training on ID data using some new technique produces more robustness than expected). In Section 3, this submission clarifies that ER does not serve this purpose when we fix the \"ID\" dataset in our definition of ER to be $data_1^{test}$, then use that ER to compare models trained on ID data ($data_1^{train}$) with models trained on OOD data ($data_2^{train}$).  However, this submission's solution also seems to lose this original purpose: models with high multi-ID ER are those with higher-than-expected $acc_{OOD}$ when predictions are made using a combination of an **OOD accuracy** (for a model trained on $data_1^{train}$, this would be $acc_{data_2^{test}}$) and an ID accuracy (for a model trained on $data_1^{train}$, this would be $acc_{data_1^{test}}$). In other words, this metric's use of an OOD accuracy (e.g., $acc_{data_2^{test}}$ for the model trained on $data_1^{train}$) to predict an OOD accuracy (i.e., $acc_{OOD}$) seems to preclude its interpretation as an ER measurement and harm usefulness/interpretability. For example, because we would expect OOD accuracies to be correlated, it seems that the calculation of multi-ID ER can lower the detected ER in a model that truly has ER: if a model has high OOD accuracy (e.g., $acc_{data_2^{test}}$ for a model trained on $data_1^{train}$), that can inflate multi-ID ER's prediction of $acc_{OOD}$, which lowers the detected ER of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their time and attention!\n\n## Two ID test sets\nIn this paper, we consider that two ID test sets are used, rather than an ID test set and an OOD test set as mentioned by the reviewer. For models trained on one of the ID datasets, even though there may be some distribution shift between their training data and the other ID test set, we assume that this distribution shift is relatively minor compared to the distribution shift to the OOD test sets (e.g., the difference between ImageNet and YFCC is much smaller, compared to the difference between ImageNet-Sketch and ImageNet/YFCC). Indeed this treatment is not perfect and can have limitations as the (potentially minor) distribution shift between the multiple ID test sets is not considered, and we will add it to the \"limitations\" section. But we believe it is still a better evaluation than the single-ID evaluation when models are trained on different data. \n\n## Choosing the second dataset\nThe choice of the second ID test set is not arbitrary. In this paper, we propose that the ID test sets need to match the training data (Line 113~114). So if we are comparing ImageNet and YFCC models, the ID test sets will only be from ImageNet and YFCC, but not LAION. When we only choose ID test sets in this way, we show that it is already enough to conclude that the considered models mainly have similar effective robustness. \nChoosing the ID test sets in some different way (e.g., as mentioned by the reviewer) will be a different setting. It is still interesting for future work to consider, but it is beyond the scope of our proposed method.\n\n## WiSE-FT\nWiSE-FT performs weight-space ensembling which edits the weights. It is not quite clear what the training distribution is for such kinds of models with post-training weight editing, compared to models that are directly trained from a known distribution without weight modification (e.g., dataset mixture).\n\n## $R^2$ in Table 5\n$R^2$ values are not reported as the models in the table are not involved in the fitting. They are directly evaluated on fitting lines/planes from Table 3.\n\n## Figures\nWe will change Table 1 into Figure 1, and add plane equations to the figures. \n"}, {"review_id": "QmIfBz23lW", "reviewer": "Reviewer_sN9Q", "summary": "This work delves into the definition of effective invariance, which is proposed to measure how the model performs beyond what we can expect from its in-distribution accuracy. This work points out that the training distribution should be considered when measuring the model's effective robustness. Specifically, models can be trained on different and diverse distributions. Then, the usage of the \"in-distribution\" test set should be changed accordingly. The new evaluation metric provides a more accurate characterization of CLIP models' robustness. ", "questions": "Please comment on the choice of OOD test set:  CLIP models are trained on diverse and large-scale training sets (e.g., LIAON). Then, the test set may already be covered by the training distribution. Namely, a test set is OOD for ImageNet models but may be so OOD for CLIP models. In such a case, the effective invariance may truly reflect model behaviors under distribution shifts.  ", "limitations": "This work includes a paragraph to discuss the limitation and potential future directions, where the access to training distribution and determination of training distribution for some models are discussed. One more point to discuss: training distributions of CLIP models may already cover test distribution, then the definition of OOD needs to be re-considered. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ [***The motivation of re-define the effective invariance is sound***]  Large-scale models trained on various datasets make it necessary to evaluate and compare models trained on different data distributions. Namely, the in-distribution dataset should be well-aligned with the training distribution. \n\n+ [***Extending the idea of effective invariance to multiple ID test sets is sound*** ] Using multiple ID test sets to check the relationship between ID accuracy and OOD accuracy is reasonable and straightforward. \n\n+ [***Good clarify***] This work is well-written and easy to read. The motivation, background knowledge of effective invariance, and the idea of multiple ID test sets are well presented. ", "weaknesses": "- [***Fitting quality is not very convincing***] In experiments, fit quality is mainly used to show that using multiple ID test sets can lead to a more accurate match between ID and OOD accuracy. If our goal is to produce more accurate fits, we can use one ID test set and one OOD test set to fit another OOD test set. Please comment on why you should care about the fitting quality.\n\n- [***Discussing ID test sets for models pre-trained on more data***] The motivation for using multiple ID test sets is clear. However, for models pre-trained with more data and finetuned on ImageNet, what is their ID test set?\n\n- [***Clarification of CIFAR model***] CIFAR images cost $32\\times32$, while ImgeNet images are much higher resolution. How to combine them for training? Upsampling or downsampling?\n\n- [***Please summarize observations***] It is unclear what the main observations and new insights are given by the multiple ID sets. It is clear that \"the effective robustness of a zero-shot CLIP model trained on language-image data is similar to that of a standard ImageNet model\". What other discoveries have been made besides this? For example, does pretraining on more data still contribute to effective invariance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their time and attention! \n\n## Fitting quality\nA relatively high fitting quality is desired, which implies that the considered models have similar effective robustness and it is a major finding in this work.\n\nFollowing previous works on effective robustness, effective robustness checks the extra performance on OOD data beyond the performance that can be predicted from ID performance (Line 1~2). Therefore, there is a constraint here that the test sets used for the prediction cannot be arbitrarily taken, and they need to be ID test sets, not an OOD dataset. We only take ID test sets that match the training data, which is also consistent with the single-ID evaluation if models are only trained on a single dataset [40]. If an OOD test set such as ImageNet-R is used for the prediction, it already measures OOD performance. Though the fitting quality may be better, it cannot reveal the effective robustness. \n\n## Models pre-trained on more data and fine-tuned on ImageNet\nFor those models, we consider that their training distribution gradually shifts from the pre-training distribution to the ImageNet distribution, as the fine-tuning proceeds, and during the fine-tuning, the training distribution is some combination of the pre-training one and ImageNet. In this case, we use two ID test sets, including ImageNet and also the one matching the pre-training data, to cover the training distribution. \n\n## Combining CIFAR and ImageNet in training\nWe upsample CIFAR images to 224x224 for combined training. We will mention it in the paper. \n\n## Main takeaways\nOne main observation is the one already mentioned by the reviewer about the effective robustness of various models including zero-shot CLIP models. In addition, we propose that the effective robustness evaluation needs to be fully aware of the training data and consider the training data, when there are models trained on different datasets. \n\nThe amount of training data does not affect effective robustness, as already mentioned in previous works [12].\n\n## Training distribution of CLIP may overlap with the test distribution\nThis is a limitation that also exists in previous works as long as they evaluate CLIP on the commonly adopted OOD datasets. Some previous works have mentioned that the likelihood of a direct data overlapping is low (e.g., see Section 6 in Schuhmann et al., 2022 or Section 5 in [30]). But it may require future work to better understand the effect. We will add it to the limitation section.\n\n\nSchuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., ... & Jitsev, J. (2022). Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35, 25278-25294. \n\nGadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., ... & Schmidt, L. (2023). DataComp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14108. \n"}, {"review_id": "k3UCWFRGVh", "reviewer": "Reviewer_MjfW", "summary": "The authors propose a modification to effective robustness evaluations that takes multiple training sets into account. This facilitates a more precise comparison of the effective robustness of models trained on different training sets. The authors demonstrate empirically that their method yields an improved predictor of OOD accuracy. The analysis of models trained on ImageNet, YFCC, and LAION reveals that CLIP models might not, in general, exhibit larger effective robustness than ResNets, further strengthening the existing empirical evidence in this direction.", "questions": "\nI am surprised how well models trained on CIFAR10 perform on ImageNet (x-axis in Figure 3b, blue dots). Was the size of the CIFAR10 images kept as usual or upscaled to 224x224 to make it more comparable to ImageNet trained models? I am surprised that training on 32x32 CIFAR10 images gives such high accuracy numbers on ImageNet, even after class-subsampling. Intuitively, I would expect the accuracy on CIFAR-10.2 to be much higher than on ImageNet, of a model trained on CIFAR10 since the distribution of CIFAR-10.2 should be closer to CIFAR10 than ImageNet. But the accuracies on CIFAR-10.2 and ImageNet are almost the same; the effect is much smaller than I would have expected. Could you maybe comment on this finding? Do you find it surprising / have any intuition or explanation for it?\n\n\nTable 3 vs Table 2: In Table 2, the R² coefficients were all above 99% while in Table 3, most numbers are generally below 99%. Why do you think that is? For ObjectNet, the R² values for both YFCC and LAION are the lowest ones among all datasets. What makes ObjectNet different from the other datasets and why does the technique work worse for this dataset? Is there an interpretation of what a higher R² value means for the different training datasets, maybe in terms of similarity to the test set? E.g., on ObjectNet, the R² is higher when training on YFCC compared to LAION. Can anything interesting be deduced from this?\n\n\nFollowing up on the previous point: Can there be a situation when your method does not work and could this be measured with the R² fitting score? Is it possible to pin it down to a threshold value when the technique should not be used? E.g. R² < 95% or R² < 90%? \n\n\nAdditionally, it is unclear to what extent the method relies on the ID sets to be similar. Do you have an intuition whether this approach would work at all for models trained on datasets that could be considered “OOD sets” of each other (e.g. a model trained on natural images vs. a model trained on sketches)? Related to this, how important was it for the fit of the baseline that models trained on a mixture of the ID sets were available? An ablation of this point would be interesting, as it might be prohibitively expensive in practice to retrain models on multiple mixtures of the ID sets simply to facilitate a comparison.\n\n\nIt seems to me as if the data would have been available to additionally facilitate a 3-way multi-ID evaluation of some models on the ID sets ImageNet, LAION, and YFCC. Could you elaborate why this was not attempted? I believe it would greatly strengthen the claim that this method should natively extend to more than 2 ID sets.\n", "limitations": "yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "\nThe paper is well written and easy to follow. In more detail:\n- The introduction is very well written and motivates the ideas clearly. The problem targeted in the paper makes sense. \n- The motivating experiment in Section 3 is very nice and illustrates the limitation of using a single ID test set very well and the provided intuition in line 104 makes sense (“We observe that when there is a mismatch in the training data and the ID test data, the models appear to have greater effective robustness (YFCC models in Figure 1a and ImageNet models in Figure 1b), as their performance on the ID test data and the OOD performance predicted from the single ID accuracy tend to be lower.”)\n\n\nThe issue that the regular definition of effective robustness is confounded by the training distribution makes sense and makes it difficult to compare the performance of models trained on different datasets. The idea of doing regression on the accuracies of multiple ID test sets is simple and seems to work well. The conclusion that CLIP models do, in general, not exhibit greater effective robustness is important and relevant.\n", "weaknesses": "While the paper is generally well written, there is an abundance of typos and grammar errors (see examples below), which need fixing before acceptance. I would strongly suggest to the authors to do a round of proof-reading the manuscript. \n\n\nL40: “In particular, previous works performed single-dimensional linear regression on a set of baseline models to predict OOD accuracy from a single ID accuracy.” citations needed\n\n\nLine 58: “Under natural distribution shifts, the OOD accuracy of a model is often correlated with the ID accuracy.” citation needed\n\n\nIn Fig.1, does “ImageNet models” refer to both standard ImageNet classifiers and CLIP models trained on ImageNet? If so, isn’t it quite remarkable that CLIP models and standard classifiers lie on the exact same line? In general, it is unclear in this figure which of the models from section 5 are depicted here.\n\n\nL128: Table 1 is a Figure not a Table\n\n\nL181: “For datasets that consist of image-text pairs for language-image pre-training without original labels, including YFCC and LAION, we automatically generate classification labels by matching captions with ImageNet classes, and we then hold out a balanced test set from the original dataset.” Please provide examples of YFCC and LAION images+captions which were accepted into the ID test sets, along with their new ImageNet label in Appendix B.2.\n\n\n\n#### Clarity:\n\nOverall, the 3d plane plots are unhelpful in gauging whether points lie close to the plane or not. In addition to the provided 2d projections _along the ID accuracy axes_ a 2d projection orthogonal to the fitted plane could be helpful to visually judge the model fit.\n\n\nL121: the sentence in line 121 is a bit hard to parse, would suggest rephrasing for clarity.\n\n\nL133: the sentence in line 133 is a bit hard to parse, would suggest rephrasing for clarity.\n\n\nL260-261: it is unclear that \"SLIP models ... achieve higher average effective robustness\" _compared to other models_, rather than _compared to single-ID setting_\n\n\nCan the used OOD sets really be considered OOD sets for all training distributions? LAION, for example, contains numerous renditions and sketches. As a result, the measured effective robustness might not be a true “OOD robustness”. A remark on this terminology could be helpful. \n\n\n#### Grammar / typos:\n\nAbstract, typo: “Our new evaluation metric provides a better estimate of the effectiveness robustness”\n\n\nL11: “effective~~ness~~ robustness”\n\n\nL18: “following ~~on~~”\n\n\nL22: “Unfortunately,” comma missing\n\n\nL63: “are … resulted from” is grammatically incorrect\n\n\nL78: “Limitation **of** the Single ID Test Set”\n\n\n\nL80: “reflective **of**”\n\n\n\nL81: “reflective **of**”\n\n\nL81: “However, as the emergence of large-scale pre-trained models,” grammar, maybe “due to” instead of “as”?\n\n\nL85: “when only one … **is** used”\n\n\nL85: “--” should be “:”, I think\n\n\nL88, 114, 117, 123, 172, 224, Table 3, Caption/L279: comma missing before respectively. Please add commas before all instances of “respectively”. In addition, if there is no full-stop after “respectively”, a comma should be added after this word as well (e.g., Table 3, caption).\n\n\nL110: “test set”, set is missing\n\n\nL121, L265: comma after thus missing\n\n\nL148: “**generalized**”, “in **principle**”\n\n\nL166: rephrase to “and subsample the combined datasets”\n\n\nL195: should “retrain” be “retain”?\n\n\nL258: “although **they** may”\n\n\nL290: “models” should be plural\n\n\nL315: may further be\n\n\nL503: “**held** out”\n\n\nL515: “**under**”\n\n\nL519 “**clearer** understanding **of**”\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their time and attention!\n\nWe will fix the writing issues accordingly and make further proof-reading. We answer the questions below:\n\n## CIFAR models on ImageNet \nWhen a model is trained on both CIFAR and ImageNet, we upsample CIFAR images to 224x224. If a model is only trained on CIFAR, 32x32 is used. \n\nThe reviewer mentioned that \"even after class-subsampling\", but class-subsampling tends to increase the accuracy when there are fewer classes. When evaluating the class-subsampled ImageNet accuracy, we only evaluate on images with ImageNet labels that can be mapped to CIFAR-10 classes [26], which has potentially made this version of ImageNet much easier. \n\n## $R^2$ values on different OOD datasets\n$R^2$ can be affected by the relationship between the ID datasets and the OOD datasets. It tends to be harder to predict the OOD performance when the OOD datasets have larger distribution shifts or are less related to the ID datasets and then $R^2$ values tend to be lower. As mentioned by the reviewer, it can have implications on the similarity between the datasets.\n\nA good fitting does not always exist. As mentioned in Line 273-275 and Line 308-311, even if the models are trained on the same dataset and the previous single-ID evaluation is used, \"accuracy-on-the-line\" does not always hold. Thus, we only consider datasets where at least \"accuracy-on-the-line\" holds for models trained on the same dataset. On these datasets, our method works well. As mentioned by the reviewer, it may be checked by $R^2$ scores and it is good to set a threshold. We think $R^2\\geq 90\\%$ is generally satisfactory, although one may set a different threshold if they have a different requirement. \n\n## Similarity between the ID test sets\nWe do not think our method is applicable when the training datasets are highly different. When we have multiple ID test sets, we are treating these test sets as ID test sets for all the models trained on different data. Thus, these ID test sets need to be relatively close compared to the OOD test sets. In our case, ImageNet and YFCC are relatively close, compared to OOD test sets that are quite different (e.g., ImageNet-Sketch). We will mention this restriction in the revision.\n\n## Including models with mixed training data in the fitting\nWe add an ablation study below. We evaluate the MAE scores of models trained on a mixture of datasets (ImageNet+YFCC models and ImageNet+LAION models), when they are excluded or included in the fitting respectively. The MAE scores (the lower the better) increase if the models with mixed training data are excluded in the fitting but are generally satisfactory. In this work, we have trained a large number of models with data mixed at various ratios and considered models with diverse combinations of accuracies, to obtain more convincing conclusions on the effective robustness.\n\n**MAE of ImageNet+YFCC models**\n| OOD test set | Excluding ImageNet+YFCC models in the fitting | Including ImageNet+YFCC models in the fitting |\n| --- | --- | --- |\n| ImageNet-V2 | 0.0071 | 0.0064 |\n| ImageNet-R | 0.0130 | 0.0121 |\n| ImageNet-Sketch | 0.0098 | 0.0094 |\n| ObjectNet | 0.0173 | 0.0095 |\n\n**MAE of ImageNet+LAION models**\n| OOD test set | Excluding ImageNet+LAION models in the fitting | Including ImageNet+LAION models in the fitting |\n| --- | --- | --- |\n| ImageNet-V2 | 0.0054 | 0.0051 |\n| ImageNet-R | 0.0156 | 0.0119 |\n| ImageNet-Sketch | 0.0262 | 0.0203 |\n| ObjectNet | 0.0291 | 0.0147 |\n\n## 3-way multi-ID evaluation\nBy comparing two datasets at a time in this work, we can already obtain conclusions that the considered models trained on any of the considered datasets have similar effective robustness, and a 3-way evaluation is not needed here. Directly comparing more than two datasets can be much more costly, as a larger number of models with diverse combinations of accuracies are needed for obtaining relatively convincing fitting planes and conclusions. We also mentioned it in Line 149-151 and Line 316-317.\n"}]

Author rebuttal:

Meta-review: This paper aims to provide a more accurate comparison of the effective robustness (ER) of models trained on different datasets. Overall, the reviewers agree on the importance of the studied problem and find this paper interesting to read. Meanwhile, several major concerns are raised, including 1) additional ablations studies are needed, and 2) some claims/statements are confusing and probably incorrect.

The rebuttal was constructive --- with several rounds of discussions between authors and reviewers, several issues were identified and fixed accordingly, particularly regarding when the proposed multi-ID evaluation is applicable and how it could supplement existing single-ID evaluation for a better understanding of ER. All reviewers reached a unanimous decision to recommend the paper for acceptance. 

For the paper to meet the rigorous publication standards of NeurIPS, the authors must undertake a significant revision of this paper, including rectifying flawed statements and incorporating the clarifications and ablation studies discussed during the rebuttal phase.
Decision: Accept (poster)