Title: Effective Robustness against Natural Distribution Shifts for Models with Different Training Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: Effective robustness

natural distribution shifts

out-of-distribution robustness
Abstract: ``Effective robustness'' measures the extra out-of-distribution (OOD) robustness beyond what can be predicted from the in-distribution (ID) performance. Existing effective robustness evaluations typically use a single test set such as ImageNet to evaluate the ID accuracy. This becomes problematic when evaluating models trained on different data distributions, e.g., comparing models trained on ImageNet vs. zero-shot language-image pre-trained models trained on LAION. In this paper, we propose a new evaluation metric to evaluate and compare the effective robustness of models trained on different data. To do this, we control for the accuracy on multiple ID test sets that cover the training distributions for all the evaluated models. Our new evaluation metric provides a better estimate of effective robustness when there are models with different training data. It may also explain the surprising effective robustness gains of zero-shot CLIP-like models exhibited in prior works that used ImageNet as the only ID test set, while the gains diminish under our new evaluation. Additional artifacts including interactive visualizations are provided at https://shizhouxing.github.io/effective-robustness.

Effective Robustness against Natural Distribution Shifts for Models with Different Training Data

Abstract

'Effective robustness' measures the extra out-of-distribution (OOD) robustness beyond what can be predicted from the in-distribution (ID) performance. Existing effective robustness evaluations typically use a single test set such as ImageNet to evaluate the ID accuracy. This becomes problematic when evaluating models trained on different data distributions, e.g., comparing models trained on ImageNet vs. zero-shot language-image pre-trained models trained on LAION. In this paper, we propose a new evaluation metric to evaluate and compare the effective robustness of models trained on different data. To do this, we control for the accuracy on multiple ID test sets that cover the training distributions for all the evaluated models. Our new evaluation metric provides a better estimate of effective robustness when there are models with different training data. It may also explain the surprising effective robustness gains of zero-shot CLIP-like models exhibited in prior works that used ImageNet as the only ID test set, while the gains diminish under our new evaluation. Additional artifacts including interactive visualizations are provided at https://shizhouxing.github.io/effective-robustness .

1 Introduction

Robustness against distribution shifts is important for machine learning models to work reliably across various environments. For natural distribution shifts on image classification datasets, Taori et al. (2020) proposed the notion of effective robustness to control for in-distribution (ID) accuracy when evaluating out-of-distribution (OOD) accuracy. Following a long line of work that has found a strong correlation between ID and OOD accuracy on many test sets (Recht et al., 2019; Yadav & Bottou, 2019), effective robustness allows researchers to assess whether an apparently improved OOD accuracy is a result of effectively improved robustness or is simply an expected outcome of enhanced ID accuracy.

Unfortunately, the current definition of effective robustness has a subtle limitation: it requires a fixed ID test set, which is typically ImageNet (Deng et al., 2009) when using ImageNet-like OOD test sets

in Taori et al. (2020) or CIFAR-10 (Krizhevsky et al., 2009) when using CIFAR-like OOD test sets in Miller et al. (2021). It is acceptable when models are trained predominately on only one dataset. However, the emergence of many large-scale models trained on significantly different datasets makes it necessary to evaluate and compare models trained on different data distributions, under which it becomes unclear which ID test set should be used.

In particular, models from Contrastive Language-Image Pre-training, such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) have recently exhibited unprecedented effective robustness gains during zero-shot inference (Radford et al., 2021; Fang et al., 2022; Nguyen et al., 2022). However these previous works simply take ImageNet as the single ID test set, even though the models are not trained on ImageNet. We demonstrate that the results of evaluating effective robustness using a single ID test set can vary drastically depending on the selection of the ID test set. Therefore, this imprecise treatment on the ID test set in existing works could end up exaggerating the effective robustness of zero-shot CLIP models compared to models that are exactly trained on ImageNet.

In this paper, we propose to more precisely evaluate and compare the effective robustness of models trained on different datasets. Instead of controlling for a single ID accuracy that may bias towards models from a particular training distribution, we propose to use multiple ID test sets that cover the training distributions of all the models. In particular, previous works performed single-dimensional linear regression on a set of baseline models to predict OOD accuracy from a single ID accuracy (Taori et al., 2020). And they then evaluate the actual OOD accuracy of the models beyond the expected value that can be predicted from the fitting line, as the effective robustness. We expand on this definition by allowing for multiple ID test sets, and perform multi-dimensional linear regression to fit a plane to predict OOD accuracy from the accuracy on multiple ID test sets.

In summary, we make the following contributions:

We reveal a limitation in the existing effective robustness evaluation when used to compare models trained on different data distributions.

We then propose a new effective robustness evaluation which uses multiple ID test sets to more precisely compare the effective robustness of models trained on different data.

We show that the OOD accuracy of various models including zero-shot CLIP models can usually be better predicted from accuracies on multiple ID test sets compared to using only one ID test set.

Our results provide new understandings on the effective robustness gains of CLIP-like models observed in prior works only using ImageNet as the ID test set, while the gains diminish under our new evaluation.

2 Background of Effective Robustness

Under natural distribution shifts, the OOD accuracy of a model is often correlated with the ID accuracy. After applying a logit transformation on the accuracy, a linear trend between the transformed ID accuracy and OOD accuracy holds across many datasets (e.g., a distribution shift from ImageNet (Deng et al., 2009) to ImageNetV2 (Recht et al., 2019), or from CIFAR-10 (Krizhevsky et al., 2009) to CIFAR-10.2 (Hendrycks & Dietterich, 2018)) and models with various architectures and training methods (Taori et al., 2020; Miller et al., 2021). This phenomenon implies that most models showing higher OOD accuracies naturally resulted from better ID performance.

To eliminate the confounding effect of ID accuracy on OOD performance, Taori et al. (2020) proposed effective robustness that measures the OOD performance beyond the expected OOD accuracy given the ID accuracy, where the expected OOD accuracy is predicted according to the fitted linear trend of baseline models. Since they only use a single ID test set, we refer to this version of effective robustness as single-ID effective robustness .

Suppose there are n baseline models f 1 , f 2 , · · · f n . A baseline function ˜ β ( x ) is constructed to predict the OOD accuracy of each baseline model, acc ood ( f i ) (1 ≤ i ≤ n ) , given the single ID accuracy of the model x =acc id ( f i ) . The baseline function is instantiated as:

where w and b are parameters, logit( x ) = ln( x 1 -x ) is the logit transformation, and expit( x ) is the inverse of logit( x ) . Since logit( ˜ β ( x )) = w logit( x ) + b , the baseline function is essentially a linear

10

25

50

70

80

90

95

10

25

50

70

80

90

95

ImageNet models

Linear fit for ImageNet models

Linear fit for YFCC models

ImageNet accuracy (css., %)

ImageNet-V2 accuracy (css., %)

10

25

50

70

80

10

25

50

70

80

ImageNet models

Linear fit for ImageNet models

Linear fit for YFCC models

YFCC accuracy (css., %)

ImageNet-V2 accuracy (css., %)

function after applying a logit transformation on the accuracies, and it can be determined by solving a linear regression. Then the single-ID effective robustness of a model f is evaluated as

which subtracts the predicted OOD accuracy based on the ID accuracy acc id ( f ) , from the actual OOD accuracy acc ood ( f ) .

3 Limitation of the Single ID Test Set

The existing effective robustness evaluation in Section 2 fixes a single ID test set for all the models, which is reflective of the ID performance only if all the models are trained on the same dataset that matches the ID test set. However, as large-scale pre-trained models emerge, it becomes necessary to compare models trained on different datasets, in order to know if the latest pre-training techniques can yield effective robustness gains. In this section, we use the comparison between zero-shot CLIP models and standard ImageNet models as an example to show the limitation of using a single ID test set: when only one ID test set is used, using different ID test sets leads to contradictory conclusions.

Following Fang et al. (2022), we compare models trained on ImageNet (Deng et al., 2009) and YFCC-15M (Thomee et al., 2016), respectively. On ImageNet, we include standard classifiers, and we also train CLIP models using templates filled with an ImageNet class name as the caption in a format of 'A photo of a {class name}'. We also train CLIP models on YFCC-15M, a dataset with image-text pairs. And we use ImageNet-V2 (Recht et al., 2019) as the OOD test set. We consider two different ID test sets. One ID test set is simply ImageNet. The other ID test set is constructed from YFCC-15M, since we have CLIP models trained on YFCC. We refer to this test set as 'YFCC test set', and we refer to the accuracy on this test set as 'YFCC accuracy'. We discuss its details in Section 5.1 and Appendix B.2. Both ID test sets we consider here match the training distribution of some of the models (ImageNet models and YFCC models respectively) but not all the models.

We then plot the ImageNet-V2 accuracy of the models against their ImageNet accuracy and YFCC accuracy, respectively. There is a strong linear trend between the scaled ID accuracy and OOD accuracy for ImageNet models and YFCC models, respectively, and we plot fitting lines for these two sets of models, respectively. When the ID test set is ImageNet, Fig. 1a shows that the fitting line for YFCC models is generally above the fitting line for ImageNet models (except for the regime with extremely low accuracies), which appears to suggest that YFCC models have effective robustness gains over ImageNet models, as also suggested in Fang et al. (2022). However, in Fig. 1b which uses YFCC as the ID test set, the fitting line of ImageNet models are now mostly above YFCC models, which instead appears to suggest that ImageNet models have greater effective robustness than YFCC models. We observe that when there is a mismatch in the training data and the ID test data, the

models appear to have greater effective robustness (YFCC models in Figure 1a and ImageNet models in Figure 1b), as their performance on the ID test data and the OOD performance predicted from the single ID accuracy tend to be lower. This makes it difficult to compare models trained on different data and leads to imprecise conclusions on effective robustness if only one ID test set is used.

4 Multi-ID Effective Robustness

Considering the limitations of using a single ID test set, we propose a new way for effective robustness evaluation using multiple ID test sets that cover the training data distributions of all the involved models. We name it multi-ID effective robustness . Specifically, for each training distribution, we propose to prepare an ID test set that matches the training distribution, respectively. In particular, we focus on comparing models trained on two different datasets at a time in this paper, and we thereby use two ID test sets, where each of them corresponds to one of the training datasets.

While we refer to them as ID test sets, each of them is only the exact ID test set for some of the considered models that are trained on the distribution matching the test set, and it is not exactly an ID test set for all the considered models. However, we assume that the training distributions of all the models are still relatively close compared to the OOD test distributions (e.g., images normally collected from social medias in ImageNet (Deng et al., 2009), YFCC (Thomee et al., 2016), and LAION (Schuhmann et al., 2021) are relatively close compared to the OOD images in ImageNetSketch (Wang et al., 2019) that consists of sketch images specifically). In this way, both ID test sets are relatively ID for all the models compared to the OOD test sets, and it can be meaningful to control for the performance on these ID test sets when comparing the OOD performance.

We still use acc ood ( · ) to denote the OOD accuracy, and we use acc 1 ( · ) and acc 2 ( · ) to denote the accuracy on the two ID test sets, respectively. In contrast to the previous baseline function ˜ β ( x ) in Eq. (1), we propose a new baseline function β ( x, y ) that predicts the OOD accuracy based on the accuracies x and y on the two ID test sets, respectively.

All the models in Figure 1 are trained on either ImageNet or YFCC. Thus, to compare their effective robustness under our new evaluation, we use two ID test sets for ImageNet and YFCC at the same time, in contrast to Figure 1a and 1b which use one ID test set separately at each time and results on the two different ID test sets lead to contradictory conclusions. As shown in Figure 2, we plot the OOD accuracy against the two ID accuracies on both two ID test sets in a 3D space. We observe that the data points approximately lie on a plane when plotted on the logit scale. This motivates us to instantiate β ( x, y ) as:

where w x , w y , b are parameters. β ( x, y ) , which is the plane in Figure 2, is also a linear function w.r.t. x and y under the logit scale, and thus it is a reasonable extension from ˜ β ( x ) by using a multi-dimensional linear function on the logit scale. We determine the parameters by solving an ordinary least squares regression to fit the accuracies. Metrics for linear regression such as the coefficient of determination, a.k.a. R 2 ,

ImageNet Models

06

ImageNet-V2 acc. (css., %)

80

70

50

25

10

90

%

ImageNet acc. (css., %)

10255070 80

25

#

10

90

YFCC acc. (css.

can be used to evaluate the fitting quality of the baseline function. A high R 2 value indicates that the OOD accuracy is accurately predicted by the baseline function from the ID accuracies, and thus the evaluated models have similar effective robustness. And our multi-ID effective robustness for a model f is defined as

Compared to the existing definition for effective robustness in Eq. (2), the major difference is the inclusion of two ID accuracies acc 1 ( f ) and acc 2 ( f ) in the baseline function, compared to using a single ID accuracy acc id ( f ) .

Generalizing to more than two training datasets. Although we focus on handling two training datasets at a time, our method may be generalized to more than two datasets in principle, by defining a baseline function based on multiple ID accuracies acc 1 ( · ) , · · · , acc k ( · ) . However, it could be costly as it would require training more models to fit a high-quality baseline function. We leave it for future works to reduce the cost when dealing with a larger number of datasets.

5 Experiments

5.1 Settings

Models. In order to fit a baseline function, we need a large amount of models with diverse accuracies. To this end, we follow Taori et al. (2020) to train models with various proportions of data by subsampling from the entire training set (namely dataset subsampling), which effectively produces models with diverse accuracies. Moreover, we also combine examples from two datasets with different sampling ratios and train models on these combined datasets. This produces models with training distributions varying between the two training datasets and it is supposed to yield different combinations of the two ID accuracies. We use models trained on each single dataset as well as the combined datasets in the same fitting process, so that the baseline functions do not bias towards models trained on certain data. Our experiments include the following models:

Standard classifiers on CIFAR-10 and ImageNet. We train standard classifiers on CIFAR-10 and ImageNet (Deng et al., 2009). We use ResNet-18, ResNet-50, and ResNet-101 (He et al., 2016). Additionally, we train models by combining CIFAR-10 and ImageNet at various ratios, where we upsample CIFAR-10 images from 32 × 32 to 224 × 224 . Furthermore, we include ViT-S/16, ViT-B/16, ViT-L/16 models (Dosovitskiy et al., 2021) pre-trained on the whole ImageNet.

CLIP models. On YFCC-15M (Thomee et al., 2016) and LAION-15M (Schuhmann et al., 2021) which consist of image-text pairs, we train CLIP models using ResNet-50 and ResNet-101. We also train models by combining ImageNet and YFCC-15M and LAION-15M, respectively. We discard models with ImageNet accuracy below 5%. Additionally, in Section 5.4, we also have downloaded ViT-based models from Mu et al. (2022); Ilharco et al. (2021) and CLIP models fine-tuned on ImageNet, which are only used for evaluation but not fitting the baseline functions. We provide additional details in Appendix B.1.

We use '{Name_of_dataset} models' to denote models trained only on the dataset, e.g., 'CIFAR-10 models'. And we use '{Name_of_dataset_A}+{Name_of_dataset_B} models' to represent models trained on a combination of two datasets, e.g., 'CIFAR-10+ImageNet models'.

ID test sets. We focus on image classification. Labeled image classification datasets such as ImageNet can be directly used for evaluating ID accuracy. For datasets that consist of image-text pairs for language-image pre-training without original labels, including YFCC and LAION, we automatically generate classification labels by matching captions with ImageNet classes, which has been similarly performed in Fang et al. (2022) for training classifiers using caption data, and we then hold out a balanced test set from the original dataset. More details are reported in Appendix B.2. Although it is possible to obtain a higher-quality test set by human labelling, we will show that using the automatically labelled test sets can already produce reasonable results.

OOD test sets. To compare the effectiveness robustness of models trained on CIFAR-10 and ImageNet, we use 3 CIFAR-like OOD test sets with natural distribution shifts, including CIFAR10.1 (Recht et al., 2019), CIFAR-10.2 (Lu et al., 2020), and CINIC-10 (Darlow et al., 2018). We use 4 ImageNet-like OOD test sets to compare models trained on ImageNet with models trained on YFCC and LAION: ImageNet-V2 (Recht et al., 2019), ImageNet-R (Hendrycks et al., 2021a), ImageNet-Sketch (Wang et al., 2019), and ObjectNet (Barbu et al., 2019). We do not use ImageNetA (Hendrycks et al., 2021b) which involves adversarial filtering and has a different behavior in effective robustness evaluation (Taori et al., 2020; Fang et al., 2022).

Class subsampling and mapping. Considering that different test sets may not have the same classes, we follow prior works (Taori et al., 2020; Fang et al., 2022) to use class subsampling 2 to

CIFAR-10 Models

ImageNet Models

CIFAR-10+ImageNet Models

90

CIFAR-10.2 acc. (%)

80

50

25

10

CIFAR-10 acc. (%)

10

25

50

70g0

90

95

10

25

ImageNet acc. (mapped, %)

ImageNet Models

ImageNet+YFCC Models

95

06

ImageNet-R acc. (css., %)

50

25

...

10

ImageNet acc. (css., %)

10

25

(%

YFCC acc. (css.

retain classes appearing in all the test sets. We also follow Miller et al. (2021) to map a subset of ImageNet classes to CIFAR-10 classes when comparing CIFAR-10 models and ImageNet models,.

5.2 Evaluation on CIFAR-like OOD Test Sets

10

25

50

70

80

90

95

10

25

50

70

80

90

95

CIFAR-10

ImageNet

CIFAR-10+ImageNet

CIFAR-10 accuracy (%)

CIFAR-10.2 accuracy (%)

10

25

50

70

80

10

25

50

70

80

CIFAR-10

ImageNet

CIFAR-10+ImageNet

ImageNet accuracy (mapped, %)

CIFAR-10.2 accuracy (%)

Figure 4: Projected views of Figure 3a. Figure 4a and Figure 4b correspond to single-ID evaluations using different ID test sets and yield contradictory conclusions on the effective robustness. Our multi-ID evaluation provides a more holistic view where all these models are approximately on a same plane and thus have similar effective robustness.

We first experiment with models trained using CIFAR-10 and ImageNet on CIFAR-like OOD test sets. We show the fitting quality in Table 1a and the effective robustness of various models in Table 1b. Compared to the single-ID evaluation, our multi-ID evaluation achieves a better fitting quality and predicts the OOD accuracy from the ID accuracies more precisely (higher R 2 and lower MAE), and thus provides a more precise understanding on the effective robustness. Specifically, while both single-ID effective robustness and multi-ID effective robustness have relatively high fitting quality on CIFAR-like test sets, using multi-ID effective robustness further improves the fitting quality. In terms of the effective robustness, under the single-ID evaluation, ImageNet models achieve 3.91 ± 2.20 (%) and 2.77 ± 1.25 (%) effective robustness on CIFAR-10.2 and CINIC-10, respectively. The positive

(a) Fitting quality evaluated by R 2 and mean absolute error (MAE).

(b) Effective robustness values (%). We report the mean and standard deviation for three groups of models with different training data, respectively.

effective robustness values seems to suggest an advantage of ImageNet models compared to CIFAR10 models, which is consistent with the findings in Miller et al. (2021). However, under the multi-ID evaluation, the advantage of ImageNet models diminishes, and the effective robustness values of both CIFAR-10 models and ImageNet models are much closer to 0. Therefore, the apparent advantage reported by prior works can be explained as the effect of training data on the single-ID evaluation, and our multi-ID evaluation resolves this confounder to provide a more precise understanding.

In Figure 3a, we visualize the multi-ID effective robustness on CIFAR-10.2, where the accuracies of all the models approximately lie on a plane (the baseline function) on the logit scale, and thus these models have similar effective robustness as the OOD accuracy of all the models can be approximately predicted using a simple plane. We also show projected views of Figure 3a in Figure 4, where Figure 4a and Figure 4b correspond to the single-ID evaluation taking different ID test sets with contradictory conclusions. In contrast, our new evaluation provides a more holistic view.

5.3 Evaluation on ImageNet-like OOD Test Sets

We then conduct evaluation on ImageNet-like OOD test sets, and we compare ImageNet models with models trained on YFCC and LAION, respectively. We show the fitting quality in Table 2 and the effective robustness in Tables 3a and 3b. Consistent with results in Section 5.2, our multi-ID evaluation improves the fitting quality over the single-ID evaluation to better predict and understand the OOD accuracies from ID accuracies. On effective robustness, single-ID evaluation leads to a perception of an effective robustness gain when there is mismatch between the training data and the single ID test set. Our multi-ID evaluation enjoys a holistic view of all the training distributions and suggests that all the models evaluated here have similar effective robustness.

Specifically, the improvement of fitting quality is particularly significant for models involving LAION on ImageNet-R ( R 2 improved from 0.216 to 0.982 and MAE reduced from 9.23% to 1.32%) and ImageNet-Sketch ( R 2 improved from 0.281 to 0.937 and MAE reduced from 7.90% to 2.10%). On the effective robustness values, under the single-ID evaluation, YFCC and LAION models have positive effective robustness values (2.59 ± 2.43 (%) on average for YFCC models and 5.96 ± 4.96 (%) on average for LAION models), which is consistent with the findings in Fang et al. (2022); Nguyen et al. (2022). In contrast, under our multi-ID evaluation, the average effective robustness becomes 0.77 ± 0.85 (%) for YFCC models, and -0.00 ± 0.52 (%) for LAION models, much closer to 0. While single-ID evaluation used by prior works (Fang et al., 2022; Nguyen et al., 2022) suggests effective

(a) Models involving ImageNet and YFCC.

(b) Models involving ImageNet and LAION.

robustness gains of YFCC models compared to ImageNet models (Figure 5a), all the models have similar effective robustness under our multi-ID evaluation. Additionally, we provide an ablation study on using models with mixed training data in Appendix A.3 and additional interactive visualization on our website at https://shizhouxing.github.io/effective-robustness .

5.4 Evaluation on Additional Models

We also evaluate additional models that are not used in fitting the baseline functions. We download models pre-trained by existing works, including OpenCLIP (Ilharco et al., 2021) and SLIP (Mu et al., 2022). OpenCLIP provides CLIP models trained on YFCC and LAION, and SLIP provides YFCC models trained using CLIP and also a combination of self-supervised learning (Chen et al., 2020a,b) and CLIP (SimCLR+CLIP namely SLIP). And we also fine-tune CLIP models on ImageNet for models we pre-train on YFCC and LAION. We use both vanilla fine-tuning and also WiSEFT (Wortsman et al., 2022b) which aims to improve the robustness after fine-tuning, using a weightspace ensembling of the pre-trained model and the fine-tuned model. Details are in Appendix B.1.

In Table 4, we show results involving YFCC and LAION, respectively. Since these models are not used in the fitting, we do not use R 2 , but we use MAE to measure the fitting quality. Our multi-ID evaluation generally reduces MAE compared to the single-ID evaluation, and thus the multi-ID evaluation can still more accurately predict the OOD accuracy from the ID accuracies for these models that are not used in the fitting. The effective robustness values of the models also generally become closer to 0, especially for the zero-shot CLIP models. The results further validate that zero-shot CLIP models, although may achieve high OOD performance if pre-trained with large-scale data (Radford et al., 2021), generally do not improve effective robustness if we control for all the ID accuracies. Among the models evaluated here, SLIP models on YFCC and WiSE-FT models from LAION achieve relatively higher average effective robustness compared to other models, under our multi-ID evaluation, although the gains are not consistently significant on all the test sets and become much smaller than those reflected in the single-ID evaluation. However, we are not certain

on whether SLIP and WiSE-FT can alter the underlying training distribution, because SLIP uses SimCLR that introduces different training images, and WiSE-FT esseentially edits the weights of the models by weight-space ensembling. Thus, we do not draw a definite conclusion for the effective robustness of SLIP and WiSE-FT models and leave further validation for future work.

6 Related Work

For natural distribution shifts, the linear correlations between the ID and OOD performance ('accuracy-on-the-line' (Miller et al., 2021) in the single-ID effective robustness) have earlier been observed in dataset reproduction works (Recht et al., 2018, 2019; Yadav & Bottou, 2019; Miller et al., 2020). Taori et al. (2020) evaluated many ImageNet models on several ImageNet-like OOD test sets, and given the widely held linear correlations, they proposed to evaluate effective robustness by controlling for ID accuracy. Miller et al. (2021) further validated accuracy-on-the-line with a broader scope. Nevertheless, accuracy-on-the-line may not hold on some distribution shifts, such as some corruption shifts (Hendrycks & Dietterich, 2018) and shifts in the wild (Miller et al., 2021), and sometimes ID accuracy and OOD accuracy can inversely correlate (Miller et al., 2021; Teney et al., 2022). Baek et al. (2022) also observed a linear correlation between ID agreement and OOD agreement for a pair of neural networks (namely 'agreement-on-the-line') for testing whether accuracy-on-the-line holds by agreement-on-the-line which does not require labeled data. We focus on distribution shifts where at least accuracy-on-the-line holds for models from each of the training datasets, and we further propose 'accuracy-on-the-plane' using multiple ID test sets.

Recently, CLIP-like models with language image pre-training which has been studied earlier in works such as Sariyildiz et al. (2020); Zhang et al. (2022); Desai & Johnson (2021), were shown to achieve exceptional effective robustness in the single-ID evaluation (Radford et al., 2021; Jia et al., 2021). Fang et al. (2022) analyzed the cause of the effective robustness gain of CLIP and concluded that the pre-training data determined the robustness. Nguyen et al. (2022) experimented on more pre-training data, and they observed difference in the single-ID effective robustness of models trained on different data. While Fang et al. (2022); Nguyen et al. (2022) both suggested that the pre-training data could determine the effective robustness gains, our evaluation suggests that zero-shot CLIP models do not have effective robustness gains. Besides, Kumar et al. (2021); Andreassen et al. (2022); Wortsman

et al. (2022b) studied the robustness of fine-tuned CLIP models. Moreover, Devillers et al. (2021); Santurkar et al. (2022) studied the transfer performance of CLIP models, which is out of our scope on the robustness against natural distribution shifts.

7 Conclusion

To conclude, we propose a new and more precise effective robustness evaluation for models with different training data. In our evaluation, the OOD accuracy can generally be better predicted from multiple ID accuracies compared to previous effective robustness evaluation with a single ID test. We find that zero-shot CLIP models pre-trained on language-image data do not have better effective robustness compared to standard image classifiers, and we provide a new understanding of the apparently significant effective robustness gains observed by prior works.

8 Limitations and Future Work

There remain several limitations that may be addressed in future works:

Our method requires fully knowing the training distributions of all the evaluated models, which is not directly applicable for large-scale pre-trained models with private training data. And this also requires the training methods not to significantly alter the training distribution beyond basic data augmentation, while some methods such as SLIP may alter training distributions more significantly, and it is unclear how we can precisely define the training distribution for a model with post-training processing, such as WiSE-FT (Wortsman et al., 2022b) and Model Soups (Wortsman et al., 2022a) with weight-space ensembling. Future work may study how these techniques may impact the ID performance evaluation (Section 5.4).

We assume that the two ID test sets have relatively close distributions compare to the OOD test sets. We have not considered how the difference between the multiple ID test sets may affect the evaluation, and how the effective robustness should be compared if models are trained on highly different distributions.

We use fixed OOD test sets to evaluate the OOD performance, following previous works (Radford et al., 2021; Fang et al., 2022; Nguyen et al., 2022; Schuhmann et al., 2021). When models are pre-trained on large-scale data, it becomes unclear if these 'OOD' test sets are still OOD, or if these test sets could be less OOD for the pre-trained models compared to standard classifiers. There may also be some distribution overlap between these test sets and the pre-training datasets, even though Radford et al. (2021) mentioned that the likelihood of direct data overlapping is low.

We focus on distribution shifts where at least 'accuracy-on-the-line' from existing works is known to hold for models trained on the same data (Taori et al., 2020; Miller et al., 2021), yet there are counterexamples where 'accuracy-on-the-line' does not hold (Section 6) and requires further study. We may set a threshold on the fitting quality and only adopt our method when the fitting quality is sufficiently good. And there are also other OOD test sets (Singla & Feizi, 2021; Rusak et al., 2021; Singla et al., 2022; Idrissi et al., 2022; Li et al., 2023; Moayeri et al., 2022; Vasudevan et al., 2022) that have not been studied in the effective robustness works yet.

While we mostly focus on comparing CLIP-like models with standard image classifiers, due to the notable OOD robustness of CLIP-like models studied in prior works (Radford et al., 2021; Fang et al., 2022; Nguyen et al., 2022), this work may further be extended to cover other types of models (Goyal et al., 2022; Singh et al., 2022) as well as other modalities such as distribution shifts on language data (Miller et al., 2020; Awadalla et al., 2022).

Our multi-ID evaluation is intended for the scenario with models trained on different data. For models trained on a single dataset, while there is often a correlation between the rankings derived from the single-ID evaluation and the multi-ID evaluation, respectively, the rankings are not necessarily consistent (see Appendix A.2), and thus our multi-ID evaluation is not intended to replace the single-ID evaluation in this case. We suggest using single-ID and multi-ID evaluation comprehensively.

Acknowledgments & Funding Disclosure

We thank Alex Fang and Jindong Gu for helpful discussions and the reviewers for constructive feedback. This work was supported in part by NSF 2008173, 2048280, 2325121, 2331966, ONR N00014-23-1-2300:P00001.

References

Andreassen, A. J., Bahri, Y., Neyshabur, B., and Roelofs, R. The evolution of out-of-distribution robustness throughout fine-tuning. Transactions on Machine Learning Research , 2022.

Awadalla, A., Wortsman, M., Ilharco, G., Min, S., Magnusson, I., Hajishirzi, H., and Schmidt, L. Exploring the landscape of distributional robustness for question answering models. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pp. 5971-5987, 2022.

Baek, C., Jiang, Y., Raghunathan, A., and Kolter, J. Z. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems , 35:19274-19289, 2022.

Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems , 32, 2019.

Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pp. 1597-1607. PMLR, 2020a.

Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33: 22243-22255, 2020b.

Darlow, L. N., Crowley, E. J., Antoniou, A., and Storkey, A. J. Cinic-10 is not imagenet or cifar-10. arXiv preprint arXiv:1810.03505 , 2018.

Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Li, F. Imagenet: A large-scale hierarchical image database. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 248-255, 2009.

Desai, K. and Johnson, J. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 1116211173, 2021.

Devillers, B., Choksi, B., Bielawski, R., and VanRullen, R. Does language help generalization in vision models? In Proceedings of the 25th Conference on Computational Natural Language Learning , pp. 171-182, 2021.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., and Schmidt, L. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning , pp. 6216-6234. PMLR, 2022.

Goyal, P., Duval, Q., Seessel, I., Caron, M., Singh, M., Misra, I., Sagun, L., Joulin, A., and Bojanowski, P. Vision models are more robust and fair when pretrained on uncurated images without supervision. arXiv preprint arXiv:2202.08360 , 2022.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 770-778, 2016. doi: 10.1109/CVPR.2016.90.

Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2018.

Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV , 2021a.

Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 15262-15271, 2021b.

Idrissi, B. Y., Bouchacourt, D., Balestriero, R., Evtimov, I., Hazirbas, C., Ballas, N., Vincent, P., Drozdzal, M., Lopez-Paz, D., and Ibrahim, M. Imagenet-x: Understanding model mistakes with factor of variation annotations. In The Eleventh International Conference on Learning Representations , 2022.

Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip, 2021. URL https://doi.org/10.5281/zenodo.5143773 .

Jia, C., Yang, Y., Xia, Y ., Chen, Y .-T., Parekh, Z., Pham, H., Le, Q., Sung, Y .-H., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , pp. 4904-4916. PMLR, 2021.

Kendall, M. Rank correlation methods. 1948.

Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical Report TR-2009 , 2009.

Kumar, A., Raghunathan, A., Jones, R. M., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations , 2021.

Li, Z., Evtimov, I., Gordo, A., Hazirbas, C., Hassner, T., Ferrer, C. C., Xu, C., and Ibrahim, M. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 20071-20082, 2023.

Lu, S., Nott, B., Olson, A., Todeschini, A., Vahabi, H., Carmon, Y., and Schmidt, L. Harder or different? a closer look at distribution shift in dataset reproduction. In ICML Workshop on Uncertainty and Robustness in Deep Learning , 2020.

Miller, J., Krauth, K., Recht, B., and Schmidt, L. The effect of natural distribution shift on question answering models. In International Conference on Machine Learning , pp. 6905-6916. PMLR, 2020.

Miller, J. P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning , pp. 7721-7735. PMLR, 2021.

Moayeri, M., Singla, S., and Feizi, S. Hard imagenet: Segmentations for objects with strong spurious cues. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Mu, N., Kirillov, A., Wagner, D., and Xie, S. Slip: Self-supervision meets language-image pretraining. In European Conference on Computer Vision , pp. 529-544. Springer, 2022.

Nguyen, T., Ilharco, G., Wortsman, M., Oh, S., and Schmidt, L. Quality not quantity: On the interaction between dataset design and robustness of clip. Advances in Neural Information Processing Systems , 35:21455-21469, 2022.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pp. 8748-8763. PMLR, 2021.

Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451 , 2018.

Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning , pp. 5389-5400. PMLR, 2019.

Rusak, E., Schneider, S., Pachitariu, G., Eck, L., Gehler, P. V., Bringmann, O., Brendel, W., and Bethge, M. If your data distribution shifts, use self-learning. Transactions on Machine Learning Research , 2021.

Santurkar, S., Dubois, Y., Taori, R., Liang, P., and Hashimoto, T. Is a caption worth a thousand images? a controlled study for representation learning. arXiv preprint arXiv:2207.07635 , 2022.

Sariyildiz, M. B., Perez, J., and Larlus, D. Learning visual representations with caption annotations. In European Conference on Computer Vision , pp. 153-170. Springer, 2020.

Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Singh, M., Gustafson, L., Adcock, A., de Freitas Reis, V., Gedik, B., Kosaraju, R. P., Mahajan, D., Girshick, R., Dollár, P., and Van Der Maaten, L. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 804-814, 2022.

Singla, S. and Feizi, S. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Learning Representations , 2021.

Singla, S., Moayeri, M., and Feizi, S. Core risk minimization using salient imagenet. arXiv preprint arXiv:2203.15566 , 2022.

Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems , 33:18583-18599, 2020.

Teney, D., Oh, S. J., and Abbasnejad, E. Id and ood performance are sometimes inversely correlated on real-world datasets. arXiv preprint arXiv:2209.00613 , 2022.

Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., and Li, L.-J. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Vasudevan, V., Caine, B., Gontijo Lopes, R., Fridovich-Keil, S., and Roelofs, R. When does dough become a bagel? analyzing the remaining mistakes on imagenet. Advances in Neural Information Processing Systems , 35:6720-6734, 2022.

Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems , 32, 2019.

Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple finetuned models improves accuracy without increasing inference time. In International Conference on Machine Learning , pp. 23965-23998. PMLR, 2022a.

Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7959-7971, 2022b.

Yadav, C. and Bottou, L. Cold case: The lost mnist digits. Advances in neural information processing systems , 32, 2019.

Zhang, Y., Jiang, H., Miura, Y., Manning, C. D., and Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference , pp. 2-25. PMLR, 2022.

A Additional Results

A.1 Projected Views

In Figure 5, we show projected views of Figure 3b.

10

25

50

70

80

90

95

10

25

50

70

80

90

95

ImageNet

ImageNet+YFCC

ImageNet accuracy (css., %)

ImageNet-R accuracy (css., %)

(a) ImageNet-R accuracy against ImageNet accuracy. YFCC models have higher ImageNet-R accuracy compared to ImageNet models when controlling for ImageNet accuracy only.

10

25

50

10

25

50

ImageNet

ImageNet+YFCC

YFCC accuracy (css., %)

ImageNet-R accuracy (css., %)

A.2 Agreement between Single-ID and Multi-ID Evaluation

We conduct an experiment to check the correlation between the single-ID evaluation and our new multi-ID evaluation, in terms of the relative ranking between different models trained on the same data. We use Kendall's rank correlation test (Kendall, 1948) and we report the τ statistic computed by scipy.stats.kendalltau in Tables 5 to 7. Results show that the rankings on the single-ID effective robustness and multi-ID effective robustness are positively correlated for CIFAR-10 and ImageNet models. There is also a weaker positive correlation for YFCC models. For LAION models, there is sometimes a negative correlation. Overall, while there is often a positive correlation between the rankings provided by the single-ID evaluation and the multi-ID evaluation, respectively, it is not necessarily consistent on all the datasets. Thus, when comparing models trained on the same data, our multi-ID evaluation is not intended to replace the single-ID evaluation. Our multi-ID evaluation is mainly for comparing models trained on different data, and may be used as a supplementary evaluation if all the models are trained on a single dataset.

A.3 Models with Mixed Training Data in the Fitting

As mentioned in Section 5.1, we train models with mixed data to obtain models with diverse accuracies. In Tables 8 and 9, we show that if we do not include models with mixed training data in the fitting, the MAE for these models can become higher, although the difference is not large.

In this work, we train models with data mixed at various ratios and consider models with diverse combinations of accuracies, to obtain more convincing conclusions on the effective robustness.

B Experimental Details

B.1 Details of Models

We use TF-Vision 3 under the Apache License Version 2.0 to train standard classifiers on CIFAR-10 and ImageNet. We follow the configurations provided in TF-Vision for vanilla ResNet training on ImageNet and we train ResNet-18, ResNet-50 and ResNet-101 models. We reuse the configurations to train models on CIFAR-10, where we only change the dataset, number of classes, and image size, without tuning hyperparameters for the training. And we load checkpoints of ViT-S/16, ViT-B/16, and ViT-L/16 models pre-trained on ImageNet, provided by TF-Vision.

For training CLIP models, we mostly follow hyperparameters provided in Fang et al. (2022) and the implementation in Open-CLIP (Ilharco et al., 2021). While Fang et al. (2022) used a batch size of 1024, we use 2048 for more parallelism. We use YFCC-15M in Radford et al. (2021), which is a subset of YFCC-100M (Thomee et al., 2016). And we use LAION-15M which we uniformly sample from LAION-400M (Schuhmann et al., 2021). For fine-tuning CLIP models, we fine-tune for 50,000 steps, using learning rates 3 × 10 -5 and 1 × 10 -4 , respectively. For WiSE-FT, we take α = 0 . 5 which is the coefficient for weight-space ensembling. For OpenCLIP models 4 , we use ViT-B/32 models trained on LAION-400M. For SLIP 5 , we use all the CLIP and SLIP models trained on YFCC-15M.

For data subsampling, we uniformly sample a proportion of training examples from the entire dataset, at ratios of { 5% , 10% , 20% , 30% , 40% , 50% } , respectively. For combining two training datasets at various ratios, given a coefficient λ (0 < λ < 1) , we uniformly sample a proportion of data from the two datasets at ratios of λ and (1 -λ ) , respectively, and then we combine the two subsets. When combining ImageNet and CIFAR-10, we take λ ∈ { 0 . 001 , 0 . 01 , 0 . 1 , 0 . 5 , 0 . 9 , 0 . 99 , 0 . 995 } ; when combining ImageNet with YFCC and LAION, respectively, we take λ ∈ { 0 . 01 , 0 . 1 , 0 . 25 , 0 . 5 } .

Models are trained using 4 × 4 or 4 × 8 TPU v2 Pods, and they are evaluated using NVIDIA V100 GPUs, on the cloud.

B.2 Details of ID Test Sets

We construct an ID test set from YFCC-15M and LAION-15M, respectively, and we automatically generate classification labels by matching text with ImageNet class names, which has also been similarly performed in Fang et al. (2022) for training classifiers using caption data. On YFCC-15M which contains metadata, we use tags for matching. On LAION-15M which does not provide metadata such as tags, we simply use the entire text for matching. We adopt a label only if a unique ImageNet class can be determined by matching for the involved image. We then construct a balanced and labelled test set by keeping 50 examples for each class that has at least 100 examples in the labelled images. The test examples are then held out from the training data. For the YFCC test set, there are 22550 examples for 451 classes; and for the LAION test set, there are 20400 examples 408 classes.

B.3 Licenses of the Datasets

We have used the following datasets:

CIFAR-10 (Krizhevsky et al., 2009). License is not clearly known.

YFCC 6 under various Creative Commons licenses.

LAION 7 under the Creative Common CC-BY 4.0 license for the metadata, and the images are under the copyright of the original authors.

CIFAR-10.1 8 under the MIT license.

CIFAR-10.2 9 . License is not clearly known.

CINIC-10 10 under the MIT license.

ImageNet-V2 11 under the MIT license.

ImageNet-R 12 under the MIT license.

ImageNet-Sketch 13 under the MIT license.

ObjectNet 14 with a license provided on the webpage.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a modification to effective robustness evaluations that takes multiple training sets into account. This facilitates a more precise comparison of the effective robustness of models trained on different training sets. The authors demonstrate empirically that their method yields an improved predictor of OOD accuracy. The analysis of models trained on ImageNet, YFCC, and LAION reveals that CLIP models might not, in general, exhibit larger effective robustness than ResNets, further strengthening the existing empirical evidence in this direction.\n\nSTRENGTHS:\nThe paper is well written and easy to follow. In more detail:\n- The introduction is very well written and motivates the ideas clearly. The problem targeted in the paper makes sense. \n- The motivating experiment in Section 3 is very nice and illustrates the limitation of using a single ID test set very well and the provided intuition in line 104 makes sense (“We observe that when there is a mismatch in the training data and the ID test data, the models appear to have greater effective robustness (YFCC models in Figure 1a and ImageNet models in Figure 1b), as their performance on the ID test data and the OOD performance predicted from the single ID accuracy tend to be lower.”)\n\n\nThe issue that the regular definition of effective robustness is confounded by the training distribution makes sense and makes it difficult to compare the performance of models trained on different datasets. The idea of doing regression on the accuracies of multiple ID test sets is simple and seems to work well. The conclusion that CLIP models do, in general, not exhibit greater effective robustness is important and relevant.\n\nWEAKNESSES:\nWhile the paper is generally well written, there is an abundance of typos and grammar errors (see examples below), which need fixing before acceptance. I would strongly suggest to the authors to do a round of proof-reading the manuscript. \n\n\nL40: “In particular, previous works performed single-dimensional linear regression on a set of baseline models to predict OOD accuracy from a single ID accuracy.” citations needed\n\n\nLine 58: “Under natural distribution shifts, the OOD accuracy of a model is often correlated with the ID accuracy.” citation needed\n\n\nIn Fig.1, does “ImageNet models” refer to both standard ImageNet classifiers and CLIP models trained on ImageNet? If so, isn’t it quite remarkable that CLIP models and standard classifiers lie on the exact same line? In general, it is unclear in this figure which of the models from section 5 are depicted here.\n\n\nL128: Table 1 is a Figure not a Table\n\n\nL181: “For datasets that consist of image-text pairs for language-image pre-training without original labels, including YFCC and LAION, we automatically generate classification labels by matching captions with ImageNet classes, and we then hold out a balanced test set from the original dataset.” Please provide examples of YFCC and LAION images+captions which were accepted into the ID test sets, along with their new ImageNet label in Appendix B.2.\n\n\n\n#### Clarity:\n\nOverall, the 3d plane plots are unhelpful in gauging whether points lie close to the plane or not. In addition to the provided 2d projections _along the ID accuracy axes_ a 2d projection orthogonal to the fitted plane could be helpful to visually judge the model fit.\n\n\nL121: the sentence in line 121 is a bit hard to parse, would suggest rephrasing for clarity.\n\n\nL133: the sentence in line 133 is a bit hard to parse, would suggest rephrasing for clarity.\n\n\nL260-261: it is unclear that \"SLIP models ... achieve higher average effective robustness\" _compared to other models_, rather than _compared to single-ID setting_\n\n\nCan the used OOD sets really be considered OOD sets for all training distributions? LAION, for example, contains numerous renditions and sketches. As a result, the measured effective robustness might not be a true “OOD robustness”. A remark on this terminology could be helpful. \n\n\n#### Grammar / typos:\n\nAbstract, typo: “Our new evaluation metric provides a better estimate of the effectiveness robustness”\n\n\nL11: “effective~~ness~~ robustness”\n\n\nL18: “following ~~on~~”\n\n\nL22: “Unfortunately,” comma missing\n\n\nL63: “are … resulted from” is grammatically incorrect\n\n\nL78: “Limitation **of** the Single ID Test Set”\n\n\n\nL80: “reflective **of**”\n\n\n\nL81: “reflective **of**”\n\n\nL81: “However, as the emergence of large-scale pre-trained models,” grammar, maybe “due to” instead of “as”?\n\n\nL85: “when only one … **is** used”\n\n\nL85: “--” should be “:”, I think\n\n\nL88, 114, 117, 123, 172, 224, Table 3, Caption/L279: comma missing before respectively. Please add commas before all instances of “respectively”. In addition, if there is no full-stop after “respectively”, a comma should be added after this word as well (e.g., Table 3, caption).\n\n\nL110: “test set”, set is missing\n\n\nL121, L265: comma after thus missing\n\n\nL148: “**generalized**”, “in **principle**”\n\n\nL166: rephrase to “and subsample the combined datasets”\n\n\nL195: should “retrain” be “retain”?\n\n\nL258: “although **they** may”\n\n\nL290: “models” should be plural\n\n\nL315: may further be\n\n\nL503: “**held** out”\n\n\nL515: “**under**”\n\n\nL519 “**clearer** understanding **of**”\n\nQUESTIONS:\nI am surprised how well models trained on CIFAR10 perform on ImageNet (x-axis in Figure 3b, blue dots). Was the size of the CIFAR10 images kept as usual or upscaled to 224x224 to make it more comparable to ImageNet trained models? I am surprised that training on 32x32 CIFAR10 images gives such high accuracy numbers on ImageNet, even after class-subsampling. Intuitively, I would expect the accuracy on CIFAR-10.2 to be much higher than on ImageNet, of a model trained on CIFAR10 since the distribution of CIFAR-10.2 should be closer to CIFAR10 than ImageNet. But the accuracies on CIFAR-10.2 and ImageNet are almost the same; the effect is much smaller than I would have expected. Could you maybe comment on this finding? Do you find it surprising / have any intuition or explanation for it?\n\n\nTable 3 vs Table 2: In Table 2, the R² coefficients were all above 99% while in Table 3, most numbers are generally below 99%. Why do you think that is? For ObjectNet, the R² values for both YFCC and LAION are the lowest ones among all datasets. What makes ObjectNet different from the other datasets and why does the technique work worse for this dataset? Is there an interpretation of what a higher R² value means for the different training datasets, maybe in terms of similarity to the test set? E.g., on ObjectNet, the R² is higher when training on YFCC compared to LAION. Can anything interesting be deduced from this?\n\n\nFollowing up on the previous point: Can there be a situation when your method does not work and could this be measured with the R² fitting score? Is it possible to pin it down to a threshold value when the technique should not be used? E.g. R² < 95% or R² < 90%? \n\n\nAdditionally, it is unclear to what extent the method relies on the ID sets to be similar. Do you have an intuition whether this approach would work at all for models trained on datasets that could be considered “OOD sets” of each other (e.g. a model trained on natural images vs. a model trained on sketches)? Related to this, how important was it for the fit of the baseline that models trained on a mixture of the ID sets were available? An ablation of this point would be interesting, as it might be prohibitively expensive in practice to retrain models on multiple mixtures of the ID sets simply to facilitate a comparison.\n\n\nIt seems to me as if the data would have been available to additionally facilitate a 3-way multi-ID evaluation of some models on the ID sets ImageNet, LAION, and YFCC. Could you elaborate why this was not attempted? I believe it would greatly strengthen the claim that this method should natively extend to more than 2 ID sets.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work delves into the definition of effective invariance, which is proposed to measure how the model performs beyond what we can expect from its in-distribution accuracy. This work points out that the training distribution should be considered when measuring the model's effective robustness. Specifically, models can be trained on different and diverse distributions. Then, the usage of the \"in-distribution\" test set should be changed accordingly. The new evaluation metric provides a more accurate characterization of CLIP models' robustness.\n\nSTRENGTHS:\n+ [***The motivation of re-define the effective invariance is sound***]  Large-scale models trained on various datasets make it necessary to evaluate and compare models trained on different data distributions. Namely, the in-distribution dataset should be well-aligned with the training distribution. \n\n+ [***Extending the idea of effective invariance to multiple ID test sets is sound*** ] Using multiple ID test sets to check the relationship between ID accuracy and OOD accuracy is reasonable and straightforward. \n\n+ [***Good clarify***] This work is well-written and easy to read. The motivation, background knowledge of effective invariance, and the idea of multiple ID test sets are well presented.\n\nWEAKNESSES:\n- [***Fitting quality is not very convincing***] In experiments, fit quality is mainly used to show that using multiple ID test sets can lead to a more accurate match between ID and OOD accuracy. If our goal is to produce more accurate fits, we can use one ID test set and one OOD test set to fit another OOD test set. Please comment on why you should care about the fitting quality.\n\n- [***Discussing ID test sets for models pre-trained on more data***] The motivation for using multiple ID test sets is clear. However, for models pre-trained with more data and finetuned on ImageNet, what is their ID test set?\n\n- [***Clarification of CIFAR model***] CIFAR images cost $32\\times32$, while ImgeNet images are much higher resolution. How to combine them for training? Upsampling or downsampling?\n\n- [***Please summarize observations***] It is unclear what the main observations and new insights are given by the multiple ID sets. It is clear that \"the effective robustness of a zero-shot CLIP model trained on language-image data is similar to that of a standard ImageNet model\". What other discoveries have been made besides this? For example, does pretraining on more data still contribute to effective invariance?\n\nQUESTIONS:\nPlease comment on the choice of OOD test set:  CLIP models are trained on diverse and large-scale training sets (e.g., LIAON). Then, the test set may already be covered by the training distribution. Namely, a test set is OOD for ImageNet models but may be so OOD for CLIP models. In such a case, the effective invariance may truly reflect model behaviors under distribution shifts.\n\nLIMITATIONS:\nThis work includes a paragraph to discuss the limitation and potential future directions, where the access to training distribution and determination of training distribution for some models are discussed. One more point to discuss: training distributions of CLIP models may already cover test distribution, then the definition of OOD needs to be re-considered.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors identify that the definition of effective robustness is ill-equipped for comparing models trained on a variety of distributions; in particular, it only uses a single data distribution to predict OOD accuracy $acc_{OOD}$. The authors propose a more precise effective robustness measurement for models trained on two different datasets ($data_1^{train}$ and $data_2^{train}$), which is based on using regression to optimize the parameters $b,w_1, w_2$ in the following equation: $acc_{OOD}(model_i) = b + w_1 acc_{data_1^{test}}(model_i) + w_2 acc_{data_2^{test}}(model_i),$ where accuracies are given in logit space. Thus, if we wish to compare the effective robustnesses of models $A$ and $B$ that were trained on $data_1^{train}$ and $data_2^{train}$, respectively, we can use this new approach to predicting $acc_{OOD}$ values without being forced to use just one of the two data distributions ($data_1$ or $data_2$) as the distribution we use to predict $acc_{OOD}$. The benefit of this approach, across a wide variety of contexts/models, is that it fits the data better and gives lower effective robustness measurements for models that appeared effectively robust under the old definition because they were trained on a dataset from a distribution different from the one used to predict $acc_{OOD}$.\n\nSTRENGTHS:\nOriginality:\n\n1. The submission's effective robustness (ER) definition is a simple and novel extension of the original definition. \n\nQuality:\n\n2. The first three sections of the paper are strong. In particular, Section 3 is a very convincing and interesting illustration of a problem with our current definition of effective robustness.\n\n3. The submission's approach (described in Section 4) provides interesting results that may be helpful for understanding when a model is effectively robust. The models and datasets used are comprehensive.\n\nClarity:\n\n4. Largely, the paper is well written and organized. \n\nSignificance\n\n5. This work offers an important insight about evaluation protocol in an active area of research.\n\nWEAKNESSES:\nMy main concern is that, despite superficial similarity in their definitions, the interpretations of single- and multi-ID ER are drastically different. Accordingly, I am concerned about whether the new ER definition (multi-ID ER) will have practical utility for researchers interested in identifying models with ER. I expand on these concerns below and in the \"Questions\" section.\n\n1. My (and seemingly the authors') understanding of the purpose of effective robustness is that it identifies models that surpass the OOD accuracy we expect of them *based on their ID accuracy* (i.e., we want to know if training on ID data using some new technique produces more robustness than expected). In Section 3, this submission clarifies that ER does not serve this purpose when we fix the \"ID\" dataset in our definition of ER to be $data_1^{test}$, then use that ER to compare models trained on ID data ($data_1^{train}$) with models trained on OOD data ($data_2^{train}$).  However, this submission's solution also seems to lose this original purpose: models with high multi-ID ER are those with higher-than-expected $acc_{OOD}$ when predictions are made using a combination of an **OOD accuracy** (for a model trained on $data_1^{train}$, this would be $acc_{data_2^{test}}$) and an ID accuracy (for a model trained on $data_1^{train}$, this would be $acc_{data_1^{test}}$). In other words, this metric's use of an OOD accuracy (e.g., $acc_{data_2^{test}}$ for the model trained on $data_1^{train}$) to predict an OOD accuracy (i.e., $acc_{OOD}$) seems to preclude its interpretation as an ER measurement and harm usefulness/interpretability. For example, because we would expect OOD accuracies to be correlated, it seems that the calculation of multi-ID ER can lower the detected ER in a model that truly has ER: if a model has high OOD accuracy (e.g., $acc_{data_2^{test}}$ for a model trained on $data_1^{train}$), that can inflate multi-ID ER's prediction of $acc_{OOD}$, which lowers the detected ER of the model.\n\nQUESTIONS:\n1. Many of the results are intuitive/interesting, but I'm not sure how they are aligned with the notion of predicting OOD performance from an ID dataset (i.e., \"effective robustness\"). My understanding of the proposed definition of multi-ID effective robustness is that it involves predicting OOD performance using an ID *and* an OOD performance, rather than using two ID performances as the proposed name (\"multi-ID\" ER) suggests. I think the submission (and my score) could be significantly improved if the authors can clarify why multi-ID ER can be interpreted as an ER measurement without issues.\n\n2. While the original ER definition is limited in ways (including a way identified by this submission), it also has clear practical utility in that I can understand whether a model outperforms on OOD data relative to models trained on the same ID data. The practical utility of this submission's proposed alternative ER definition is unclear to me. For instance, given a model trained on $data_1$, is it possible to get a different understanding of its multi-ID ER simply by changing the second dataset used in the ER definition (e.g., from YFCC to LAION)? If so, how many dataset pairs ($data_1$, $data_i$) do I need to check my model on before I understand its expected OOD performance (or its ER)? \n\n3. Line 264: Why would WiSE-FT alter the underlying training distribution in a way that makes it unsuitable for multi-ID ER? It seems like its train distribution would be similar to that of the models that are trained on dataset mixtures.\n\n4. Why not report the R2 in some tables (e.g., Table 5)?\n\n5. Table 1 should be labeled as a Figure.\n\n6. It might be nice to see the plane equations on the plane plots.\n\nLIMITATIONS:\nPotential limitations are well addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends the work \"Measuring Robustness to Natural Distribution Shifts in Image Classification\" to more than one ID test set scenario. This paper proposes a robustness evaluation metric that uses more ID test sets covering the evaluated models' training distributions.\n\nSTRENGTHS:\n1. The problem of Effective Robustness is important.\n\n2. The paper's findings are interesting: vision-language model evaluation needs future explored given the unique training setting.\n\nWEAKNESSES:\n1 **The motivation is not sound**. Intuitively, the effective robustness evaluation is ID training dataset driven. If we want to compare different model architectures, comparing the domain shift OOD performance of different models trained on the same training dataset is meaningful. In that case, using a single ID test set is enough. \n\nFor models trained on large-scale image-language pair datasets (CLIP pretrained on LAION), it is unclear how to define a domain-shift OOD evaluation: how to find a \"domain shift\" OOD dataset of LAION? If the model is pertained on LAION and then finetuned on ImageNet, how to guarantee ImageNet-v2 is not overlapped with LAION? The accuracy of ImageNet-v2 itself is meaningful. Comparing different vision-language models that both trained on LAION is also meaningful. However, it is hard to understand why we must compare two models trained on different training sets. Which expected conclusion are you trying to get? The conflicting findings and conclusion on L95~L108 do not make sense if the previous motivation is unclear.\n\nUsually, if we want to see the OOD performance of vision-language models, we show a list of performances on different test sets. \n\n2 **The Figure-1 and conclusion are confusing** In Figure1, how many models are you using as ImageNet models and YFCC models? Why do you want to compare these two groups of models? What is your expected possibility of conclusion? \nAlso, in Figure1(a), the ImageNet accuracy overlap between the two groups is limited, so it is hard to directly compare the two fitted lines.\n\n3 **Limited Novelty** If we skip the motivation problem and focus on the proposed method, it is straightforward to extend equation 1 to equation 3 if we want to consider two ID test sets.\n\n4 **Generalization to more than two ID tests** The paper only conducted experiments on two ID test sets, but claimed the effectiveness on multiple ID test sets. It is not straightforward to extend to more datasets. For instance, how to build the sampling classes in Section 5.2? How to guarantee the sampled overlap class is good enough to evaluate\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}