The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
Authors: Anders Johan Andreassen
Yasaman Bahri
Behnam Neyshabur
Rebecca Roelofs
Venue: Accepted by TMLR ()

Abstract: Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across a testbed of models. 
Models that are more accurate on the out-of-distribution data relative to this baseline exhibit “effective robustness” and are exceedingly rare. 
Identifying such models, and understanding their properties, is key to improving out-of-distribution performance. 
We conduct a thorough empirical investigation of effective robustness during fine-tuning and surprisingly find that models pre-trained on larger datasets exhibit  effective robustness during training that vanishes at convergence. 
We study how properties of the data influence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difficulty of the dataset. 
We also find that models that display effective robustness are able to correctly classify 10\% of the examples that no other current testbed model gets correct. 
Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.

Published in Transactions on Machine Learning Research (09/2022)
The Evolution of Out-of-Distriuttion R Through- out Fine- Tuning
Anders Andreassen
sjendraasen@@goggee.com
Google Research
Yasaman Bahri
yasamanbikggoolle.cm
Google Research
Behnam Neyshabur
neyshabur@ggoogle.com
Google Research
Rebecca Roelofs Google Research
rofls@google.com
Reviewed on OpenReview: https: opw.reveee.....e../mm net/ forum? id- Qs3EfpieOh
Abstract
Although machine learning models typically experience a drop in performance on out-of- distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across testbed of models. Models that are more accurate the out-of-distribution data relative this baseline exhibit "effective robustness" and are exceedingly rare. Identifying such models, and understanding their properties, is key performance. We study datasets with naturally occurring
to improving distribution shifts, and we conduct thorough empirical investigation of effective robustness during fine-tuning and surprisingly find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We study how properties of the data influence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difficulty of the dataset. We also find that models that display effective robustness are able to correctly classify 10% of the examples that other current testbed model gets correct. Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.
1 Introduction
The ability to generalize to data not seen during training is essential for the widespread trust and adoption of machine learning models. In practical applications machine learning, we typically train and fine-tune on a dataset which are in-distribution (ID) with each other, but when deployed the model will face shifts from this distribution, and empirically the vast majority of models show a significant drop performance from ID data to out-of-disttribttion (OOD) data (Quiñonero- -Quiienero-Candelaa et al., 2009 Recht al. 2019 Biggio & Roli 2018 Hendrycks & Dietterich 2019 Azulay & Weiss 2018 Shankar al. 2019 Gu al.
Szegedy et al. 2013
2019 Torralba & Efros 2011p. Common examples include data captured in different environment, like time
of day or geographical location (Koh et al.]
2020); noise or small corruptions of the input data (Hendrycks & Dieuterich 2019 Geirhos el. al. (018); or ad adversarial examples created to explicitly fool neural networks into
making incorrect predictions SSzegddy et al. 2013 Biggio & Roli 2018).
Although the performance gap on OOD shifts is pervasive, it follows an intriguing pattern: there clear lnner relotionship between model's final performance ID and 0OD data Taori et al. 2020 Recht et al.

---

Published in Transactions on Machine Learning Research (09/2022)
Oat of Dittrimm monioo Ace (%)
Sae
Ellontive pollinntttnnesss (%))
Image NotV (top-1
Effective Rooustness
90
CUP
Suar Assan La
36-
Nos Siali
20
Dit. A7Tttaake ky
CLDP krow Peide
5G In- Distribattio: Test Ace (98)
45
10
60
ImageeNet (top-1. X]
ImaggNett (sop- 80
(a) Illustrating the definition of effective robustness (ER)
(b) Accuracies on ImageNet and ImageNetV2
(c) ImageNet accuracies vs, ER on Ima- geNet
Figure 1: During fine-tuning on ImageNet, pre-trained models exhibit effective robustness (ER) while randomly-initialicn models do not. However, the ER of pre-trained models diminishes throughout the fine-tuning process and vanishes altogether at the end of training.
(a) illustrates the definition of ER that we define more precisely Section 3 Subfigure (b) and (c) show ImageNet test accuracy on the x-axis for either randomly initialized or pre-trained models throughout ImageNet training (we evaluate accuracy on checkpoints every epoch of training). (b) plots the Image test accuracy on the y-axis while (c) shows the effective robustness for the same models. ER can also be achieved by zero-shot models. In (b) and (c) we also show BiT model that was trained on JFT and evaluated on ImageNet using a class-map bet ween the two sets of targets (see Appendix C.7). and we show the best performing zero-shot and linear probe CLIP models from Radford et al. (2021a). The BiT and CLIP zero-shot models have comparable ER, but at different ID accuracy, while the linear probe CLIP model has no ER. Standard testbed models (see Section fully trained on ImageNet are in blue and are shown with 95% Clopper-Peane Pearson confidence intervals.
Yadav & Bottou 2019 Miller et al. 2020). In other words, given model's performance on an ID test set, a linear fit can accurately predict what the performance drop will be on the OOD test set. This also implies that a certain amount of improvement on OOD accuracy can be explained by an improvement on ID accuracy. The linear relationship holds across a wide range of models and is well-established for several robustness benchmarks, including image classification on both synthetic and natural distribution shifts (Recht et al. 2018 Yadav & Bottoul 2019 Recht et al. [001 Taori et al., 2020). 2D object detection (Shankar et al. 2019), and question-answer models in NLP (Miller et al. 2020). Since even the nighest-performing models will still have a gap between ID and OOD accuracy, the lincar relationship revcals that our current methods are insufficient for addressing OOD shift.
2019:
Models which lie above the linear fit are said to exhibit effective robustness (ER) (Taori et al. 2020 which measures the model's OOD accuracy relative to the fit (see Figure [[a] for an illustration). Models with nonzero ER deviate in their OOD behavior in a manner that qualitatively and quantitatively different. from what we can currently achieve. We seek to thoroughly investigate when the pre-training and fine-tuning paradigm gives rise to models with nonzero ER, and understand factors that control this behaviour.
Models with high ER (> 1%) are exceedingly rare. First, the most notable example is the recently proposed zero-shot CLIP model (Radford et al. 2021b). and subsequent work (Wortsman et al. 2021 2022 has found that weight-space ensembling of zero-shot and fine-tuned models can improve both ID and OOD accuracy and produce models with high ER. Second, only a handful of the 204 ImageNet models evaluated in Taori ct al. (2020) were found to have non-zero ER. Though most of these effectively robust models identified so lar were pre-trained on large datasets, the majority of models pre-trained this way exhibit no ER, and we currently do not know when additional data helps; what effect the choice of architecture has; well as the effects of dataset and distribution shift on these findings.
In this work, we present an empirical study of the evolution of ER throughout fine-tuning on naturally occurring distribution shifts. By studying the evolution, we find intriguing properties that are missed by focusing only on models at convergence: pre-trained models exhibit ER, while handomly-intiialin models do not (see Figure D.
Summary of Contributions:
2

---

Published in Transactions on Machine Learning Research (09/2022)
We identify pre-trained, fine-tuned models entire class of effectively robust models (that match the ER of CLIP (Radford et al. 2021a)) and investigate how details such model size, dataset size, and example difficulty influence ER. We find that the vanishing of ER at convergence depends on the distribution shift, and on some datasets pre-trained, fine-tuned models may still exhibit non-zero ER convergence.
(Section 5.1 5.3
We analyze properties of pre-trained models at the peak of their ER, including particular metrics defined in previous theorctical work (Mania & Sra 2020). We find that effectively robust models make remarkably dissimilar predictions compared to slandard models, and are able to correctly classify 10% of the examples that no other model gets correct.
(Section 5.4)
(Section 5.5) We find that pre-trained models gradually lose their ER throughout fino-tuning, oven as both the ID and OOD accuracies of the model simultaneously increase. We discuss several potential solutions to mitigate this problem, but find that none of them are able to maintain high ER high ID accuracy.
2
Related Work
Here we review some prior findings that are key for understanding the context of our results.
Linear trends under distribution shift. In recent replication efforts, researchers carefully recreated new test. sets for the CIFAR-10 and ImageNet classification benchmarks (Recht et al., 2018 2019). Despite following the original collection procedures as closely as possible, some shift was introduced. Evaluating an extensive testbed of trained models on the original and new tests revealed a clear relationship between accuracy on the original and new lests that is well-captured by linear fit. with positive slope. Follow-up work also found linear trends for new test sets MNIST (Yadav & Bottou 2019 and the SQUAD question-answer dataset (Miller et al. 2020), geographic distribution shifts for 3D Object Detection in self-driving cars (Sun ct al. 2019), and synthetic distribution shifts (Tori et al. 2020). We find that models pre-trained on larger and more diverse datasets can break the linear trend of accuracy under distribution shift in the middle of the fine-tuning process, and we investigate factors that affect this.
Examples of effectively robust models. Models that lie off the linear trend (effectively robust models) are historically extremely rare. In recent extensive empirical study, researchers evaluated 204 trained ImageNet models spanning wide range of architectures and training techniques on several popular natural and synthelic robustness benchmarks for ImageNet. (Taori et al. 2020). Across several natural robustness benchmarks, the main outliers with positive ER and high accuracy on ImageNet were all models that were pre-trained on larger and more diverse data than the ImageNet training set. They included a ResNet152 model trained on 11,000 NetgeNet classes (Wul 2016), several ResNeXt models trained on billion images from Instagram (Mahajan et al., 2018), and the Neft (NoisyStudent) model trained on a Google-internal JFTT-300M dataset of 300 million images (Xie et al. 2020). However, not all models that were pre-trained on larger datasets showed effective robustness.
Another recent work, RRdford et al. 2021a), observed that zero-shot CLIP classifiers have larger ER than CLIP models that had been fine-tuned to the downstream task. While the CLIP model was pre-trained with a contrastive loss that combined components from natural language processing and image classification, our results show that pre-trained ImageNet models that are evaluated in a zero-shot manner also have high ER, and, similar to the CLIP model, these image classification models lose their ER once they are finc-tuned the downstream task. Since the completion of this paper, subsequent work (Wortsman et al., 2021) has proposed weight-space ensembles for fine-tuning (WiSE-FT) that interpolates between zero-shot and fine-tuned weights to find models with high ER as well as high ID and OOD accuracy. This method is currently limited to CLIP-style models, and in this work we mostly focus on traditional supervised image classification models.
Model similarity. Recent work offers a theoretical explanation, relying on an assumption of model similarity MMani & Sra 2020). for why classifier accuracies follow a lincar trend under distribution shift. Empirically, Mania et al. (0019) observed that the labeling assignments across wide range of ImageNet models are significantly more similar to each other than would be expected by chance, which suggests that the size
3

---

Published in Transactions on Machine Learning Research (09/2022)
CIFAR-10 Eost 0000 (55)
Torlec
Tociel Middl
CIEAR- 10.100111 too nax (30
Figure 2: Testbed model accura- cies (Recht et al,, 2018) show a linear relationship in rescaled logit-space (Taori et al., 2020). (a) shows the linear fit in logit-space, and (b) shows the same fit transformed back to linear-space.
80
00
CIEAR-10 test arc (80)
CIPAR-10 10 test acn (%)
of the class of function approximators that neural networks learn is smaller than what may be expected a priori. Under the assumption of model similarity, Mania et al. (2019) proves that models' accuracies, when evaluated on two distributions, must be approximately collinear, unless the size of the distribution shift is large in a certain sense. In our work, we identify and analyze models that are not collinear with other models when evaluated on two distributions, and we find that such models break the assumption model similarity sufficient to prove the existence of the linear fit.
3 Effective Robustness
The linear fit to testbed model accuracies on ID and OOD data provides baseline for how improvements in OOD accuracy are tied to improvements in ID accuracy based on current methods. Effective robustness (ER), a metric for robustness proposed in Taori et al. 2020)) measures the difference between model's OOD accuracy and that predicted from the baseline. (See Figure ((a) for an illustration.)
To define ER, we fix an ID and OOD lest set and let B(x) be the baseline predicted OOD test set accuracy for given accuracy on the ID test set. Given a set of models evaluated on both the ID and OOD test sets, I can be computed by performing log-linear fit between the models' accuracies the two test scts. Similar to Taori et al. 22020), empirically find that the best linear fit comes from rescaling the accuracies using the logit function, logit(a) = log (1=)). before performing the lincar fit. In Figure 2 we show the log-linear for CIFAR-10 vs. CIFAR-10.1 in logit-space as well as transformed back to linear-space. See Appendix A for more details.
The effective robustness of model f is then defined
p() acCout (J) - B(acc
(1)
where (f) and accout()) (f) represent the model's accuracics on the ID and OOD test sets, respectively. Graphically, ER represents the distance above the testbed line to model's OOD accuracy, shown in Figure ((a)
We note that ER is distinct from the absolute accuracy model can achieve the OOD data. For instance, the effectively robust model (green point) in Figure I(a) still has low absolute OOD accuracy relative to other models. An ideal model achieves both high ER and high absolute OOD accuracy.
In this work, we consider any observed ER that is higher than the the 95% Clopper Pearson confidence intervals for the models in our testbed (see Section { to be significant. To make the significance explicit, we have included these confidence intervals to all the plots throughout the paper.
4 Experimental setup
In this section, we discuss which fine-tuning datasets, pre-trained models, robustness benchmarks, and linear fits we use in our experimental evaluation.
Fine-tuning datasets. We evaluate ER throughout fine-tuning (see Appendix B.2 for our definition of fine-tuning) on both ImageNet and CIFAR-10 since rescarchers commonly fine-tune pre-trained models to these popular benchmarks. For CIFAR-10 particular, there are several widely available pre-trained ImageNet models that we easily transfer to the CIFAR-10 dataset.
4

---

Published in Transactions on Machine Learning Research (09/2022)
Pre-trained models. The majority of the pre-trained models study are Big Transfer (BiT) models Kolesnikov et al. 2020), which use ResNet-v2 architecture (e et al. 2016b). with Group Normalization (Wu & He 2018) and Weight Standardization (Qiao et al. 2019). The BiT models come in three different flavors: BiT-S, BiT-M and BiT-L, where S, M and L indicate if the pre-training was done on ILSVRC-2012 (which we denote as ImageNet-Tk or just ImageNet) (Russakovsky
al.) 2015). ImageNNe-221k (Deng el al. 2009), or JFT (Sun et al. 2017); respectively. In addition, these models may be further fine-tuned, and 'Ik', 21k' or 'JFTT indicate the dataset it was fine-tuned on. For example, BiT-M-R152x4-Ik is the ResNet-15224 model that was pre-trained on ImageNet-21k ("M') and fine-tuned on ImageNet-1k ("1k"). We also study series of BiT models that were pre-trained in Djolonga et al. (2021) on different amounts of data from JFT.
We also study six ImageNet pre-trained PYTORCII (Paszke et al. 2019 models: AlexNet (Krizhevsky 2014 ResNet-18, -50 and -122 (He et al. 2016a). VGG-11 with Batch Normalization Simonyan Zisserman 2014), and WidcResNet-50022 (Zagoruyko ko Komodakis 2016
Robustness benchmarks. Both CIFAR-10 and ImageNet have several well-established robustness bench- marks. We choose to focus on naturally occurring distribution shifts, rather than synthetic shifts which modify images, because they are more realistic and because there are several robustness interventions that work well on synthetic shifts but do not transfer to natural distribution shifts Taori et al. (2020). Thus, for CIFAR-10, we evaluate robustness on CIFAR-10.1, which was created in the replication study of Recht et al. 2018 and is one of the few natural distribution shift benchmarks available for it; and for ImageNet, we use the ImageNetV2 (Recht et al. 2019), ObjectNeet (Barbu et al. 2019), and ImageNet-R (Hendrycks et al., 2020 test sets.
Testbed models and lincar fits. Mcasuring ER requires establishing lincar relationship between model's accuracy ID and OOD data. To define the linear relationship wet the original test accuracy and the OOD test accuracy on CIFAR-10 and ImageNet, we require a testbed of models that span the range from low to high accuracy. For ImageNet, we use the publicly available testbed from Recht et al. (2019), and for CIFAR-10 we use the testbed from Recht et al. 2018
5 Results
We overview our main experimental results. First, we establish that pre-trained models exhibit ER during fine-tuning, and we examine how properties of the pre-training setup, such as model size, datasel size, and example difficulty, influence the magnitude of the ER. Finally, we explore ideas that attempt to maintain high ER at the end of fino-tuning.
Table 1: The range of experiments we run are represented with the cross product within each row of the models, OOD dataset and pre-training dataset listed below.
Model
OOD
Pre-training dataset
AlexNet, Res,et-118,50, -152, VGG11 BN, Wede-RessNet-50-2
CIFAR-10.1
ImageNet-Ik
CIFAR-10.1
ImageNet-lk
ImageNet V2
ImageNet-21k, JFT
BIT-R101x3
ObjectNet, ImageNet-R
JFT
5

---

Published in Transactions on Machine Learning Research (09/2022)
Badeadl Redkace
VOGEL Wid- NecTie
Zie Tealei Wilkh
R.G Kitted
CUP Zoe Shet
R.T. JFT
CLIP Loves Pade
Effective Rolrust (%6
Baaken kit. Mia
Torted bholts
Eifletive Robnutnss
00
90
40
60
60
CIPAR-10 (%6
InnageNet (top-1, %)
(a) CIFAR-10.1
(b) ImageNet
Image-Ntt R
ObjertNet
Eliectio Robustass (96)
R.T. RI
Ellecdiv Robustuss (96)
BeT iittos a01 Thuthe Mtiles
Troled Se
-00
401
90
300
20
40
80
1001
ImagoNet (top-1, 96), subelassed
ImageNet
(top-1, 96,, subelassed
(c) ImageNet-R
(d) ObjectNet
Figure 3: Pre-trained models consistently have high ER across different choices of urchitectures on both ImageNet and CIFAR-10. We plot ER (%) versus ID accuracy (%) at every epoch of training as we fine tune (a) pre-trained ImageNet models on CIFAR-10 or (b,c,d) pre-trained JFT models on ImageNet. Figure (a) shows the ER. on CIFAR-10.1 while Figures (b,c,d) show the ER on ImageNet ImageNet- and Net,ct respectively. For each plot, we also show the ER and accuracy achieved by converged testbed models in blue with 95% Clopper Pearson confidence intervals. For (a) CIFAR-10.1 and (b) Image.Net V2, we trained randomly initialized version of all the model architectures and we report the model with the highest maximum ER during training (solid black curve). We observe that the pre-trained models all exhibit high ER in the middle of fine-tuning, which eventually decreases as fine-tuning proceeds. In contrast, the randomly initialized models do not have significant ER relative to the testbed models.
5.1 Trainre Models Have High Effective Robustness
We find that models pre-trained on large and diverse datasets exhibit high effective robustness during fine- tuning, while randomly initialized models do not at any point during training (Figure 3) As described below, this observation is invariant to the choice of robustness benchmark (CIFAR-10.1, ImageNet- V2, Object and ImageNet-RR) model architecture; and the details of the fine-tuning, such as learning rate, optimization algorithm, type of fine-tuning (last-layer versus whole model), and loss function.
CIFAR-10. We evaluate several architect including AlexNet, ResNet-18, ResNet-50, ResNet-152 and VGG-11-BN, throughout fine-tuning CIFAR-10 using both ImageNet pre-trained initialization and random initialization. At every epoch of training, measure ER using CIFAR-10.1 as the OOD test set and CIPAR-10 as the ID test set. Figure plots the ER (%) versus CIFAR-10 test accuracy (%) throughout fine-tuning for all of the ImageNet pre-trained architectures, as well as the randomly initialized architecture that achieved the maximum ER (for the full comparison see Appendix 10 We see that all of the ImageNet pre-trained models have significantly higher maximum ER than either the testbed models or the best performing randomly initialized model. Moreover, we observe that ER generally increases throughout fine-tuning, peaks, and then gradually disappears towards the end of fine-tuning.
II initial experiments also studied DenseNet-121, 121, -161, -169, -201; GoogLeNet; MobileNet ResNet-34, -101; ResNext-50 32x4d, -101_ 32x8d; ShuflcNetV22xx0. -x1.0; and SqueezeNet-1.0, -1.1, VGG-11, -13 BN,_BNN-10 -16 BN,, -19, -19 BN, and selected the subset of six models presented this paper based the widest range of observed ER. 2 AAdditionally, we add several low-accuracy scikit-learn models which received personal communication and which expect be publicly available soon.

---

Published in Transactions on Machine Learning Research (09/2022)
ImageNet. For models fine-tuned on ImageNet, we measure ER for JFT pre-trained models at every epoch of training using either ImageNet Nbject or ImageNet-R as the OOD test set and ImageNet as the ID test set. We report ImageNet top-1 accuracy, and for ImageNet- R and ObjectNet, we compute accuracy over the respective subset of classes for the two benchmarks. Figure 3(b] shows the ImageNet ER, and Figures 3(c] and 3(d] shows ER on ImageNet-R and ObjectNet respectively. In all cases, the pre-trained model reaches maximum ER during fine-tuning which then decreases as we continue fine tuning.
For the ObjectNet and ImageNet-R evaluation, we fine-tune BYT-L-R152x4-JFT model on ImageNet and evaluate the same model on ImageNet ObjectNet, and ImageNet-R. Though the model's ImageNet accuracy is the same in all cases, for Vma the ER vanishes (OOD accuracy drops to the bascline set by the testbed models), while for ImageNet-Re and ObjectNet there is still a non-zero ER at the end of training. This observation is consistent with Taori et al. 2020): which found that converged pre-trained models are more likely to exhibit nonzero ER on ObjectNet or ImageNet-R than ImageNetV2.
Model architecture. We find that pre-trained models with larger architectures exhibit. higher ER. For ImageNet V2 in Figure 3(b) we evaluate multiple BiT ResNets with increasing depth and width (R150x1, R101x3, R152x4) that were pre-trained on JFT, and we find that models with more parameters have higher peak in ER. Similarly, for CIFAR-10.1. Figure 3(a) shows that larger, more recent architectures such as the Reste-ResN exhibit higher effective robustness than smaller architectures like AlexNet. Moreover, Figure 4(b) shows that max ER on CIFAR-10. increases for BiT models of increasing depth and width as we vary the pre-training dataset.
Details of training. We performed a detailed study of a variety of components of the optimization process, and we found that the ER is not sensitive to these details. See Appendix C for variations in learning rate, loss function, lype of optimization (full model vs. last layer).
5.2 Zero-Shot Evaluation
Recent work (Radford et al. 2021a) introduced the CLIP architecture that allowed for zero-shot evaluation image classification datasets like ImageNet and the ImageNet robustness benchmarks. The zero-shot CLIP models has high amounts of ER, but it is unclear if the ER comes from the new and very large dataset used for training, the natural language model component, the contrastive training objective, or ils zero-shot evaluation. In Figure 3[b] we show the ER of the best zero-shot and linear probe CLIP models from Radford ct al. (021a)). We also show results from zero-shot evaluation on a BiT model that was pre-trained on JFT. This is done by creating a mapping between similar classes of JFT and ImageNet. While the BiT model has about 5% lower ImageNet top-1 accuracy, it matches the amount of ER that the CLIP model has. This is surprising, and suggests that the zero-shot component of CLIP plays significant role in the high value of ER CLIP achieves.
5.3
The Impact of Data on Effective Robustness
We find that the maximum ER depends on both the pro-training dataset and the data the model is fine-tuned on. As we discuss below, larger and more diverse pre-training datasels, as well as more difficult fine-tuning examples, all increase the magnitude of the ER peak.
Dataset size and diversity. In Figure [[a) we pre-train models on randomly sampled fractions of JFT. We observe that maximum ER increases as the amount of pre-training data increases but eventually plateaus for the largest datasct sizes. We suspect the reason for the plateau the fixed number of iterations used across those experiments, which eventually serves bottleneck for performance improvements as reported by prior work (Hernandez et al. 2021 Nakkiran et al. 2020 Figure 4(b) shows that maximum ER increases we pre-train on larger and more diverse datasets.
We also find that pre-training first on a larger dalaset and then switching to a smaller dataset leads to lower ER during the final fine-tuning task than a setup that only pre-trains on the larger dataset. To see this, consider the BiT-M-Ik and BiT-M-21k in Figure 4(b] Both models were initially pre-trained on ImageNet-21k (as denoted by the 'M'), but the BiT-M-1k model was additionally fine-tuned on ImageNet- 1k as part of the pre-training procedure. We then evaluated the ER of both models as we fine-tuned them to
7

---

Published in Transactions on Machine Learning Research (09/2022)
Destcaed
Elflction Rollnstrres
Max Elfective Pollustonn
Ellletiive Rolretttn
Itf
B.T-S-4k
BrT-3-4k
B-T-31211a aIF-LIFFT
JFT Pre-Triinng Dataset size
Pre Pre-Thiining Dataset
CIPAR-10 Tistt (90
(a) Max ER vs. fraction of JFT dataset size
(b) Max ER vS.
(c) Varying the difficulty of the fine-
different: pre-training datasets
tuning dataset
Figure 4: The maximum ER depends both the pre-training and the fine-tuning dataset. (a) We evaluate BiT-R101x3 models pre-trained on varying fractions of the JFT dataset throughout fine-tuning on the CIFAR-10 dataset, and we find that maximum CIFAR-10.1 ER increases with the size of the pre-training dataset. Error bars represent standard deviation over runs. (b) We evaluate several BiT models of increasing size (R50x1, R101x3, R152x4) that were pretrained on either ImageNet (S), ImageNet- 21K (M), or JFT (L) throughout fine-tuning on CTFAR-10. (The BiT-M-1K was first pro-trained on ImageNet-21K and then ImageNet-1kk. Maximum CIFAR-10.1 ER increases as the model size and pre-training dataset size and diversity increase. (c) The maximum CIFAR-10.1 ER also depends the difficulty of the fine-tuning dataset. We select examples based on the C-score Jiang et al. (2020) of CIFAR-10 and train on either the 5000 casiest, hardest or random images in the training set. Training on harder examples yields more ER for both pre-trained and randomly initialized models, but lower CIFAR-10 test accuracy than training on random examples.
CIFAR-10. Figure 4(b) shows that the extra fine-tuning step ImageNet-1k 1k gives about 1% drop in ER relative to the -iT-M-21k model.
Example difficulty. The ER during training also depends on the data the model is being fine-tuned on. In Figure [(c) we select the 5000 casicst, hardest or random examples from CIFAR-10 training set and only fine-tune on these examples. The difficulty of each example measured by the C-score (Jiang el al., 2020j. Training on easier (harder) examples gives less (more) ER, but training on only the hardest. or easiest. examples gives lower final accuracy than training on the examples of random difficulty. See Appendix C.9 for more details.
5.4 Analysis at Maximum Effective Robustness
understand the differences between models that exhibit ER and testbod models, we compare the predictions from the former both al the peak ER and at the end of training and the latter.
Dominance probability. Recent work (Mania & Sra 2020 showed that the linear trend between the ID and OOD accuracies will arisc if two assumptions are satisfied: low dominance probability and specific notion of distributional closeness. The dominance probability a similarity metric between the predictions of two models, measuring what fraction of examples the model with lower overall accuracy gets correct that the model with higher overall accuracy gets wrong.
Formal definition (Mania & Sra. 2020). Let X' be covariate space, y be discrete label space, and P be a probability distribution over X' x V. Suppose model f maps X' + y and has accuracy / = FE (f((x) = y)]- Given any two models fi and fj with μi S M4y the dominance probability is given by
((ff(())=))) n + +
(2)
In this work, we find that the dominance probability assumption does not hold for models with ER, but that the assumption of distributional closeness still holds.

---

Published in Transactions on Machine Learning Research (09/2022)
Dominance probability InageNel
.44
Accuacy en cooappll les ot
test bed modells gats WT WOTO (36)
Trated Maald
Balls Ricdus
Depdinare prebabilttio
6.125
Trcled wask Sitiiitte Mae 3D BitGiiitei Chensipe CLP-L1NGS
-1
CUP LESTOs
6.100
-0
6
Raciaa Carss
6.075
4-
6.000
0.025
6.000
-2
0.0
30
40
50
80
70
80
P1 P1
ImaggNNet (top-1, 90)
(a) Dominance prob. distribution
(b) Pairwise dominance proh.
(c) Difference from testbed predictions
Figure 5: Models with high effective robustness have higher dominance probabilities than models with zero effective robustness BiT-L-R152x4 Max ER (A), CLIP zero-shot (B), CLIP-LBEGS (C), and BiT-L-R152xx Converged (D). (a) For each pair (i,j) of models in the testbed plus these four models, we plot their dominance probability versus their accuracy difference (μ4 Mg)) We find that the CLIP zero-shot model and the pre-trained BiT-L-RR12x4 model have higher dominance probabilities on ImageNet than any of the models in the testbed, but once the CLIP and BiT-I, models are fine-tuned on ImageNet, they align more with the dominance probability distribution of the testbed models. (b) We show the pairwise comparison of the dominance probability on ImageNet between all testbed models (not labeled) in addition to our four models. We sort models in order of increasing top-1 accuracy ImageNet (see Appendix C.10 for bigger version with labels for all models), We see that the four labeled models make very different predictions relative to the testbed. (c) For each of the plotted testbed models, we plot the percent of examples that none of the other testbed models get correct. Considering the 4.8% of all examples that none of the testbed models get right, the best BiT model pre-trained on JFT gets 10% of them correct and the zero-shot CLIP model gets approximately 8% correct.
ImageNet. We analyze the predictions of the following four models:
In Figure we see that the dominance probability of a pre-trained BiT model carly-stopped at its maximum ER shifted relative to the models the testbed, while the pre-trained BiT model that was fine-tuned to convergence (and has low ER) has dominance probability similar to the models in the testbed. We also find that the dominance probability for the pre-trained BiT model is very similar to that of zero-shot CLIP model (sce Appendix B.3 details fine-tuning). Even though the fine-tuned CLIP model has close to zero ER (see Figure ((b)), we see in Figure 5(a) and 5(b) that its dominance probability is also higher than that of standard testbed models. Unlike the BiT JFT pre-trained model, the CLIP model was pre-trained with contrastive learning and text embeddings, and we hypothosize that these elements may contribute to this
effect. For additional details see Appendix C.10
Examples that only effectively robust models get right. Models that have high ER are able correctly classify images that no other testbed model gets correct. We select the 4.8% of images that none of the testbed models get correct, and we find that the ER model with the best ID performance gets 10% of these examples correct, i.e. 237 examples, divided among 198 different classes, with a maximum number of three examples in each class, After manual inspection of all examples, we do not see any pattern among the 237 images.
Remarks. Since testbed models all have relatively low pairwise dominance probabilities, there are a set of easy examples correctly classified by all models, but harder examples are only classified correctly by increasingly higher accuracy models. This suggests that images this notion of difficulty is shared by all the testbed models. In contrast, the high dominance probability of the effectively robust models suggests that they have a different notion of difficulty compared to that of the testbed models, i.e. some images that are difficult for lestbed models are not as difficult for effectively robust models, As further evidence, effectively robust models are able to classify some images correctly that no other testbed model gets correct. We hypothesize that the different notion of difficulty of the effectively robust models comes from pre-training on larger and more diverse data, which could expose them to examples that are dissinilar from those in the fine-tuning dataset.
be placed scale of difficulty, and that
9

---

Published in Transactions on Machine Learning Research (09/2022)
5.5 Maintaining Effective Robustness at High Accuracy
In the previous sections, we have seen that models pre-trained on larger and more diverse datascts exhibit robustness properties during fine-tuning, but that, depending on the choice of OOD dataset, the ER can vanish as the model converges towards the end of training. In this section we explore several ideas for ways to maintain the high ER as the model reaches high accuracies, but ultimately we find that none of the techniques are able avoid the loss of ER.
Replay buffer. Deep neural networks that are subject to training two distinct tasks sequentially commonly forget learned information about the first task after training on the second, a phenomenon known as catastrophic forgetting [Kirpptrrick et al., 2017 Toneva et al.) 2019). A possible hypothesis, therefore, is that retaining learned information about the pre-training dataset- - for instance, by maintaining high accuracy on this dataset during the course of fine-tuning to downstream tasks may have a beneficial effect on ER. We explore this possibility by implementing a popular technique for mitigating catastrophic forgetting the use of a "replay buffer" (Ratclifi 1990: Rolnick et al. 2019
Without a replay buffer, we find that ResNet-18 model pre-trained ImageNet and fine-tuned CIFAR-10 has drop in ImageNet test accuracy from 69% to 39% when early stopped at 92% CIFAR-10 Lest accuracy.
When simultaneously fine-tuning on CIFAR-10 with a replay buffer, we retain 63% ImageNet test accuracy, but we observe no improvement in retained ER. See Appendix C.11 for more experimental details and results.
Mapping between different class sets. Using the same constructed map of classes used for zero-shot evaluation (c.f. Section 5.2 and Appendix C.7D, we can use the target classes of the pre-training dataset for fine-tuning. We use the mapping from CIFAR-10 to ImageNet classes, and we use this for finc-tuning on CIFAR-10 initializing with the pre-trained prediction head.
First, we explored fine tuning on just CIFAR-10 images with the 1000 ImageNet target classes, Second, we used a replay buffer for combined fine-tuning on CIFAR-10 and ImageNct using the same prediction head. In both setups we see no significant change in ER at the end of fine-tuning. See Appendix C.12 for further details on the experiments and results.
Example difficulty. In Section 5.3 we saw that training on harder examples during fine-tuning increases the ER, but prohibits reaching high final accuracy. We study different training schedules where we vary the difficulty of examples during training, but we find that none of the proposed methods are sufficient to maintain high ER and reach high test accuracy at the end of training. See Appendix C.9 for more details on these experiments.
6 Conclusion
Machine learning models currently experience an undesirable yet predictable drop in accuracy on OOD shifts. The exceptions from this trend models with ER experience less of drop in OOD accuracy compared to standard testbed models at the same ID accuracy. The ultimate goal is to produce ER models at state-of-the-art accuracies; however, none of the known models with high ER are in the high-accuracy regime, and the models with the highest OOD accuracy have no ER. One way to approach this problem is to understand why some models have ER so that we can replicate this behavior for shate-of-the-ars models and decrease the OOD performance gap.
In this work, we demonstrate that pre-trained models in the middle of fine-tuning, as well as zero-shot pre-trained models, represent an entire class of models that exhibit high amounts of ER. With this rich set of effectively robust models, we take some initial steps towards understanding what is unique about them that causes them to outperform testbed models on OOD shifts. Our results suggest that properties of the pre-training or fine-tuning datasets, such as the size, diversity, or difficulty, cause effectively robust. models to learn an entirely different difficulty scale than that of standard testbed models. Moreover, because ER models make different prodictions than standard models and are able to correctly classify examples that no standard models get right, these models are valuable for applications such as model ensembling, which requires prediction diversity.
10

---

Published in Transactions on Machine Learning Research (09/2022)
From our work, we know that pre-training provides clear benefit to ER, but that fine-tuning the ID task reduces F.R The future challenge lies in constructing mechanisms to enhance ER in the high accuracy regime. We hope our initial implementations of and investigations into such techniques will be launching point for rescarchers for further experimentation.
References
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint arXiv: 181777277177 2018.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled datasct for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Ahhéé-u,,, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https //proceedings. neur ips httppp///w.....o.aaa....o.o.o.a.
.dd
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the risc of adversarial machine learning. Poltern Recognition, 84:317 331, 2018.
Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey. Cinic-10 is not imagenet or cifar-10. arxiv preprinl ar Xr:: 03505, 2018.
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255, 2009. doi: 10.1109/CVPR. 2t..22000922244.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander Nicholas D' D'Amour, Dan Moldovan, Sylvain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lučié. On robustness and transferability of convolutional neural networks. Conference on Computer Vision and Pattern Recognition, 2021. URL /ttt::://arxt..orr/bbbsss 08558
Robert Geirhos, Carlos R. Medina Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. orXiv preprial arxiv:1808 08750, 2018.
Keren Gu, Brandon Yang, Jiquan Ngiam, Quoe Le, and Jonathon Shlens. Using videos to evaluate image model robustness. arXie preprint 19X4 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770 778, 2016a.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. European conference on computer vision, pp. 330-645. Springer, 2016b.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural netw robustness to common corruptions and perturbations. In International Conference Learning Representations (ICLR), 2019. https
01697
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint orXic:2006.162 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.0 01298, 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456 456.. PMLR, 2015.
11

---

Published in Transactions on Machine Learning Research (09/2022)
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Exploring the memorization-
generalization continuum in deep learning. CoRR, abs/2002.03206,, 2020. URL httpp://arxiv.org abs/2002.03206
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kicran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 11 (14(13):3521-33 -3526, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, et al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:2012. 074421, 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcorver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning, 2020.
Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. ar preprinl arXiv:1404.5997, 2014.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised protraining. In Proceedings of the Europeon Conference on Computer Vision (ECCV), pp. 181-196, 2018.
Horia Mania and Suvrit Sra. Why do classifier accuracies show linear trends under distribution shift? arXiv preprint Xrx::200 15489, 2020.
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. arviv preprint Xiv::1900.1120 121199055.180, 2019.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1905-6916. PMLR, 13 18 Jul 2020. URL htttp://proceeednggg mmlr wtt..pprree//119/mmieee22200
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap: Good online learners are good offiine generalizers. arXiv preprint arXiv:2010.08127, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http://papers. newrsss...nrrpscccccceoooo Leateep--eernnnnnllbbbrrr pdf
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arxiv preprinl arXiv:1903.1052) 2019.
Joaquin Quiionero-Candele Candela,, Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer. Dataset shift in machine learning. Mit Press, 2009.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2021a. URL https:://cdn.opeaai.. papers/Leearingg Transferable,
BiiiaalMMddes_FFaa Nartom_aatura___ Langruage_ wtw.uaggggugeegggsiiiiiiiooo
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Xiv preprint arXin:2103 x:Xx:22003.0000, 2021b.
12

---

Published in Transactions on Machine Learning Research (09/2022)
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting
functions. Psychological review, (7(2))285, 1990.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.004555 00451, 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? 2019. URL httt:://axiv..et wttp:///wwwr.abb//./oea ats// 10811
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advonces in Neural Information Processing Systems, volume 32, pp- 350 360, Curran Associates, Inc., 2019. URL https://prroceedings www.oneeerrreeppeeeoeee /at.r/ppp..c/ppperr////////e pdf
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255) 252, 2015.
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. A systematic framework for natural perturbations from vidcos. CoRR, Abs/1106.02165, 2019. URL
02168
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xix: 1409. 1556, 2014.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843 008-852,240 2017.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. CoRR, abs/1912.04833. 2019. URL
Christian Szegedy, Wojeiech Zaremba, Tlya Sutskever, Joan Bruna, Dumitru Erhan, Tan J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks In International Conference Learning Representations (ICLR), 2013. http:://arrdv..orr/bbb//22.2..
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. When robustness doesn't promote robustness: Synthetic vs. natural distribution shifts on imagenet, 2020. URL
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example lorgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL https:://pperrevie..n
Antonio Torralba and Alexei Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 2011.
IEEE,
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Robecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schuidt, Robust fine-tuning of zero-shot models, 2021. URL /ttps://arxiva 2ttps//.00r//a////200000000
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint 05444 2022.
13

---

Published in Transactions on Machine Learning Research (09/2022)
W Wu. Classifying images into 11k classes with pretrained model, 2016. http:///w....oooototeeeee/eeeeee htg.
URL https
py#1441.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp. 3-19, 2018.
Qizhe Xie, Minh- Minh-Thang Luong, Eduard Hovy, and Quoe Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. or preprinl 1rXi::1905.14490 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ar preprint rrXt:1605.0011. 2016.
14

---

Published in Transactions on Machine Learning Research (09/2022)
A Scaling for Linear Fit
When modeling the linear relationship between the original and new test set for CIPAR-10 and ImageNet, applying nonlinear scaling to the accuracy a has shown to improve the linear fit, especially when considering both very high and low performing models. Recht et al. (2018 2019) applies a probit scaling to the accuracies, and Taori et al. 2020) also considers a logit scaling.
Probit scaling uses the probit function
probit(a) = $ (a),,,
(3)
where (a) is the inverse of the cumulative distribution function of the standard normal distribution; the logit scaling uses the logit function
logit(a) = log 1 -
(4)
We compare the goodness of the fit for each of the scalings in Table 2
We conclude that the logit scaling gives the best linear fit for both CIFAR-10 and ImageNet and will use this scaling throughout this work. In Figure 2 we show the linear fit in logit-scaled space and transformed back to linear space for CIFAR-10.
Table 2: R2 values for linear fit for different scaling functions
SCALING
CIFAR-10
ImageNet
LINEAR
0.981
0.984
PROBIT
0.996
0.995
LOGIT
0.998
0.996
Numerical fit values. All linear fits throughout this work were computed by first transforming to logit space (see Section A) and then fitting using scipy. stats. lints..inregress. Eq. gives the functional form for our fit, and the numerical values are listed in Table g
(New Acc) logit
IA-logit(Orig. Acc) B]
(5)
Table 3: Numerical fit values for Eq.
A
B
CIFAR-10/10.1 MMAGENET/IMAGENETT
0.8318
-0.4736
0.9225
-0.4896
B Experimental Setup
B.1 Pre-trained Image Classification Models
We conduct our experiments using diverse set of model architectures and sizes make sure the observed phenomenon are general and not tied to any particular choice. We study two groups of models.
ImageNet pre-trained. The first group consists of AlexNet Krizhevsky (2014). ResNet-18, ResNet-50. Res- He el al. (a), VGG11 BN Simonyan & Zisserman (2014), and WideRtesNet-50-- Zagoruyko & Komodakis 20166), where we use the PYTORCH Paszke et al. (2019) implementation and weights coming from torchvision. monels v0.8.2 where the models were pre-trained on ImageNet. When fino-tuning pro-trained ImageNet models on CIFAR-10, we always replace the original classification layer with a 10-class randomly initialized classification layer.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Big Transfer (BiT). The second group of models consists of series of models from Big Transfer (BiT)
Kolesnikov et al. (2020). The BiT models all use ResNet-v2 architecture He et al. (2016b). except that they replace all Batch Normalization Loffe & Szegedy (2015) layers with Group Normalization Wu & He (2018) and use Weight Standardization Qiao ct al. (2019) in all convolutional layers. Following Kolesnikov et al. (2020), we label the ResNet architectures by their depth and the widening factor for the hidden layers, for example R50x1, R101x3 and R152x4, which are the sizes we will consider in this work.
The BiT model come in three different flavors: BiT-S, BiT-M and BiT-L, where S, M and L indicate if the pre-training was done ILSVRC-2012 (which denote ImageNet-lk or just ImageNet) Russakovsky ct al. (015)) ImageNet-21k Deng et al. (2009) or JFT (un et al. (2017), respectively. In addition, these models may be further fine-tuned, and 'Ik', '21k' or "JFT" indicates the dataset it was fine-tuned on. For example, BiT-M-R152x4-11 1k is the ResNet-1522 model that was pre-trained on ImageNet-214: ('M") and fine-tuned on ImageNet-1kk ("1k").
We also consider a series of BiT models that were pre-trained by Djolonga al. 2021 on different random fractions of the JFT dataset with additional fine-tuning in Section 5.3
B.2 Fine- Tuning
Fine-tuning refers to a training setup where the weights learned by training model on one data distribution are reused when training the model on a second data distribution. Fine-tuning can either be end-to-end, where the weights learned on the first distribution are only used for initialization when training on the second distribution, or last-layer, where only the last layer weights are updated when training on the second distribution. Fine-tuning can occur in multiple rounds using multiple successive data distributions. Typically, the carlier training datasets are larger in size and class sets and more diverse.
B.3
CLIP model
Our CLIP zero-shot model uses the publicly released pre-trained weights for the ViT-B-32 model from https://githube wttptth.ccooooaaaa//PPP Following the methodology from Radford et al. (2021a), we fine-tune the zero-shot CLIP model using logistic regression classifier optimized with L-BFGS, and we determine the L2 regularization strength À using a hyperparamctor sweep on the validation sets with ten logarithmically-spacao values between 10-6 and 107.
B.4 Data Preprocessing
CIFAR-10. In our experiments on CIFAR-10, we are interested in testing models that were pre-trained on ImageNet-like datasets. Since the ImageNet images are typically preprocessed to 224x224 during training, we also preprocess the CIFAR-10/10.1 images by resizing from 32x32 to 224x224 pixcls. To keep the experimental setup the same, we also resize the CIFAR-10/ 10/R11011.11111 images to 224x224 pixels when using randomly initialized models 3 All CIFAR images are normalized using the mean [0.4914, 0.4822, 0.4465] and std [0.2023, 1994,, 0.2010].
When evaluating on test set, when extracting the features from the penultimate ResNet layer when training only the last layer, we do not apply any preprocessing beyond the upsampling and normalization. When training the full model on the CIFAR-10 training set (with image size 224x224), we additionally apply RandomCrop with padding-28 and RandonfoorizontalFlii
ImageNet. The ImageNet training images are preprocessed with RandomResizedCrrop to 224 pixels and RandomHorizootalFLip and the test images are resized to 256 pixels and preprocessed with CenterCrop 224 pixels. Both are normalized with mean= (0.455, 0.456, 0.406) and A.d=(0.229) 0.224, 0.225).
TThe testbed models for CIFAR-10 and CIFAR-10.1 using the standard 32x32 image size, but as in Figure [(b] and [0:b] the ER of our randomly initialized models align with the testbed models, which indicates that the image size does not affect results. However, we have independently confirmed that the get the same results our setup with image size 32x32 as in Figure 10(b
16

---

Published in Transactions on Machine Learning Research (09/2022)
AleSNet
ResNee-152 VGGUL BN
Widl-ReNNas-0-
Zeco Testhed Models
RoNNet-18 BesNict-00
Tiethed Muriels
ICT- RIOLG--FF
Effective Robustness (%)
Effective Robustress (%)
BIT-L- -IT---RF
4
24
2-
0
-2
20
40
60
80
100
100
CIFAR-10 Test Acc (%)
ImageNet (top-1, 1, %)
(a) CIFAR-10
(b) ImageNet
Figure Multiple runs are combined by computing the ER as binned average across the ID accuracy (using 100 bins). The shaded around each curve shows the standard deviation per bin over 5 runs. (a) is the same data as in Figure 3(a) and (b) is the same runs Figure 3(b)
B.5 Confidence Intervals, Averaging Runs, and Error Bars
Testbed confidence intervals. For the ImageNet and CIFAR testbed models, described in Section we report confidence intervals on the accuracies calculated using Clopper-Pearse Peaeson intervals confidence level 95% Recht al. 2018 2019).
Averaging runs. For all measurements of ER that were averaged over multiple runs, report the binned average value, which we calculate using scipy.stee stats. binned caaned_statistie using 100 bins.
Error bars. For most of the rosults throughout this work, we have left out the crror bars to make the plots more readable. Here, we show the typical error bars for models fine-tuned on ImageNet and CIFAR-10 10 in Figure The reported error bars are the standard deviation from the binned average across runs.
When measuring the maximum ER, like in Figure 4(a) have the choice between using the standard deviation in the bin with maximum ER, or the maximum ER across all the bins. Since the number of data points in each bin can vary, we observe that the standard deviation in the maximum ER bin has more variation across different runs. So, to be conservative, we report the maximum ER across all the bins. A comparison between the two choices of standard deviation can be seen in Figure
B.6 Optimizer and Hyperparameters
Here we give the details used in the training of all our results. For all models we ran five runs and all were trained using stochastic gradient descent with momentum 0.9 and weight decay 10-4, unless otherwise specified.
The default scttings train the full PYTORCH models (soc Soction B.1 was using batch size 64 for 250 epochs using learning rates 0.01] for the randomly initialized model, and for 100 epochs using learning rates [0.01, 0.001, 0.001, for the pre-trained models. The main results shown in the main text and the appendix are using learning rate 0.01 and 0.001 for the randomly initialized and pretrained model, respectively, but we have checked that the general conclusions are the same across the different learning rates.
When training the BIT models, we extract the features from the pemultimate layer and only train the prediction head (except for the randomly initialized BiT models where we train the full model). Unless otherwise specified, the models are trained using learning rate 0.001 with batch size 32768 for 100 epochs with weight, decay.
17

---

Published in Transactions on Machine Learning Research (09/2022)
Max Effective Robustnes (%)
6-
2-
107
10a
JFT Pre- Pre-Training Dataset size
(a) Max std. across all bins
Max Effective Robust meess (%)
6-
2
10a
10a
JFT Pre- Training Dataset size
(b) Std. bin with ER
Figure 7: Comparison of two different ways of reporting the standard deviation (std.) for the maximum ER for the experiments in Figure 4(a) (a) shows the maximum standard deviation across all bins of CIFAR-10 ID test accuracy, and (b) uses the standard deviation in the bin that had the maximum ER. (b) has higher
variation in the std. across different runs because each bin contains different number of data points, so we use (a) as a more conservative estimate as our main reported results.
18

---

Published in Transactions on Machine Learning Research (09/2022)
C Additional Experiment Details and Results
This section describes we additional experimental results and details of training not described in the main paper.
C.1 Pre-trained versus Randomly Initialized Models
In Section 5.1 we discussed how pre-trained models exhibit ER during fine-tuning, while randomly initializod models have no ER during training. Here add additional experimental details that did not fit into the main paper.
ImageNet. In Figure 8(a) we show the ER during training on ImageNet for BiT-L-R101x3 model that was pre-trained on JFT in comparison to a randomly initialized model the same architecture. The pre-trained model is averaged across 5 runs and only training the last layer. For the randomly initialized model we train the full model, but due to the high computational cost, we only perform one run. The pre-trained model is using syper-paramette trained following the setup as described in Kolesnikov et al. (2020
as described in Appendix B.6 while the randomly initialized model was
CIFAR-10. In Figure 8(b) we show the exact samne experiment, as described in the previous paragraph, on CIFAR-10. Again, the randomly initialized model is only run a single time due to the computational cost.
For CIFAR-10 we also perform extensive comparisons between several different architectures where we cither use pre-trained or randomly initialized weights. The results are shown in Figure 10 and the hyper-paramelers are as described in Appendix [.6]
C.2 ImageNet Robustness Benchmarks
For models that are fine-tuned on ImageNet, we study the ER on ImageNel V2, ImageNet-RR and ObjectNet. We train one BFT-L-R152x4 JJTT benchmarks. The ER for ImageNet-R and Object was shown in Figure 3(c) and 3(d) In Figure 9 we show the same data in Figure ((c) and 3(d) but in term of accuracies in stead of ER. While the ER ImageNet V2 goes to zero at the end of fine-tuning, the ER for ImageNet-R and ObjectNet remains non-zero. As discussed in Section 5.1 this is consistent with previous observations in the literature. Besides observing that the accuracy on IetgeNet-R and ObjectNet stops at around 60%, we do not know why this is happening for these two datasets but not for ImageNet V2.
_J'T model on ImageNet and evaluate the same model on all three robustness
C.3 Architecture Independence
We observe ER of pre-trained models across different model architectures in Figure 10 and none of these architectures have any significant ER when randomly initialized. We used PYTORCH models (c.f. Section B.1 pre-trained on ImageNet and replaced the prediction head with a randomly initialized layer for CIFAR-10.
Both the pre-trained and randomly initiaked models were trained with SGD with learning rates 0.01, 0.001 and 0.0001, and batch size 64, momentum 0.9, and weight-decay 5 10-4, We show the results from averaging five runs using learning rate 0.001 in Figure 10 but the results are similar using the other learning rates (learning rate dependence is studied in Appendix C.5 The randomly initialized models were trained for 250 epochs, while the pre-trained models were trained for 100 epochs.
C.4 Loss function
In this section we show that our observations about ER are not unique to models trained using cross-entropy loss, and we get the same behaviour using mean square error (MSE).
For evaluation of the MSE error loss, we compute the MSE between the output logits and a onc-hot target vector, We train the model using stochastic gradient descent with constant learning rate 0.001 and batch size 4096 for 100 epochs.
19

---

Published in Transactions on Machine Learning Research (09/2022)
Effective Robustness (%)
DIT L-Rioxd-
Testbed Medil
20
40
ImageNet (top-1, 1, %)
(a) ImageNet
Effective Robustness (%)
DTT RI0EE Rand. Init
20
40
GO
80
100
CIFAR-10 test ace (%)
(b) CIFAR-10
Figure Comparison of the ER of BiT-R101x3 model that is either pre-trained on JFT or randomly initialized. (a) shows the ER on ImageNet when trained ImageNet, and (b) shows the ER CIFAR- 10.1 when trained on CIFAR-10. For both datasets, the randomly initialized models show no ER while the pre-trained models do exhibit ER during training, but it vanishes the model converges.
In Figure we compare the ER duing fine-tuning of the last layer of a JFT pre-trained BiT-L-R152x4 model where one is trained using MSE and the other is trained using cross-entropy loss. In this specific experiment, the MSE trained model has higher ER at its maximum, but due to the other similarities between the evolution we do not believe this would always the case.
C.5 Learning Rate
The observed ER. during fine tuning is largely independent of the learning rate used during optimization. we show the ER during training CIFAR-10 for pre-trained and randomly initialized Wide-ResNNet-00-2 using the same hyper-parameters in Appendix C.3.
In Figure 12
C.6 Full Model vs. Last Layer Fine- Tini-Tunigg
The ER during fine-tuning of pro-trained models largoly insensitive whether the full network is trained or if only train the last layer. In Figure 13 we compare the ER of full model versus last layer fine-tuning for the six PyTORCH models described in Appendix B.T
20

---

Published in Transactions on Machine Learning Research (09/2022)
ImageNet-
ObjectNet
ImageeNtt ---R (top-1, %%
RIR2--IFT
ObjectNet (top-1, %)
40-
20-
20-
0-
20
100
20
100
ImageNet (top-1, 1, %), subclassed
ImageNet (top-1, %), subclassed
(a) ImageNet-R
(b) ObjectNet
Figure 9: Evaluation on ImageNet-R and ObjectNet for BiT-I-R152x4_JU JFT model that is finc-tuned on ImageNet. Where the ImageNet ER goes to zero at the end of fine-tuning, the pre-trained model still has non-zero ER.
We start both last layer and full model fine-tuning with the same pre-trained weights for that architecture. For training of the full model, we use SGD with learning rate 0.001, and batch size 64, momentun 0.9, and weight-decay 5 10-4 for 250 epochs. When training only the last layer, we use SGD with learning rate 0.0001, and batch size 4096, and without momentum or weight-docay, and we trained for 1000 epochs. Wo experimented with learning rale schedulers, but saw no significant change. Despite the difference training and hyper-parameters, we observe that both have about the same amount of FRR The biggest observed difference is that the only last-layer trained models don't reach the same final accuracy when training the full model, which is to be expected the last-layer model cannot change its pre-learnt features.
C.7 Zero-Shot Evaluation
Zero-shot evaluations on traditional image classifiers is made possible with hand curated maps between the classes of one dataset to another. To map from CIFAR-10 to ImageNet, we utilize the mapping from ImageNet. CIFAR-10 classes used CINIC-10 Darlow et al. (2018), and we utilize a previously constructed map from ImageNet to JFT evaluate the performance of BiT-L models on ImageNet in a zero-shot manner.
These maps were used for zero-shot evaluation ImageNet and ImageNet in Figure I and discussed in Section 5.2 The map from CIFAR-11 10 to ImageNet was used in Appendix C.12
In this section we investigate different ways of combining the multiple logits corresponding to each CIFAR-10 class to make predictions using a model pre-trained on ImageNet. Given a CIFAR-10 class, we collect the corresponding ImageNet softmax scores and combine the probabilities using three different functions: max, mean and sum. We show the results Figure 14 where we see that most of the zero-shot models have some amount of ER, but are not significantly more robust than pre-trained models in the middle of fine-tuning, We see no significant difference between the three combination functions test, and use the max as the default in all other plots throughout our work.
C.8 Dataset Size and Diversity
In Section
5.3 we summarized our experiments where we vary the pre-training dataset for different fixed BiT architectures. Here, we will include additional experimental details and observations.
All models in this section was fine-tuned using SGD with learning rate 0.001, batch size 4096, with no momentum weight decay, and trained for 1000 epochs.
21

---

Published in Transactions on Machine Learning Research (09/2022)
Effective Robustness (%)
VOGUL BN
10.0
RosNet--
RosNect-00
+
Testbed Models
7.5
RoaNeet 50
5.0
2.5
2.5
20
100
CIFAR-10 Test Acc (%)
(a) Pre-trained models
Effective Robustness (%)
10.0
AleaNes
RosNett-8
VOGUL BN
ReaNet-30
+ Totted Modals
7.5
RowNet-
2.5
0.0
-2.5
5.0
20
40
60
80
100
CIFAR-10 Test Ace (%)
(b) Randomly initialized models
Figure 10: Pro-trained models show ER for all six architectures studied, while none of the randomly initializod models do. The amount of ER does vary across different pre-trained models, but since did not do the pre-training, we cannot control for the differences between architecture and optimization in this experiment (See Section 5.3 and Appendix C.8 and C.& for more controlled experiments on the BiT architectures.) However, the randomly initialized models show that pre-training is crucial to get any ER at all for any of the architecture choices.
JFT dataset size. Using series of BIT-R10133 that were pre-trained on different randomly sampled fractions of the JFT-300M dataset Sun et al. (017)). show that the maximum ER inereases with the pre-training dataset size. Since the pre-training images were randomly sampled from one dataset, it gives some control for the dataset diversity as opposed to our experiments where we compare models trained on ImageNet- ImageNet-21k and JFT. However, since the models were trained on different amounts of data, we cannot rule out any effects from the pre-training optimization on the maximum ER.
Different pre-training datascts. When using different pre-training datasets, we find that larger and more diverse datasets give higher maximum ER during fine-tuning on CIFAR-10.
We show the ER during fine-tuning for three different BiT model sizes in Figure 17 For R152x4 and R-101x3 we see that the BiT-L models that were pre-trained on JFTT has the highest ER and the BiT-S models that were pre-trained on ImageNet-lk has the lowest. We can also make an interesting observation by looking at the two BiT-M models: The BiT-M-21k model (pre-trained on ImageNet- 21k) has higher ER than the BiT-M-1k model (pre-trained on ImageNet-211 and then finc-tuned on ImageNet-1k)) In other words, fino-tuning on the smaller and less diverse ImageNet- 1k dataset before training on CIFAR-10 gives about a percentage point drop in maximum ERR
22

---

Published in Transactions on Machine Learning Research (09/2022)
JFT
Effective Robustness (%)
7.5
5.0
0.0
-2.
20
40
60
80
100
CIFAR-10 Test Ace (%)
Figure 11: The rise and fall of ER occurs for different loss functions used in training. Hcre, that the ER during tine-tuning is incredibly similar between model trained with MSE and cross-entropy loss.
Eifective Robustness (%)
Wade- ReaNact-0-2 Pre- Prainrained
Effective Robustaess (%)
Wide-ReaNit-003
Randoan Exit.
Modale
0.003
Movils
0.00
5-
0-
-5
20
40
60
80
100
20
40
60
80
100
CIFAR-10 Test Acc (%)
CIFAR-10 Test Ace (%)
(a) Pre-trained
(b) Randomly initialized
Figure 12: Same setup as in 10 but with varying learning rate for a fixed architecture, Wide-Res.Ne-500e The left (right) plot shows the results using pre-trained (random) weights. While there some small variations from different settings, the conceptual difference between pro-trained model and randomly initialized weights holds true.
While the observed trend that the maximum ER is higher for larger and more diverse pre-training datasets seems very robust, observe a deviation from this trend for the BiT-S-R50x1 model. The N-T-S-R50xl model has higher maximum ER than the BiT-L-R50x1 and BiT-M-R50x1 models. Also, it has higher ER than the comparable model with larger model architecture (BiT-S-R10133 and BiT-S-R152x4), thus breaking yet another trend. However, we do not know if this is caused by an interesting phase transition when using the combination of the smallest pre-training dataset and the smallest model, or it simply caused by some part of the pre-training optimization that we do not have access to.
Model size. In Figure 18 we compare the ER during training for three different BiT architecture sizes for fixed pre-training dataset. For all but the BiT-S model (as discussed above), we find that the ER increases with increased model size. This is a plausible outcome since bigger models can build more expressive hidden representations.
23

---

Published in Transactions on Machine Learning Research (09/2022)
Dilective Rolrastoos (56)
Eifertiv Ratrasmec (55)
Fut Moak Los Buddk
Elleciive Raleaninness (8)
Pat bhatt los Trukde 3labl
Inded
10
10
10
20
60
100
20
6C
20
100
20
50
CIFAR-10 (56)
CIFAR-10 (96)
CIFAR-10 (%6
(a) AlexNet
(b) ResNet-18
(c) ResNet-50
Effectivv Robuettnce (36)
Dooile Coateed Kaabia
Effectivn Robustnns (26)
Voltas Iet Lod
Indied
Ellestive Robontuuee (x)
Gain- Tondend
10
10
10-
CIVAR-1G test acc (56)
CHEAR- test acc (30)
CIEAR- best ace (%5
(d) ResNet-152
(o) VCG11 with batch norm
(f) Wide-ResNett5002
Figure 13: The ER of pre-trained models is mostly insensitive to whether the full model is trained during fine-tuning or just the last layer. We observe this across six different model architectures, and the only significant observed difference is that the full-model training reaches higher final accuracies, which to be expected.
Elfeerive Rolnotaess
10
Eifeerivve Rohnnsteee
10
Effocttvv Roboutuce
10
CIEAR-10 best (XE)
CIEAR-10 bost (%)
CIFAR-I0 nce (56
(a) Max
(b) Mean
(c) Sum
Figure 14: Zero-shot predictions using pre-trained BiT models fine-tuned to ImageNet-1lk 1k. We combine the ImageNet predictions that match the relevant CIFAR-10 class by choosing either the max, mean, or sum of the probabilities. The majority of models show some amount of ER.
C.9 Example Difficulty
In this section explore the ways in which the fine-tuning dataset affects the ER OOD dataset. We study ResNet-18 model that is either randomly initialized or pre-trained on ImageNet. Both sets of models are trained with a single run for 100 epochs using batch size 64, and the pre-trained and randomly initialized models use learning rate 0.001 and 0.01, respectively.
Finc-tuning with casy, random or hard examples. Using the C-score Jiang et al. (2020) as a metric for example difficulty, we select 5,000 of the casiest, hardest or random examples while maintaining class balance (i.e. 500 examples from each class).
24

---

Published in Transactions on Machine Learning Research (09/2022)
1.28M
2.00M
13:00M
32.00M
125.00M
Effective Robustness (%)
2
20
40
60
80
100
CIFAR-10 Test Ace (%)
Figure 15: ER of serics of BiT-R101x3 models pre-trained different amounts of data from JFT and fine-tuned on CIFAR-10. The maximum ER and the final accuracy increases with the JFT dalasel size. The model with the maximum ER used 32 million images from JFT, and discussed in Section 5.3 we suspect the plateau after this point is caused by the fixed number of iterations used during pre-training, which eventually serves a bottleneck for performance improvements.
BIT--IFFT
BrE-M-2tk
Max Effective Robustness
DIT 34 11
R50x1
R101x3
R152x4
Model Architecture (size)
Figure 16: The maximum ER increases as a function of model architecture size for all pre-training datasets. The only exception is the BiT-S- model that has the higest maximum ER among all the R50x1 models, and it also higher than the BiT-S-R101x3 and BiT-S-R152x4 models.
In Figure [(c) show the ER during training, and we make two observations: First, training more difficult, examples gives higher ER than when training easy random examples. Second, the models that train on only hard examples reaches lower ID accuracy than the easy or random models.
Phasing out easy examples during fine-tuning Since the fine-tuning dataset affects the ER during fine-tuning, there might be way to choose what. examples from the fine-tuning dataset to train on that achieves both high ID accuracy and high ER. In Figure 19 we show the result from experiments on randomly initialized and pre-trained models where start the fine-tuning process with the full CIFAR-10 10 training set, and we gradually phase out the easy examples until there are only 1000, 5000 10000 examples left in the last epoch of training. However, we can see this does not seem to have any effect on the ER.
An alternative approach we attempted was swilching between random, easy and hard, or first fine-tuning on the data and switching to only the hard examples at the end. Generally, we found that when a model that reached high ID accuracy was being further finc-tuned on difficult examples, the ER does go up, but the ID accuracy also goes down. The gradual phase-out described the previous section was the most consistent. way reach high ID accuracy while only training on hard examples by the end of training.
25

---

Published in Transactions on Machine Learning Research (09/2022)
10.0
Regresden
Edfeetivee Robusttess (%)
Engleie Regresien
Elfective Robust
7.5
5.0
2.5
2.5
0.0
0.0
-2.5
5.0
CIFAR-10 test acc (%)
CIFAR-10 (%6
(a) R112x4
(b) R101x3
10.0
Logietic Begrasien on Pretrained Prature
2CI-S-DDRd-1
Effective Robustness
Br/-A5-00tmi-ik
BIT Rotv 211
Toeled Modile
5.0
0.0
2.5
20
10
60
80
100
CIFAR-10 test acc (%)
(c) R50x1
Figure 17: The ER during fine-tuning on CIFAR-10 is mostly increasing for larger pre-training dataset. We show this effect across three different architecture sizes, where each plot compares models pre-trained on different datasets.
C.10 Dominance probability
The dominance probability Mania & Sra (0200) compares the predictions of two models and is the probability that the lower accuracy model correctly classifies given example which the higher accuracy model incorrectly classifies.
Pairwise dominance probability comparison. We compute the pairwise dominance probabilities on the ImageNet validation set for the following set of models: the ImageNet testbed models, BTT-L models (BiT-L-R152x4 early stopped at maximum ER, and the same model at end of training when the model has converged), and our wero-shot and fine-tuned CL.P models (see Appendix B.3
Figure 21 shows a heatmap of the full set of pairwise dominance probabilities (a smaller version appears in Figure the main paper). The models are sorted by increasing ImageNet validation accuracy and the lighter colors the heatmap correspond to models with higher dominance probabilities. Dominance probability not symmetric, but make the heatmap readable using either the row or column models the more accurate model, mirror the dominance probabilities across the axis.
We observe that the models with the highest dominance probabilities, the BiT-L-R152x4 Max ER model and the CLIP zero shot model, also have the highest effective robustness. However, the CLIP L-BFGS model has lower effective robustness than the -255- converged model ( 0% versus 1%, respectively), but slightly higher average dominance probability. This suggests that effective robustness alone is not the sole influencer of dominance probability, and some component of the CLIP pre-training, such as the natural language supervision, contrastive learning objective, large and diverse pre- dataset, contribute to the high dominance probabilities for even the converged CLIP model.
26

---

Published in Transactions on Machine Learning Research (09/2022)
Eiffective Robustness (%)
10.0
Logletic Regreedon on Protrolned Reatures
Effective Robustness (%)
10.0
Logletic Regreadon on Protrolnnd Fearures
BIT-M-RRICO.4231
7.5
BrE-GBi300ckIFT Tothed
7.5
BIE-36-21000-2214
5.0
5.0
2.5
2.5
0.0
2.5
2.5
-5.0
-5.0
20
40
60
80
100
20
10
60
80
100
CIFAR-10 test acc (%)
CIFAR-10 test acc (%)
(a) BIT-L-FFT
(b) BTT-M-21k
Elfecttvve Robustress (%)
10.0
Logistic Pretrained
Effective Robustness (%)
10.0
Pretenined
Bri-I-IliS2e--1
(2T-S-850dd-1
Teathed Madkls
Tosted Madels.
5.0
5.0-
2.5
2.5
0.0
0.0
-2.5
-2.
-5.0
-5.0-
20
40
100
20
40
60
100
CIFAR-10 test acc (%)
CIFAR-10 10 test acc (%)
(c) B/T-M-Ik
(d) BIT-S-Ik
Figure 18: Larger model architectures gives higher maximum ER during finetuniin on CIFAR-10. We show this effect for four different BiT architecture sizes for three different fixed pre-training datasets.
Distributional closeness. To characterize distributional closeness, Mania & Sra 2020 also introduce another metrie which we call the "triplee-robaablitiees,, measuring how close the predictions bet ween triplet. of models are (see Mania & Sra (2020) for formal definition and discussion). In Figure 20 show the triplet-probabilitte for the same set of models for the dominance probabilities in the last section. We observe that the effectively robust models do not violate the notion of distributional closeness as defined by the triplet probabilities, i.c. the majority of the triplet probabilities for the effectively robust models are in the same cone as the standard testbed models.
Discussion. Mania & Sra (2020) proved that models' accuracies are collinear under distribution shift under the assumptions of (a) low dominance probabilities and (b) distributional closeness. Conversely, if deviations from the linear fit, at least one of these assumtions must be violated. Our results show that effectively robust. models (which do not exhibit collinear accuracies under distribution shift) break the dominance probability assumption rather than the distributional closeness assumption. We note that ImageNet and ImageNet-V2 are explicitly constructed as to similar, so for other OOD test sets one might expect that the distributional closeness will also be violated.
C.11 Replay Buffer: Replaying Images from the Pre-trained Dataset
Our experiments using a replay bufler discussed in Section 5.5 were conducted on a ResNet-18 model that. was pre-trained on ImageNet.
As a baseline, we use the pre-trained ResNet-18 18 model fino-tuned on only CIFAR-10 in Appendix C.3 Next, fine-tune the pre-trained ResNet-18 model on both CIFAR-10 and ImageNet using batch size 256 and
27

---

Published in Transactions on Machine Learning Research (09/2022)
Effective Robust (%)
100D
Elfective Robust (%)
1000
10020
10000
2-
-2
20
40
60
80
100
20
40
60
80
100
CIFAR-10 Test Acc (%)
CIFAR-10 Test Ace (%)
(a) Randomly initialized
(b) Pre-trained
Figure 19: Since models fine-tuned on more difficult examples show more ER, we start training with the full CIFAR-10 training We then gradually eliminate the easiest examples (where difficulty is determined by the C-score) until we only have 1,000, 5,000 or 10,000 examples left by the last training epoch. (a) shows the FR during training for ResNet-18 randomly initialized model, and (b) show the same model initialized with ImageNet pre-trained weights. In both cases this dataset schedule does not help maintain higher ER at high ID accuracies.
0.7
Tesbed
ImageNet Probabilities
0.6
BIT-L-R152dd max ERt
BiT-L-RtSSid conwrgged
CLIP amo-hot
0.5
CLIP-LBFGS
0.4
0.3
0.2
0.1
0.0
0.2
0.4
0.6
ImageNet Probabilities
Figure 20: piilllt-robabbilttes between all models in our testbed in addition to two CLIP models and two BiT-L models. 99.8% of all triplet events are inside the code as defined in Mania & Sra 2020)) and there is no observable violation of the distributional closeness for the effectively robust models.
28

---

Published in Transactions on Machine Learning Research (09/2022)
pitaanes possnetSlaggs
14
Basnet
DTT L. R15204 Converg
cabnee smntl Et
polyut
inception yearat Ieeeennt atteld)
lbcepttiene
dpat
opads
no.respecnco
pespext 6424d
se reancl 52
reonet 52
10
inceptiin vd
se (nommi33))
resnetlo]
g
dousao
8
rosee
denseno120)
I. 101 Lt
calfermeentll
CLIP--LRFCC
dpuds
demomett
respet 20031
denseneti
basnetammbbll
nassact paobic
vog (il los
var 1t.ft
Ve to
Inception.
CLIP
M Se
Lo ooe
2
Figure 21: Pairwise dominance probabilities on ImageNet between all models in the testbed in addition to the two CLIP models, and two BiT-L modcls.Th The effectively robust modcls (CLIP zero-shot and BiT-L Max ER) have high dominance probability relative all other models, which means that they make different predictions than all other models in the testbed. Even after fine-tuning of the CLIP and BiT-L model, see from the heatmap that these models still make more different predictions than other models despite being different from the testbed models in Figure]
29

---

Published in Transactions on Machine Learning Research (09/2022)
learning rate 0.001 for 100 epochs. The replay buffer model usee two prediction heads, one for ImageNet and one for C1FAR-10. For ImageNet, we kept the original 1,000 class prediction head from the pre-trained model, and the CLFAR-10 prediction head was randomly initialized. During fine-tuning, we varied the ratio of ImageNet to CIFAR-10 images to be 1, 10, or 100. The results are shown in Figure 22
We observe that none the replay buffer models are able to maintain high ER at high ID accuracy. This is despite the fact that such a training procedure still maintains by design high accuracy on the pre-training dataset. Before any fine-tuning, the top-1 accuracy of the pre-trained ResNet-18 on the ImageNet test set is 69%. This drops to 39% during tunetunnig CIFAR-10 when early stopped at 92% CIFAR-10 test accuracy; however, with the use of replay buffer, the final ImageNet test accuracy is 63% at the end of fine-tuning on CIFAR-10.
This is intriguing since it demonstrates that a model that is forced to maintain high accuracy on a different OOD dataset during the course of fine-tuning on the target dataset does not do better on the (closer) OOD dataset of interest. For this particular setup, one might have imagined that a model that continues to perform well on ImageNet during fine-tuning CIFAR-10 has learned more general properties of the image data manifold than it otherwise would have, and that this might be beneficial in ER on CIFAR-10.1.
In conclusion, using a replay buffer does help prevent catastrophic forgetting, but it does not prevent the ER on CIFAR-10 from disappearing towards the end of fine-tuning.
Effective Robustness (%)
10
RoNNee-18
Training
Preirainod
RooNee-18 ReplayPuffer <100 Testhed Models
ResNer 1s Beply xt
ResNec BopleeButie
20
60
80
CIFAR-10 test ace (%)
Figure 22: We study the effect of using buffer replay on the ER during fine-tuning on CIFAR-10 for a
pre-trained ResNet-18 model. We compare a standard ResNet-18 to training a ResNet-18 where we see 1, or 100 TmageNet batches for every CIFAR-10 batch. Despite minor differences the x1, x10 and x100 curves, none of them are able to retain ER beyond the standard ResNet-18 model.
C.12 Class Mapping
Instead of starting from a randomly initialized prediction head when fine-tuning a pre-trained model on C-F,R-10, we can use a pre-trained prediction head we map the ten CIFAR classes the 1,000 ImageNet. classes. For the map from CIFAR-10 to ImageNet, we use the map as defined by DDallww et al. (2018) in the construction of the CINIC-10 dataset. With this setup, conduct several experiments to test if there is any benefit from the pre-trained prediction head in maintaining ER at high ID accuracies.
Since each CIFAR-10 class can map to multiple ImageNet classes, we use a cross-entropy loss where we select. the max logit across the multi-target classes. We train the last layer of a BiT-M-R152x4 1K model on CIFAR-10 using learning rate 0.001, batch size 32,768 for 1000 epochs. All results are shown in Figure 23
Replay buffer with class mapping. Since CIFAR-10 and ImageNet now have the same target classes, we repeat the buffer replay experiments using the same prediction head for both datasets. Tn Figure 23
30

---

Published in Transactions on Machine Learning Research (09/2022)
Effective Robustness (%)
CIFAR-10
Elfective Robustness (%)
CIFAR-IO,
CIFAII:10
ImsgeNet CIEAR-10 ImageNit x100
CHFARI- A=
CIEAR-10, A
3-
2
2-
60
70
80
90
100
90
100
CIFAR-10 Test Ace (%)
CIFAR-10 Test Acc (%)
(a) Replay Buffer
(b) Weight regularizer
Figure 23: Using class map from CIFAR-10 to ImageNet target classes, we can finc-tune CIFAR-10 by initializing the prediction head to the pre-trained ImageNet. model. In (a) we fine-tune the last layer of a BiT-M-R152x4 1k model when training on only CIFAR-10, or using a replay buffer with 1 or 100 times as many ImageNet images. In (b) we fine-tune on only CIFAR-10, but using a L2 regularization term around the pre-trained weights, where À is the pre-factor of the L2 term in the loss function. However, none of them are able to maintain high ER at high ID accuracy.
we show the results from training using the same number of images from CIFAR-10 and ImageNet, and also with 100 times more ImageNet images, but we see no improvement in maintaining high ER.
Weight regularization. To test if any of the benefit the pre-training comes from the pre- pre-trained prediction head, we introduce 1.2 regularization term to keep the weights close to the pre-trained weights of the prediction head. We introduce pre-factor À to the L2 regularization in the loss, but in ER from small À despite a small reduction in ID accuracy relative to the results in Figure (3(a) For large X, the L2 regularization dominates, and we do not even reach high ID accuracies.
improvement
Shape of the Effective Robustness Curve
In our experiments, observe that the ER of pre-trained models starts out near zero, increases during training, reaches a peak, and then diminishes towards the end of training.
Interestingly, we find that the existence of the peak is in part cansed by the linear fit in logit space and the definition of ER. To understand this, we plot the ER of the y = x line (i.e. the line "Same accuracy (y = x)' in Figure and 2 along with the ER of pre-trained BiT models in the left plot of Figure 24 (CIFAR-10) and 25 (ImageNet). The ER of the y curve also has the same observed shape with a peak. Hence, isolate the effect of deviating from the linear relationship from the natural shape of any ER line also show the ER as fraction of the accuracy gap between the linear fit and the y line in the right plot of Figure 24 for CIFAR-10 and Figure 25 for ImageNet. We observe that these curves do not have the same maximum as observed in the ER, and instead they have a mostly decreasing trend during fine-tuning.
D.1 Mixed Classifier
Due to the convex shape of the testbed model accuracies (without the logit scaling), one way to construct a model that has ER is to randomly sample predictions from a low accuracy bestbed model with probability a and from a high accuracy testbed model with probability 1 1-a a. We call this model a mixed classifier with parameter a. As a varies from to 1, the accuracies on the ID and OOD test sets traces out a straight line between the accuracy of the random and highest performing model, which will have ER due to the convexity of the logit-scaled linear fit. While mixed classifiers hold theoretical interest and should be analyzed more in
31

---

Published in Transactions on Machine Learning Research (09/2022)
15
Effective Robustiess (%)
Lngletic Regression Protrained Features
10
of Accuracy Gap
DIT DIT-L-REsess-IF
DIU-M-RR1524
80
BIT: 36-33--8 M-R-MMRG6606-21 21k
BTT 02--S-30 S----11224411 1k
5-
40
-5
Logistic Regresiio
Postrsiond Festures
No Accuracy Gap
20
-10-
R162x4
B-T R1122ed
BIT 2-12262--1
Tistbed Models
15
60
80
20
100
CIFAR-I0 test ace (%)
CIFAR- 10 test (%)
Figure 24: Left: The ER of pre-trained BIT models during finettning on CIFAR-10. We include curve indicating what the ER would be for a model that has no accuracy gap between the original and new test sets. Right: The ER as a percentage of the accuracy gap. As a percentage of the accuracy gap, the pre-trained models are slowly decreasing during fine-tuning.
100
Effective Robustees (%)
Loglstic Regression Pra-Tralned
Legistic Regevssion Pre-Trained BeT-L-RISN:s-JFT
No Accutacy Cap
15-
BIT-L-RIG2x-IIFT
% of Aceuracy Gap
80
RIDUL-
BTT-L-FOod-IFT
BIT-L-R5SI--FTT
10
Zaro ER
60
Tretbed
20
20
40
60
80
100
20
40
60
80
ImageNet (top-1, 1, %)
ImageNet (top-1, %)
The ER of pre-trained BIT models during fine-tuning ImageNet. We include a curve indicating what the ER would be for model that has no accuracy gap between the original and new test sets. Right: The ER as a percentage of the accuracy gap. We note that as a percentage of the accuracy gap the pro-trained models are slowly docreasing during finc-tuning.
Figure 25: Same in Figure 24 for ImageNet. Left:
future work, their in practical applications is limited since cannot construct mixed classifter that has both high ER and high ID accuracy.
32


Reviews:
[{"review_id": "sU9JMtcXWu", "reviewer": "Reviewer_Ctd4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ This work provide an extensive analysis for OOD robustness during pre-training and fine-tuning, which may provide new insights for understanding the dynamics of OOD generalization in large-scale machine learning.\n\n+ The paper is well-written and easy to follow. The problem is well-motivated, and the descriptions of the analytical experiments are very clear.\n\n+ This work raises new challenges regarding how to mitigate the reduction of effective robustness during fine-tuning. This challenge is well-motivated and also has practical value, given that the “pre-training then fine-tuning” paradigm has been widely adopted in large-scale machine learning. The authors provide initial exploration with several potential solutions, which is a good launching point for future research.\n\nWeaknesses:\n\n- Most conclusions from the analysis in this work are somewhat obvious. Some observations might be known to the community before this work, for example, pretraining with a larger and more diverse dataset promotes better OOD generalization.\n\n- In this work, the authors only present empirical observations but don’t provide further analysis for the reason. For example, they observe that the effective robustness would be reduced during fine-tuning, but don’t provide more analysis to show _why_ it can happen. More in-depth insights on this would strengthen the work. \n\n- The conlusions in this work are primarily supported by empirical results without theoretical guarantee. It is unclear whether the results hold under different types of distribution shifts or different modalities of inputs (such as language models). It somewhat limits the contribution of this work although the empirical analysis is meaningful.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors aim to investigate how the pre-training and fine-tuning processes affect the “effective robustness”, which measures the performance gap between in- and out-of-distribution data. First, they identify that pre-trained models have high effective robustness and analyze how properties of the pre-training process affect it. Second, the authors empirically show the advantage of effectively robust models over other models on hard example predictions. Lastly, the authors present the issue of losing effective robustness during fine-tuning and provide an initial discussion with several potential solutions. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "In my view, this work is overall thorough and convincing enough in its current form. The authors may improve their contributions by providing in-depth analysis with experiments or theoretical results, as described in the weaknesses. \n\nMinor suggestion: The meaning of the blue error bar in Figure 4c is not very clear. The authors may need to add more descriptions to clarify.", "broader_impact_concerns": "I do not have ethical or broader impact concerns for this paper.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TFft0dscTO", "reviewer": "Reviewer_QbE1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Primary claims of the paper\n\nThe primary reviewing criterion given by TMLR is to determine whether the paper’s claims are justified. In this spirit, I identified four main groups of claims in the paper, and have evaluated their justification below.\n\n- “Pre-trained models consistently have high ER across different choices of architectures on both ImageNet and CIFAR-10.” Figure 3 confirms this. \n    - “High” ER is a bit subjective, but potentially merited given the poor performance of the baselines. This claim is also backed up by prior work (e.g. Radford et al.). \n\n- “We find that pre-trained models with larger architectures exhibit higher ER” and “Figure 4(b) shows that maximum ER increases as we pre-train on larger and more diverse datasets.”\n    - Both claims are weakly confirmed by Figure 4. However, the error bars on Figure 4a are quite large relative to the effect size, and the ambiguous effects of R50x1 in Figure 4b suggest that the story may not be as cut-and-dried as the one-line “more diversity is better” narrative suggests. \n- “...we find that the dominance probability assumption does not hold for models with ER” \n    - This is stated to be backed up by Figure 5, which appears to confirm that the dominance probabilities of effectively robust models are greater than those of testbed models, though there are some clarity issues that make me a bit unsure of this. The $\\mu_i$ and $\\mu_j$ terms in the x-axis of figure 5a are not defined in the main body, which makes this figure somewhat more difficult to interpret, and it is not clear what the indices $i$ and $j$ refer to, again limiting my ability to evaluate the figure. Additionally, the non-zero accuracy of some testbed models on “examples all testbed models get wrong” is counter-intuitive. Explanations of these two points would improve my confidence in this claim.\n- “…ultimately we find that none of the techniques are able to avoid the loss of ER [at high ID accuracies].”\n    - In each of the three methods considered, the authors’ claims are backed up by their empirical findings in the class of models considered, though the set of model architectures evaluated are not necessarily sufficiently broad to rule out this possibility in _all_ architectures.\n\t\n\n### Strengths\n- The paper provides a comprehensive empirical analysis of a previously-observed phenomenon. In particular, the study of the dose-response effects of dataset/model size on ER provides interesting information beyond prior point observations of improved robustness in pretrained models.\n- The focus on natural distribution shifts is well-motivated.\nI thought the following observation was particularly interesting: “the high dominance probability of the effectively robust models suggests that they have a different notion of difficulty compared to that of the testbed models”. My interpretation of these results is that pre-training a large network on a diverse set of visual images builds more ‘human-like’ inductive biases into the network, which yield higher ER, but that over the course of fine-tuning a neural network will override these biases in order to attain high accuracy on a smaller dataset. I would be curious to see whether the analysis found in [1] might be relevant here to shed light on this phenomenon (e.g. by comparing a feature’s usefulness on in- and out-of-distribution data).\n- The negative results concerning fine-tuning, replay buffers, and zero-shot adaptation may be informative and potentially save other researchers time and resources.\n- The experiment protocols are for the most part easy to follow, though as I note below it would be useful to provide more detail on how the large number of baseline models are generated.\n- The figures (bar figure 5, see comment above) are easy to parse, clearly labeled and straightforward to interpret.\n\n[1] Adversarial Examples Are Not Bugs, They Are Features. Ilyas et al., NeurIPS 2019\n\t\n\n### Weaknesses\n\n- The effect sizes measured in many experiments are relatively small, suggesting only modest improvements in robustness from the methods studied in this paper. In contrast, the paper often uses terms such as “high” ER without qualification. A 4% improvement in accuracy when the generalization gap is 5%, for example, is quite different from one for a model whose generalization gap is 20%, and it’s not always obvious how close the ER models come to closing the gap.\n- Some terms are described at a high level but not explicitly defined: in particular, the use of “testbed models” in the introduction and abstract and “dominance probability”, which makes it difficult at times to understand and contextualize the corresponding results. \n- The overall finding that pre-trained models exhibit effective robustness early in training but then exhibit robustness similar to testbed models once they have been fine-tuned to match the ID accuracy of the baselines hasn’t been shown before to my knowledge. It does however present a somewhat limited improvement in our understanding of deep neural network generalization when taken in the context of prior work studying effective robustness. A deeper understanding of what is happening in the networks as they lose their robustness and why this occurs would strengthen the relevance of this contribution to the community.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies out-of-distribution generalization to distributionally similar datasets (e.g. CIFAR-10 and CIFAR-10.1). It identifies models that exhibit “effective robustness” (ER), whereby the model’s accuracy on out-of-distribution data relative to its in-distribution performance lies above a linear trend exhibited by baselines. These models share the feature of being large neural network architectures pre-trained on large and diverse datasets. These findings consider a slightly broader class of models than previous work identifying the effective robustness phenomenon, but largely present a replication and confirmation of prior work.\n\nThe paper goes on to conduct an empirical analysis of various properties of effectively robust models, including their behaviour on data points misclassified by baseline neural networks and the role of both the pre-training and fine-tuning datasets on effective robustness. This analysis identifies a dose-response effect between some training variables and ER, whereby increasing architecture size and dataset diversity increases effective robustness.\n\nFurther analysis reveals that effectively robust models produce qualitatively different predictions on challenging data points than those from the testbed. This is characterized by the `dominance probability', which measures the fraction of data points misclassified by baseline models that the ER models classify correctly. \n\nThe paper concludes with an exploration of a variety of methods which could plausibly increase effective robustness, finding that none of these methods produce significant improvements in effective robustness at high in-distribution accuracies. \n\n The principal findings of the paper can be summarized as follows: models pre-trained on large and diverse datasets are more robust to natural distribution shifts than models trained one-shot on the fine-tuning dataset; however, this increased robustness declines at high in-distribution accuracies and intuitive approaches to preserving effective robustness fail to do so in this high-accuracy regime.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "### Main requests\n- I would like the authors to provide more comments on how and why the testbed models were selected. This is important information, as the paper’s contribution depends on the comparison between ER models and this testbed. Additionally, the large number of data points in the figures relative to the number of listed architectures suggests that several variations on each architecture or random seeds were evaluated in order to generate the plots in the paper. This should be clarified.\n- Relatedly, could the authors comment on whether the set of testbed models could be made more diverse? It looks like only up to 6 distinct models were used in any given evaluation, and these tend to be drawn from similar families of convolutional architectures. Would a family of vision transformer architectures exhibit similar behaviour? This seems important for the generality of the claims made in the paper, to ensure they are not specific to convolutional architectures.\n- The claims of high ER should be justified where the effect size from pretraining is small, or it should be explained why the small percentage improvements are actually significant. \n- I would be particularly interested in seeing a deeper analysis of the different notions of example difficulty exhibited by ER vs baseline models. The ER models and the baselines tend to agree on ~90% of mislabeled samples even in the highest-disagreement models, which is large relative to the disagreement within the testbed but not in absolute terms. Right now the claim that the ER models have a different notion of example difficulty seems a bit strong given this relatively low disagreement, but I think additional qualitative or quantitative analysis of the learned features could be quite informative. I don't have any concrete ideas for this beyond e.g. studying the evolution of the correlation of different feature activations with the target classes on the in- vs out-of-distribution sets over the course of fine-tuning, so this request would be nice to have but isn't essential.\n- The negative results on replay buffer training would be more compelling if a) the findings in Figure 22 were replicated in a larger model (as previous results suggest larger models will exhibit greater ER) and b) the results were compared against the idealized robustness curve shown in Figure 24. This is important for the generality of the claim of the negative result. \n\n### Minor requests \n- The definition of dominance probability should be included in the main body (the x-axis in figure 5 uses notation only discussed in the appendix).\n- \"Testbed models” is often used without explicit definition of and motivation for the exact form of these models. Why these ones? Are they motivated by prior work? Do they represent the current state of the art?\n- In section 5.5, the paragraph at the bottom of page 9 refers to the results of section 5.3, and is worded to suggest that additional results are provided but does not provide a reference to where these results can be found.\n- Table 1: what is the significance of the horizontal bars? Do these indicate that the cross-product is only taken within a row, or also between rows?\n\n### Errata\n- In the caption of figure 2 Taori et al should be \\citep not \\citet \n- Figure 23 is missing a label in the legend for what I assume are the testbed models. The caption also doesn’t specify how the error bars are obtained.\n- p26 \"Oour results\"\n", "broader_impact_concerns": "I do not have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8Ut01Q3fVs", "reviewer": "Reviewer_VHU9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow. \n- The authors did comprehensive empirical studies about the evolution of OOD robustness throughout the training. This alone is an important contribution to the field given ML with distribution shift is such a challenging problem.\n- I like the replay buffer experiments in Section 5.5. When reading the paper, the hypothesis in my head is about catastrophic forgetting, so this experiment is exactly what I hope to see in the paper. Training with a replay buffer does not help is surprising to me, but I would take that with a grain of salt as there could be many other confounding factors. I encourage the authors to try harder on this one.\n\n\nWeaknesses:\n- As a paper on OOD robustness, the authors fail to define what is OOD data. White noise is also OOD data in my opinion, and in this case, the OOD performance has nothing to do with ID performance. Without clarifying the data distribution, all the arguments made in the paper are misleading. That being said, I understand that people in the community have been struggling to get a satisfying definition of OOD data as it is hard to reason about data distribution, and it is not just this work.\n- Though some of the observations made in the paper are quite interesting, they are not surprising. For example, it is easy to understand that a larger and more diverse dataset could boost OOD robustness if pretraining is helpful. It would be more impressive if the authors could dig deeper into the underlying mechanisms rather than just presenting some results. In Section 5.4, the authors try to under why some models yield stronger robustness w.r.t OOD data in the middle of training, but again the conclusions are quite superficial. What I learned from that section is that dominance probability is not a good metric to predict OOD robustness, which I don't find very interesting from the understanding perspective.\n- The authors put too much emphasis on effective robustness, but what we really care about is the OOD accuracy. As long as the model achieves a high OOD accuracy, effective robustness doesn't really matter. As in standard ML, we care more about the test accuracy rather than the generalization gap.\n- All the experiments are about image classification. To be honest, I think the experiments are quite comprehensive, but it would be a big plus if the authors could also try on other domains.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the evolution of out-of-distribution robustness in neural network training. With careful and extensive experiments, the authors made an interesting observation that pre-trained models tend to exhibit \"effective robustness\" in the middle of training while randomly initialized networks do not. In addition, the authors showed that pretraining with a larger dataset, more diverse, or harder training examples boost out-of-distribution robustness. Unfortunately, the benefit of using pre-trained models in terms of out-of-distribution robustness vanishes at convergence.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I like this sort of empirical \"science\" paper and I thank the authors for their efforts in conducting comprehensive experiments. But I hope such a paper could be deeper in understanding. In general, I think we should do research with three questions \"what? how? and why?\" This paper did very well in terms of \"what\" as it presents solid results, but I don't think the authors explained \"how\" and \"why\".\n\nOverall, I think this paper is good enough to warrant acceptance. But I hope the authors take this further with experiments in other domains and explore the aspects of how and why. I believe this may eventually lead to models with both high accuracies and effective robustness.", "broader_impact_concerns": "I don't have any concerns regarding the broader impact of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents an in-depth empirical study of "effective robustness" (ER), that is, exhibit stronger OOD performance than baselines that suffer a drop, but otherwise follow a linear trend. Through large scale experiments, the probe how fine tuning, the amount of data, and example difficulty affect ER. The chief criticism of reviewers was that the work did not shed much light on the mechanisms underlying the phenomena they documented. Despite this limitation, the reviewers and I agree that the work provides a solid empirical foundation for other work.
Decision: Accept (poster)