The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
Authors: Anders Johan Andreassen
Yasaman Bahri
Behnam Neyshabur
Rebecca Roelofs
Venue: ICLR 2022 Submitted (2022)
Keywords: out-of-distribution
generalization
robustness

Abstract: Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across a testbed of models. Models that are more accurate on the out-of-distribution data relative to this baseline exhibit “effective robustness” and are exceedingly rare. Identifying such models, and understanding their properties, is key to improving out-of-distribution performance. We conduct a thorough empirical investigation of effective robustness during fine-tuning and surprisingly find that models pre-trained on larger datasets exhibit  effective robustness during training that vanishes at convergence. We study how properties of the data influence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difficulty of the dataset. We also find that models that display effective robustness are able to correctly classify 10% of the examples that no other current testbed model gets correct. Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.

Under review as a conference paper at ICLR 2022
THE EVOLUTION OF OUT-OF-DISTRIBUTION ROBUST- NESS THROUGHOUT FINE-TUNING
Anonymous authors Paper under double-bling review
ABSTRACT
Although machine learning models typically experience a drop in performance on out-o--distriuution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across testbed of models. Models that are more accurate on the out-of-distribution data relative this baseline exhibit "effective robustness" and are exceedingly rare. Identifying such models, and understanding their properties, is key to improving out-of-distribution performance. We conduct a thorough empirical investigation of effective robustness during fine-tuning and surprisingly find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes convergence. We study how properties of the data influence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difficulty of the dataset. We also find that models that display effective robustness are able to correctly classify 10% of the examples that no other current testbed model gets correct. Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the -iitt--idittiittiinn accuracy of state-of-the-art models.
INTRODUCTION
The ability to generalize to data not seen during training is essential for the widespread trust and adoption of machine learning models. In practical applications of machine learning, we typically train and fine-tune on dataset which are in-distribution (ID) with each other, but when deployed the model will face shifts from this distribution, and empirically the vast majority of models show significant (OOD) data (Quioonero-Candel et al. 2009
drop in performance from ID data to out-of-distributee
Recht et al. 2019 Biggio & Roli 2018 Szegedy et al. 2013 Hendrycks & Dietterich) 2019 Azulay
& Weiss 2018 Shankar et al. 2019 1 Gu et al. 2019 Torralba & |fros]20111 2011). Common examples
include data captured in a different environment, like time of day or geographical location (Koh
et al. 2020); noise or small corruptions of the input data (Hendrycks & Dietterich/ 2019 Geirhos
et al. 2018): or adversarial examples created to explicitly fool neural networks into making incorrect
predictions (Szegedy al. 2013 Biggio & Roli 2018).
Although the performance gap on OOD shifts is pervasive, it follows an intriguing pattern: there is clear linear relationship between a model's 's final performance on ID and OOD data (Taori et al.) 2020 Recht et al. 20199 Yadav & Bottou, 2019 Miller et al., 2020). In other words, given a model's performance on an ID test set, a linear fit can accurately predict what the performance drop will be on the OOD test set. This also implies that a certain amount of improvement OOD accuracy can be explained by an improvement on ID accuracy. The linear relationship holds across wide range of models and is well-established for several robustness benchmarks, including image classification on both synthetic and natural distribution shifts (Recht et al.] 2018 Yadav & Bottou] 2019) Recht et al.. 2019; Taori et al., 2020), 2D object detection (Shankar et al. 2011), and question-anse answer models in NLP Miller et al. 2020).. Since even the nighest-performen
models will still have a gap between ID and OOD accuracy, the linear relationship reveals that our current methods are insufficient for addressing OOD shift.
Models which lie above the linear fit are said to exhibit effective robustness (ER) Taori et al. 2020).
which measures the model's OOD accuracy relative to the fit (see Figure [(a) for an illustration). Models with nonzero ER deviate in their OOD behavior in manner that is qualitatively and

---

Under review as a conference paper at ICLR 2022
Oot-od- Thorriattion Avn (36
Eifoctree Reloutooes (PF)
ELL MELEE
Effeciive
Robustr ese
Losiiiiiaiioon Tost Jun (86
6n
ImageNSit (lrg-) 15)
ImsgeeSet (trg-) (8)
(a) Illustrating the definition of effective robustuess (ER)
(b) Accuracies on ImageNet and ImageNetV2
(c) ImageNet accuracies vs, ER on ImageNetV2
Figure 1: During fine-tuning on ImageNet, pre-trained models exhibit effective robustness (ER) while aanomly-initialinn
models do not. However, the ER of pre-trained models diminishes throughout the fine-tuning process and vanishes altogether at the end of training.
(a) illustrates the
definition of ER that we define more precisely in Section 3 Subfigure (b) and (c) show ImageNet test accuracy the x-axis for either randomly initialized pre-trained models throughout ImageNet training (we evaluate accuracy on checkpoints every epoch of training). (b) plots the ImageNet-V V2 test accuracy the y-axis while (c) shows the effective robustness for the same models. ER also be achieved by zero-shot models. In (b) and (c) we also show a BiT model that was trained on JFT and evaluated on ImageNet using a class-map between the two
of targets (see Appendix C.7 and we show the best performing zero-shot and linear probe CLIP models from |Rafford et al. (202 la). The BiT and CLIP zero-shot models have comparable ER, but at different ID accuracy, while the linear probe CLIP model has no ER. Standard testbed models (see Section 4 fully trained on ImageNet are in blue and are shown with 95% Clopper- Pearson confidence intervals.
quantitatively different from what we can currently achieve. We seek to thoroughly investigate when the pre-training and fine-tuning paradigm gives rise to models with nonzero ER, and understand factors that control this behaviour.
Models with high ER (> 1%) are exceedingly rare. First, the most notable example is the recently proposed sero-shot CLIP model (Radford et al., (2021b), and subsequent work (Wortsman et al. (021) has found that weight-space ensembling of zero-shot and fine-tuned CLIP-style models can improve both ID and OOD accuracy and produce models with high ER. Second, only a handful of the 204 ImageNet models evaluated in Taori et al. (2020) were found to have non-zero ER. Though most of these effectively robust models identified so far were pre-trained on large datasets, the majority of models pre- mre-trained this way exhibit ER, and currently do not know when additional data helps; what effect the choice of architecture has; as well as the effects of dataset and distribution shift on these findings.
In this work, we present an empirical study of the evolution of ER throughout fine-tuning. By studying the evolution, we find intriguing properties that missed by focusing only on models at convergence: pre-trained models exhibit ER, while randomly-initialin models do not (see Figure T).
Summary of Contributions:
(Section 5.1 5.3) We identify pre-trained, fine-tuned models as an entire class of effectively robust models (that match the ER of CLIP (Radford et al. (021a)) and investigate how details such model size, dataset size, and example difficulty influence ER. We find that the vanishing of ER at convergence depends on the distribution shift, and on some datasets pre-trained, fine-tuned models may still exhibit non-zero ER at convergence.
(Section 5.4p
We analyze properties of pre-trained models at the peak of their ER, including
particular metrics defined in previous theoretical work (Mania & Sral 2020). We find that effectively robust models make remarkably dissimilar predictions compared to standard models, and are able to correctly classify 10% of the examples that no other model gets correct.
(Section 5.5) We find that pre-trained models gradually lose their ER throughout fine- tenintnnngg even as both the ID and OOD accuracies of the model simultaneously increase. We discuss several potential solutions to mitigate this problem, but find that none of them are able to maintain high ER at high ID accuracy.
2

---

Under review as a conference paper at ICLR 2022
2
RELATED WORK
Here we review some prior findings that are key for understanding the context of our results.
Linear trends under distribution shift. In recent replication efforts, researchers carefully recreated new test sets for the CIFAR-10 and ImageNet classification benchmarks (Recht et al. 2018 2019). Despite following the original collection procedures as closely as possible, some shift was introduced. Evaluating an extensive testbed of trained models on the original and new tests revealed a clear relationship between accuracy on the original and new tests that well-captured by linear fit with positive slope. Follow-up work also found linear trends for new test sets on MNIST (Yadav & Bottou 2019) and the SQUAD question-ane guestion-answer answer dataset (Miller et al. 2020), geographic distribution shifts for 3D Object Detection in self-driving cars (Sun al., 2019). and synthetic distribution shifts (Taori ct al. 2020). We find that models pre-trained on larger and more diverse datasets can break the linear trend of accuracy under distribution shift in the middle of the fine-tuning process, and we investigate factors that affect this.
Examples of effectively robust models. Models that lie off the linear trend (effectively robust models) are historically extremely rare. In recent extensive empirical study, researchers evaluated 204 trained ImageNet models spanning wide range of architectures and training techniques on several popular natural and synthetic robustness benchmarks for ImageNet (Taori et al. 2020). Across several natural robustness benchmarks, the main outliers with positive ER and high accuracy on ImageNet were all models that were pre-trained on larger and more diverse data than the ImageNet training set. They included a ResNet152 model trained on 11,000 ImageNet classes (Wu 2016). several ResNeXt models trained on billion images from Instagram (Mahajan et al.) 2018). and the EfficientNet-1.22 (NoisyStudent) model trained on a Google-internal JFT-300M dataset of 300 million images (Xie et al. 2020). However, not all models that were pre-trained on larger datasets showed effective robustness.
Another recent work, (Radford et al.) 2021a), observed that zero-shot CLIP classifiers have larger ER than CLIP models that had been fine-tuned to the downstream task. While the CLIP model was pre-trained with a contrastive loss that combined components from natural language processing and image classification, our results show that pre-trained ImageNet models that are evaluated in zero-shot manner also have high ER, and, similar to the CLIP model, these image classification models lose their ER once they are fine-tuned to the downstream task. Since the completion of this paper, subsequent work (Wortsman et al. (202T) has proposed weight-space ensembles for fine-tuning (WiSE-FT) that interpolates between zero-shot and fine-tuned weights to find models with high ER as well high ID and OOD accuracy. This method is currently limited to CLIP-style models, and in this work we mostly focus on traditional supervised image classification models.
Model similarity. Recent work offers a theoretical explanation. relying on an assumption of model similarity (Mania & Sral 22200), for why classifier accuracies follow a linear trend under distribution shift. Empirically, Mania al. (2019) observed that the labeling assignments across wide range of ImageNet models are significantly more similar to each other than would be expected by chance, which suggests that the size of the class of function approximators that neural networks learn is smaller than what may be expected priori. Under the assumption of model similarity, Mania et al. (2019) proves that models' accuracies, when evaluated on two distributions, must be approximately collinear, unless the size of the distribution shift is large in certain sense. In our work, we identify and analyze models that are not collinear with other models when evaluated on two distributions, and we find that such models break the assumption of model similarity sufficient to prove the existence of the linear fit.
3 EFFECTIVE ROBUSTNESS
The linear fit to testbed model accuracies on ID and OOD data provides a baseline for how improve- ments in OOD accuracy are tied to improvements in ID accuracy based on current methods. Effective robustness (ER), metric for robustness proposed in TTori et al. (2020), measures the difference between a model's OOD accuracy and that predicted from the bascline. (See Figure [(a) for an illustration.)
3

---

Under review as a conference paper at ICLR 2022
Figure 2: Testbed model accura- cies (Recht et al. (2018)) show linear relationship in rescaled logit-space caattsspcceTTrriiii et al. (2020), (a) shows the linear fit in sppce, and (b) shows the same fit trans- formed back to linear-space.
Looks
CUPUU--00 lool
CIPAB-10
U100 last (95)
CLEAR-1 (90)
To define ER, fix an ID and OOD test set and let ((r) be the baseline predicted OOD test set accuracy for a given accuracy x on the ID) test set. Given a set of models evaluated on both the II) and OOD test sets, 3 can be computed by performing a log-linear fit between the models' accuracies on the two test sets. Similar to Taori et al. (020), we empirically find that the best linear fit comes
from rescaling the accuracies using the logit function, logit(a) log I-a
before performing the a
linear fit. In Figure 2 as transformed back to linear-space. See Appendix A for more details.
we show the log-lincat fit for CIFAR-10 vs. CIFAR-100 10.1 in logit-space well
The effective robustness of model is then defined
p(f) acC (J) ((accin (f)),
where (fC and (f) represent the model's accuracies on the ID and OOD test sets, respec- tively. Graphically, ER represents the distance above the testbed line to a model's OOD accuracy, as shown in Figure ((a)
We note that ER is distinct from the absolute accuracy a model can achieve on the OOD data. For instance, the effectively robust model (green point) in Figure [t)]l has low absolute OOD accuracy relative to other models. An ideal model achieves both high ER and high absolute OOD accuracy.
EXPERIMENTAL SETUP
In this section, discuss which fine-tuning datasets, pre-trained models, robustness benchmarks, and linear fits we use in our experimental evaluation.
Fine-tuning datasets. We evaluate ER throughout fine-tuning (see Appendix B.2 for our definition of fine-tuning) on both ImageNet and CIFAR-10 since researchers commonly fine-tune pre-trained models to these popular benchmarks. For CIFAR-10 in particular, there are several widely available pre-trained ImageNet models that we can easily transfer to the CIFAR-10 dataset.
Pre-trained models. The majority of the pre-trained models study are Big Transfer (BiT) models (Klesnikov et al., 2020), which use a ResNet-v2 architecture (He et al. (016b). with Group Normalization (Wu& & He 2018) and Weight Standardization (Qiao al. 2019). The BiT models come in three different flavors: BiT-S, BiT-M and BiT-L. where S, M and L indicate if the pre-training was done on ILSVRC-2012 (which we denote as ImageNet- 1k or just ImageNett (Russakovsky et al. 2015). ImageNet-21k (Deng et al. 2009). or JFT (Sun et al. (017), respectively. In addition, these models may be further fine-tuned, and "Ik" "21k" on. For example, BiT-M-R152s0 15x44-kk k is the ResNet-152x4 model that was pre-trained on ImageNet- 21k ("M") and fine-tuned on ImageNet-Ik ("Ik"). We also study a series of BiT models that were pre-trained in Djolonga et al. 2021) on different amounts of data from JFT.
or "JFT" indicate the dataset was fine-tuned
Table 1:
The range of experiments we run are represented with the cross product of the models, OOD dataset and pre-training dataset listed below.
Model
OOD
Pre-training dataset
AlexNet, 18.. -152, VGGII_BN, Wide-ResNNet-0--2
CIFAR-10.1
ImageNet-1k
CIFAR- ImageNetV2
ImageNet-I1 1k
BITTR[550x.101x3,15524]
ImageNet-21k, JFT
BiT-R101x3
ObjectNet, ImageNet-R
JFT
4

---

Under review as a conference paper at ICLR 2022
We also study six ImageNet pre-trained PY Paszke et al. 2019) models: AlexNet (Krizhevsky 2014), ResNet-18, 18, -50 and -152 (He et al. 2016a), VGG-11 with Batch Normalization (Simonyan & Zisserman 2014). and Net Zagoruyko & Komodakis 2016).
Robustness benchmarks. Both CIFAR-I0 and ImageNet have several well-established robustness benchmarks. We choose focus naturally oceurring distribution shifts, rather than synthetic shifts which modify images, because they are more realistic and because there are several robustness interventions that work well on synthetic shifts but do not transfer to natural distribution shifts Taori et al. 2020). Thus, for CIFAR-10, we evaluate robustness on CIFAR-10.1, which was created in the replication study of Recht al. (2018) and is one of the few natural distribution shift benchmarks available for it; and for ImageNet, we use the ImageNetV2 (Recht et al. 2019), ObjectNet (Barbu
et al. 2019). and ImageNet-R (Hendryeks et al. 2020 test sets.
Linear fits. Measuring ER requires establishing linear relationship between a model's accuracy on ID and OOD data. To define the linear relationship between the original test accuracy and the OOD test accuracy on 11FAR-10 and ImageNet, we require a testbed of models that span the range from low to high accuracy. For ImageNet, we use the publicly available results from Recht et al. (2019). and for CIFAR-10 10 we use the results from Recht et al. (2018)
5 RESULTS
We overview our main experimental results. First, we establish that pre-trained models exhibit ER during fine-tuning, and we examine how properties of the pre-training setup, such as model size, dataset size, and example difficulty, influence the magnitude of the ER. Finally, we explore ideas that attempt to maintain high ER at the end fine-tuning.
5.1 PRE-TRAINED MODELS HAVE HIGH EFFRCTIVE ROBUSTNESS
We find that models pre-trained on large and diverse datasets exhibit high effective robustness during fine-tuning, while randomly initialized models do not at any point during training (Figure 3 As described below, this observation is invariant to the choice of robustness benchmark (CIFAR-10.1, VV, ObjectNet, and ImageNet-R); model architecture; and the details of the fine-tuning, such as learning rate, optimization algorithm, type of fine-tuning (last-layer versus whole model), and loss function.
CIFAR-10. We evaluate several architectures, including AlexNet, ResNet- 18, ResNet-50, ResNet- 152 and VGG-11-BA throughout fine-tuning on CIFAR- 10 using both ImageNet pre-trained initial- ization and random initialization. At every epoch of training, we measure ER using CIFAR-10. 10.1 as the OOD test set and CIFAR-10 10 as the II) test set. Figure 3(a) plots the ER (%) versus CIFAR-10 test accuracy (%) throughout fine-tuning for all of the ImageNet pre-trained architectures, as well as the randomly initialized architecture that achieved the maximum ER (for the full comparison Appendix TO).. We see that all of the ImageNet pre-trained models have significantly higher maximum ER than either the testbed models or the best performing randomly initialized model. Moreover, observe that ER generally increases throughout fine-tume tt-iinning peaks, and then gradually disappears towards the end of fine-tuning.
ImageNet. For models fine-tuned on ImageNet, we measure ER for JFT pre-trained models at every epoch of training using either ImageNetV2, ObjectNet, ImageNet-R as the OOD test set and ImageNet as the ID test set. We report ImageNet top-1 accuracy, and for ImageNet-R and ObjectNet, we compute accuracy over the respective subset of classes for the two benchmarks. Figure 3(b) shows the I2 ER, and Figures (3c] and [(d] shows ER on ImageNet-R and ObjectNet respectively. In all cases, the pre-trained model reaches maximum ER during fine-tuning which then decreases as we continue fine tuning.
For the ObjectNet and ImageNet-R evaluation, we fine-tune a BiT-L-R1552x-JFTT model on ImageNet and evaluate the same model on 2m ObjectNet, and ImageNet-R. Though the model's s ImageNet accuracy the same in all cases, for ImageNetV2, the ER vanishes (OOD accuracy drops to the baseline set by the testbed models), while for ImageNet-R and ObjectNet there is still non-zero
Additionally, we add several low-accuracy scikit-lcam models which received in personal communication and which we expect he publicly available soon.
5

---

Under review as a conference paper at ICLR 2022
WELLBOOAA CLe Prde Talett Saae
Fillctiv Relaustuee (36)
Elferttre Holnurnes
20
40
6P
03 ImngeeNes (top-1. [6)
110
CIFAR-10 (%)
(a) CIFAR-10.1
(b) ImageNerV2
ImongeNeet-R -R
Oboject Net
ESSction Bobustunce (15)
The and Manilo
ElGectvve Boknatinne
40
InsigeNet (top-1, 90), subelosadd
ImageeNot (top=), 56), subschuerd
(c) ImageNet-R
(d) ObjectNet
Figure 3: Pre-trained models consistently have high ER across different choices of architectures on both ImageNet and CIFAR-10. We plot ER (%) versus ID accuracy (%) at every epoch of training as fine tune (a) pre-trained ImageNet models CIFAR-10 (b,c,d) pre-trained JFT models on ImageNet. Figure (a) shows the ER on CIFAR-10. while Figures (b,c,d) show the ER on ImageNetV2, ImageNet-R, and ObjectNet, respectively. For each plot, we also show the ER and accuracy achieved by converged testbed models in blue with 95% Clopper Pearson confidence intervals. For (a) CIFAR-10.1 and (b) ImageNetV2, we trained a randomly initialized version of all the model architectures and we report the model with the highest maximum ER during training (solid black curve). We observe that the pre-trained models all exhibit high ER the middle of fine-tuning. which eventually decreases fine-tuning proceeds. In contrast, the randomly initialized models do not have significant ER relative to the testbed models.
ER at the end of training. This observation is consistent with Taori et al. (2020), which found that converged pre-trained models are more likely to exhibit nonzero ER on ObjectNet or ImageNet-R than ImageNetV2.
Model architecture. We find that pre-trained models with larger architectures exhibit higher ER. For ImageNetV2 in Figure (3bb we evaluate multiple BiT ResNets with increasing depth and width (R150x1, R101x3, RI R152x4) that were pre-trained on JFT, and we find that models with more parameters have higher peak in ER. Similarly, for CIFAR-10. 10. Figure [[a] shows that larger, more recent architectures such as the Wide-ResNet-50-22 exhibit higher effective robustness than smaller architectures like AlexNet. Moreover, Figure 4(b shows that max ER on CIFAR-10 increases for BiT models of increasing depth and width as we vary the pre-training dataset.
Details of training. We performed detailed study of variety of components of the optimization process, and we found that the ER is not sensitive to these details. See Appendix C for variations in learning rate, loss function, type of optimization (full model vs. last layer).
5.2 ZERO-S SHOT EVALUATION
Recent work (Radford al. 2021a) introduced the CLIP architecture that allowed for zero-shot evaluation on image classification datasets like ImageNet and the ImageNet robustness benchmarks. The zero-shot CL.IP models has high amounts of ER, but unclear the ER comes from the new and very large dataset used for training, the natural language model component, the contrastive training objective, or its zero-shot evaluation. In Figure ((b) we show the ER of the best zero-shot and linear probe CLIP models from Radford et al. (2021a). We also show results from zero-shot evaluation on a BiT model that was pre-trdi on JFT. This is done by creating a mapping between similar classes of JFT and ImageNet. While the BiT model has about 5% lower ImageNet top- I accuracy, it matches the amount of ER that the CLIP model has. This is surprising, and suggests that the shro component of CLIP plays significant role in the high value of ER CLIP achieves.

---

Under review as a conference paper at ICLR 2022
No. Elfection Rolrasians (80
Mor Eifectto Teskeatnne
Effortive Tolcuutonns
Traiiigg LT
Fre Traning
CIRAR-1O Ave
(a) Max ER
(b) Max ER vs.
(c) Varying the difficulty of the fine-tuning dataset
Vs. fraction of JFT dataset size
different pre-training datasets
Figure 4: The maximum ER depends both on the pre-training and the fine-tuning dataset. (a) We evaluate BiT-RI01x3 models pre-trained on varying fractions of the JFT dataset throughout fine-tuning the CIFAR-10 dataset, and we find that maximum CIFAR-10.1 ER increases with the size of the pre-training dataset. Error bars represent standard deviation over 5 runs. (b) We evaluate several BiT models of increasing size (R50x1, RI0Ix3, R152x4) that were pretrained on either ImageNet (S), ImageNet-2IIK (M), or JFT (L) throughout fine-tuning on CIFAR-10, (The BiT-M-1K was first pre-trained on ImageNet-21K and then ImageNet-1k) Maximum 10.A ER increases as the model size and pre-training dataset size and diversity increase. (c) The maximum CIFAR-10.1 ER also depends on the difficulty of the fine-tuning dataset. We select examples based on the C-score Jiang et al. (2200) of CIFAR-10 and train on either the 5000 easiest, hardest or random images in the training set. Training on harder examples yields more ER for both pre-trained and randomly initialized models, but lower CIFAR- 10 test accuracy than training random examples.
5.3 THE IMPACT OF DATA ON EFFECTIVE ROBUSTNESS
We find that the maximum ER depends on both the pre-training dataset and the data the model is fine-tuned on. As we discuss below, larger and more diverse pre-training datasets, as well as more difficult fine-tuning examples, all increase the magnitude of the ER peak.
Dataset size and diversity. In Figure 4(a) we pre-train models on randomly sampled fractions of JFT. We observe that maximum ER increases the amount of pre-training data increases but eventually plateaus for the largest dataset sizes. We suspect the reason for the plateau is the fixed number of iterations used across those experiments, which eventually serves a bottleneck for performance improvements as reported by prior work (Hernandez et al.) 2021 Nakkiran et al.) (2020). Figure (bb) shows that maximum ER increases as we pre-train on larger and more diverse datasets.
We also find that pre-training first on a larger dataset and then switching to smaller dataset leads to lower ER during the final fine-tuning task than setup that only pre-trains on the larger dataset. To see this, consider the BiT-M-IK Ik and BiT-M-2IK in Figure 4(b) Both models were initially pre-trained on ImageNet-21k (as denoted by the 'M'), but the BiT-M-Ik model was additionally fine-tuned on ImageNet-II 1k part of the pre-training procedure. We then evaluated the ER of both models fine-tuned them to CIFAR- 10. Figure (bb) shows that the extra fine-tumin fine-tuning step on ImageNet- 1k gives about a 1% drop in ER relative to the BiT BiT-M-22 M-21k model.
Example difficulty. The ER during training also depends on the data the model is being fine-tuned In Figure we select the 5000 easiest, hardest or random examples from CIFAR-10 training set and only fine-tune on these examples. The difficulty of each example is measured by the C-score (Jiang et al., 2020). Training on easier (harder) examples gives less (more) ER, but training on only the hardest or easiest examples gives lower final accuracy than training on the examples of random difficulty. See Appendix C.9 for more details.
5.4 ANALYSIS AT MAXIMUM EFFECTIVE ROBUSTNESS
To understand the differences between models that exhibit ER and testbed models, we compare the predictions from the former both at the peak ER and at the end of training and the latter.
Dominance probability. Recent work (Mania& & Sra 2020 showed that the linear trend between the ID and OOD accuracies will arise two assumptions are satisfied: low dominance probability and a specific notion of distributional closeness. Dominance probability (Mania & Sra 2020) is similarity
7

---

Under review as a conference paper at ICLR 2022
Dominanee probccility Image
peerplle all Loctral midkle wnimt
331 a.l REE-itiu CUT LEROE
01:23
Asomeny conerples
0.75
0.35
AB
ImageNec (a0g- 56)
(a) Dominance prob. distribution
(b) Pairwise dominance prob. (c) Difference from testbed predictions
Figure 5: Models with high effective robustness have higher dominance probabilities than mod- els with zero effective robustness on ImageNet. We analyze the predictions of the following four models: BiT-L-RR52x: Max ER (A), CLIP zero-shot (B), CLIP-LBFGS (C), and BiT-L-R1524 Converged (D). (a) For each pair of models in the testbed plus these four models, we plot their dominance probability versus their accuracy difference. We find that the CLIP zero-shot model and the pre-trained BiT-L-R152x4 model have higher dominance probabilities on ImageNet than any of the models in the testbed, but once the CLIPP and BiT-I. models are fine-tuned on ImageNet, they align more with the dominance probability distribution of the testhed models. (b) We show the pairwise comparison of the dominance probability on ImageNet between all testbed models (not labeled) in addition to our four models. We sort models in order of increasing top-1 accuracy on ImageNet (see Appendix C.10 for a bigger version with labels for all models). We see that the four labeled models make very different predictions relative to the testbed. (c) For each of the plotted testbed models, we plot the percent of examples that none of the other testbed models get correct. Considering the 4.8% of all examples that none of the testbed models get right, the best BiT model pre-trained on JFT gets 10% of them correct and the zero-shot CLIP model gets approximately 8% correct.
metric between the predictions of two models, measuring what fraction of examples the model with lower overall accuracy gets correct that the model with higher overall accuracy gets wrong. (See formal definition in Appendix ) In this work, we find that the dominance probability assumption does not hold for models with ER, but that the assumption of distributional closeness still holds.
we see that the dominance probability of pre-trained BiT model early-stopped at its maximum ER is shifted relative to the models in the testbed, while the pre-trained BiT model that was fine-tuned to convergence (and has low ER) has dominance probability similar to the models in the testbed. We also find that the dominance probability for the pre-trained BiT' model is very similar to that of a zero-shot CLIP model (see Appendix B.3 details on fine-tuning). Even though the fine-tuned CLIP model has close to zero ER (see Figure 3(b).. we see in Figure 5(a) and 5(b) that its dominance probability is also higher than that of standard testbed models. Unlike the BiT JFT pre- pre-trained model, the CLIP model was pre-trained with contrastive learning and text embeddings, and we hypothesize that these elements may contribute to this effect.
In Figure 5)
Examples that only effectively robust models get right. Models that have high ER are able to correctly classify images that no other testbed model gets correct. We select the 4.8% of images that none of the testbed models get correct, and we find that the ER model with the best II) performance gets 10% of these examples correct, i.e. 237 examples, divided among 198 different classes, with maximum number of three examples in each class. After manual inspection of all examples, we do not see any pattern among the 237 images.
Remarks. Since testbed models all have relatively low pairwise dominance probabilities, there are set of easy examples correctly classified by all models, but harder examples are only classified correctly by increasingly higher accuracy models. This suggests that images can be placed on scale of difficulty, and that this notion of difficulty is shared by all the testbed models. In contrast, the high dominance probability of the effectively robust models suggests that they have different notion of difficulty compared to that of the testbed models, i.e. some images that are difficult for testbed models are not difficult for effectively robust models. As further evidence, effectively robust models are able to classify some images correctly that no other testbed model gets correct. We hypothesize that the different notion of difficulty of the effectively robust models comes from pre-training larger and more diverse data, which could expose them to examples that are dissimilar from those in the fine-tuning dataset.
8

---

Under review as a conference paper at ICLR 2022
5.5 MAINTAINING EFFECTIVE ROBUSTNESS AT HIGH ACCURACY
In the previous sections, we have seen that models pre-trained on larger and more diverse datasets exhibit robustness properties during fine-tuning, but that, depending on the choice of OOD dataset, the ER can vanish the model converges towards the end of training. In this section we explore several ideas for ways to maintain the high ER as the model reaches high accuracies, but ultimately we find that none of the techniques are able to avoid the loss of ER.
Replay buffer. Deep neural networks that are subject to training on two distinct tasks sequentially commonly forget learned information about the first task after training on the second, phenomenon known as catastrophic forgetting (Kirkpatrick et al. 2017; Toneva et al. 2019). A possible hypothesis, therefore, that retaining learned information about the pre-training dataset for instance, by maintaining high accuracy this dataset during the course of fine-tuning downstream tasks may have a beneficial effect on ER. We explore this possibility by implementing a popular technique for mitigating catastrophic forgetting the use of "replay buffer" (Ratclifi) 1990. Rolnick et al. 2019).
Without a replay buffer, we find that a ResNet- 18 model pre-trained on ImageNet and fine-tuned on CIFAR-10 has a drop in ImageNet test accuracy from 69% to 39% when early stopped at 92% CIFAR- 10 test accuracy. When simultaneously fine-tuning on CIFAR- 10 with a replay buffer, we retain 63% ImageNet test accuracy, but we observe improvement in retained ER. See Appendix C.1 for more experimental details and results.
Mapping between different class sets. Using the same constructed map of classes used for zero- shot evaluation (c.f. Section 5.2 and Appendix C.7). J. we can use the target classes of the pre-training dataset for fine-tuning. We use the mapping from CIFAR-IO to ImageNet classes, and we use this for fine-tuning on CIFAR- 10 initializing with the pre-trained prediction head.
First, we explored fine tuning on just CIFAR-I0 images with the 1000 ImageNet target classes. Second, we used replay buffer for combined fine-tuning on CIFAR-10 and ImageNet using the same prediction head. In both setups we see no significant change in ER at the end of fine-tuning. See Appendix C.12 for further details on the experiments and results.
Example difficulty. In Section 5.3 we saw that training on harder examples during fine-tuning increases the ER, but prohibits reaching high final accuracy. We study different training schedules where we vary the difficulty of examples during training, but we find that none of the proposed methods are sufficient to maintain high ER and reach high test accuracy at the end of training.
CONCLUSION
Machine learning models currently experience an undesirable yet predictable drop in accuracy on OOD shifts. The exceptions from this trend - models with ER experience less of a drop in OOD accuracy compared to standard testbed models at the same ID accuracy. The ultimate goal is to produce ER models at state-of-the-art accuracies; however, none of the known models with high ER in the high-accuracy regime, and the models with the highest OOD accuracy have no ER. One way to approach this problem is to understand why some models have ER so that we can replicate this behavior for thth-atf models and decrease the OOD performance gap.
In this work, we demonstrate that pre-trained models in the middle of fine-tuning, as well as zero-shot pre-trained models, represent an entire class of models that exhibit high amounts of ER. With this rich set of effectively robust models, we take some initial steps towards understanding what is unique about them that causes them to outperform testbed models on OOD shifts. Our results suggest that properties of the pre-training or fine-tuning datasets, such as the size, diversity, or difficulty, cause effectively robust models learn an entirely different difficulty scale than that of standard testbed models. Moreover, because ER models make different predictions than standard models and are able to correctly classify examples that no standard models get right, these models are valuable for applications such as model ensembling, which requires prediction diversity.
From our work, we know that pre-training provides a clear benefit to ER, but that fine-tuning on the ID task reduces ER. The future challenge lies in constructing mechanisms to enhance ER in the high accuracy regime. We hope our initial implementations of and investigations into such techniques will be a launching point for researchers for further experimentation.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? arXiv preprint 1805. 2018.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfrrend, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL http:ttppppoooceeaiiiiiia www.eeeiiiinnnunee..eu.ee neurips httpp...cppppee cc/paper//211/ Fi 6ea
pdff
Battista Biggio and Fabio Roli, Wild patterns: Ten years after the rise of adversarial machine learning. Pattern Recognition, 84:317-331, 2018.
Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos Storkey. Cinic- 10 is not imagenet or cifar-10. arXiv preprint arXiv: 1810.03505, 2018.
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 1T1.0///VV..20099522
Josip Djolonga, Jessica Yung. Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander Nicholas D' Amour, Dan Moldovan, Sylvain Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lučić. On robustness and transferability of convolutional neural networks. Conference on Computer Vision and Pattern Recognition, 2021.
URL httth:://aaxiii org/ 08052
Robert Geirhos, Carlos R Medina Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. arXiv preprint T000::888007755 2018.
Keren Gu, Brandon Yang, Jiquan Ngiam, Quoc Le, and Jonathon Shlens. Using videos to evaluate image model robustness. arXiv preprint arXiv: 1904. 10044.0007 /0076, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Earopean conference on computer vision, pp. 630-645. Springer, 2016b.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distributi generalization. arXiv preprint arXiv:2006. 16241, 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293 2021.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Exploring the memorization- generalization continuum in deep learning. CoRR, abs/2002.03206, 2020. URL hitps:// arxiv.org/abor 2002..00222032033
10

---

Under review as a conference paper at ICLR 2022
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3522) 2017.
Grabska-Barwinske Barwinska, et al. Overcoming
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips. Sara Beery, et al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:20/2.07421, 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning, 2020.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv: 2014.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. Proceedings of the European Conference on Computer Vision (FCCV), pp. 181-196, 2018.
Horia Mania and Suvrit Sra. Why do classifier accuracies show linear trends under distribution shift? arXiv preprint 201i:: /5483, 2020.
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. arXiv preprint arXiv:1905. 2019.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 691666916. PMLR, 13-18 Jul 2020. URL mlr www.prees/vee httss//119/mmmmeeeee httml
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap: Good online learners are good offline generalizers. arXiv preprint arXiv:2010.08127, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL UtLL/tttpp//pppers.eeeiieo...e
901 5 oggp://ccc.......aaa...aaaaaaaro
Library.
pdf
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv preprint arXiv:1903. 2019.
Joaquin Quinonero-Candela, Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer. Dataset shift in machine learning. Mit Press, 2009.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervi- sion. 2021a. URL httttttp:... //tp::////./n///
ng "ggTTrasfeerblle Visual Models From Natural Language Supervision. pdf
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 2021b.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2)2285, 1990.
11

---

Under review as a conference paper at ICLR 2022
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.0045. 2018.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? 2019. URL htttt://aaxxiiii www.aiiii......a.ooooooo
wwtp///gggga.ooooooo.oooooe abs/ 10811
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32, pp. 350-360. Curran Associates, Inc., 2019. URL httpitps://rrccceednggg
cc/
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 1-22, 2015.
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. A systematic framework for natural perturbations from videos. CoRR, abs/1906.02166 2019. URL/attp://arxiv. org/ 9966 C2168
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint IrXivv 1556, 2014.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843 003,852,227 2017.
Pei Sun, Henrik Kretzsehmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability perception for autonomous driving: Waymo open dataset. CoRR, abs/1912.04838, 2019. URL http::://wwxww.......o 1912.04838
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2013. httt:://axxv..rrg/// abs 1112/1111111 6199
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt.
When robustness doesn't I promote robustness: Synthetic vs. natural distribution shifts on imagenet, 2020.
IyrFvH
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019. URL RRR[stttps://opereeile feet/Eorum?idse 3mcmmmmmm
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 20/1, pp. 1521-1528. IEEE, 2011.
Mitchell Wortsman, Gabriel Ilharco, Mike Li, Jong Wook Kim, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models, 2021.
W Wu. Classifying images into 11k classes with pretrained model, 2016. URL
https::/giitie http:///....tltccc.c.....ooaaaes http::/eeetttttttst//s/s/ttttaat com/ iisssiiiiiionn
aws 1 hat.a///..eeeeeeeraaaaaaaaac///a
Common/modeeloo..py
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
12

---

Under review as a conference paper at ICLR 2022
Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. arXiv preprint 190. 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 2016.
13

---

Under review as a conference paper at ICLR 2022
A SCALING FOR LINEAR FIT
When modeling the linear relationship between the original and new test set for CIFAR-10 and ImageNet, applying a nonlinear scaling the accuracy has shown to improve the linear fit, especially when considering both very high and low performing models, Recht et al. (2018) 22199) applies a probit scaling the accuracies, and Taori al. (2020) also considers logit scaling.
Probit scaling uses the probit function
probit(a) $ (a),
where & (ax) is the inverse of the cumulative distribution function the standard normal distribu- tion; the logit scaling uses the logit function
logit(a:) = log
(3)
We compare the goodness of the fit for each of the scalings in Table 2 scaling gives the best linear for both CIFAR- and ImageNet and will use this scaling throughout this work. In Figure 2 we show the linear fit in logit-scaled space and transformed back to linear space for CIFAR-10.
We conclude that the logit
Table 2: R² values for linear fit for different scaling functions
SCALING
CIFAR-10
IMAGENET
LINEAR
0.981
0.984
PROBIT
0.996
0.995
LOGIT
0.998
0.996
Numerical fit values. All linear fits throughout this work were computed by first transforming to logit space (see Section[A] and then fitting using scipy. stat Linnegress.. Eq, gives the functional form for our fit, and the numerical values are listed in Table
(New Acc) logit 1 [A logit(Orig. Acc) +
Table 3: Numerical fit values for Eq
CIFAR-10/10.1 IMAGENEET/IAGENNETV
0.8318
-0.4736 -0.4896
0.9225
B EXPERIMENTAL SETUP
B. PRE-TRAINED IMAGE CLASSIFICATION MODELS
We conduct our experiments using diverse set of model architectures and sizes to make sure the observed phenomenon are general and not tied to any particular choice. We study two groups of models.
ImageNet pre-trained. The first group consists of Alex Krizhevskss (2014), ResNet-18, ResNet- 50, R22 al. (2016a), VGG11 BNJSimonyan & Zisserman| (2014), and WideResNet-50-2 Zagoruyko Komodakis (0066), where we use the PYTORCH Paszke et al. (2019) implementation and weights coming from forchvision m ImageNet. When fine-tuning pre-trained ImageNet models on CIFAR-10, we always replace the original classification layer with 10-class randomly initialized classification layer.
models s v0.8.2 where the models were pre-trained on
Big Transfer (BiT). The second group of models consists of a series of models from Big Transfer (BiT) Kolesnikov et al. (020). The BiT models all use ResNet-v2 architecture He et al. (001bb).
14

---

Under review as a conference paper at ICLR 2022
except that they replace all Batch Normalization loffe & Szegedy (2013) layers with Group Nor- malization Wu & (2018) and use Weight Standardization Qiao et al. (2019) in all convolutional layers. Following Kolesnikov al. (2020), label the ResNet architectures by their depth and the widening factor for the hidden layers, for example R50x1, R101x3 and R R152x4, which are the sizes will consider in this work.
The BiT model come in three different flavors: BiT-S, BiT-M and BiT-L, where S, M and indicate done ILSVRC-2012 (which denote as ImageNet-IK or just ImageNet) Russakovsky et al. (2015), ImageNet-21K Deng et al. 2009) or JFT Sun et al. (2017). respectively. In addition, these models may be further fine-tuned and "Tk". "21k" "JFT" indicates the dataset was fine-tuned on. For example, BiT-M-R 1-5-M-R152x---1 1k is the ResNet-1122 model that was pre-trained on ImageNet-21k ("M"") and fine-tuned on ImageNet- 1k ("Ik").
the pre-training
We also consider series of BiT models that were pre-trained by Djolonga et al. (2021) on different random fractions of the JFT dataset with no additional fine-tuning Section
B.2 FINE-TUNING TUNING
Fine-tuning refers to training setup where the weights learned by training a model on one data distribution are reused when training the model on second data distribution. Fine-tuning can either be end-to-end, where the weights learned on the first distribution are only used for initialization when training the second distribution, or last-layer, where only the last layer weights when training on the second distribution. Fine- Fine-tuning can occur in multiple rounds using multiple successive data distributions. Typically, the earlier training datasets larger in size and class sets and more diverse.
updated
B.3 CLIP MODEL
Our CLIP zero-shot model uses the publicly released pre-trained weights for the ViT-B-33 -2 model from https: /ettps:/gqitann //tttubgittbb..t com/ Openeeeee/a//.m..a CLIP Following the methodology from Radford et al.) (222aa), we fine-tune the zero-shot CLIP model using logistic regression classifier optimized with LFBGGS and we determine the 1.2 regularization strength À using hyperparameter sweep on the validation sets with ten logarithmically-span values between 10 and 10".
B.4 DATA PREPROCESSING
CIFAR-10. In our experiments CIFAR-10, we are interested in testing models that were pre-
trained on ImageNet-like datasets. Since the ImageNet images are typically preprocessed to 224x224 also preprocess the CIFAR-10/10. images by resizing from 32x32 to 224x224
during training, experimental setup the same, we also resize the EIFAR-10/10, images to 224x224 pixels. To keep the pixels when using randomly initialized models 2 All CIFAR images are normalized using the mean 40,4914,0 0.4822, 0.4465) and std = [0.2023, 0. 1994, 0.2010).
When evaluating on test set, or when extracting the features from the penultimate ResNet layer when training only the last layer, we do not apply any preprocessing beyond the upsampling and normalization. When training the full model the CIFAR-I0 training set (with image size 224x224), we additionally apply RandomCrop op with padding=28 and RandomHor zontalFlip
ImageNet. The ImageNet training images are preprocessed with RandomRest zedCrop to 224 pixels and RandemHorizen wwwrnninaaaaaiaa cessed with CenterCres to 224 pixels. Both are normalized with mean=(0.448, 0.456, 0.406) and sid=(0.229, 0.224, 0.22.. 0.255).
and the test images are resized to 256 pixels and prepro-
"The testbed models for CIFAR- 10 and CIFAR-101 10.1 were using the standard 32x32 image size, but as we see in Figure [(b) and [0(b) the ER of our randomly initialized models align with the testbed models, which indicates that the image size does not affect our results. However, we have independently confirmed that the we get the same results in our setup with image size 32x32 in Figure T0(h)
15

---

Under review as a conference paper at ICLR 2022
Aindie Bodice-1e Brabaao
Wide Hombee 20-3
Bee ER
VOGAI T88
Teshhe Madzh
Tiithed Mosta
Difrctive Robustness (%)
Effective Robustress (%)
4-
2-
0-
40
100
20
10
80
100
CIFAR-10 Testt Ace (%)
ImageNet (top-1, %)
(a) CIFAR-I0
(b) ImageNet
Figure 6: Multiple accuracy (using 100 bins). The shaded area around each curve shows the standard deviation per bin over runs. (a) is the same data in Figure 3(a) and (b) is the same runs Figure 3(b)
combined by computing the ER binned average across the ID
B.5 CONFIDENCE INTERVALS, AVERAGING RUNS, AND ERROR BARS
Testbed confidence intervals. For the ImageNet and CIFAR testhed models, described in Section report confidence intervals the accuracies calculated using Clopper-Pearson intervals confidence level 95% Recht et al. (2018) 2019).
Averaging runs. For all measurements of ER that were averaged over multiple runs, we report the binned average value, which we calculate using scipy. stats binned_statisss ist ic using 100 bins.
Error bars. For most of the results throughout this work, we have left out the eтror bars to make the plots more readable. Here, we show the typical error bars for models fine-tuned on ImageNet and CIFAR-10 10 in Figure 6 The reported error bars are the standard deviation from the binned average across 5 runs.
When we are measuring the maximum ER, like in Figure ((a) we have the choice between using the standard deviation in the bin with maximum ER, or the maximum ER across all bins. Since number of data points in each bin can vary, observe that the standard deviation in the maximum ER bin has more variation across different runs. So, to be conservative, we report the maximum ER across all the bins. A comparison between the two choices of standard deviation can be seen in Figure
B.6 OPTIMIZER AND HYPERPARAMETERS
Here we give the details used in the training of all our results. For all models we ran five runs and all were trained using stochastic gradient descent with momentum 0.9 and weight decay 10-4 unless otherwise specified.
The default settings train the full PYTORCH models (see Section B.I was using batch size 64 for 250 epochs using learning rates 10.1,0.01 for the randomly initialized model, and for 100 epochs using learning rates [0.01, 0.001, 0.00010 for the pre-trained models. The main results shown in the main text and the appendix are using learning rate 0.01 and 0.001 for the randomly initialized and pretrained model, respectively, but we have checked that the general conclusions are the same across the different learning rates.
When training the BiT models, we extract the features from the penultimate layer and only train the prediction head (except for the randomly initialized BiT models where train the full model). Unless otherwise specified, the models are trained using learning rate 0.001 with batch size 32768 for 100 epochs with weight decay.
16

---

Under review as a conference paper at ICLR 2022
Max Elfective Robustues (%)
Max Elfective Robustress (%)
10
108
10'
108
JFT Pre-Training Dataset size
JFT Pre-Training Dataset size
(a) Max std. across all bins
(b) Std. at bin with max ER
Figure 7: Comparison of two different ways of reporting the standard deviation (std.) for the maximum ER for the experiments in Figure 4(a) (a) shows the maximum standard deviation across all bins of CIFAR-10 10 II) test accuracy, and (b)
the standard deviation in the bin that had the maximum ER. (b) has higher variation in the std. across different runs because each bin contains
different number data points, we use (a) more conservative estimate as our main reported results.
C ADDITIONAL EXPERIMENT DETAILS AND RESULTS
This section describes we additional experimental results and details of training not described the main paper.
C.I PRE-TRAINED VERSUS RANDOMLY INITIALIZED MODELS
In Section 5.1
we discussed how pre-trained models exhibit ER during fine-tuning, while randomly initialized models have no ER during training. Here we add additional experimental details that did not fit into the main paper.
NetgeNet. In Figure 8(a) we show the ER during training on ImageNet for a BiT-L-R101x3 model that was pre-trained on JFT in comparison to a randomly initialized model the same architecture. The pre-trained model is averaged across 5 runs and is only training the last layer. For the randomly initialized model we train the full model, but due to the high computational cost, we only perform The pre-trained model is using ppram-earameters randomly initialized model was trained following the setup as described in Kolesnikov et al. (2020).
described in Appendix [B.6 while the
CIFAR-10. In Figure 8(b) show the exact same experiment, described in the previous paragraph, on CIFAR-10 10. Again, the randomly initialized model is only run a single time due to the computational cost.
For CIFAR-10 10 we also perform extensive comparisons between several different architectures where either use pre-trained randomly initialized weights. The results are shown in Figure 10 and the hyper-parameters are as described in Appendix B.6
C.2 IMAGENET ROBUSTNESS BENCHMARKS
For models that are fine-tuned on ImageNet, we study the ER on ImageNetV2, ImageNet-R and ObjectNet. We train one HiT-L-R152xx JIT model on ImageNetV2,
evaluate the same model on all three robustness benchmarks. The ER for ImageNet-R and ObjectNet was shown in Figure 3(c) and ((d) In Figure we show the same data as in Figure 3(c) and 3(d) but in term accuracies in stead of ER. While the ER ImageNetV2 goes zero the end of fine-tuning, the ER for ImageNet-R and ObjectNet remains non-zero. As discussed in Section 5.1 this consistent with previous observations in the literature. Besides observing that the accuracy ImageNet-R and ObjectNet stops at around 60% we do not know why this is happening for these two datasets but not for ImageNetV2
17

---

Under review as a conference paper at ICLR 2022
Dffective Robustees (%6
Effective Robustness (%)
0-
2-
-2-
-2
3T RiBIsS-Rast Band Tns
R.T-StotsSBBoo Rood
Torled Nordds
Ztatioat Madiea
60
80
20
80
100
ImageNet (top-1, %)
CIEAR-10 test acc (%)
(a) ImageNet
(b) CIFAR-10
Figure Comparison the ER of a BiT-R101x3 model that is either pre-trained on JFT or randomly initialized. (a) shows the ER on ImageNerV2 when trained on ImageNet, and (b) shows the ER on CIFAR-10.1 when trained on CI no ER while the pre-trained models do exhibit ER during training, but it vanishes the model converges.
both datasets, the randomly initialized models show
ImageNet-R
ObjectNet
ImageNet- R. [top-1. %)
80
Thetled Morkls
ObjoctNet (top11 %0
60
Toethve Medie
60
40
60
80
20
40
100
ImageNet (top- 1, %), subclassed
ImageNet (top-1, X), subelassed
(a) ImageNet-R
(b) ObjectNet
Figure 9: Evaluation on ImageNet-R and ObjectNet for BÍT-L-RI55244JFT model that is fine-tuned on ImageNet. Where the ImageNetV2 ER goes to zero the end of fine-tuning, the pre-trained model still has non-zero ER.
C.3 ARCHITECTURE INDEPENDENCE
We observe ER of pre-trained models across different model architectures in Figure 10 and none of these architectures have any significant ER when randomly initialized. We used PYTORCH models (c.f. Section B. pre-trained on ImageNet and replaced the prediction head with randomly initialized layer for 11.
Both the pre-trained and randomly initialzed models were trained with SGD with learning rates 0.01, 0.001 and 0.0001, and batch size 64, momentum 0.9, and weight-decay 5 10-4, We show the results from averaging five runs using learning rate 0.001 in Figure 10 but the results are similar using the other learning rates (learning rate dependence is studied in Appendix C.5). The randomly initialized models were trained for 250 epochs, while the pre- traiiid models were trained for 100 epochs.
C.4 Loss FUNCTION
In this section we show that our observations about ER are not unique to models trained using cross-entropy loss, and get the same behaviour using mean square error (MSF).
18

---

Under review as a conference paper at ICLR 2022
Effective Robostress (%)
Effective Robustness (%)
Aincle
VEGEE BN
Whewe Berrre
Wide flnohve
Rocher 20
Testhed Madile
Hlacbee 30
Teatiod Mindek
Rechet..i
7.5
Reshet-1122
5.0
5.0
2.5
2.5m
2.5
5.0
5.0
20
40
60
80
100
80
100
CIFAR-10 Test Ace (%)
CIFAR-10 Test Aoc (%)
(a) Pre-trained models
(b) Randomly initialized models
Figure 10: Pre-trained models show ER for all six architectures studied, while none the randomly initialized models do. The amount of ER does vary across different pre-trained models, but since we did not the pre-training, we cannot control for differences between architecture and optimization in this experiment (See Section 5.3 and Appendix C.8 and C.8 for more controlled experiments on the BiT architectures.) However, the randomly initialized models show that pre-training crucial to get any ER at all for any of the architecture choices.
BIT-L-R152x1 JIT
7.5
XSS
Dieo estrggy
Effective Robustness
5.0
2.5
0.0
-5.0
20
60
100
CIFAR-10 Test Ace (%)
Figure 11: The rise and fall of ER occurs for different loss functions used training. Here, we that the ER during fine-tuning is incredibly similar between model trained with MSE and cross-entropy loss.
For evaluation of the MSE error loss, we compute the MSE between the output logits and a one-hot target vector. We train the model using stochastic gradient descent with constant learning rate 0.001 and batch size 4096 for 100 epochs.
In Figure II we compare the ER duing fine-tune fine-tuning of the last layer of JFT pre-trained BYT-L-R152x4 model where one is trained using MSE and the other is trained using cross-entropy loss. In this specific experiment, the MSE trained model has higher ER at its maximum, due the other similarities between the evolution we do not believe this would always be the case.
C.5 LEARNING RATE
The observed ER during fine tuning is largely independent of the learning rate used during optimiza- tion. In Figure 12 we show the ER during training on CIFAR- 10 for a pre-trained and randomly initialized Wide-ResNet-Ske using the same hyper-parameters in Appendix C.3
19

---

Under review as a conference paper at ICLR 2022
Ellective Robustnnss (%
Hllective Robustaess (9)
Midel
10
Teated
10
20
60
60
80
100
20
10
60
80
100
CIFAR-10 Test Acc (%6
CIFAR-I0 10 Test Ace (%)
(a) Pre-trained
(b) Randomly initialized
Figure 12: Same setup in 10 but with varying learning rate for fixed architecture, Wide-ResNet- 50-2. The left (right) plot shows the results using pre-trained (random) weights. While there is some small variations from different settings, the conceptual difference between pre-trained model and randomly initialized weights holds true.
C.6 FULL MODEL VS. LAST LAYER FINE-TUNING
The ER during fine-tuning of pre-trained models largely insensitive to whether the full network is trained or if we only train the last layer. In Figure 13 we compare the ER of full model versus last layer fine-tuning for the six PYTORCH models described in Appendix B.
We start both last layer and full model fine-tuning with the same pre-trained weights for that architecture. For training of the full model, we use SGD with learning rate 0.001, and batch size 64, momentum 0.9, and weight-decay 5 100 for 250 epochs. When training only the last layer, we use SGD with learning rate 0.0001, and batch size 4096, and without momentum or weight-decay, and we trained for 1000 epochs. We experimented with learning rate schedulers, but saw no significant change. Despite the difference in training and syper-parameters, we observe that both have about the same amount of ER. The biggest observed difference is that the only last-layer trained models don't reach the same final accuracy as when training the full model, which is to be expected as the last-layer model cannot change its pre-learnt features.
C.7 ZERO-SHOO SHOT EVALUATION
Zero-shot evaluations on traditional image classifiers is made possible with hand curated maps between the classes of one dataset to another. To map from CIFAR-10 to ImageNet, we utilize the mapping from ImageNet to CIFAR-10 classes used in CINIC-10Darlon a previously constructed map from ImageNet to JFT to evaluate the performance of BiT-L models on ImageNet in a zero-shot manner.
al. (2018), and utilize
These maps were used for zero-shot evaluation on ImageNet and ImageNerV2 in Figure and discussed in Section 5.2
The map from CIFAR- to ImageNet was used Appendix C.12
In this section we investigate different ways of combining the multiple logits corresponding to each CIFAR-10 class to make predictions using a model pre-trained on ImageNet. Given a CIFAR- 10 class, we collect the corresponding ImageNet softmax scores and combine the probabilities using three different functions: max, mean and sum. We show the results in Figure [4 where we see that most of the zero-shot models have some amount of ER, but are not significantly more robust than pre-trained models in the middle of fine-tuning.
see no significant difference between the three combination functions we test, and we use the max as the default in all other plots throughout our work.
C.8 DATASET SIZE AND DIVERSITY
In Section we summarized our experiments where we vary the pre-training dataset for different fixed BiT architectures. Here, will include additional experimental details and observations.
20

---

Under review as a conference paper at ICLR 2022
Eilekte Pallett (15)
Inded Blik
Eilective Stalicutionn (25)
bode lut
Telediick
Eifectre Bolluuttoo (35)
Tedeetick
109
GB
100
330
CIPARi-1 teet are (%)
CIPAT--O sext are (3)
CITAD-10 test aec (X)
(a) AlexNet
(b) ResNet-18
(c) ResNet-50
Eifelio Eodoutaci (56
Ei TraMing
Effecture Richanttonn (56)
Eilective Redoaattacn
163
100
CIRAR-10 uet are (%)
CIPAR-ID tand arc (3
CIFAR--O reve ann (3)
(d) 15s
(e) VGG11 with batch norm
(f) Wide-Res
Figure 13: The ER of pre-trained models is mostly insensitive whether the full model is trained during fine-tuning or just the last layer. We observe this across six different model architectures, and the only significant observed difference is that the full-model training reaches higher final accuracies, which is to be expected.
Eliedie Puboottioe 195
Elieiie Paak celtanee (91)
Bivertee Doboutteen E92)
100
100
CIPAR-10 bad ox (36)
CIPAR-10 bot ax: (1))
CIFAR-10 toll sax (56)
(a) Max
(b) Mean
(c) Sum
Figure 14: Zero-shot predictions using pre-trained BiT models fine-tuned ImageNet-1k, 1k. We combine the ImageNet predictions that match the relevant CIFAR-10 10 class by choosing either the max, mean, sum of the probabilities. The majority of models show some amount of ER.
All models in this section fine-tuned using SGD with learning rate 0.001, batch size 4096, with no momentum or weight decay, and we trained for 1000 epochs.
JFT dataset size. Using series of BiT-R101x² that were pre-trained on different randomly sampled fractions of the JFT-300M dataset Sun et al. (2017). we show that the maximum ER increases with the pre-training dataset size. Since the pre-training images were randomly sampled from one dataset, it gives us some control for the dataset diversity as opposed to our experiments where we compare models trained on ImageNet-Ik, 1k, ImageNet-211 and JFT. However, since the models were trained on different amounts of data, we cannot rule out any effects from the pre-training optimization on the maximum ER.
Different pre-training datasets. When using different pre-training datasets, find that larger and more diverse datasets give higher maximum ER during fine-tuning on CIFAR-10.
We show the ER during fine-tuning for three different BiT model sizes in Figure 17 For R152x4 and R-101x3 we see that the BiT-L models that were pre-trained on JFT has the highest ER and the BiT-S models that observation by looking at the two BiT-M models: The BiT-M-21K model (pre-trained on ImageNet- 21k) has higher ER than the BiT-M-1K model (pre-trained ImageNet-21 21k and then fine-tuned
pre- pre-trained
ImageNet-Ik 1k has the lowest. We also make interesting
21

---

Under review as a conference paper at ICLR 2022
1.201
10.000
1.0M
1090M
27 10N
125
Elficctive Robustness (%)
1.31
0
80
CIFAR-10 Test Ace (%)
Figure 15: ER of series of BiT-RI01x3 models pre-trained different amounts of data from JFT and fine-tuned on CIFAR-10. The maximum ER and the final accuracy increases with the JFT dataset size. The model with the maximum ER used 32 million images from JFT, and as discussed in Section 5.3 we suspect the plateau after this point is caused by the fixed number of iterations used during pre-training, which eventually serves bottleneck for performance improvements.
Max Efffective Robustness (%)
BT-SSII
6-
BTT-S-Kk
5-
4-
3
R50x1
R101x3
R152x1
Model Architecture (size)
Figure 16: The maximum ER increases as a function of model architecture size for all pre-training datasets. The only exception is the BiT-S-R50x1 model that has the higest maximum ER among all the R50x1 models, and it is also higher than the BiT-S-RI01x3 and BiT-S-R152x4 models.
ImageNet-IIe 1k). In other words, fine-tunine tne-tuning on the smaller and less diverse ImageNet- Ik dataset before training on CIFAR-10 gives about percentage point drop in maximum ER.
While the observed trend that the maximum ER is higher for larger and more diverse pre-training datasets seems very robust, we observe a deviation from this trend for the BiT-S-R50x model. The BiT-S-R50x model has higher maximum ER than the BiT-L-R50x and BiT-M-R50xl models. Also, it has higher ER than the comparable model with larger model architecture (BiT-S-RI01> and B1T-S-R152x4), thus breaking yet another trend. However, we do not know if this caused by an interesting phase transition when using the combination of the smallest pre-training dataset and the smallest model, or if is simply caused by some part of the pre-training optimization that we do not have access
Model size. In Figure 18 we compare the ER during training for three different BiT architecture sizes for fixed pre-training dataset. For all but the BiT-S model (as discussed above), find that the ER increases with increased model size. This is plausible outcome since bigger models can build more expressive hidden representations.
22

---

Under review as a conference paper at ICLR 2022
10.0-
legaie Regrniee Proceend Fremerm RE St
10:1
Segaire Reginiies Pontment Pedtemm
Elfeetive Robbustaess
7.5
Ellettive Roboustass
7.
6.-
5.0
2.5
100
100
CIFAR-10 Sel act (56)
CIFAR-10 wel acc (56)
(a) R152x4
(b) R1O1X3
Ellective Robustuss (6)
1000
Legattc Baprosbon Frotrdaed Dattoes
Twrbed Micbo
000
2.5
-500
44)
80
100
CIFAR-10 10
(96)
(c) R50x1
Figure 17: The ER during fine-tuning on CIFAR-10 is mostly increasing for larger pre-training dataset. We show this effect across three different architecture sizes, where each plot compares models pre-trained on different datasets.
C.9 EXAMPLE DIFFICULTY
In this section explore the ways in which the fine-tuning dataset affects the ER OOD dataset. We study a ResNet- 18 model that either randomly initialized or pre-trained on ImageNet. Both sets of models are trained with single for 100 epochs using batch size 64, and the pre-trained and randomly initialized models use learning rate 0.001 and 0.01, respectively.
Fine-tuning with easy, random hard examples. Using the C-score et al. (2020) metric for example difficulty, we select 5,000 the easiest, hardest or random examples while maintaining class balance (i.e. 500 examples from each class).
In Figure [ we show the ER during training, and we make two observations: First, training on more difficult examples gives higher ER than when training on easy random examples. Second, the models that train on only hard examples reaches a lower ID accuracy than the easy or random models.
Phasing out easy examples during fine-tuning Since the fine-tuning dataset affects the ER during fine-tuning, there might be way choose what examples from the fine-tuning dataset train on that achieves both high ID accuracy and high ER. In Figure 19 we show the result from experiments on randomly initialized and pre-trained models where we start the fine-tuning process with the full CIFAR-10 training set, and we gradually phase out the easy examples until there are only 1000, 5000 or 1 10000 examples left the last epoch of training. However, as we can see this does not seem to have any effect on the ER.
An alternative approach we attempted was switching between random, easy and hard, or first fine- tuning on all the data and switching to only the hard examples at the end. Generally, found that when a model that reached high ID accuracy was being further fine-tuned on difficult examples, the ER does go up, but the ID accuracy also goes down. The gradual phase-out described in the previous section was the most consistent way to reach high ID accuracy while only training on hard examples by the end of training.
23

---

Under review as a conference paper at ICLR 2022
Elleetive Robustress (%)
10.0
Effective Robustrness (%
Nade
7.5
BBT
Uoaled Naalle
5.--
5.0
0.0
0.0
-25
-2.
20
d0
60
80
100
20
40
60
80
100
CIFAR-10 test acc (X)
CIFAR- 10 test arc (%)
(a) BiT-L-JFT
(b) BiT-M-21K
Fifective Robusttess (96)
Proteirest
Effective Robustness (96)
10.0
Preteskeed
Rist Tealed Maddd
I.T. R12b4
7.5
7.5-
Tocted Haak
3.0
5.0
2.5
0.0
0.0
-2.5
-23
40
60
100
100
CIFAR-10 10 test acc (%)
CIFAR-10 acc (%)
(c) BiT-M-Ik
(d) BiT-S-Ik
Figure 18: Larger model architectures gives higher maximum ER during fine-tuning on CIFAR 10. We show this effect for four different BiT architecture sizes for three different fixed pre-training datasets.
Effective Robostness (%)
I000
Effective Robostress (%)
2
-2
40
100
80
CI Test Ace (%)
CIFAR-10 Testt. Ace (%)
(a) Randomly initialized
(b) Pre-trained
Figure 19: Since models fine-tuned on more difficult examples show more ER, we training with the full CIFAR-I0 training set. We then gradually eliminate the easiest examples (where difficulty determined by the C-score) until we only have 1,000, 5,000 or 10,000 examples left by the last training epoch. (a) shows the ER during training for a ResNet- 18 randomly initialized model, and (b) show the same model initialized with ImageNet pre- mraitreind weights. In both cases this dataset schedule does not help maintain higher ER at high ID accuracies.
C.10 DOMINANCE PROBABILITY
The dominance probability Mania & Sra (020) compares the predictions of two models and is the probability that the lower accuracy model correctly classifies given example which the higher accuracy model incorrectly classifies.
24

---

Under review as a conference paper at ICLR 2022
0.7
Toibel modils
ImageNet Probaaaliiie
0.6-
BiT-L-Riibsl
RiT-- coneged
0.5
CLI aroohhe
GSPS
0.3
0.2
0.1
0.1
0.2
0.3
0.4
0.5
ImageNet Probabilities
Figure 20: Triplet-roobabilttee between all models in our testbed in addition to CLIP models and two BiT-L models. 99.8% of all triplet events are inside the code as defined in Mania & Sra (2020), and there is observable violation of the distributional closeness for the effectively robust models.
Formal definition Mania & Sra 2020). Let X' be covariate space, y be discrete label space, and P be a probability distribution over x y. Suppose model maps - y and has accuracy μ Ep [p[1(((x) y)] Given any models f. and f, with μt S Mý, the dominance probability is given by
((ff(()==)) = v} (f(x)) + v1)
(5)
Pairwise dominance probability comparison. We compute the pairwise dominance probabilities on the ImageNet validation set for the following of models: the ImageNet testbed models, two BiT-L models (BiT-L-R152x4 early stopped at maximum ER, and the same model at end of training when the model has converged), and our zero-shot and fine-tuned CLIP models (see Appendix BB3)
Figure 21 shows a heatmap of the full set of pairwise dominance probabilities (a smaller version appears Figure in the main paper). The models are sorted by increasing ImageNet validation accuracy and the lighter colors in the heatmap correspond to models with higher dominance probabil- ities. Dominance probability is not symmetric, but to make the heatmap readable using either the row column models as the more accurate model, we mirror the dominance probabilities across the y = x axis.
We observe that the models with the highest dominance probabilities, the BiT-L-R152x4 Max ER model and the CLIP zero shot model, also have the highest effective robustness. However, the CLIP L-BFGS model has lower effective robustness than the BiT-L-R 112x5 converged model ( 0% versus 1 %6 respectively), but slightly higher average dominance probability. This suggests that effective robustness alone is not the sole influencer of dominance probability, and some component of the CLIP pre-training, such as the natural language supervision, contrastive learning objective, or large and diverse pre-training dataset, must contribute the high dominance probabilities for the converged CLIP model.
Distributional closeness. To characterize distributional closeness, Mania & Sra (020) also introduce another metric which we call the "triplet-robbabilttes"" measuring how close the predictions between triplet of models (see Mania & Sra (2020) for formal definition and discussion). Figure 20 we show the cipplet-probbille
for the same set of models as for the dominance probabilities in the last section. We observe that the effectively robust models do not violate the notion of distributional closeness as defined by the triplet probabilities, i.e. the majority of the triplet probabilities for the effectively robust models are the same cone as the standard testbed models.
Discussion. Mania & Sra (2020) proved that models' accuracies are collinear under distribution shift under the assumptions (a) low dominance probabilities and (b) distributional closeness. Conversely, if we see deviations from the linear fit, at least one of these assumtions must be violated. Oour results show that effectively robust models (which do not exhibit collinear accuracies under distribution shift) break the dominance probability assumption rather than the distributional closeness assumption. We note that ImageNet and ImageNet- V2 are explicitly constructed as to be similar, so for other OOD test sets might expect that the distributional closeness will also be violated.
25

---

Under review as a conference paper at ICLR 2022
14
BIT-C-RLssoE
polyrnt
12
10
menurt vi s
CLIP-
gosnett
10t of
Man
15
Figure 21: Pairwise dominance probabilities on ImageNet between all models in the testbed in addition to the two CLIP models, and two BiT-L models. The effectively robust models (CLIP zero- shot and BiT-L Max ER) have high dominance probability relative to all other models, which means that they make different predictions than all other models in the testbed. Even after fine-tuning of the CLIP and BiT-L model, predictions than other models despite being different from the testhed models in Figure 5
from the heatmap that these models still make more different
26

---

Under review as a conference paper at ICLR 2022
C.11 REPLAY BUFFER: REPLAYING IMAGES FROM THE PRE- PRE-TRAINED DATASET
Our experiments using replay buffer discussed in Section 5.5 were conducted on a ResNet-18 model that was pre-trained on ImageNet.
As baseline, use the pre- pre-trained ResNet-18 18 model fine-tuned on only CIFAR-10 in Appendix C.3 Next, we fine-tune the pre- pre-trained ResNet- 18 model on both CIFAR-10 and ImageNet using batch size 256 and learning rate 0.001 for 100 epochs. The replay buffer model uses two prediction heads, one for ImageNet and one for CIFAR- 10. For ImageNet, we kept the original 1,000 class prediction head from the pre-trained model, and the CIFAR-10 prediction head was randomly initialized. During fine-tuning, we varied the ratio of ImageNet to CIFAR- 10 images to be 1, 10, or 100. The results are shown in Figure 22
We observe that none of the replay buffer models are able to maintain high ER at high ID accuracy. This is despite the fact that such training procedure still maintains - by design - high accuracy on the pre-training dataset. Before any fine-tuning, the top-1 accuracy of the pre-trained ResNet-1 8 on the ImageNet test set is 69%. This drops 39% during fine-tuning on CIFAR-10 when early stopped at 92% CIFAR- 10 test accuracy; however, with the use of a replay buffer, the final ImageNet test accuracy 63% the end of fine-tuning on CIFAR-10.
This is intriguing since it demonstrates that a model that is forced to maintain high accuracy on different OOD dataset during the course of fine-tuning on the target dataset does not do better on the (closer) OOD dataset of interest. For this particular setup, one might have imagined that model that continues to perform well on ImageNet during fine-tuning on CIFAR- 10 has learned more general properties of the image data manifold than it otherwise would have, and that this might be beneficial in ER on 10....10.1.
In conclusion, using replay buffer does help prevent catastrophic forgetting, but it does not prevent the ER on CIFAR- 10 from disappearing towards the end of fine-tuning.
Elfective Robustaess (%)
Pull Modid Training
Pestenitaed
10
Huddee Bepcaptiette 2003
Beskeiifforst
BooNeist Repke
20
10
60 80
100
CIFAR-10 test acc (%)
Figure 22: We study the effect of using buffer replay on the ER during fine-tuning on CIFAR-10 for a pre-trained ResNet-18 model. We compare standard ResNet-18 training a ResNet-18 where we see 1, 10 or 100 ImageNet batches for every CIFAR-10 10 batch. Despite minor differences in the xl, x10 and x100 curves, none of them are able retain ER beyond the standard ResNet- model.
C. CLASS MAPPING
Instead of starting from a randomly initialized prediction head when fine-tuning a pre-trained model on CIFAR-10, we use a pre-trained prediction head if map the ten CIFAR classes to the 1,000 ImageNet classes. For the map from CIFAR-I0 to ImageNet, we use the map as defined by Darlow et al. (2018) in the construction of the CINIC-10 10 dataset. With this setup, conduct several experiments to test if there is any benefit from the pre-trained prediction head in maintaining ER at high ID accuracies.
Since each CIFAR- 10 class can map to multiple ImageNet classes, we use a cross-entropy loss where select the max logit across the multi-target classes. We train the last layer of SHT-M-R112x4
27

---

Under review as a conference paper at ICLR 2022
Dffective Robustness (%)
CIFAR 10
Difective Robustress (96
CIEAR A-31
CIFAB-I0 InnaeSe.
CIEAS-M
CIPAR-10 TaegeSe
CIPAS
60
70
90
100
70
80
90
100
CIEAR-10 Test Ace (%)
CIFAR-10 Test Ace (%)
(a) Replay Buffer
(b) Weight regularizer
Figure 23: Using a class map from CIFAR-10 to ImageNet target classes, we can fine-tune on CIFAR-10 by initializing the prediction head to the pre-trained ImageNet model. In (a) fine-tune the last layer of a BiT-M-R1522 152x4 1k model when training on only CIFAR- 10. or using replay buffer with or 100 times as many ImageNet images. In (b) we fine-tune on only CIFAR-10 10, but using a L2 regularization term around the pre-trained weights, where À is the pre-factor of the L2 term the loss function. However, none of them are able maintain high ER at high accuracy.
model on CIFAR-10 using learning rate 0.001, batch size 32,768 for 1000 epochs. All results shown in Figure 23
Replay buffer with class mapping. Since CIFAR-10 and ImageNet now have the target classes, we can repeat the buffer replay experiments using the same prediction head for both datasets. In Figure 23(a) we show the results from training using the same number of images from CIFAR- 10 and ImageNet, and also with 100 times more ImageNet images, but we see no improvement in maintaining high ER.
Weight regularization. To test if any of the benefit of the pre- pre- training comes from the pre-trained prediction head, we introduce L2 regularization term keep the weights close to the pre-trained weights of the prediction head. We introduce pre-factor to the L2 regularization in the loss, but we see no improvement in ER from small À despite small reduction in ID accuracy relative to the results in Figure 23(a) For large X, the 1.2 regularization dominates, and we do not even reach high ID accuracies.
D SHAPE OF THE EFFECTIVE ROBUSTNESS CURVE
In our experiments, we observe that the ER of pre-trained models starts out near zero, increases during training, reaches peak, and then diminishes towards the end of training.
Interestingly, we find that the existence of the peak is in part caused by the linear fit in logit space and the definition of ER. To understand this, plot the ER of the
line (i.e. the line "Same accuracy (y = x)" in Figure I and 2 along with the ER of pre-trained BiT models in the left plot of Figure 24 (CIFAR-10) and 25 (ImageNet). The ER of the y curve also has the
observed shape with peak. Hence, to isolate the effect deviating from the linear relationship from the natural shape of any ER line we also show the ER a fraction the accuracy gap between the linear fit and the y = x line in the right plot of Figure for CIFAR-10 and Figure 25 for ImageNet. We observe that these curves do have the same maximum as observed in the ER, and instead they have a mostly decreasing trend during fine-tuning.
D.1 MIXED CLASSIFIER
Due to the convex shape of the testbed model accuracies (without the logit scaling), one way to construct model that has ER to randomly sample predictions from low accuracy testbed model with probability a and from a high accuracy testbed model with probability 1 a. We call this model mixed classifier with parameter As varies from 0 to the accuracies the ID and OOD
28

---

Under review as a conference paper at ICLR 2022
15
100
Elfective Robustness (%)
Logitic Regnnssion on Pestruiued Faslures
% of Accuracy Cap
377-1-R10266-777
BTT B/T-X-RI R2sk1
T877 Ridbet 511
60-
0-
BIT.E JPT
DTT 93 Ri133e 3ix
Teatbed Miodle
15
0
20
40
600
80
100
20
40
601
80
100
CIFAR- 10 test acc (%)
CIFAR 10 test acc (%)
Figure 24: Left: The ER of pre-trained BiT models during fine-tuning CIFAR-10. We include curve indicating what the ER would be for model that has no accuracy gap between the original and new test sets. Right: The ER as a percentage of the accuracy gap. As a percentage of the accuracy gap, the pre-trained models are slowly decreasing during fine-tuning.
Effective Robustrees (%)
Logasic Regeeskon Pre-Triied
100
Logisic Regevssion Pre-Ininsdd
15
% of Accuracy Cap
BIT-L-RiiSst.R
101-4-11000 JFT
RITEE RF0
-.. Zen Fit
Tailbal Modh
0
10
60
80
100
10
60
80
ImageNet (top 1, 16)
ImageNet (top 1, 1%)
Figure 25: Same as in Figure 24 for ImageNet. Left: The ER of pre-trained BiT models during fine-tuning on ImageNet. We include curve indicating what the ER would be for model that has no accuracy gap between the original and new test sets. Right: The ER as percentage of the accuracy gap. We note that as percentage of the accuracy gap the pre-trained models are slowly decreasing during fine-tuning.
test sets traces out straight line between the accuracy of the random and highest performing model, which will have ER due to the convexity of the logit-scaled linear fit. While mixed classifiers hold theoretical interest and should be analyzed more in future work, their in practical applications is limited since we cannot construct a mixed classifier that has both high ER and high ID accuracy.
29


Reviews:
[{"review_id": "kqHAA0FV4bI", "reviewer": "Reviewer_AB8m", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is very clearly written and has a thorough experimental section validating the authors' claims.\nThe authors have a thorough selection of experiments that validate their claims.\n\nWeaknesses:\nOne weakness of this paper is that the authors do not properly define fine-tuning. While its meaning is implicit, fine-tuning is a key concept in this paper, so having a clear definition of the term seems necessary. This is especially true when considering multiple fine-tuning steps such as when fine-tuning a fine-tuned model BiT-M-21k on CIFAR-10.\n\nThe authors use a pre-trained or randomly initialized model on a large dataset, fine-tune on a smaller dataset and measure OOD accuracy on an analogous dataset to the fine-tuned dataset. It would help if the authors give some examples of when such a training procedure would be useful. Usually fine-tuning is carried out on the distribution that the model is going to be evaluated on.\n\nAn analysis of the relation between the fine-tuning dataset and the OOD test set would be useful. Right now the relationship is alluded to based on natural distribution shifts, but it's not clear how this might generalize to other types of distribution shifts.", "summary_of_the_paper": "This paper highlights important variables impacting the effective robustness (ER) of a pre-trained, fine-tuned model. \nThe authors identify that increasing model size, dataset size, and example difficulty improves the ER of a pre-trained, fine-tuned model.\nThe experiments suggest that the zero-shot component of CLIP plays a significant role in the high value of ER CLIP achieves.\nThe investigation of ER on dominance probability shows that models with high ER have high dominance probability.\nThe authors also present a negative result showing that several reasonable approaches to maintaining high ER while fine-tuning fail.", "summary_of_the_review": "Overall this paper is very thorough. The authors set out to investigate the role fine-tuning has on OOD robustness and they successfully identify several key variables to consider. There are many experiments in the main paper as well as in the appendix that validate their claim. This work will be very valuable to the community as it provides some insight into what variables lead to OOD robustness for pre-trained, fine-tuned models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ifpIjM6pki3", "reviewer": "Reviewer_kcNT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Apologies to the authors for what may sound like a rather glib judgement of this submission (and for which I acknowledge through the confidence scores below that my opinion is not absolute as I do not work directly in the space of image classification), but the results and conclusions of this paper seem remarkably obvious.  Namely, that models that have been pre-trained on a large collection of different datasets tend to lose their strengths at predicting out of distribution as they are progressively fine-tuned towards predicting a specific type of data.  And that when these models are performing in the 'effective robustness' mode the types of in sample problems they find easy (alt. hard) are different to those that models trained on the dataset at hand find easy (alt. hard).  The perils of over-fitting to a particular training set are well known and strategies to avoid this and improve generalisation are a major component of ongoing work in the machine learning (see e.g. Roger Grosse's comp sci lecture notes: https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/readings/L09%20Generalization.pdf ).  To change my mind on this point would require additional discussion by the authors to connect this work to general principles of machine learning and establish the novelty of the insights reached from these numerical experiments.\n\nThat said, my many years in research have taught me that sometimes results that seem to me to be 'remarkably obvious' are actually not so for the general audience, and that 'simple' examples demonstrating such principles can actually have a large impact and generate huge citation indices.  I mean this genuinely; not trying to be cynical here.  So for that reason I would respect the decision of other reviewers and the AEs if this paper was in fact recommended for the conference series.  Certainly, having a reference to point to for e.g. the fact that self-driving cars probably shouldn't spend too much time refining their algorithms to over-fit to a commuter's every day journey to work (this being inevitably at the expense of performance when he/she wants to take a drive in the countryside), could actually be very useful.", "summary_of_the_paper": "In the manuscript entitled, \"The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning\", the authors present an empirical investigation of model exhibiting a property known as 'effective robustness'.  In particular, their focus is on how 'effective robustness' changes during fine tuning and on the characteristics of these models.", "summary_of_the_review": "Conclusions seem 'obvious' to this reviewer, but willing to consider other opinions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iTwhMw28rWC", "reviewer": "Reviewer_cHRW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Summary:\n\nThe paper conducts an empirical study into an interesting problem of robustness of deep models on out of distribution data. The paper finds that the pre-trained models exhibit better effective robustness during training which will disappear upon convergence of the same models.\n\n### Pros:\n\nThe paper is well-written, easy to understand and follow along.\n\nMost significant of all is that this paper has an extensive study of various initializations with pretrained models for vision problems.\n\nThe breadth of explorations such as pre-trained models ER during training, data set size, example difficulty, \n\n### Cons:\n\nare there proper bounds for the ER values, what would it really mean to have higher value, lower value etc, can you briefly explain?\n\nER ~ 0 for CIFAR-10 (Figure 3a) at the end of training, exactly the point at which any of the models are having the corresponding best accuracies on IN set. The only difference at that point is, the accuracy of various models is different which is already known and well-studied in literature. Similar for Imagenet, it is visible at low accuracy, and the trends are visible as the accuracy gets better similar to CIFAR. The question is, why should anyone care, if the ER is high in the middle of training at low accuracy? This is not well-justified in the current version of the paper. Also, the reasons for the peaks in ER during training are not justified, why are they intriguing? -- is it because the pre-trained models change significantly to the down-stream tasks, or something else? The random initializations don’t fluctuate that much, why not investigate these observations in detail?\n\nIn Figure 4b, why further fine-tune only the BiT-M-1k model, what happens if you further fine-tune all the models? This experiment is not a fair comparison, not all models see the same amount of data.\n\nAgain, in Figure 4c and the corresponding appendix, why would anyone use a low accuracy classifier when one knows it will perform bad on the hard to classify examples, in that case ER is not even a thing to worry in the first place, accuracy becomes the first concern. Fine, at least the ones that the classifier can classify, there is better robustness but not entirely convincing though. \n\nThis paper relies heavily on Taori et al. (2020), which seem to have a number of unresolved concerns, most important of all is that the paper is a bit short on novelty, however, the empirical study in itself is interesting.\n\nShow the same findings hold for at least one more domain, for example, NLP.\n\nOverall, the paper has breadth in the number of experiments and the directions that it explores without enough depth and justifications to a majority of findings.\n\n", "summary_of_the_paper": "\nThe paper conducts an empirical study into an interesting problem of robustness of deep models on out of distribution data. The paper finds that the pre-trained models exhibit better effective robustness during training which will disappear upon convergence of the same models.\n", "summary_of_the_review": "Overall, the paper has breadth in the number of experiments and the directions that it explores without enough depth and justifications to a majority of findings. Also, the paper lacks novelty or detailed analysis of the proposed concepts. I would give it a score of 4.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AJfZMLnWW0t", "reviewer": "Reviewer_XPbn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper has the following strengths: \n\n1. I think identifying models that have effective robustness and understanding their properties is an important and interesting problem. This paper has some empirical observations under this direction. \n\n2. Enough details are included for the experiments. \n\n3. Overall, the paper is well-written and the related work is properly discussed. \n\nHowever, I think this paper has the following weaknesses: \n\n1. My major concern is that the contribution is not very significant. The authors have some empirical observations, but those observations are not very useful and don't help us understand the problem better. For the models in the middle of fine-tuning, although they exhibit a high amount of effective robustness, the accuracy of those models on the in-distribution dataset is not high and thus such kinds of models may not be useful. Also, when the fine-tuning converges, the models have high accuracy on the in-distribution dataset but don't have effective robustness. Thus, the models obtained via fine-tuning don't have clear advantages over previous models. Besides, although the authors discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy, none of those methods work. So such a discussion may not be useful.\n\n2. They only have some empirical observations, but don't have analysis for them. For example, they only show that the effective robustness generally increases throughout fine-tuning, peaks, and then gradually disappears towards the end of fine-tuning empirically, but don't analyze or explain why such a phenomenon exists. It is unclear whether such a phenomenon is general or it just exists on some datasets. It seems Figure 3 in the paper shows that such a phenomenon doesn't exist on ImageNet-R and ObjectNet when using ImageNet as in-distribution. \n\n3. Some claims are not well supported by results. For example, the authors claim that the pre-trained models in the middle of fine-tuning, as well as zero-shot pre-trained models, represent an entire class of models that exhibit high amounts of effective robustness. I think this claim may not be true. There might be other training methods that could lead to better effective robustness and also high accuracy. The authors only explore the methods of fine-tuning and zero-shot evaluation. Thus, it is hard to claim that they represent an entire class of models that exhibit high amounts of effective robustness. The claim that the models with effective robustness make different predictions than standard models and are able to correctly classify examples that no standard models get right, is also not well supported by the results. They only select 4.8% of images that none of the testbed models get correct and show that the model that has effective robustness with the best in-distribution performance gets 10% of these examples correct. I think such results cannot support the claim that the models with effective robustness are able to correctly classify examples that no standard models get right since only 10% of these examples are predicted correctly by the model that has effective robustness with the best in-distribution performance. Also, it seems these results could not demonstrate that the models with effective robustness have prediction diversity. \n\n4. Some observations may already be known to the community. For example, the observation that the effective robustness increases with the larger size and more diversity of the dataset seems obvious. \n\n", "summary_of_the_paper": "In this paper, the authors conduct a thorough empirical investigation of effective robustness during fine-tuning and have several observations: 1. models pre-trained on larger datasets in the middle of fine-tuning, as well as zero-shot pre-trained models, exhibit high amounts of effective robustness, but the effective robustness vanishes at convergence; 2. the effective robustness increases with the larger size, more diversity, and higher example difficulty of the dataset; 3. models that have effective robustness make different predictions than standard models and are able to correctly classify examples that no standard models get right. Besides, they discuss several potential solutions to mitigate the problem of vanishing of effective robustness during fine-tuning, but find that none of them are able to maintain high effective robustness at high in-distribution accuracy. ", "summary_of_the_review": "I think this paper doesn't make enough contributions and the claims are not well supported by results. Also, they don't provide analysis for the observations and the observations may not be helpful in understanding the problem. Thus, I think this paper is not ready for publication. \n\n***[Post Rebuttal]***\n\nAfter reading the rebuttal, I think my major concerns still remain: the contributions are not very significant and the findings may not be useful. I still think that the models studied in this paper are not enough to represent all models that exhibit ER. The authors need to explore other kinds of models that have ER (and also have high accuracy). Thus, I keep my original rating and think the paper is not ready for publication. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to NeurIPS.  The reviewers are quite split on this paper, but some remain substantially negative even after discussion.  I'm a bit more optimistic about the paper: the observed increase then decrease in ER during fine-tuning _does_ strike me as a fundamentally interesting phenomenon, and I believe that papers that present such phenomena can be valuable contributions even without more fundamental "explanations" of the observations.  My recommendation, therefore, ultimately rests largely on the fact that I think (as is honestly evidenced by the reviews to a large degree), the presentation and contextualization of these results can be substantially improved in a future revision of the paper.  Specifically, the fact that several reviewers found the results obvious and/or not sufficiently substantiated suggests that the basic premises here are still failing to land.  I would strongly suggest revisions that clarified these points in a resubmission.
Decision: Reject