The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning

1 INTRODUCTION . The ability to generalize to data not seen during training is essential for the widespread trust and adoption of machine learning models . In practical applications of machine learning , we typically train and fine-tune on a dataset which are in-distribution ( ID ) with each other , but when deployed the model will face shifts from this distribution , and empirically the vast majority of models show a significant drop in performance from ID data to out-of-distribution ( OOD ) data ( Quiñonero-Candela et al. , 2009 ; Recht et al. , 2019 ; Biggio & Roli , 2018 ; Szegedy et al. , 2013 ; Hendrycks & Dietterich , 2019 ; Azulay & Weiss , 2018 ; Shankar et al. , 2019 ; Gu et al. , 2019 ; Torralba & Efros , 2011 ) . Common examples include data captured in a different environment , like time of day or geographical location ( Koh et al. , 2020 ) ; noise or small corruptions of the input data ( Hendrycks & Dietterich , 2019 ; Geirhos et al. , 2018 ) ; or adversarial examples created to explicitly fool neural networks into making incorrect predictions ( Szegedy et al. , 2013 ; Biggio & Roli , 2018 ) . Although the performance gap on OOD shifts is pervasive , it follows an intriguing pattern : there is a clear linear relationship between a model ’ s final performance on ID and OOD data ( Taori et al. , 2020 ; Recht et al. , 2019 ; Yadav & Bottou , 2019 ; Miller et al. , 2020 ) . In other words , given a model ’ s performance on an ID test set , a linear fit can accurately predict what the performance drop will be on the OOD test set . This also implies that a certain amount of improvement on OOD accuracy can be explained by an improvement on ID accuracy . The linear relationship holds across a wide range of models and is well-established for several robustness benchmarks , including image classification on both synthetic and natural distribution shifts ( Recht et al. , 2018 ; Yadav & Bottou , 2019 ; Recht et al. , 2019 ; Taori et al. , 2020 ) , 2D object detection ( Shankar et al. , 2019 ) , and question-answer models in NLP ( Miller et al. , 2020 ) . Since even the highest-performing models will still have a gap between ID and OOD accuracy , the linear relationship reveals that our current methods are insufficient for addressing OOD shift . Models which lie above the linear fit are said to exhibit effective robustness ( ER ) ( Taori et al. , 2020 ) , which measures the model ’ s OOD accuracy relative to the fit ( see Figure 1 ( a ) for an illustration ) . Models with nonzero ER deviate in their OOD behavior in a manner that is qualitatively and quantitatively different from what we can currently achieve . We seek to thoroughly investigate when the pre-training and fine-tuning paradigm gives rise to models with nonzero ER , and understand factors that control this behaviour . Models with high ER ( > 1 % ) are exceedingly rare . First , the most notable example is the recently proposed zero-shot CLIP model ( Radford et al. , 2021b ) , and subsequent work ( Wortsman et al. , 2021 ) has found that weight-space ensembling of zero-shot and fine-tuned CLIP-style models can improve both ID and OOD accuracy and produce models with high ER . Second , only a handful of the 204 ImageNet models evaluated in Taori et al . ( 2020 ) were found to have non-zero ER . Though most of these effectively robust models identified so far were pre-trained on large datasets , the majority of models pre-trained this way exhibit no ER , and we currently do not know when additional data helps ; what effect the choice of architecture has ; as well as the effects of dataset and distribution shift on these findings . In this work , we present an empirical study of the evolution of ER throughout fine-tuning . By studying the evolution , we find intriguing properties that are missed by focusing only on models at convergence : pre-trained models exhibit ER , while randomly-initialized models do not ( see Figure 1 ) . Summary of Contributions : . • ( Section 5.1 , 5.3 ) We identify pre-trained , fine-tuned models as an entire class of effectively robust models ( that match the ER of CLIP ( Radford et al. , 2021a ) ) and investigate how details such as model size , dataset size , and example difficulty influence ER . We find that the vanishing of ER at convergence depends on the distribution shift , and on some datasets pre-trained , fine-tuned models may still exhibit non-zero ER at convergence . • ( Section 5.4 ) We analyze properties of pre-trained models at the peak of their ER , including particular metrics defined in previous theoretical work ( Mania & Sra , 2020 ) . We find that effectively robust models make remarkably dissimilar predictions compared to standard models , and are able to correctly classify 10 % of the examples that no other model gets correct . • ( Section 5.5 ) We find that pre-trained models gradually lose their ER throughout fine-tuning , even as both the ID and OOD accuracies of the model simultaneously increase . We discuss several potential solutions to mitigate this problem , but find that none of them are able to maintain high ER at high ID accuracy . 2 RELATED WORK . Here we review some prior findings that are key for understanding the context of our results . Linear trends under distribution shift . In recent replication efforts , researchers carefully recreated new test sets for the CIFAR-10 and ImageNet classification benchmarks ( Recht et al. , 2018 ; 2019 ) . Despite following the original collection procedures as closely as possible , some shift was introduced . Evaluating an extensive testbed of trained models on the original and new tests revealed a clear relationship between accuracy on the original and new tests that is well-captured by a linear fit with positive slope . Follow-up work also found linear trends for new test sets on MNIST ( Yadav & Bottou , 2019 ) and the SQUAD question-answer dataset ( Miller et al. , 2020 ) , geographic distribution shifts for 3D Object Detection in self-driving cars ( Sun et al. , 2019 ) , and synthetic distribution shifts ( Taori et al. , 2020 ) . We find that models pre-trained on larger and more diverse datasets can break the linear trend of accuracy under distribution shift in the middle of the fine-tuning process , and we investigate factors that affect this . Examples of effectively robust models . Models that lie off the linear trend ( effectively robust models ) are historically extremely rare . In a recent extensive empirical study , researchers evaluated 204 trained ImageNet models spanning a wide range of architectures and training techniques on several popular natural and synthetic robustness benchmarks for ImageNet ( Taori et al. , 2020 ) . Across several natural robustness benchmarks , the main outliers with positive ER and high accuracy on ImageNet were all models that were pre-trained on larger and more diverse data than the ImageNet training set . They included a ResNet152 model trained on 11,000 ImageNet classes ( Wu , 2016 ) , several ResNeXt models trained on 1 billion images from Instagram ( Mahajan et al. , 2018 ) , and the EfficientNet-L2 ( NoisyStudent ) model trained on a Google-internal JFT-300M dataset of 300 million images ( Xie et al. , 2020 ) . However , not all models that were pre-trained on larger datasets showed effective robustness . Another recent work , ( Radford et al. , 2021a ) , observed that zero-shot CLIP classifiers have larger ER than CLIP models that had been fine-tuned to the downstream task . While the CLIP model was pre-trained with a contrastive loss that combined components from natural language processing and image classification , our results show that pre-trained ImageNet models that are evaluated in a zero-shot manner also have high ER , and , similar to the CLIP model , these image classification models lose their ER once they are fine-tuned to the downstream task . Since the completion of this paper , subsequent work ( Wortsman et al. , 2021 ) has proposed weight-space ensembles for fine-tuning ( WiSE-FT ) that interpolates between zero-shot and fine-tuned weights to find models with high ER as well as high ID and OOD accuracy . This method is currently limited to CLIP-style models , and in this work we mostly focus on traditional supervised image classification models . Model similarity . Recent work offers a theoretical explanation , relying on an assumption of model similarity ( Mania & Sra , 2020 ) , for why classifier accuracies follow a linear trend under distribution shift . Empirically , Mania et al . ( 2019 ) observed that the labeling assignments across a wide range of ImageNet models are significantly more similar to each other than would be expected by chance , which suggests that the size of the class of function approximators that neural networks learn is smaller than what may be expected a priori . Under the assumption of model similarity , Mania et al . ( 2019 ) proves that models ’ accuracies , when evaluated on two distributions , must be approximately collinear , unless the size of the distribution shift is large in a certain sense . In our work , we identify and analyze models that are not collinear with other models when evaluated on two distributions , and we find that such models break the assumption of model similarity sufficient to prove the existence of the linear fit . 3 EFFECTIVE ROBUSTNESS . The linear fit to testbed model accuracies on ID and OOD data provides a baseline for how improvements in OOD accuracy are tied to improvements in ID accuracy based on current methods . Effective robustness ( ER ) , a metric for robustness proposed in Taori et al . ( 2020 ) , measures the difference between a model ’ s OOD accuracy and that predicted from the baseline . ( See Figure 1 ( a ) for an illustration . ) 10 30 40 50 60 70 80 90 95 99 CIFAR-10 test acc ( % ) 10 20 30 40 50 60 70 80 90 95 C IF A R -1 0 . 1 te st ac c ( % ) Same Accuracy ( y=x ) Fit to Testbed Models Testbed Models 20 40 60 80 100 . CIFAR-10 test acc ( % ) 20 40 60 80 100 C IF A R -1 0 . 1 te st ac c ( % ) Same Accuracy ( y=x ) Fit to Testbed Models Testbed Models Figure 2 : Testbed model accuracies ( Recht et al . ( 2018 ) ) show a linear relationship in a rescaled logit-space Taori et al . ( 2020 ) . ( a ) shows the linear fit in logit-space , and ( b ) shows the same fit transformed back to linear-space . To define ER , we fix an ID and OOD test set and let β ( x ) be the baseline predicted OOD test set accuracy for a given accuracy x on the ID test set . Given a set of models evaluated on both the ID and OOD test sets , β can be computed by performing a log-linear fit between the models ’ accuracies on the two test sets . Similar to Taori et al . ( 2020 ) , we empirically find that the best linear fit comes from rescaling the accuracies using the logit function , logit ( α ) = log ( α 1−α ) , before performing the linear fit . In Figure 2 we show the log-linear fit for CIFAR-10 vs. CIFAR-10.1 in logit-space as well as transformed back to linear-space . See Appendix A for more details . The effective robustness of a model f is then defined as ρ ( f ) = accout ( f ) − β ( accin ( f ) ) , ( 1 ) where accin ( f ) and accout ( f ) represent the model ’ s accuracies on the ID and OOD test sets , respectively . Graphically , ER represents the distance above the testbed line to a model ’ s OOD accuracy , as shown in Figure 1 ( a ) . We note that ER is distinct from the absolute accuracy a model can achieve on the OOD data . For instance , the effectively robust model ( green point ) in Figure 1 ( a ) still has low absolute OOD accuracy relative to other models . An ideal model achieves both high ER and high absolute OOD accuracy .

Summary: This paper highlights important variables impacting the effective robustness (ER) of a pre-trained, fine-tuned model.  The authors identify that increasing model size, dataset size, and example difficulty improves the ER of a pre-trained, fine-tuned model. The experiments suggest that the zero-shot component of CLIP plays a significant role in the high value of ER CLIP achieves. The investigation of ER on dominance probability shows that models with high ER have high dominance probability. The authors also present a negative result showing that several reasonable approaches to maintaining high ER while fine-tuning fail.