An Empirical Study of Pre-trained Models on Out-of-distribution Generalization

Generalizing to out-of-distribution ( OOD ) data – that is , data from domains unseen during training – is a key challenge in modern machine learning , which has only recently received much attention . Some existing approaches propose leveraging larger models and pre-training on larger datasets . In this paper , we provide new insights in applying these approaches . Concretely , we show that larger models and larger datasets need to be simultaneously leveraged to improve OOD performance on image classification . Moreover , we show that using smaller learning rates during fine-tuning is critical to achieving good results , contrary to popular intuition that larger learning rates generalize better when training from scratch . We show that strategies that improve in-distribution accuracy may , counter-intuitively , lead to poor OOD performance despite strong in-distribution performance . Our insights culminate to a method that achieves state-of-the-art results on a number of OOD generalization benchmark tasks , often by a significant margin . 1 INTRODUCTION . Most machine learning ( ML ) models assume that test data is drawn from the same distribution as training data . However , this assumption does not hold in many real-world applications . As a result , ML models often fail to generalize to out-of-distribution ( OOD ) data encountered during their deployment and suffer from significant performance drops compared with the model performance on in-distribution ( ID ) data ( Quiñonero-Candela et al. , 2009 ; Torralba & Efros , 2011 ) . For example , common distribution shifts prevalent during test time include variation in locations ( Koh et al. , 2021 ) and weather ( Volk et al. , 2019 ) , noise and blur corruptions ( Hendrycks & Dietterich , 2018 ) , and small adversarial perturbations ( Szegedy et al. , 2013 ) . As ML models are increasingly deployed in safety-critical applications , it is becoming ever more critical to ensure strong OOD generalization for such models , i.e. , the models robustly generalizing to relevant OOD data not seen during training . While this problem is indeed difficult since the goal is to generalize to data that are not seen during training , there have been a handful of methods recently proposed to improve OOD generalization . Some methods propose specialized training methods , such as simulating OOD data during training ( Li et al. , 2018a ) , learning invariant representations ( Arjovsky et al. , 2019 ) , and performing adversarial data augmentation ( Volpi et al. , 2018 ) . Intriguingly , Gulrajani & Lopez-Paz ( 2020 ) conducted an extensive empirical evaluation on domain generalization benchmark datasets , and demonstrated that classical empirical risk minimization ( ERM ) approach achieves nearly state-of-the-art OOD generalization performance compared with these specialized methods . On the other hand , most of the approaches apply small or medium size networks that are usually pre-trained on the ImageNet-1k dataset , such as pre-trained ResNet50 ( Gulrajani & Lopez-Paz , 2020 ) . On the other hand , recent works find that pre-training on larger and more diverse data is one of the most effective paths toward generalizing to out-of-distribution data on ImageNet ( Taori et al. , 2020 ) . In this paper , we systematically investigate the importance of pre-trained models for OOD generalization . Specifically , we conduct extensive experiments on models with different model sizes that are pre-trained on large datasets . The pre-trained models are then fine-tuned on the training data for the underlying task . Instead of focusing on achieving state-of-the-art results on benchmark OOD datasets , our empirical study aims to develop a better understanding of the critical role that pre-trained models along with different design choices for fine-tuning such models play in ensuring good OOD generalization . This provides novel insights towards closing the gap between ID and OOD generalization for future research . The main contributions of our work are as follows : • When leveraging larger models for OOD generalization , we find that both large pre-training dataset sizes and large model sizes are critical . Missing either one of the two components may hurt OOD generalization . • We show that using a small learning rate generalizes better for OOD when we leverage a pretrained model . This is a complementary argument to Li et al . ( 2019 ) that suggested to use a large learning rate for better generalization in the case of no pre-training . • We show cases where improving in-distribution performance actually leads to worse OOD performance suggesting that ID performance is not a reliable indicator of OOD performance . • Our insights culminate in a method that achieves SOTA , often by a significant margin , on a number of OOD generalization benchmark tasks including PACS , VLCS , and Office-Home . 2 PRELIMINARIES AND EXPERIMENTAL SETUP . We begin this section by introducing the out-of-distribution generalization problem , with the primary focus on a special case of this broader issue , namely domain generalization . Subsequently , we describe the experimental setup adopted in our empirical study and various parameter choices we make throughout the paper . In this paper , we study a general multi-class classification setting , with all input instances and their labels belonging to X and Y : = { 1 , . . . , K } , respectively . Let D denote the training dataset which may potentially comprises data belong to k different training domains { Dj } j∈ [ k ] : = { 1 , ... , k } , i.e. , D = ∪j∈ [ k ] Dj . We assume that the training data from the j-th domain Dj = { ( xji , y j i ) } i∈ [ nj ] ⊂ X × Y is sampled from the distribution P jid , i.e. , ( x j i , y j i ) ∼ P j id . At test time , we evaluate a trained model for both its in-distribution and out-of-distribution performance . The in-distribution performance is evaluated on a test dataset Tid that consists of instances belonging to the domains encountered in the training dataset D. On the other hand , we utilize an OOD test dataset Tood from an unseen domain with distribution Pood to assess the model ’ s OOD performance . Given a family of candidate classifiers F = { f : X → Y } and the underlying training dataset D , we primarily employ standard empirical risk minimization ( ERM ) ( Vapnik , 1998 ) to learn a classifier f̂ as follows : f̂ = argmin f∈F 1 |D| ∑ i∈D ` ( f ( xi ) , yi ) , ( 1 ) where ` ( · , · ) denotes the cross-entropy loss function . For a classifier f , we define its in-distribution accuracy Accid , f and out-of-distribution accuracy Accood , f as follows : Accid , f = E ( x , y ) ∈Tid [ 1 { f̂ ( x ) 6= y } ] ; Accood , f = E ( x , y ) ∈Tood [ 1 { f̂ ( x ) 6= y } ] , ( 2 ) where 1 { · } denotes the standard indicator function . Often , models that achieve large in-distribution accuracy Accid only achieve relatively small out-of-distribution accuracy Accood , i.e. , Accid Accood ( Torralba & Efros , 2011 ; Hendrycks & Gimpel , 2016 ; Gulrajani & Lopez-Paz , 2020 ) . Thus , under the domain generalization problem , one particularly focuses on designing training methods that result in classifier with good performance on both out-of-distribution data and in-distribution data . Pre-trained models . We mainly focus on fine-tuning pre-trained models on the training dataset D by using ERM . We explore four classes of pre-trained models for OOD generalization : ( 1 ) . ResNetbased models ( He et al. , 2016a ; Xie et al. , 2017 ) pre-trained on ImageNet ( Russakovsky et al. , 2015 ) ( ResNet50 , ResNext50-32x4d , and ResNext101-32x8d ) ; ( 2 ) . ( BiTm ) -ResNet-v2-based models ( He et al. , 2016b ) pre-trained on ImageNet-21k ( Deng et al. , 2009 ) ( ResNetV2-50x1 , ResNetV2-50x3 , and ResNetV2-101x1 ) ( Kolesnikov et al. , 2020 ) , where group normalization ( Wu & He , 2018 ) and weight standardization ( Qiao et al. , 2019 ) are used in ResNetV2 ; ( 3 ) . ( SWSL ) -ResNet-based semi-weakly supervised ImageNet models pre-trained on IG-1B-Targeted data ( Yalniz et al. , 2019 ) ; and ( 4 ) . Vision transformer ( ViT ) pre-trained on ImageNet-21k ( Dosovitskiy et al. , 2020 ) . A detailed description of these pre-trained models can be found in Table 3 ( in Appendix ) . OOD datasets . In our experiments , we use four vision datasets used to benchmark domain generalization algorithms ( Gulrajani & Lopez-Paz , 2020 ) : ( 1 ) . PACS dataset ( Li et al. , 2017 ) ; ( 2 ) . Office-Home dataset ( Venkateswara et al. , 2017 ) ; ( 3 ) . VLCS dataset ( Fang et al. , 2013 ) ; and ( 4 ) . TerraIncognita dataset ( Beery et al. , 2018 ) . Each of these datasets contains 4 different domains . We train the models on 3 domains and treat the examples from the remaining domain as the out-ofdistribution data Tood . For the three training domains , we use 80 % data as training dataset and the remaining 20 % data for evaluation . We add the test domain information after the dataset to specify the test domain , for example , PACS ( S ) means the training domains are ‘ P ’ , ‘ A ’ , and ‘ C ’ and the test ( OOD ) domain is ‘ S ’ . Fine-tuning and model selection . We use stochastic gradient descent ( SGD ) with a momentum of 0.9 for fine-tuning all pre-trained models considered in this paper . The default weight decay for SGD is set to be 0 . We use a cosine learning rate decay ( Loshchilov & Hutter , 2016 ) as learning rate scheduler for SGD . For initial learning rates η we compare the following set { 0.05 , 0.02 , 0.01 , 0.005 , 0.002 , 0.001 , 0.0005 , 0.0002 , 0.0001 } . For evaluation , we pick the five checkpoints from each model with highest in-distribution accuracy . We then compute the average OOD accuracy of these five checkpoints . 3 MAIN RESULTS . We present our main experimental results in this section . First , we highlight the importance of models pre-trained on large and diverse datasets for OOD generalization . Next , we investigate the effect of fine-tuning learning rates , especially for models pre-trained on diverse datasets . Then , we perform a systematic examination of several components of pre-trained models for OOD generalization , including model size , pre-training dataset , and model architecture . Finally , we evaluate whether techniques used for improving ID accuracy can also enhance OOD generalization . 3.1 IMPORTANCE OF A BETTER PRE-TRAINED MODEL . Models pre-trained on more diverse datasets have been shown to achieve better OOD generalization on real-world distribution shifts ( Taori et al. , 2020 ; Hendrycks et al. , 2020a ) . As a warm-up for understanding the properties of pre-trained models on OOD data , we first study the OOD generalization performance of a ResNet-based model that is pre-trained on a large and diverse pre-training dataset . In particular , we focus on an SWSL-ResNext101-32x4d model pre-trained on the IG-1B-Targeted data ( Yalniz et al. , 2019 ) , which is a much larger and more diverse dataset than ImageNet . The fine-tuning approach described in Section 2 with different learning rate significantly outperforms the baseline results from Gulrajani & Lopez-Paz ( 2020 ) ( cf . Table 1 ) . This shows that a better pretrained model indeed improves OOD generalization without using specialized algorithms for domain generalization . For example , the OOD accuracy improves from 79.1 % to 91.3 % on PACS ( S ) , and 62.7 % to 76.4 % on Office-Home ( A ) . Overall , Table 1 suggests that using a larger model pre-trained on more data can be very effective for better OOD generalization . Next , we conduct more detailed experiments to better understand the OOD generalization performance of various pre-trained models .

Summary: This paper provides a thorough and relatively in-depth analysis of how pre-trained models affect OOD generalization performance. Specifically, the authors dissect this issue and perform extensive ablations on learning rates, model architecture, dataset size, model size, etc. By studying large-scale pretraining on vision tasks, the paper claims that 1) pre-trained models typically have better OOD generalization performance (which prior works have somewhat discussed as well); 2) small learning rates during the fine-tuning phase tend to create more robust models; and 3) some of the previous understandings (e.g., of how ID and OOD accuracies are related; or of the effect of training data size) might need to be revisited.