An Empirical Study of Pre-trained Models on Out-of-distribution Generalization

Generalizing to out-of-distribution ( OOD ) data – that is , data from domains unseen during training – is a key challenge in modern machine learning , which has only recently received much attention . Some existing approaches propose leveraging larger models and pre-training on larger datasets . In this paper , we provide new insights in applying these approaches . Concretely , we show that larger models and larger datasets need to be simultaneously leveraged to improve OOD performance on image classification . Moreover , we show that using smaller learning rates during fine-tuning is critical to achieving good results , contrary to popular intuition that larger learning rates generalize better when training from scratch . We show that strategies that improve in-distribution accuracy may , counter-intuitively , lead to poor OOD performance despite strong in-distribution performance . Our insights culminate to a method that achieves state-of-the-art results on a number of OOD generalization benchmark tasks , often by a significant margin . 1 INTRODUCTION . Most machine learning ( ML ) models assume that test data is drawn from the same distribution as training data . However , this assumption does not hold in many real-world applications . As a result , ML models often fail to generalize to out-of-distribution ( OOD ) data encountered during their deployment and suffer from significant performance drops compared with the model performance on in-distribution ( ID ) data ( Quiñonero-Candela et al. , 2009 ; Torralba & Efros , 2011 ) . For example , common distribution shifts prevalent during test time include variation in locations ( Koh et al. , 2021 ) and weather ( Volk et al. , 2019 ) , noise and blur corruptions ( Hendrycks & Dietterich , 2018 ) , and small adversarial perturbations ( Szegedy et al. , 2013 ) . As ML models are increasingly deployed in safety-critical applications , it is becoming ever more critical to ensure strong OOD generalization for such models , i.e. , the models robustly generalizing to relevant OOD data not seen during training . While this problem is indeed difficult since the goal is to generalize to data that are not seen during training , there have been a handful of methods recently proposed to improve OOD generalization . Some methods propose specialized training methods , such as simulating OOD data during training ( Li et al. , 2018a ) , learning invariant representations ( Arjovsky et al. , 2019 ) , and performing adversarial data augmentation ( Volpi et al. , 2018 ) . Intriguingly , Gulrajani & Lopez-Paz ( 2020 ) conducted an extensive empirical evaluation on domain generalization benchmark datasets , and demonstrated that classical empirical risk minimization ( ERM ) approach achieves nearly state-of-the-art OOD generalization performance compared with these specialized methods . On the other hand , most of the approaches apply small or medium size networks that are usually pre-trained on the ImageNet-1k dataset , such as pre-trained ResNet50 ( Gulrajani & Lopez-Paz , 2020 ) . On the other hand , recent works find that pre-training on larger and more diverse data is one of the most effective paths toward generalizing to out-of-distribution data on ImageNet ( Taori et al. , 2020 ) . In this paper , we systematically investigate the importance of pre-trained models for OOD generalization . Specifically , we conduct extensive experiments on models with different model sizes that are pre-trained on large datasets . The pre-trained models are then fine-tuned on the training data for the underlying task . Instead of focusing on achieving state-of-the-art results on benchmark OOD datasets , our empirical study aims to develop a better understanding of the critical role that pre-trained models along with different design choices for fine-tuning such models play in ensuring good OOD generalization . This provides novel insights towards closing the gap between ID and OOD generalization for future research . The main contributions of our work are as follows : • When leveraging larger models for OOD generalization , we find that both large pre-training dataset sizes and large model sizes are critical . Missing either one of the two components may hurt OOD generalization . • We show that using a small learning rate generalizes better for OOD when we leverage a pretrained model . This is a complementary argument to Li et al . ( 2019 ) that suggested to use a large learning rate for better generalization in the case of no pre-training . • We show cases where improving in-distribution performance actually leads to worse OOD performance suggesting that ID performance is not a reliable indicator of OOD performance . • Our insights culminate in a method that achieves SOTA , often by a significant margin , on a number of OOD generalization benchmark tasks including PACS , VLCS , and Office-Home . 2 PRELIMINARIES AND EXPERIMENTAL SETUP . We begin this section by introducing the out-of-distribution generalization problem , with the primary focus on a special case of this broader issue , namely domain generalization . Subsequently , we describe the experimental setup adopted in our empirical study and various parameter choices we make throughout the paper . In this paper , we study a general multi-class classification setting , with all input instances and their labels belonging to X and Y : = { 1 , . . . , K } , respectively . Let D denote the training dataset which may potentially comprises data belong to k different training domains { Dj } j∈ [ k ] : = { 1 , ... , k } , i.e. , D = ∪j∈ [ k ] Dj . We assume that the training data from the j-th domain Dj = { ( xji , y j i ) } i∈ [ nj ] ⊂ X × Y is sampled from the distribution P jid , i.e. , ( x j i , y j i ) ∼ P j id . At test time , we evaluate a trained model for both its in-distribution and out-of-distribution performance . The in-distribution performance is evaluated on a test dataset Tid that consists of instances belonging to the domains encountered in the training dataset D. On the other hand , we utilize an OOD test dataset Tood from an unseen domain with distribution Pood to assess the model ’ s OOD performance . Given a family of candidate classifiers F = { f : X → Y } and the underlying training dataset D , we primarily employ standard empirical risk minimization ( ERM ) ( Vapnik , 1998 ) to learn a classifier f̂ as follows : f̂ = argmin f∈F 1 |D| ∑ i∈D ` ( f ( xi ) , yi ) , ( 1 ) where ` ( · , · ) denotes the cross-entropy loss function . For a classifier f , we define its in-distribution accuracy Accid , f and out-of-distribution accuracy Accood , f as follows : Accid , f = E ( x , y ) ∈Tid [ 1 { f̂ ( x ) 6= y } ] ; Accood , f = E ( x , y ) ∈Tood [ 1 { f̂ ( x ) 6= y } ] , ( 2 ) where 1 { · } denotes the standard indicator function . Often , models that achieve large in-distribution accuracy Accid only achieve relatively small out-of-distribution accuracy Accood , i.e. , Accid Accood ( Torralba & Efros , 2011 ; Hendrycks & Gimpel , 2016 ; Gulrajani & Lopez-Paz , 2020 ) . Thus , under the domain generalization problem , one particularly focuses on designing training methods that result in classifier with good performance on both out-of-distribution data and in-distribution data . Pre-trained models . We mainly focus on fine-tuning pre-trained models on the training dataset D by using ERM . We explore four classes of pre-trained models for OOD generalization : ( 1 ) . ResNetbased models ( He et al. , 2016a ; Xie et al. , 2017 ) pre-trained on ImageNet ( Russakovsky et al. , 2015 ) ( ResNet50 , ResNext50-32x4d , and ResNext101-32x8d ) ; ( 2 ) . ( BiTm ) -ResNet-v2-based models ( He et al. , 2016b ) pre-trained on ImageNet-21k ( Deng et al. , 2009 ) ( ResNetV2-50x1 , ResNetV2-50x3 , and ResNetV2-101x1 ) ( Kolesnikov et al. , 2020 ) , where group normalization ( Wu & He , 2018 ) and weight standardization ( Qiao et al. , 2019 ) are used in ResNetV2 ; ( 3 ) . ( SWSL ) -ResNet-based semi-weakly supervised ImageNet models pre-trained on IG-1B-Targeted data ( Yalniz et al. , 2019 ) ; and ( 4 ) . Vision transformer ( ViT ) pre-trained on ImageNet-21k ( Dosovitskiy et al. , 2020 ) . A detailed description of these pre-trained models can be found in Table 3 ( in Appendix ) . OOD datasets . In our experiments , we use four vision datasets used to benchmark domain generalization algorithms ( Gulrajani & Lopez-Paz , 2020 ) : ( 1 ) . PACS dataset ( Li et al. , 2017 ) ; ( 2 ) . Office-Home dataset ( Venkateswara et al. , 2017 ) ; ( 3 ) . VLCS dataset ( Fang et al. , 2013 ) ; and ( 4 ) . TerraIncognita dataset ( Beery et al. , 2018 ) . Each of these datasets contains 4 different domains . We train the models on 3 domains and treat the examples from the remaining domain as the out-ofdistribution data Tood . For the three training domains , we use 80 % data as training dataset and the remaining 20 % data for evaluation . We add the test domain information after the dataset to specify the test domain , for example , PACS ( S ) means the training domains are ‘ P ’ , ‘ A ’ , and ‘ C ’ and the test ( OOD ) domain is ‘ S ’ . Fine-tuning and model selection . We use stochastic gradient descent ( SGD ) with a momentum of 0.9 for fine-tuning all pre-trained models considered in this paper . The default weight decay for SGD is set to be 0 . We use a cosine learning rate decay ( Loshchilov & Hutter , 2016 ) as learning rate scheduler for SGD . For initial learning rates η we compare the following set { 0.05 , 0.02 , 0.01 , 0.005 , 0.002 , 0.001 , 0.0005 , 0.0002 , 0.0001 } . For evaluation , we pick the five checkpoints from each model with highest in-distribution accuracy . We then compute the average OOD accuracy of these five checkpoints . 3 MAIN RESULTS . We present our main experimental results in this section . First , we highlight the importance of models pre-trained on large and diverse datasets for OOD generalization . Next , we investigate the effect of fine-tuning learning rates , especially for models pre-trained on diverse datasets . Then , we perform a systematic examination of several components of pre-trained models for OOD generalization , including model size , pre-training dataset , and model architecture . Finally , we evaluate whether techniques used for improving ID accuracy can also enhance OOD generalization . 3.1 IMPORTANCE OF A BETTER PRE-TRAINED MODEL . Models pre-trained on more diverse datasets have been shown to achieve better OOD generalization on real-world distribution shifts ( Taori et al. , 2020 ; Hendrycks et al. , 2020a ) . As a warm-up for understanding the properties of pre-trained models on OOD data , we first study the OOD generalization performance of a ResNet-based model that is pre-trained on a large and diverse pre-training dataset . In particular , we focus on an SWSL-ResNext101-32x4d model pre-trained on the IG-1B-Targeted data ( Yalniz et al. , 2019 ) , which is a much larger and more diverse dataset than ImageNet . The fine-tuning approach described in Section 2 with different learning rate significantly outperforms the baseline results from Gulrajani & Lopez-Paz ( 2020 ) ( cf . Table 1 ) . This shows that a better pretrained model indeed improves OOD generalization without using specialized algorithms for domain generalization . For example , the OOD accuracy improves from 79.1 % to 91.3 % on PACS ( S ) , and 62.7 % to 76.4 % on Office-Home ( A ) . Overall , Table 1 suggests that using a larger model pre-trained on more data can be very effective for better OOD generalization . Next , we conduct more detailed experiments to better understand the OOD generalization performance of various pre-trained models .

Summary: Within the now common pre-train/fine-tune training paradigm, this paper provide an empirical examination of the influence of pre-training on out-of-distribution accuracy *after* fine-tuning, specifically for computer vision. The authors showcase evidence suggesting that the choice of the pre-training dataset, the size of the model and the fine-tuning learning rate directly influence downstream OOD accuracy throughout a series of tasks. In addition, they provide some analysis of the effect that various training techniques (eg. label smoothing, augmentation etc..) have on OOD accuracy when they are used at fine-tuning time.