An Empirical Study of Pre-trained Models on Out-of-distribution Generalization
Authors: Yaodong Yu
Heinrich Jiang
Dara Bahri
Hossein Mobahi
Seungyeon Kim
Ankit Singh Rawat
Andreas Veit
Yi Ma
Venue: ICLR 2022 Submitted (2022)
Keywords: out-of-distribution generalization
domain generalization
pre-training

Abstract: Generalizing to out-of-distribution (OOD) data -- that is, data from domains unseen during training -- is a key challenge in modern machine learning, which has only recently received much attention. Some existing approaches propose leveraging larger models and pre-training on larger datasets. In this paper, we provide new insights in applying these approaches. Concretely, we show that larger models and larger datasets need to be simultaneously leveraged to improve OOD performance. Moreover, we show that using smaller learning rates during fine-tuning is critical to achieving good results, contrary to popular intuition that larger learning rates generalize better when training from scratch. We show that strategies that improve in-distribution accuracy may, counter-intuitively, lead to poor OOD performance despite strong in-distribution performance. Our insights culminate to a method that achieves state-of-the-art results on a number of OOD generalization benchmark tasks, often by a significant margin.

Under review as a conference paper at ICLR 2022
AN EMPIRICAL STUDY OF PRE-TRAINED VISION MOD- ELS ON OUT-OF-DISTRIBUTION GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Generalizing to out-of-distribution (OOD) data - that is, data from domains unseen during training- is key challenge in modern machine learning, which has only recently received much attention. Some existing approaches propose leveraging larger models and pre-training insights in applying these approaches. Concretely, we show that larger models and larger datasets need to be simultaneously leveraged improve OOD performance on image classification. Moreover, we show that using smaller learning rates during fine-tuning is critical to achieving good results, contrary to popular intuition that larger learning rates generalize hetter when training from scratch. We show that strategies that improve in-distribution accuracy may, counter-intuitive poor OOD performance despite strong in-distribution performance. Our insights culminate to a method that achieves state-of- offte-of-theaarr results on a number of OOD generalization benchmark tasks, often by significant margin.
larger datasets. In this paper, we provide new
lead to
INTRODUCTION
Most machine learning (ML) models assume that test data is drawn from the same distribution as training data. However, this assumption does not hold in many real-world applications. As result, ML models often fail to generalize to out-t--distribution (OOD) data encountered during their deployment and suffer from significant performance drops compared with the model performance on in-distribution (ID) data (Quiñonero-Candel, et al., 2009; Torralba & Efros, 201 1). For example, common distribution shifts prevalent during test time include variation in locations (Koh et al., 2021) and weather (Volk et al., 2019), noise and blur corruptions (Hendryeks & Dietterich, 2018), and small adversarial perturbations (Szegedy et al., 2013). As ML models are increasingly deployed in safety-critical applications, it is becoming ever more critical to ensure strong OOD generalization for such models, i.e., the models robustly generalizing to relevant OOD data not seen during training.
While this problem is indeed difficult since the goal is to generalize to data that not seen during training, there have been a handful of methods recently proposed to improve OOD generalization. Some methods propose specialized training methods, such simulating OOD data during training (Li et al., 2018a), learning invariant representations (Arjovsky et al., 2019), and performing adversarial data augmentation (Volpi et al., 2018). Intriguingly, Guirajani & Lopez-Paz (2020) conducted an extensive empirical evaluation on domain generalization benchmark datasets, and demonstrated that classical empirical risk minimization (ERM) approach achieves nearly state-of-the-art OOD generalization performance compared with these specialized methods. On the other hand, most of the approaches apply small or medium size networks that are usually pre-trained on the ImageNet 1k dataset, such as pre-trained ResNet50 (Gulrajani & Lopez-Paz, 2020). On the other hand, recent works find that pre-training on larger and more diverse data is one of the most effective paths toward generalizing out-of distriibbui data on ImageNet (Taori et al., 2020).
In this paper, we systematically investigate the importance pre-trained models for OOD gen- eralization. Specifically, we conduct extensive experiments on models with different model sizes that are pre-trained on large datasets. The pre-trained models are then fine-tuned on the training data for the underlying task. Instead of focusing on achieving oate-of-the-all results on benchmark OOD datasets, our empirical study aims to develop better understanding of the critical role that pre-trained models along with different design choices for fine-tuning such models play in ensuring

---

Under review as a conference paper at ICLR 2022
Eine daka
1magellet-1t Pre-train
line tune
OOD data Trxt
OOD Accuracy
IG-18 Prestrain
0.65
(alono)
(snziooa)
pre-iriied model
generaline
0.80-
larger
pro training dataset
model
(kots))
0.70
0.980
0,985
0.996 0.995
In-distribution Accuracy
(a) Fine-tuning procedure for OOD generalization.
(b) ID V.S. OOD.
Figure 1: (Left) Illustration our fine-tuning procedure used throughout this paper, i.e., we fine-tune pre pre-trained model on training dataset DD and evaluate the model performance on OOD data Tood data domain not seen during training. (Right) Evaluating ID and OOD accuracies for two classes of pre-trained models. Orange squares represent the ImageNet-Il k pre-trained models and blue circles represent the IG- pre-trained models, where IG-1B is a larger dataset than ImageNet- lk. Larger marker size means the model size is larger.
good OOD generalization. This provides novel insights towards closing the gap between ID and OOD generalization for future research.
The main contributions of our work are follows:
When leveraging larger models for OOD generalization, we find that both large pre-training dataset sizes and large model sizes are critical. Missing either one of the two components may hurt OOD generalization.
We show that using a small learning rate generalizes better for OOD when we leverage a pre- trained model. This is complementary argument Li al. (2019) that suggested to use large learning rate for better generalization in the case of no pre-training.
We show cases where improving in-distribution performance actually leads to worse OOD performance suggesting that ID performance is not reliable indicator of OOD performance.
Our insights culminate in a method that achieves SOTA, often by a significant margin, on number of OOD generalization benchmark tasks including PACS, VLCS, and Office-Home.
2 PRELIMINARIES AND EXPERIMENTAL SETUP
We begin this section by introducing the out-of-distribbttinn generalization problem, with the primary focus on a special case of this broader issue, namely domain generalization. Subsequently, we describe the experimental setup adopted in our empirical study and various parameter choices make throughout the paper.
In this paper, study general multi-class classification setting, with all input instances and their labels belonging to and y == (1,.... .....,, K). respectively. Let 'D denote the training dataset which may potentially comprises data belong to & different training domains {Di {D* [=].M) i.e., D
Uje(R)") DJ. We assume that the training data from the j-th domain D = {(x) vz (e)))) C X' x y
sampled from the distribution Pá i.e., (w//u/) Pil At test time, we evaluate trained model for both its in-distribution and out-of out-of- of-distribution performance. The in-distribution performance is evaluated on a test dataset Taa that consists of instances belonging the domains encountered in the training dataset 'D. On the other hand, we utilize an OOD test dataset Tood from an unseen domain with distribution Pood to assess the model's OOD performance.
Given family of candidate classifiers F = {f : X + y) and the underlying training dataset D, we primarily employ standard empirical risk mininization (ERM) (Vapnik, 1998) to learn classifier as follows:
argmin JEF
1
Σ
(ff())) u.),
(1)
D
iED
2

---

Under review as a conference paper at ICLR 2022
where E(.,-) .) denotes the cross-entropy loss function. For classifier f. define its in in-distributtin
accuracy / and outtoo-diistibutiio accuracy ACCood.f as follows:
Accia,5 ()). [ [1x()) # ut|; AcCood.f = E( [1{[(x) + v}},
(2)
where 1()) denotes the standard indicator function. Often, models that achieve large in-distribution ricuracyy Accia only achieve relatively small out-of-distribution accuracy AcCood. i.c., Accia > AcCood (Torralba & Efros, 2011; Hendrycks & Gimpel. 2016; Gulrajani & Lopez-Paz, 2020). Thus, under the domain generalization problem, one particularly focuses designing training methods that result in classifier with good performance on both duttof-distiibution data and in-distribution data.
Pre-trained models. We mainly focus on fine-tuning pre-trained models on the training dataset 'D by using ERM. We explore four classes pre-trained models for OOD generalization: (1). ResNet- based models (He et al., 2016;; Xie et al., 2017) pre-trained on ImageNet (Russakovsky et al., 2015) (ResNet50, ResNext50-32x44 and ResNext101-32x800 (2). (esTTm)--ReNNet2--bbeeeeee models (He al., 2016b) pre-trained on ImageNet-21k (Deng al., 2009) (ResNetV2-50xL, ResNetV2-50x3. and ResNetV2-101x.) (Kolesnikov et al., 2020), where group normalization (Wu & Hc, 2018) and weight standardization (Qiao al., 2019) used in ResNetV2; (3). (SWSL)--ReNNet-basee) based semi-weakly supervised ImageNet models pre-traing pre-trained on IG-IB- --Tageeeed data (Yalniz et al., 2019); and (4). Vision transformer (ViT) pre-trained ImageNet-21k (Dosovitskiy et al., 2020). A detailed description of these pre-trained models can be found in Table 3 (in Appendix).
OOD datasets. In our experiments, we use four vision datasets used to benchmark domain gen- eralization algorithms (Gulrajani & Lopez-Paz, 2020): (I). PACS dataset (Li et al., 2017); (2). Office-Home dataset (Venkateswara et al., 2017); (3). VLCS dataset (Fang et al., 2013); and (4). Terralncognita dataset (Beery et al., 2018). Each of these datasets contains 4 different domains. We train the models on 3 domains and treat the examples from the remaining domain as the out-of- distribution data Tood- For the three training domains, we use 80% data as training dataset and the remaining 20% data for evaluation. We add the test domain information after the dataset to specify the test domain, for example, PACS (S) means the training domains are 'P,, 'A', and 'C' and the test (OOD) domain is 'S'.
Fine-tuning and model selection. We use stochastic gradient descent (SGD) with a momen- tum of 0.9 for fine-tuning all pre-trained models considered in this paper. The default weight decay for SGD is set to be O. We use a cosine learning rate decay (Loshchilov & Hutter, 2016) as learning rate scheduler for SGD. For initial learning rates n we compare the following set (0.05, 0.02, 0.01, 0.005, 0.002, 0.001, 0.0005, 0.0002, 0.0001}- }. For evaluation, we pick the five checkpoints from each model with highest in-distribution accuracy. We then compute the aver- . age OOD accuracy of these five checkpoints.
3 MAIN RESULTS
We present our main experimental results in this section. First, we highlight the importance of models pre-trained on large and diverse datasets for OOD generalization. Next, investigate the effect of fine-tuning learning rates, especially for models pre-trained on diverse datasets. Then, we perform systematic examination of several components of pre-trained models for OOD generalization, including model size, pre-training dataset, and model architecture. Finally, we evaluate whether techniques used for improving ID accuracy can also enhance OOD generalization. I
3.1 IMPORTANCE OF A BETTER PRE- PRE-TED MODEL
Models pre-trained on more diverse datasets have been shown to achieve better OOD generalization on real-world distribution shifts (Taori et al., 2020; Hendrycks et al., 2020a). As a warm-up for un- derstanding the properties of pre-trained models OOD data, we first study the OOD generalization performance of a ResNet-basee model that is pre-trained on a large and diverse pre-training dataset. In particular, we focus an SWSL-ResNextl00 01-32x4d model pre-trained the IG- 1--TTrgete data (Yalniz et al., 2019), which a much larger and more diverse dataset than ImageNet.
The fine-tuning approach described in Section 2 with different learning rate significantly outperforms the baseline results from Gulrajani& & Lopez-Paz (2020) (cf. Table 1). This shows that a better pre- trained model indeed improves OOD) generalization without using specialized algorithms for domain
3

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison with ERM baseline results from Guirajani & Lopez-Paz (2020). We compare our approach with the baseline in terms of the OOD accuracy AcCood. Note that our approach (Yalniz al., 2019) with different learning rates
amounts fine-tuning SWSL-ResNex(001-92. and employing the models selection procedure described in Section 2. Note that, for each benchmark, we treat one of the four domains as the OOD domain and fine-tune the model on the remaining three domains. We report results for all four choices for the OOD domain on each benchmark.
OOD Domain
OOD Domain
PACS
A
C
P
S
VLCS
C
L
S
V
Baseline
88.1
78.0
97.8
79.1
Baseline
97.6
63.3
72.2
76.4
Ours
96.2
94.6
99.4
91.3
Ours
98.2
66.1
77.0
80.5
Office-Home
A
C
P
R
TerraIncognita
L100
L38
L43
L46
Baseline
62.7
53.4
76.5
77.3
Baseline Ours
50.8
42.5
57.9
37.6
Ours
76.4
68.5
86.5
87.6
48.3
47.5
57.2
43.7
Test
1..00
PHOUDS
COO ACCUFBOY
1--331
2.44
PH0000S
0.63
0.05
0.3
ICCS
0.75
433
0.79
2.
616 2.75
3.0 5a% 2.16
0.7
Accnncy
Accursty
Acouncy
(a) PACS (C).
(b) OfficeHome (C).
(c) VLCS (S).
(d) Terra (L38).
NACS That Comaees
Officttom Teat Comercs
PUCS Teat dammerd
Tanddresgela Test d0m
Buidre
65
Baadin
000
000
Learning mit
Learting rate
Lewriing Date
Learting neee
(e) PACS (C).
OfficeHome (C).
VLCC (S).
(h) Terra (I.38).
fine-tuned using different learning rates on ID and OOD data. (Top row) Scatter plot of ID accuracy (X-axis) and OOD accuracy (Y -axis) (Bottom row) Compare ID accuracy with OOD accuracy w.r.t. learning rate (X-axis). The green dashed line corresponds to the baseline OOD accuracy, and the blue dash-dotted line represents the selected model (by selecting the model with best ID accuracy).
Figure 2: Evaluating models SSWSL-RsNNxx101-22222
generalization. For example, the OOD accuracy improves from 79.1% to 91.3% on PACS (S), and 62.7% to 76.4% on Office-Home (A). Overall, Table suggests that using larger model pre- pre-trained on more data can be very effective for better OOD generalization. Next, we conduct more detailed experiments to better understand the OOD generalization performance of various pre-trained models.
3.2 EFFECT OF FINE-TUNE LEARNING RATE
Given the fact that simply fine-tuning the SWSL model leads to compelling improvement in the OOD generalization, we now take closer look at models trained with different learning rates. We evaluate models trained with different learning rates on both ID and OOD test data. Figure 2 summarizes the results for fine-tuning SWSL-ResNext101-222 -3 with different learning rates. In Figure 2(a)-2(d), each point in the plot corresponds to a model trained with a distinct learning rate. As mentioned in Section 2, we select models that achieves the top-5 ID accuracy, and depict the standard deviation of both Accia and AcCood for each model. We only show results for models that achieve > 95% training accuracy to better compare model performance.
Regularization effect of small learning rate. Based on Figure 2, our main finding is that the fine-tuning learning rate plays key role in determining both ID and OOD accuracy. In particular,
4

---

Under review as a conference paper at ICLR 2022
PACS, Test dorrainc
officeomee. Test damaindC
VIE. Test domainss
0.8
Accuracy
0.9
O0D Accuracy
OOD Accuracy
0.6
0.8
0.7
0.7
Baselne
RR-0.01
LR=0.01
Beselime
IRw0.01
Beseine
L0=0.005
LR 2.005
LR-0.000
0.6
IR-0.001
LR-0.001
IR-0.001
16-1
10-1
10-
10 102
10-1
20-1
10
10-3
10
200
In-sistribution Training Loss
In-distribution Training Loss
In-distributiin Training
(a) PACS (C).
(b) OfficeHomme (C).
(C) VLCS (S).
during training. We visualize models trained with three different learning rates in terms of OOD accuracy vs. training loss. Each point in the above plots represents the model evaluated at one iteration during training
Figure 3: OOD accuracy of models (SWSI-ResNex1101-92200
observe that, in most of the settings considered in this paper, the models trained with smaller learning rates achieve much better OOD generalization', even when the II) accuracy does not change much. This is different from the fact that larger learning rates generalize better when the model is directly trained on the training data D without a pre-training phase (Li et al., 2019; Lewkowyez et al., 2020; Nakkiran, 2020). We also observe similar behavior with SWSL-ResNext101 10x3xxxx11 models, where smaller learning rates lead to better OOD performance (see Figure in Appendix). Intriguingly, we do not observe this phenomenon for models pre-trained on ImageNet. There, shown in Figure 12 in Appendix, when the ID accuracy is similar, a model trained with larger learning rate achieves better OOD generalization (c.g.. see Figure 12(c) and 12())). To summarize, results indicates that when models are pre- trained on more diverse datasets, smaller learning rates can lead to better OOD generalization.
To obtain a better understanding of the effect of learning rate, we also study the OOD accuracy during training. In Figure 3, we visualize the OOD accuracy vs. training loss as measured every 100 SGD iterations for models trained with three different learning rates E {0.01, 0.005, 0.001 J. Note that all three learning rates eventually achieve similar training loss, but the models trained with smaller learning rates have better OOD generalization. Figure 3 also suggests that models trained with larger learning rates cannot achieve similar OOD accuracy by using early stopping. Meanwhile, the figure also confirms the regularization effect of small learning rates for fine-tuning pre-trained models.
Overall, we find that the learning rate is a key parameter for achieving good OOD generalization. While different learning rates may not affect ID accuracy much, OOD accuracy can be very sensitive to the choice of learning rate. Our results also highlight a limitation of performing model selection based on ID accuracy as models with similar ID accuracy may have very different OOD performance.
3.3 PRE-TRAINING FOR BETTER OOD GENERALIZATION
Now, we systematically explore the role that pre-trained models play in improving OOD gener- alization. Specifically, we study three aspects of pre- pre-trained models: pre- pre-training dataset, model architecture, and model size. Furthermore, we compare the models trained from scratch (i.e., without pre-training) with pre-trained models with respect to OOD generalization.
Effect of pre-training dataset. We study the OOD performance of models pre-trained on different pretrained pre-trained on IG-IB-Tarre Targeted. The SWSL model
datasets, inlcuding ResNext101-32xx6 pre-trained on ImageNet, GiTm-ResNett222 on ImageNet-21k, and Nestsl11 and BiTm model achieve similar Top-1 accuracies ImageNet, 84.2% and 84.0%. respectively, and the standard ResNext101 -1-x3x8d achieves a slightly worse in terms of Top-1 accuracy of 79.3%. The results for this comparison are summarized in Figure 4 and Figure 15 (in Appendix).
Our first observation is that the pre-training dataset has a big impact on OOD generalization perfor- mance. With same architecture and model size, the SWSL pre-trained model consistently outperforms the standard ImageNet pre-trained model across. Secondly, we find that SWSL generally performs the best among all three models as it is pre-trained on the largest and the most diverse pre-training dataset. Furthermore, we find that the BiTm model pre- trained on ImageNet-21 k also generally
Among models achieve similar ID generalization, models trained with smaller learning rates achieve better OOD generalization.
5

---

Under review as a conference paper at ICLR 2022
PACS, Test domain:C
offichomee, Test domain:C
VLCS, Test domain:
Baseline
0.7
Baseline
0.80
Biseinn
1esnest101_ 32dhd
resnectt 01 12xld
rec3eat101, __1_12
Accuracy
OOD Accurasy
Accuracy
reshext101_ 32s5
resnext10__22x68
0.70
0.65
0.88 In-distribution Accuracy
in-distribution Accuracy
distribution Accuracy
(a) PACS (C).
(b) OfficeHomme (C).
(c) VLCS (S).
Figure 4: Evaluating OOD and ID performance of models pre-trained on different datasets. Each color corresponds the models pre-trained on distinct dataset and the dash-dotted line represents the model picked by our model selection procedure. For each model we report the accuracy across different fine-tuning learning rates.
PACS. Test d
Officellome. Test domain:C
VLCS. Test domain
000 Accuracy
OOD Accuracy
0.6
ODD Accuracy
0.75
0.75
0.70
0.70
0.65
Baseine
Baseine
B1se inc.
vit
wit
0.60
reaneiv2 bitm
reameto2 bilm
1winati2 skm
0.775
distribution Accuracy
mrddistribbbe bution Accuracy
In distributicn Accuracy
(a) PACS (C).
(b) OfficeHome (C).
(c) VLCS (S).
Figure 5: comparison of four VIT models and three BiTm models on OOD accuracy and ID) accuracy. The orange squares represent ViT models and the blue circles represent BiTm models. The dash-dotted lines represent the selected models. We do not distinguish the model architectures within the same model class.
performs better than the ResNext model pre-trained on ImageNet (except for the PACS dataset, where both models have similar OOD performance).
Another interesting finding is that the OOD accuracy can be significantly improved by chosing the right fine-tuning learning rate, while the improvement the accuracy is small. For example, in Figure 4(a), the ID) accuracy improves from 98% to 99%, whereas the OOD accuracy increases 10%. Our results suggest that the "accuracy on the line" phenomenon (Miller et al., 2021) does not always hold for the benchmark datasets used in domain generalization. This also echoes the observation from D' Amour et al. (2020), that OOD generalization can be very different among models with similar ID accuracies.
Effect of model architecture: VITs vs. CNNs. We now investigate the role that the model archi- tecture plays in ensuring good OOD generalization. Compared with convolutional neural networks (CNNs), the recently proposed Vision Transformers (ViT) achieves similar or even better performance on image classification tasks (Dosovitskiy et al., 2020). This raises the question whether vision transformers behave differently from CNNs in terms of their OOD performance? Towards this, we explore three Bee-V2 pre-trained models (Kolesnikov al., 2020) and four ViT pre-trained models (Dosovitskiy et al., 2020). In particular, we compare BiTm- Ves2 with ViT- ViT- patch33, amall-patch16 6, pasee, base- patch16). We present the comparison between ViTs and BiTms OOD benchmarks in Figure 5 and Figure 16 (in Appendix).
101x1, 50x3)
We find that the OOD generalization accuracy of VIT models is similar to that of BiTm models. In some cases, BiTs slightly outperform ViT's on OOD generalization, for example, results shown in Figure 5(b), 5(c), 16(a). Since both classes of models are pre-trained on the same pre- pre-training dataset (i.e., ImageNet-21k) and achieve similar ImageNet Top-1 accuracies (84.5% for ViTs vs. 84.0% for BiTms), our results indicate that replacing convolution operation with self-attention operation does not bring additional benefits in terms of OOD generalization for the settings consider in this paper.
Effect of model size. It is evident from Figure 4, where both the baseline model (i.e., ResNet50) and ResNext101- 01133xxdd are pre-trained on the ImageNet- k dataset, that increasing the model

---

Under review as a conference paper at ICLR 2022
ImageNet Pretrain
BiTm Pretrain
SWSL Pretrain
Resket50
5ilm AeSNiiv) 50xt
SWSL Redeetso
040
Rusherboo
517h Aallii02 10te:
SWIL ResMee:00 120dd
Redtestin
577m Noa 5003
SWSL Aedetar 101 32
Accuracy
0OD Accuracy
6.7
6.4
PACS
OfficeHome
VLCS
PACS
OfficeHome
VLCS
PACS
OfficeHome
VLCS
ImageNet pre-training.
(b) BiTm pre-training.
SWSL pre-training.
Evaluating OOD generalization for three classes of models with different model sizes. Left:: Results for ResNe(x)t models pre-trained on ImageNet- Ik. Middle: Results for BiTm-ResNetVV models pre-trained on ImageNet-21k, Right: Results for (WSL-ResNe(x))) models pre-trained on IG- 1e--B--TTttee For given model class, each model size is represented by distinct color.
Figure 6:
size from ResNe150 to ResNext101-32x42 can improve OOD generalization. For example, OOD generalization is improved from 78.0% 84.9% in Figure 4(a). Motivated by these promising results, we conduct experiments to understand the role of model size for OOD generalization. We consider increasing model size on three classes of models, including ResNe(x)t pre-trained on ImageNet- Ik, BiTm-ResNetV2 pre-trained on ImageNet-21 21,,, k, and ResNe(x)t pre-trained on IG-IBBTargeen Targetee. We investigate three model sizes for each class of pre-trained models and the results are summarized in Figure and Table 4-6 (in Appendix).
For all three classes of models, we find that increasing the model size can improve OOD generalization in many settings. Overall, this shows that model size is a crucial design choice for improving OOD generalization. Furthermore, we note that SWSL-ResNeer diverse dataset than
is pre-trained on a larger and more and both models achieve 100% training accuracy, but Ret-ReeNNe
on OfficeHome (C). This suggests that both the pre-training dataset and the model size play key role in determining the OOD generalization, and ideally it is preferable to employ larger models pre-trained on larger and more diverse pre-training data.
SWSL-ResNNeL50 has lower OOD accuracy than
OOD performance of models trained from random initialization. To further validate the impor- tance of pre- pre-training data along with the model size for better OOD generalization, train models with increasing model sizes from random initialization, i.e., without employing pre-training phase. Our results for this experiment (ef. Table show that larger models do not significantly improve the OOD generalization without the use of pre-training. For example, increasing ResNext50-33 ResNext101-32xx only improves the OOD accuracy on OfficeHome (C) by less than 2%. Thus, our results suggest that without pre-training, only increasing model size is not very effective in improving the OOD performance.
to
3.4 ANALYSIS OF TECHNIQUES USED FOR IMPROVING ID ACCURACY
In this subsection, examine various techniques that have been shown to improve ID accuracy to assess their utility in achieving good OOD generalization. First, we study the impact of training data size the OOD generalization, as increasing the number of training samples is an effective approach to improve ID generalization. Then we evaluate four techniques in our OOD setting, including label smoothing (Szegedy et al., 2016), AutoAugment (Cubuk et al., 2018), PatchGaussian (Lopes et al., 2019), and Sharpness-Awan Minimization (SAM) (Foret et al., 2020).
Utility of more training data. We consider fine-tuning with datsets of four different sizes, i.c., 100%, 50%, 25% and 12.5% of the total training data for a given benchmark. Our results in Figure 7 show that increasing the training data from 12.5% 100% does not significantly improve OOD generalization. In fact, on VLCS (S), a better OOD generalization is realized when we utilize less training data train a ResNext101-32x80 model (cf. Figure 7(c)). This suggests that increasing ID training samples is not as effective as using larger and better pre-trained models.
Methods for improving ID accuracy. We find that applying the methods listed in Table does not significantly improve the OOD generalization across four datasets compared with scaling model size and pre-training dataset size, and utilizing augmentations/reggllaiiaz
potentially even hurt
7

---

Under review as a conference paper at ICLR 2022
PACS, Test domalins
OfficeHome,, Test domain:C
VLCS. Test doma IncL
0.70
000 Accursi
00.90
OD0 Acturacy
0.85
0.60
000
0.66
pnst resnort101 32xad
seal resnexibo
26s1 resmext101. 32xad
reanextoo 32x4d
0.64
2n5. resnexcion 32x8d
reanextoo 22:40
40
8D
80
100
20
60
no
100
20
80
100
Training Data Size (9)
Training Data Size (%)
Training Data Size 1%)
(a) PACS-OOD (S).
(b) OfficeHome-OOD (C).
(c) VLCS-OOD (L).
domain:s
domain:C
VLCS, domaind
0.93
0.995
Accuracy
Accurary
0.985
0.980
0.91
DMs resseetto, 32wld
SM5, resnext5o 3budd resnextio 32x64
sws resneat5o, 32wdd
0.955
SMa resnexti01 32x88
resneatiol 32x8d
Training
Training Data
Training
(d) PACS-ID (S).
(c) OfficeHome-ID (C).
(f) VL.CS-ID (I.).
Figure 7: Evaluating OOD generalization performance of models trained with different number of training samples. X-axis represents the number of training samples. We use SWSL-ResNex550- 32x4d and DeSS-RReNext001-333 as the pre-trained models. For each pre-trained model, we visualize the OOD accuracies of the top-3 models selected by ID accuracy.
Table Evaluation of four techniques (label smoothing, AutoAugment, PatchGaussian, and SAM) across all settings. The number inside the parentheses after the method name represents the value of the sechnique-specffie hyperparameter, e.g., PatchGaussian (1.0) corresponds to employing PatchGaus- sian (Lopes et al., 2019) with o 1.0. We highlight the best two OOD accuracics for each dataset with bold text.
for OOD generalization. We use the same pre-trained model (SWSL- W
Method
PACS (C)
Office (C)
VLCS (L)
Terra (L46)
ERM (in Table 1)
94.6
68.5
66.1
43.7
Label Smoothing (0.1)
91.6
70.6
65.2
42.5
Label Smoothing (0.2)
93.5
70.8
66.3
44.6
Augomnt
93.5
70.7
65.2
38.1
PatchGaussian (1.0)
92.8
65.8
64.6
13.1
PatchGaussian (0.5)
94.4
69.3
63.6
9.7
SAM (0.02)
93.5
72.2
67.1
44.7
SAM (0.05)
92.8
71.3
67.5
42.3
OOD generalization. For example, applying PatchGaussian decreases 0OD accuracy Terra (L.46 from 43.7% to 13.1% and 9.7% with a = 1.0 and 0.5, respectively. In constrast, PatchGaussian has very minimal impact on the ID performance. where it achieves Acc₁ 95.8% and Accia 95.6%. respectively; ERM attains Acci = 95.9%. Contrary to our results, Lopes et al. (2019) notice that PatchGaussian can improve both the clean (ID) accuracy and robustness common corruptions (OOD accuracy). This suggests that one should employing augmentattion/rggulaizzaalt carefully so as to not harm OOD generalization as a side effect. On the other hand, SAM with parameter 0.02 improves the OOD generalization on three benchmarks. Overall, we find that methods used for improving accuracy do not necessarily improve OOD accuracy, when compared with the simple ERM-based fine-tuning approach.
techniques
Thhe perturbation parameter p is defined in Foret al. (2020).
8

---

Under review as a conference paper at ICLR 2022
4 RELATED WORK
Domain adaptation and domain generalization. Ben-David et al. (2010) proposed HAH- divergence and applied it to develop error bounds on new test data distribution that are different from the training distribution. Motivated by the theoretical results in Ben-David et al. (2010), a large body of work was devoted learning Nomain-invariann representations for domain adaptation where unlabeled data from the target domain are available during training (Ajakan et al., 2014; Ganin et al., 2016; Tzeng et al., 2017: Zhao et al., 2018). Different from the domain adaptation problem. no information from the target domain is available in the domain generalization problem. Recent works proposed various algorithms to improve domain generalization, including learning invariant representations across training domains (Li et al., 201 8h; Arjovsky et al., 2019), distributionally robust optimization Sagawa ct al. (2020), data augmentation (Volpi et al., 2018; Zhou et al., 2020), and causal framework (Heinze-Deni) & Meinshausen, 2017: Mahajan et al., 2021). Zhou et al. (2021); Wang et al. (2021) provide a more detailed review of the topic of domain generalization.
OOD generalization and model robustness. Recent works developed new datasets evaluate model robustness to out-of-distribution data, including ImageNet- V2 (Recht et al.. 2019), CIFAR- 10.1 (Recht et al., 2018), ImageNet-C and CIFAR- 10-C (Hendrycks & Dietterich, 2018), ImageNet- R (Hendrycks et al., 2020a), WILDS (Koh et al., 2021), etc. A line of work proposed methods to improve model robustness to commons corruptions (Lopes et al., 2019; Xie et al., 2020; Hendrycks et 2020b; Calian et al., 2021; Yi et al., 2021). Xie et al. (2021) investigated how to leverage On multiple datasets, the researchers have observed near linear correlation between the OOD accuracy and the ID accuracy (Recht et al., 2019; Miller et al., 2020; Mania & Sra, 2020; Miller et al., 2021). However, Taori et al. (2020) found that most approaches, including the ones that improve robustness to synthetic distribution shifts, do not improve model robustness to natural distribution shifts, except for the models that are pre- pre-trained on larger datasets. Similar observation has been made in Hendryeks et al. (2020a). Our work focuses on the domain generalization benchmark datasets as well as how to perform fine-tuning one various pre-trained models for better OOD generalization.
auxiliary information and pre-training to improve OOD generalization.
DISCUSSION AND FUTURE WORK
Pre-training is one of the most effective approaches for improving model performance in wide range of machine learning tasks. In this work, we perform an empirical study of fine-tuning a diverse set of pre-trained models and evaluate their OOD generalization. We find that simply fine-tuning larger models pre- pre-trained on more (diverse) data can significantly improve model performance on OOD data, Additionally, and we also identify the regularization effect of small learning rates that important for achieving better OOD generalization. Further, through extensive experimentation, we demonstrate that, while relying on pre-trained models, model size and pre-training dataset play a key role in ensuring good OOD generalization. We hope our results can further inspire future research on bridging the gap between in-distribution accuracy and out-of -is-iiirritttionn accuracy. There are multiple interesting immediate directions explore in future work that we discuss next.
Scaling model size and pre-training dataset size. Our empirical results indicate that model size and the pre-training dataset size are two essential factors for improving OOD performance. Even though we primarily focus on image classification benchmarks, similar behavior has been observed in NLP domain (Brown al., 2020), where increasing model size leads to monotonic improvements in zero-shot performance on unseen tasks for example. It is worth noting that while increasing model size pre- pre-training dataset size only marginally improves the in-distribution accuracy, the increase in OOD accuracy can be much larger. It is an interesting direction to explore the differences in ID and OOD generalization in other domains.
How to perform model selection? Our study indicates that models trained via different methods can exhibit large variation in terms of their OOD performance, even when their in-distribution accuracy is very similar, e.g., see Figure 4(c) and Figure 5(c). We thus believe that the development of better model selection approaches for OOD generalization will be key direction of future work.
Limitations of ERM. Despite impressive results for OOD generalization, we find that ERM-based fine- tuning of trentrdi models is unlikely to close the in-distribution to qut-of-distribbtion generalization gap. even when the model size or the pre-training dataset becomes much larger. To generalize to unseen domains, it might be necessary to have access to extra information about the OOD dataset (e.g., unlabeled OOD data as available in the domain adaptation setting (Pan & Yang. 2009)) and/or design novel algorithms (e.g., updating the model parameters during test time (Sun et al., 2020)).
9

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement. In Section and Appendix A.2, We provide detailed descriptions the implementations used this paper, including about the pre-trained models, datasets, models selection, hyperparameters, and how train the models.
REFERENCES
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain- adversarial neural networks. arXiv preprint arXiv:|412.4444 2014.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 190.:v:90700200000 028993 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Shai Bven-David. John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1)1151-175, 2010.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint arXiv:2005.14165 2020.
Dan A Calian, Florian Stimberg, Olivia Wiles, Sylvestre- Alvise Rebuffi, Andras Gyorgy, Timothy Mann, and Sven Gowal. Defending against image corruptions through adversarial augmentations. arXiv preprint arXiv:2104.01080 2021.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:/805.09500 090011 2018.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint a:X3v:2011.03335. 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint 2:Xi::2002.06305, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.111 11929, 2020.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Proceedings of the IEEE International Conference on Computer Vision, pp. 1657-1664, 2013.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware aware minimization for efficiently improving generalization. In International Conference on Leurning Representations, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Nomain-adversarte training of neural networks. The journal of machine learning research, 17(1):2096-200 2030, 2016.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434. 2020.
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer. 2016b.
Christina Heinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustmess. arXiv preprint arXiv: 1710. 11469, 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2018.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint 1610.02 2016.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of diitr---ittiiuttonn generalization. arXiv preprint arXiv:2006. 1624|, 2020a.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representutions (ICLR), 2020b.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08330 003601,,..866.22200 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo. Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A benchmark of wild distribution shifts. In International Conference Machine Learning (ICML), 2021.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 491-507. Springer, 2020.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022.1. 2020.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pp. 5542-5550, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning, arXiv preprint arXiv:2002.11770 2020.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial nerworks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624 634, 2018b.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. arXiv preprint arXiv: 19000044999 2019.
11

---

Under review as a conference paper at ICLR 2022
Raphael Gontijo Lopes, Dong Yin, Ben Poole, Justin Gilmer, and Ekin D Cubuk. Improving robustness without sacrificing accuracy with patch gaussian augmentation. arXiv preprint (rXfi::106602611, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:/608.03340 2016.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference Machine Learning, pp- 7313-7324, PMLR, 2021.
Horia Mania and Suvrit Sra. Why do classifier accuracies show linear trends under distribution shift? arXiv preprint arXiv:2012.1548,, 2020.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect natural distribution shift question answering models. In International Conference on Machine Learning, pp- 69166 PMLR, 2020.
John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy the line: the strong correlation between -itto--distrrbutionn and in-distribution generalization. International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021.
Preetum Nakkiran. Learning annealing can provably help generalization, even for convex problems. arXiv preprint arXiv:2005.07366 2020.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):135-1135, 2009.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint 1rXXv::903.115220 2019.
Joaquin Quinonero-Candela. Masashi Sugiyama, Neil D Lawrence, and Anton Schwaighofer. Dataset shift in machine learning. Mit Press, 2009.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00453 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389-54800 PMLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25 15222,, 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference Learning Representations, 2020.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexci Efros, and Moritz Hardt. Test-time training with self-super for generalization under distribution shifts. In International Conference on Machine Learning, pp. 9229-92488 PMLR, 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv:1312.6199 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-28266 2016.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. arXiv preprint 0rXiv:2007.00644 2020.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528.
IEEE, 2011.
12

---

Under review as a conference paper at ICLR 2022
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Vapnik. Statistical learning theory new york. NY: Wiley, 1:2, 1998.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018-500 1088-5077210 20027,011 2017.
Georg Volk, Stefan Müller, Alexander von Bernuth, Dennis Hospach, and Oliver Bringmann. Towards robust cnn-based object detection through augmentation with synthetic rain variations. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 292. IEEE, 2019.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. arXiv preprint arXiv:1805.12018 2018.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: survey on domain generalization. arXiv preprint arXiv:2103.0388 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 819-828, 2020.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Sang Michael Xie, Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. In- n-out: Pre-training and self- self-training using auxiliary information for out-of-distribution robustness. In International Conference on Learning Representations, 2021. URL https: / httpt::/t/peeeeeee /opp/r/eiee.....e net/ forum? iid wiwimmddddjjiizgzaeee zziizqvr15.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification. arXiv preprint arXiv: 00556, 2019.
Mingyang Yi, Lu Hou, Jiacheng Sun, Lifeng Shang. Xin Jiang, Qun Liu, and Zhiming Ma. Improved ood generalization via adversarial training and pretraing. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11987-11997. PMLR, 18-24 Jul 2021.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106. 045600 2021.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao Costeira, and Geoffrey Gordon. Adversarial multiple source domain adaptation. Advances in Neural Information Processing Systems, 31:8559-85700 2018.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference Learning Representations, 2020.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A 2021.
survey. arXiv preprint
13

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 OTHER RELATED WORKS
Pre-training and fine-tuning, Li et al. (2020) examined how to select hyperparameters for fine- tuning on a various benchmark vision datasets. Dodge al. (2020) performed an extensive study on fine-tuning pre-trane pre-trained language models on NLP benchmarks, and investigated the impact of weight initialization and training data order on model performance. Kaplan et al. (2020) studied the scaling laws, including pre-training dataset size, model size, and compute budget for training, for language models. They found that increasing model size for pre-training can improve model performance on OOD data. More recently, Zhai et al. (2021) explored how to improve VIT model performance ImageNet and few-shot learning tasks through scaling the model size and data size.
A.2 ADDITIONAL EXPERIMENTAL DETAILS
Implementation details. Our implementation is mainly adapted from DomainBed (developed in Gulrajani & Lopez-Paz (2020)). Before we fine-tune the pre-trained models, replace the final linear layer of the pre-trained model with random initialized linear layer with output dimension equals to the number of classes for the dataset. Most of the pre-trained models are from PyTorch Image Models (t (tim The default training iteration is 5, 000. For the training from scratch setting (i.e., without pre-training), the fine-tuning, we save model checkpoint every 100 iterations and select five checkpoints that achieves the top-5 in-distribution accuracies. The input size of the ViT models is (3, 224, 224). In Table 3, we summarize the top-1/top-5 ImageNet accuracies of pre-trained models used in this paper.
the training iteration as 10, 000 and weight decay 10-5 During
Table 3: Top-1 accuracy and Top-5 accuracy of pre-trained models considered in this paper. (ImageNet) Resset-bbsed models pre-trained on ImageNet- (BiTm) ResNetV2-based mod- els pre-trained on Image (SWSL) ResNet-based models pre-trained IG-IB-Targeted. (ViT) Vision transformer-based models pre-trained on ImageNet-21k.
ImageNet
ResNet50)
ResNext50-32x4d
ResNexr101-32x68
Top-1 Acc 76.13
77.62
79.30
Top-5 Acc
92.86
93.69
94.51
BiTm
ResNetV2-50xl
ResNetV2-101xl
ResNetV2-50x3
Top-1 Acc
80.34
82.33
84.01
Top-5 Acc
95.68
96.51
97.12
SWSL
ResNet50
ResNext50-3225
ResNext101-32x4d
ResNext101-32xxd
Top-1 Acc
81.16
82.18
83.23
84.28
Top-5 Acc
95.97
96.23
96.76
97.17
VIT
Small-Patchi
Base-Patch32
Top-1 Acc
75.99
80.72
81.40
84.53
Top-5 Acc
93.27
95.56
96.13
97.29
A.3 ADDITIONAL EXPERIMENTAL RESULTS
In this subsection, present additional experimental results in Section 3.
Effect of fine-tune learning rate in Section 3.2. In addition to Figure 2 and Figure 3, we provide more experimental results on the effect of fine-tune learning rate. For fine-tuning results with different learning rates, we present more results for fine-tuning WWSL-ResNext101-33 for fine-tuning SWSL-RResNexx101-3333 -32 in Figure 9, results for fine-tuning SWSL-ResNNxt500- 32x4d in Figure 10, results for fine-tuning BiTm-Resseev22500 in Figure 11, results for fine-tuning Rest 101-32xdd (ImageNet pre-trained) in Figure 12, and results for fine-tuning ResNext50- 32x4d (ImageNet pre-trained) in Figure 13. Meanwhile, in Figure A.3, we provide more results visualizing the OOD accuracy vs. training loss for
in Figure 8. results
14

---

Under review as a conference paper at ICLR 2022
Effect of pre-training dataset. In Figure 15, we provide additional experimental results study- ing the effect of pre-training dataset.
Effect of model architecture: ViTs vs. CNNs. In Figure 16, we provide additional experimental results on comparing the performance of ViT' and CNNs.
Effect of model size. We provide additional results on comparing models with different model sizes in Table (ImageNet-Il Ik pre-trained models), Table 5 (BiTm pre-trained models (Kolesnikov et al.. 2020)), Table 6 (SWSL pre-trained models (Yalniz et al., 2019)), Table 7 (ViT pre-trained models (Dosovitskiy et al., 2020)), and Table 8 (without pre-training).
Utility of more training data. In Figure 17, we provide additional experimental results on investi- gating the impact of the number of training samples the Terralncognita dataset, including the ID/OOD accuracy results.
Methods for improving ID accuracy. In Table 9, we provide the in-distribution accuracy evalua- tions of the methods described in Table 2. Also, in Table 10 and 11, we provide additional results on the ID/OOD accuracy evaluations of different methods (on more datasets).
Texk demurrs
off darehri
Tex: domatci
0.30
Reudiar
Raidi
0:45
P-0.3
Iv-
1-331
0.25
0.5
5mhh
91
ECD AECT
C.43
PH0J0ES
1-23005
0.40
04
1-2300
0.92
16:460 butlon Acorrcy
1-412 bution AccU oay
In:422 (iution Acol Doy
In.d2 buctinniionnoo Acorocy
(a) PACS (S).
(b) OfficeHome (A).
(c) VL.CS (L)).
(d) Terra (T.46..
P4C5 Tal Smatee
officiion, Test dorsieck
PU.S Teit donaitt
Tarminesget Tee 146
6.50
4200000
6:00
Soethe the 500 siand
000 icis no/
000
64
Surdin
36
38
10
15
(c) PACS (S).
OfficeHome (A).
g VLCS (L).
(h) Terra (L46).
Figure 8: (Additional results) Evaluating models SWWSL-ResNex1001-922ee fine-tuned by different learning rates on in-distribution and out-of-distributio data. (Top row) Scatter plot of in-distribution accuracy (X-axis) and -iit---diitritttion accuracy (a-axis). (Bottom row) Compare in-distribution accuracy with corresponds to the baseline OOD accuracy, and the blue dash-dotted line represents the selected model (by selecting the model with best in-distribution accuracy).
accuracy w.r.t. learning rate (X (X-axis). The green dashed line
15

---

Under review as a conference paper at ICLR 2022
PACS, Text demaltic
Offielions. dentdrraacc
VLCS, This dorrancs
Terabesgets, Text deme 1:
Eare ha
0.325
0.30
OCD ACCUTIC
7:775-
00D Accurecy
0.35
0:475
6:723
PH0000s
0.1
0:42
Indistrbutiinn. Accuncyy
Ind on Accuraty
indsirite on AccurNT
Indstritue Arcurcty
(a) PACS (C).
(b) Home (C).
(c) VLCS (S).
(d) TerraIncognita (L38).
3.66
Basdin Indtrttuce eccence 000
Ird 07 acaraay
Ind accddna
acorecy
376
14 letton accassy
teaming
(c) PACS (C).
(f) OfficeHome (C).
VLCC (S).
(h) Terralncognita (1.38).
1ACS dasaaa
Oftedioms Tortirranc
Vucs, Tex somart
Teme
0.675
150.35
I-3
+-0.00
O20 Acsurscy
030 Acousey
0.025
I-1.22
I-0.00
1-3300
k-1
2.500
P-23382
P-2 K32
P-6-005
0:35
2.5
10
0.7
375
6.30
6.5 Accuracy
6.32
2.15
3.55
3.00
6965
1.60
6,555
3.505
1945
Acousey
(i) PACS (S).
i OfficeHome (A).
(k) VLCS (L).
Terralncognita (L46).
PACS Teat Comaaes
Officatom Test dorEa
PUCS, Test domaitt
Tamdineggn Test demuls466
0.
RUCLENO
000 scomnyy
scnuney
Burdre
Burdne
043 scodlay
Learning nie
Learting rate
Learning ofte
Learring roee
(m) PACS (S).
(n) OfficeHome (A).
(0) VLCS (L).
(p) Terralncognita (L46).
fine-tuned by different learning rates on in-distribution and out-of-distribution data. (Top row) Scatter plot of in-distribution accuracy (X-axis) and disto--distribution accuracy (Y-axis). (Bottom row) Compare in-distribution accuracy with out-of-distribution accuracy w.r.t. learning rate (X-axis). The green dashed line corresponds to the baseline OOD accuracy, and the blue dash-dot line represents the selected model (by selecting the model with best in-distribution accuracy).
Figure 9: Evaluating models (SWSL-ResNext100-32226
Table 4: Comparing ImageNet pre-trained models with different model sizes. We evaluate both in-distribution accuracy and out-of-iistribution (left: ID accuracy, right: OOD accuracy).
Models
PACS (C)
PACS (S)
ResNet50
0.977 + 0.000 / 0.785 + 0.006
0.980 + 0.001 / /0.99 = 0.008
ResNext55-32x4d
0.977 + 0.001 / 0.825 + 0.005
0.982 + 0.001 / 0.698 = 0.002
ResNext101-32x8d
0.981 + 0.000 0.849 + 0.005
0.988 + 0.000 / 0.765 = 0.013
Models
Office-Home (A)
Office-Home (C)
ResNet50
0.869 + 0.001 0.662 + 0.002
0.874 + 0.002 0.542 + 0.003
ResNNex55-32x4d
0.874 + 0.000 / 0.645 + 0.003
0.887 + 0.002 / 0.558 = 0.003
ResNext101-32x8d
0.887 + 0.000 0.717 = 0.002
0.903 + 0.000 0.624 = 0.003
Models
VLCC (I.)
VLCS (S)
ResNet50
0.898 + 0.000 / 0.645 + 0.003
0.885 + 0.001 /.745 + 0.003
ResNext55-32x4d
0.905 + 0.000 0.646 + 0.007
0.891 + 0.002 0.733 + 0.003
ResSeex101-32x8d
0.909 + 0.001 0.643 + 0.002
0.888 + 0.001 0.732 = 0.005
16

---

Under review as a conference paper at ICLR 2022
PiCS, Tert domainc
Offcclions, Test Somalic
Text derderaahs
Terabeegets, Text deme
Eare ba
0.522
OCD AcCUriE
0.75
I-335
F-2.05
0.70
203
0-423
Indstriuuion Accuricy
Ind Accuntty
Indikt tndiiributionno Acouncy
Ind ACcurNTy
(a) PACS (C).
(b) Home (C).
(c) VLCS (S).
(d) TerraIncognita (L38).
basdiw bndtrtbutt eccente 000
scnouey
04
Ind wwo.
26-1
36
20
Learning
ning rote
Loanning fats
(c) PACS (C).
(f) OfficeHome (C).
VLCCS (S).
(h) Terralncognita (1.38).
PACS TeL stmard
Orictiore: Tex: demerck
VuCs Text dareert
Tea Tarnerareete Te
knch 35
O3D Aosursyy
0.57
100.02
030 Acousey
b-3322
1-0300
E.43
1-3300
P-23002
K32
0.44
P-3-00
335 0.0 eds 6:00 3.5
6.75
8.00
3.00
0.12
3.81
0.55
3.95
650
Accurwcy
PACS (S).
OfficeHome (A).
(k) VLCS (L).
Terralncognita (L46).
39C5, Tesk COREET:
Cifcalioree Tert dorrbrck
V9.55 Test doment
Tamdineagn ta. Test derdl1s66
ACOFOCY
Iecettiiin 000 srm.
scnuney
Eaidin
000
Learsing rote
Learning ruts
Learring
Loarning roee
(m) PACS (S).
(n) OfficeHome (A).
(0) VLCS (L).
(p) Terralncognita (L46).
Figure 10: Evaluating models WWWS--ResNexttO-22xxx fine-tuned by different learning rates on in-distribution and out-of-distribution data. (Top row) Scatter plot of in-distribution accuracy (X-axis) and diito--distribution accuracy (Y-axis). (Bottom row) Compare in-distribution accuracy with oul-of-distribution accuracy w.r.t. learning rate (X-axis). The green dashed line corresponds to the baseline OOD accuracy, and the blue dash-dot line represents the selected model (by selecting the model with best in-distribution accuracy).
Table 5: Comparing BiTm models with different model sizes. We evaluate both in-distribution accuracy and out-of-distribution. (left: ID accuracy, right: OOD accuracy).
Models
PACS (C)
PACS (S)
BiTm-ResNeeV2250k
0.980 + 0.001 / 0.793 + 0.005
0.983 - 0.001 / 0.618 + 0.002
BiTm-RResNetV2-10kx
0.984 + 0.000 / 0.831 + 0.002
0.990 = 0.001 / 0.669 + 0.007
BiTm-RReseeV225003
0.984 +:2 0.000 0.805 + 0.004
0.989 + 0.000 0.704 + 0.013
Models
Office-Home (A)
Office-Home (C)
0.887 +: 0.001 / /0.21 + 0.001
0.895 + 0.001 0.609 + 0.002
0.904 + 0.001 / 0.755 = 0.001
0.910 = 0.000 / 0.651 = 0.000
0.912 + 0.000 / 0.792 = 0.001
0.928 = 0.000 / 0.685 + 0.002
Models
VL.CS (I.)
VLCS (S)
0.922 + 0.000 / 0.647 + 0.010
0.896 + 0.000 / 0.759 + 0.006
0.923 + 0.002 0.656 + 0.001
0.900 - 0.000 0.783 + 0.008
BiTm- V
0.931 + 0.001 / 0.655 + 0.009
0.906 = 0.001/0.777 + 0.005
17

---

Under review as a conference paper at ICLR 2022
PACS, Text demaltoc
O'teilerre Text demalnc
Text domaitss
Tarabesgets Text demen
0.35
Base IN
Ease
Base
6.530
0.40
050 Accureyy
0.00
OSD
0.80
0.50
Indktribttinn.o Acouncy
Inckdriuution. Acouncy
Indiitbbttinn Acouncy
Indstritur on Arcurncy
(a) PACS (C).
(b) Home (C).
(c) VLCS (S).
(d) TerraIncognita (L38).
domwee133
Basdite Indertiti sounc 000
1.35 000
0.0
3.
nnd aroddaaaccee
estraey
1- lu accrsoy
10-
tneming
PACS (C).
(f) OfficeHome (C).
VLCCS (S).
(h) Terralncognita (1.38).
PACL Tealdaaarss
O'tacttere Texl demairs
VL.% Tex: demunci
Temuncarrta Texk demaaaee
0.775
PHEDES
030 Acsurna
0:95
030 Acousey
In- kng
0.750
0.772
I-0000
6.65
1-3.001
2012
0.30
P-0000L
0.544
0.573
6,15
3,066
5.500
6.00
0.86
2.3
3.30
0.30
3.55
E.0
0.32
0.3
Gole Terraincognita (L46).
2:34
Accurwcy
Accumsy
PACS (S).
OfficeHome (A).
(k) VLCS (L).
PACS Teat Comates
Offictio Tos Teat dorea
NUCS, Test domattt
Tamdnesge Teat domu
10
Inlootatie eccuncy
000 mmm
Burdin
66
Learning net
Learting rate
Learring nete
Learring rote
(m) PACS (S).
(n) OfficeHome (A).
(0) VLCS (L).
(p) Terralncognita (L46).
Figure 11: Evaluating BiTm models ReTT--ReNNeV2255033 fine-tuned by different learning rates on in-distribution and -ito---distribtiion data. (Top row) Scatter plot of in-distribution accuracy (X-axis) and duitof-distribution accuracy (Y-axis). (Bottom row) Compare in-distribution accuracy with out-of-distribution accuracy w.r.t. learning rate (X-axis). The green dashed line corresponds to the baseline OOD accuracy, and the blue dash-dot line represents the selected model (by selecting the model with best in-distribution accuracy).
Table 6: Comparing SWSL models with different model sizes. We evaluate both in-distribution accuracy and out-of- (left: ID accuracy, right: OOD accuracy).
Models
PACS (C)
PACS (S)
SWSL-ResNet50
0.987 + 0.000 / 0.870 + 0.008
0.992 0.000 / 0.856 + 0.005
SWSSL-ResNex500-22444
0.987 + 0.000 0.902 + 0.002
0.994 = 0.000 / 0.887 + 0.003
SWSL-ResNexx101-33x888
0.990 + 0.000 0.951 + 0.002
0.997 0.000 / 0.919 + 0.006
Models
Office-Home (A)
Office-Home (C)
SWSL-ResNet50
0.886 + 0.000 / 0.729 + 0.003
0.908 0.001 / 0.658 + 0.002
SWSL-ResNext50-32244
0.898 = 0.000/ 0.760 = 0.001
0.914 0.001 / 0.663 + 0.001
SWSL-ResNex1001-322888
0.908 = 0.000 / 0.818 = 0.001
0.936 = 0.002 / 0.719 + 0.002
Models
VLCCS (I.)
VLCS (S)
SWSL ResNee50
0.920 + 0.000 / 0.662 +: 0.006
0.900 = 0.000 / 0.782 + 0.003
SWSI-ResNNxt50-92x44
0.926 + 0.003 0.664 + 0.008
0.903 0.001 0.792 0.001
SWSL-ResNex(I0-32x888
0.930 + 0.000
0.673 + 0.002
0.903 0.001 / 0.787 + 0.004
18

---

Under review as a conference paper at ICLR 2022
0.35
PACS, Test Semaanc
0,925
Offistore OMnaarC
0.75
VUCE Test OOTTRIS
Terdhroogeta Text daramii1
CCD ROSLEWCC
0.30
F+4.00
910
0.75
0,30
6.70
0:43
1ed
0.92
0.34
2.36
3.56
3.00
1.2
6.31
0.56
3.00
8:305
3,573
3.55
0.300
6,65
232
332
134
6.25
(a) PACS (C).
OfficeHome (C).
(c) VLCS (S).
(d) TerraIncognita (L38).
Tedt d
offeri dorain
dorales
Tershnmenn donoirt
ACOERTY
2.
Aecuety
Indoressin sscancy
Inder
Soctre
GOD Braraay
Leaming rabe
Leaming rile
Leaningrate
Leaming ng rol0
(e) PACS (C).
(f) OfficeHome (C).
VLCS (S).
(h) TerraIncognita (L38).
X2
CCD ACC AECURETT
0.7
Sardina
0.05
1-0.11
Bacelie
8m
0.63
1-3300
1-2300
0.350
P-0.203
PH0.000
5-12%
0.950
Indistiutiinn Acounicy
stdist Accuracy
bnst2/button Acouncy
Inddsrbudinn ACCuroS
PACS (S).
OfficHHome (A).
(k) VLCS (L).
(1 TerraIncognita (L46).
Offictomme, Test
0.5
Exsding ne
Kouaay
Dtaeline
1833 aornn
ocn
e.deertaton scoueyy
65
I-deritatinn stcueey
16
20
22
16
10
26-7
(m) PACS (S).
(n) OfficeHome (A).
(0) VLCS (L).
(p) TerraIncognita (L46).
Figure 12: Evaluating models (ResNext101-32x8d pre-trained on ImageNet) fine-tuned by different learning rates on in-distribution and dittt---distiitttion data. (Top row) Scatter plot of in-distribution accuracy (X-axis) and -iit----iitritttion accuracy (axix..). (Bottom row) Compare in-distribution accuracy with out-of-distribution accuracy w.r.t. learning rate (X -xis). The green dashed line corresponds to the baseline OOD accuracy, and the blue dash-dot line represents the selected model (by selecting the model with best in-distribution accuracy).
19

---

Under review as a conference paper at ICLR 2022
PASS TexL domanns
Dikcetare, Text derranc
VLOS Text CORPARSS
Turaheegnte, Teak delddommnnt
Eardine
030 Accuracy
0.55
P-005
0.70
Semineyy
P:325
6.0
5.00
P-3202
0.53
0:50
6:42
Indietniue
Inchtriuuttnn
Indier button Accuncyy
1.50
530
Indetriue
2.44
(a) PACS (C).
OfficeHome (C).
(c) VLCS (S).
Terralncognita (L38).
Gerrete
Oitacfome, demenr
Tell omneed
0.0 stoosey
0.00 actrsey
Learning rate
Lear ring rate
Leoming rate
Losming rote
(c) PACS (C).
(f) OfficeHome (C).
VLCS (S).
(h) Terralncognita (L38).
AAC% demanes
eitire, Text
Teardarakn
Ternternaytt Teuzdwrain
6.70
0.55
$-3.55
P-0.5
P-2.2
P-2-22
DCD Accurn
P-031
1-221
9-001
020
6.35
1--3305
0.333
1-0 390
0:35
8:50 0.35 1993 6.92 3350 ESTD
0.32
6.35 Indbd: Acorsey
0.35 0.09
1835
3500 1995
0.54
0.35
335
057
in-dsinitet AcCuiacy
Inickt Beulion Acontcc
hi.datrbuioon Acod way
(i) PACS (S).
Offic (A).
(k) VLCS (L).
(I) TerraIncognita (6.6).
NMCS, Texk domaees
Cifccftone Tel deruree
V115. Teal demuin
2.0
Tenddreagete Teak dem
0.3
Juariie onn sin
000 ar
budiee
65
33
3011
25
Learning
Leariin
Learzing
(m) PACS (S).
(n) OfficeHome (A).
(0) VLCS (L).
(p) TerraTncognita (L46).
Figure 13: Evaluating models (ResNext50-32xx4 pre-trained on ImageNet) fine-tuned by different learning rates on in-distribution and out-of ditt----iitiibutionn data. (Top row) Seatter plot of in-distribution accuracy (X-axis, and diitr---iitribution accuracy (Y-axis). (Bottom row) Compare in-distribution accuracy with ofist--diitriuution accuracy w.r.t. learning rate (X-axis). The green dashed line corresponds to the baseline OOD accuracy, and the blue dash-dot line represents the selected model (by selecting the model with best in-distribution accuracy).
PACS, Test domain:S
OfficeHomee Test domain:A
VLCS, Test domaind
0.8
0.9
0.70
000 Accuracy
000 Accuracy
0.7
Accuract
0.005
0.8
0.60
0.7
(R=001
Baswin
LR-0.005
0.6-
Baseine
(Ra0.1
L0-0.005
0.50
0.55
(Rub.00
Baswine
L8-0.055
0.6
0.45
10
10
10-T
10t7
10"
10
10
10'
10
10-2
10
10'
In Institirbuttion Training Loss
In dn-titributtin Training Loss
In distribuuiion Training Loss
(a) PACS (S).
(b) OfficeHome (A).
(c) VLCS (L).
Figure 14: OOD accuracy of models models trained with three different learning rates in terms of OOD accuracy v.s. training loss. Each point in the above plots represents the model evaluated at one iteration during training.
during training. We visualize
20

---

Under review as a conference paper at ICLR 2022
PACS, Test domain:5
Officeliome, Test domain:A
VLCS. Test domaidd
Beseiie
0.5
Bose tie
0.8
reaneii2 50x bit
(e5ncwtL01 32xdd
0OD Accuracy
nCs nesnextl11 32x66
netve 50xd bit
000 Accuracy
0.650
resneuti 22dd0__0_
resnextt 12dl0
0.6
0.000
0.4
0.4
0.575
resnetr2 bm restwedd 3238d
0.8 In dustiibuttion Accuracy
0.9
1.0
0.70
0.75
0.80
0.85
0.90
0.75
0.50
0.65
0.50
In distr bution Accuracy
In distribution Accuracy
(a) PACS (S).
(b) OfficeHomme (A).
(c) VLCS (L).
Figure 15: Evaluating out-of- -ff.iiibbiitttionnnnn and in no-distribution performance of models pre-trained on different datasets. Fach color corresponds to models pre-trained on one dataset and the dash-dot line represents the selected model.
domains
Officetomme,
dome
0.66
DOD Accurat)
DOD Accuracy
0.7
GOD Accuracy
0.64
Baseline
635
Basdline
Baseline
vk.
sllm
2im
0.85 Inrisstribttion Accuracy
In-distribution Accuracy
In-distribution Accuracy
(a) PACS (S).
(b) OfficeHome (A).
(c) VLCS (L).
Terraincoonta, Test dom
Terraincagria,, Test 446
0.5
OOD Accuracy
OOD Accuraty
0.3
Sazeline
0.3
Basdline
resnetv2_pimg
0.91
0.92
0.93
0.34
0.55
0.92
0.33
0.94
0.95
0.96 0.97
In-distribution Accuracy
Indiittiiution Accuracy
(d) TerraIncognita (L.38).
(e) TerraIncogaita (I.46).
Figure 16: A comparison of four VIT models and three BiTm models out-of-distributtinn accuracy and in-distribution accuracy. The orange squares represent ViT models and the blue circles represent BiTm models. The dash-dot lines represent the selected models. We do not distinguish the model architectures within the same model class.
Terraincognits Test domaincLtb
Terraincognta, Test doma 146
0.475
0.95
GOD Accuracy
0.450
Accuracy
0.425
0.90
0.400
Baseine
Lrsienss0 32r4d
awsl resnootto 3Ex44
0.375
vani restint101 32cld
snal reanect101 52xid
40
60
80
1.00
40
100
Training Data Size (9)
Training Data
(a) Terra-OOD (L46).
(b) Terra-ID (L46).
Figure 17: Evaluating OOD generalization performance of models trained with different number of training samples on the Terralncognita dataset. X-axis represents the number of training samples. We use WWSL-RReNNxtt0--222 pre-trained model, we visualize the OOD accuracies of the top-3 models selected by ID accuracy.
and WWSL-RReNNex1001333
as the pre-trained models. For each
21

---

Under review as a conference paper at ICLR 2022
Table 7: Comparing ViTs models with different model sizes. We evaluate both in-distribution accuracy and out-of-distribttion. (left: ID accuracy, right: OOD accuracy).
Models
PACS (C)
PACS (S)
ViT-small-patchh22
0.951 + 0.001 /0.746 + 0.001
0.964 + 0.001 / /0.404 + 0.005
ViT-smaal-ppttch 16
0.982 + 0.000 0..25 + 0.005
0.985 + 0.000 0.494 0.004
ViT-base-patch322
0.971 + 0.000 0.789 + 0.018
0.982 + 0.000 0.478 + 0.013
ViT-base-patch16
0.985 + 0.000 0.832 + 0.001
0.992 +: 0.000 /.500 0.530 + 0.023
Models
Office-Home (A)
Office-Home (C)
ViT-small-patth32
0.856 0.001 0.6618 = 0.001
0.867 + 0.000 0.506 = 0.001
1iT-sh
0.888 + 0.001 0..72 = 0.001
0.907 + 0.001 / 00.584 0.003
ViT- patch322
0.895 + 0.001 0..558 = 0.004
0.909 + 0.001 0.596 = 0.001
ViT-hase-patch11
0.907 + 0.000 0.796 + 0.001
0.927 + 0.001 /0.647 + 0.001
Models
VLCS (L)
VLCS (S)
ViT-smmallpaathh22
0.897 + 0.001 0.628 0.001
0.883 + 0.003 / 0.721 = 0.004
ViT-small-patch16
0.918 + 0.001 0.664 - 0.015
0.894 +: 0.001 0..759 0.004
ViT-bass-patch322
0.910 + 0.001 0.631 + 0.005
0.892 + 0.002/ 0.733 = 0.001
DaT-base-patch16
0.928 + 0.001 0.659 = 0.003
0.904 + 0..00000.7755
0.000
Table 8: Comparing models trained from scratch with different model sizes. We evaluate both in-distribution accuracy and out-of-distribution. (left: ID accuracy, right: OOD accuracy).
Models
PACS (C)
PACS (S)
ResNet50
0.812 + 0.003 / 0.470 + 0.005
0.817 + 0.005 0.294 = 0.004
ResNext50-32x4d
0.816 0.003 0.497 0.004
0.785 0.001 0.19 0.011
ResNext101-32x8d
0.681 + 0.004 0.505 + 0.010
0.770 + 0.002 / 0.287 + 0.007
Models
Office-Home (A)
Office-Home (S)
ResNer50
0.643 + 0.002 0.201 + 0.001
0.557 0.004 0.226 0.004
ResNexx55-32244
0.656 + 0.000 0.225 + 0.001
0.560 + 0.002 0.233 = 0.001
ResNext101-32x8d
0.654 + 0.002 0.216 + 0.001
0.575 0.001
0.249 0.001
Models
VLCS (L)
VLCS (S)
ResNet50
0.758 + 0.003 0.570 + 0.012
0.747 + 0.001 0.500 = 0.000
ResNext55-32x4d
0.757 +: 0.000 0.567 +: 0.013
0.751 0.001 /.5113 0.005
ResNext101-32x8d
0.760 + 0.002 0.567 +: 0.004
0.739 + 0.003 0.502 + 0.002
Table 9: In-discribution accuracy evaluation of four techniques (label smoothing, AutoAugment, PatchGaussian, and SAM) for OOD generalization. We use the same pre-trained model (SWSL- ResNext101-32x4d) across all settings. The number inside the parentheses after the method name represents the value of the technique-specific hyperparameter, e.g., PatchGaussian (1.0) corresponds
employing PatchGaussian (Lopes et al., 2019) with 1.0.
Method
PACS (C)
Office (C)
VLCS (L)
Terra (L46)
ERM (in Table 1)
99.1
92.1
92.4
95.9
Label Smoothing (0.1)
99.0
92.4
92.9
95.8
Label Smoothing (0.2)
99.0
91.8
93.1
95.8
Augonnt
98.4
90.6
91.9
93.8
PatchGaussian (1.0)
99.1
92.9
92.6
95.8
PatchGaussian (0.5)
99.0
92.3
92,6
95,6
SAM (0.02)
99.2
92.9
93.3
96.1
SAM (0.05)
99.0
93.2
93.0
95.1
22

---

Under review as a conference paper at ICLR 2022
Table 10: Out-of-discribution accuracy evaluation of four techniques (label smoothing, AutoAugment, PatchGaussian, and SAM) for OOD generalization. We use the same pre-trained model (SWSL- ResNex(101-32x4c) across all settings. The number inside the parentheses after the method name represents the value of the technique-specific hyperparameter, e.g., PatchGaussian (1.0) corresponds employing PatchGaussian (Lopes al., 2019) with 1.0. We highlight the best two OOD
accuracies for each dataset with bold text.
Method
PACS (S)
Office (A)
VLCS (S)
Terra (L38)
ERM (in Table 1)
91.3
76,4
77.0
47.5
Label Smoothing (0.1)
91.7
78.0
78.0
48.3
Label Smoothing (0.2)
91.7
78.6
78.6
50.3
AutoAugment
91.3
78.1
77.1
46.9
PatchGaussian (1.0)
90.3
66.4
71.6
12.4
PatchGaussian (0.5)
90.0
73.5
75.9
6.2
SAM (0.02)
89.7
80.3
79.2
42,0
SAM (0.05)
90.7
80.4
79.7
43,9
Table 11: In-discribution accuracy evaluation of four techniques (label smoothing, AutoAugment, PatchGaussian, and SAM) for OOD generalization. We the same pre-trained model (SWSL- ResNext101-322e4 across all settings. The number inside the parentheses after the method name represents the value of the technique-speciff hyperparameter, e.g., PatchGaussian (1.0) corresponds
to employing PatchGaussian (Lopes et al., 2019) with 1.0.
Method
PACS (S)
Office (A)
VLCS (S)
Terra (L38)
ERM (in Table 1)
99.6
89.7
90.1
94.6
Label Smoothing (0.1)
99.5
89.8
90.6
94.8
Label Smoothing (0.2)
99.5
89.8
90.5
94.5
AutoAugment
99.6
88.6
89.9
92.2
PatchGaussian (1.0)
99.6
90.1
90.1
94.0
PatchGaussian (0.5)
99.6
90.2
90.3
94.1
SAM (0.02)
99.2
91.0
90.9
94.5
SAM (0.05)
99.7
91.2
90.9
93.3
23

---

Under review as a conference paper at ICLR 2022
PACS Test domatec
Offcelionr Tes domalic
deetdrrahs
Tamercogrte True demal
0.00
0.5
000 Accoeee
0.35
335
0:475
Ind
0.48
000
1-3 1385
2.21
0-430
Aedlarbution wattrrbbbonnnnnnnn Accuracy
In-datr on Accurssy
Ibstit-bbttion Acourcc
Indisinbut on Accuracy
(a) PACS (C).
(b) Oofe (C).
(c) VLCS (S).
(d) Terralncognita (L38).
ACCURCY
Baidi
000
0.60
InGItIazion acursy
1nd araddmaccaarg
10-4
(c) PACS (C).
(f) OfficeHome (C).
VLCC (S).
(h) Terralncognita (1.38).
PRCS, Semm11
Oracstine, Tex: derrancc
VICL Test daraeci
Terchhoggeii To
CCO CECLLTN
1-23300 932
6.42
33
03
315
333
030
031
3.5g
6.33
0.30
033
6.54
033
035
PACS (S).
OfficeHome (A).
(k) VLCS (L).
TerraIncognita (L46).
ADC%, Text denee
Officdiene Test derolred
VSC% Text demont
Terosocayss Text tes
CO.
0.
64
Tudne
000 pusttry
043 acoiticy
Learning rite
Learting rite
Learting oete
Learting rore
(m) PACS (S).
(n) OfficeHome (A).
(0) VLCS (L).
(p) TerraIncognita (L46).
Figure 18: Evaluating models SSWSL-ResNext5--22400 fine-tuned by different learning rates (with stage-wise learning rate decay) on in-distribution and ofitt--distiitttionn data. (Top row) Scatter plot of in-distribution accuracy (X (X-axis) and out-of distriiibtt accuracy (Y-axis). (Bottom row) Compare in-distribution accuracy with out-of-distribution accuracy w.r.t. learning rate (X (X-xxi)). The green dashed line corresponds to the baseline OOD accuracy, and the blue dash-dot line represents the selected model (by selecting the model with best in-distribution accuracy).
24

---

Under review as a conference paper at ICLR 2022
PACS, Test domainiC
PACS, Test domain.5
OfficeHome Test comaincA
1.0
Random INt
Random Ini
0.8
Random tnit
000 Acturati
0.8
SWSL Pre-tran
Accurary
0.8
SWIL Pre-cran
0OD Accuracy
SWIL Pre-raan
6.6
0.6
0.6
0.4
0.4
0.2
0.2
0.6
0.8
1.0
0.5
0-6
0.7
0.5
0.9
1.0
0.4
0.6
0.8
In-distribution Accuracy
In-distribution Accuracy
In-distribution Accuracy
(a) PACS (C).
(b) PACS (C).
(c) OfficeHome (A).
Office Test domaliuC
VLCS, Test domainiL
VUCS, Test domainis
Random Intt
0.70
Rando in Int:
0.8
Random Int
ODD Accurate
SWSL Pre-taan
Accurar
3WSL Predre
oap Accuracy
SWSL Pre-cran
0.00
0.6
0.50
0.4
0.4
0.6
0.8
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
in-distribution Accuracy
In-distribation Accuracy
In-distribution Accuracy
(d) OfficeHome (C).
(c) VLCS (L.)
(f) VLCS (S).
Terraincoonita,
Terraincognita,
fondom
Random
DGD Accuracy
0.4
00D
0.2
In-distribution Accuracy
In-distribttion Accuracy
(g) TerraIncognita (L.8)..
(h) TerraIncognita (L46).
Figure 19: A comparison of ResNext101-3224 -1-32x8d models pre-trained on (G-1B-Targeted (SWSL- ResNext101-32xx8l v.s. MesNext101-32xx6 without pre-training out-of-distribution accuracy and in-distribution accuracy. The orange squares represent ResNext101-322 the blue circles represent models pre-trained IG- IG-IB-Targeted We evaluate the models on both ID and OOD data every 100 iterations, and we visualize all the ID/OOD results in the above figures.
without pre-training and
25

---

Under review as a conference paper at ICLR 2022
OfficeHomme domainuA
0.70
0.6
0.44
00D Accurac)
Accuracy
0.05
0.5
0.42
0.60
0.55
0.50
0.850
In-distribution Accuracy
In-distribution Accuracy
In-distribution Accuracy
(a) PACS (C).
(b) PACS (C).
OfficeHome (A).
OfficeHHmme, Test domaincC
VLCS, Test domainiL
VLCS. Test domain:s
0.64
0.66
0.40
00D Acturacy
0.38
0.36
0.63
0.34
2/5 classes
samplss
samples
2/5 samples
0.70
0.72
0.76
0.76
0.84
0.85
0.86
0.83
0.8
0.85
In-distribution Accuracy
In-distribution Accuracy
Inviistribttion Accuracy
(d) OfficeHome (C).
(c) VLCS (L.).
(f) VLCS (S).
Terraincogrits. Test d3m
Terraincognita, Test domaini.66
0.350
0OD Accuracy
0.325
GOD
0.275
1/5 Canses
1/5 clanses
1B8 samp
0.250
1/B samp
0.20
0.35
0.85
In-distribution Accuracy
In-distribution Accuracy
(g) Terralncognita (L.38).
(h) TerraIncognita (L46).
Figure 20: A comparison of ResNet50 pre-trained on 1/8 classes of the ImageNet- v.s. ResNet50
pre-trained on 1/8 training samples of the ImageNet-1k on out-of-diitriibutio accuracy and in- distribution accuracy. The orange squares represent models pre-trained on 1/8 classes and the blue circles represent models pre-trained on 1/8 training samples.
26

---

Under review as a conference paper at ICLR 2022
PACS, Test domainiC
PACS, Test domain
OfficeHomm e, Test domaliiA
1.0
1.0
Arcurssy
Accuracy
0.8
Accuracy
0.6
0.4
accuracy
accuracy
accurscy scouracy
0.4
scoursty
sccuracy
10
10
10
10
10t4
10-t
10
10-h
10-
10
Distance to initial pre-trained weights
Distance to iritial pre-trained weights
Distance to initial pre-trained weights
(a) PACS (C).
(b) PACS (C).
(c) OfficeHome (A).
offic domairvC
domaini
VLCS, domaln:5
0.9
0.8
0.8
Accurocy
Accuracy
Accurocy
0.6
accurscy
0.7
accurscy
0.4
Baselin
Baseline
In inttiirbutionn accuracy
dstribuutinnn accuracy
0D0 accuracy
0.5
GD0 accuracy
10
10t3
10
10:3
10-4
10t3
10.3
10-4
10-2
10-2
10
Distance initial pre-trained weights
Distance to iritial pre-trained weights
Distance to initial pre-trained weights
(d) OfficeHome (C).
(e) VLCS (L).
(f) VLCS (S).
Teraincognta,, damainoL38
Terraincognita, Test domains
1.0
0.8
Saseline
Accuracy
Baseline
In distiiib but on accuracy 050 arturary
in distdir but on accurasy 000 arcuney
0.6
0.6
0.4
Distance
weights
pre-trained weights
(g) Terralncognita (L388..
(h) Terralncognita (L46).
Figure 21: Evaluating models (SWSL-ResNexx55--222cc fine-tuned by different learning rates on in-distribution and out-of-distribution data. X-axis represents the distance between the fine-tuned model weights and initial pre-trained model weights (i.e., [[@Finettneed 0nxil Each point in the scatter plot corresponds to one learning rate. The green dashed line corresponds to the baseline OOD accuracy. The orange squares represent the in-distribution accuracy results and the blue circles represent out-of-distribution accuracy results.
27

---

Under review as a conference paper at ICLR 2022
000 200
3
14:0
g
(a) PACS (C).
OfficeHomee (C).
(c) VLCS (S).
(d) Terralncognita (L38).
Tomascennts
1.3
Accurscy
Indertstinttrer sccevry
Inderriatin
Ioderiitti secuncy
Inderiiati
02
000 scureey
32
000 scurncy
28 18 15 2 3
10 20
10- ning ruñe
10 22
16
10:1
10
10
10
15-
22 28
Learning rate
Leariig rale
Leamiing rale
(e) PACS (C).
(I) OfficeHome (C).
VLCS (S).
(h) TerraIncognita (L38).
DECCAlITaa THIT SUP
YLCB THE BUrER
beterbutt Acrurry mey
Techtrbation Acourey ney
nncerbation Acouney
PACS (S).
OfficeHome (A).
(k) VLCS (L).
1) TerraIncognita (L46).
2WC5, Text damars
cificalimne Tan demalrk
VICS. Tee demani
Terrainesgri ta. damahs 145
Baadinn ne
000
000 actamey
60
0.0 accuite
000 ectumoy
Learning mate
Learning rate
Learning rate
Learting rate
(m) PACS (S).
(n) OfficeHomm (A).
(0) VLCS (L).
(p) TerraIncognita (L46).
Figure 22: Evaluating models (SWSL-ResNext50-52x4a fine-tuned by different learning rates (with 4, 10-4, 10-4,5 x
more learning rates, i.e., 7) (5 x 10 2, 2,1 10 2, 2,5x 10-3, 1.11.xx1 10
10-5,1 x 10-5, 5, x 10 x 10 5 x 10 7,1 x 10-7}) data. (Top row) Scatter plot of in-distribution accuracy (X-axis) and diitt--iistibuttin accuracy (Y-axis). (Bottom row) Compare in-distribution accuracy with out-of-distribution accuracy w.r.t. learning rate (X -aiis.. The green dashed line corresponds to the baseline OOD accuracy, and the blue dash-dot line represents the selected model (by selecting the model with best in-distribution accuracy). Here we visualize the results for all learning rates, and do not only show results for models that achieve > 95% training accuracy.
}) on in-distribution and out-of -uittf-disttrbuttion
28


Reviews:
[{"review_id": "m_syxD6G3u9", "reviewer": "Reviewer_5Ldh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the methodology side, the paper is thorough, well-executed, with (mostly) clear results and well-designed experiments. Unfortunately, I think it lacks a strong selling point as most of the proposed insights are not novel enough.\n\nFor example, it is well-known that fine-tuning a pretrained model improves performance on downstream tasks, especially when those downstream tasks differ from upstream ones. See, for example, the work of Li et al. (2020) for an in-depth study on pretrained vision models, and Dodge et al. (2020) on language models. Similarly, the conclusion that larger models and datasets improve transfer is also well-studied — see the literature around \"neural scaling laws\", e.g., Kaplan et al. (2020) or Zhai et al. (2021) and references therein.\n\nMaybe the most interesting contribution is the insight that smaller learning rates improve OOD accuracy even thought they perform on par with larger learning rates in ID data. But here the results are muddied, since this seems to be true for some settings but not all (p. 5, 1st paragraph). Could the authors expand on why ImageNet-pretrained models benefit from larger learning rates but not other models? What is their intuition for when practitioners should choose larger or smaller learning rates for OOD fine-tuning?\n\nReferences:\n\n- Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, Stefano Soatto, \"Rethinking the Hyperparameters for Fine-tuning\", 2020.\n- Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, Noah Smith, \"Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping\", 2020.\n- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, \"Scaling Laws for Neural Language Models\", 2020.\n- Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, Lucas Beyer, \"Scaling Vision Transformers\", 2021.", "summary_of_the_paper": "This paper empirically studies simple techniques to improve out-of-domain (OOD) generalization. First, they show that fine-tuning a pretrained model significantly improves OOD accuracy over the non-fine-tuned baseline. Then, they show that fine-tuning with smaller learning rates outperforms larger ones on OOD data, even though both get approximately identical in-domain (ID) accuracy. Finally, the conclude that larger models, larger datasets, or (preferably) both improve OOD generalization.", "summary_of_the_review": "Although the experimental study of the paper is well executed, it provides few novel insights.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "l_i3bpY1iN", "reviewer": "Reviewer_mKe9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will preface by saying that I am not intimately familiar with the computer-vision literature and since, despite its unnecessarily general title and abstract, this paper is very computer-vision specific, I may be overlooking some finer details.\n\nThat being said, I think that the paper places itself nicely within the existing literature by casting the focus on OOD generalization specifically. While the conclusions (that larger models and more diverse datasets are better) are not unexpected, such detailed empirical studies are worthwhile for practitioners and researchers alike. However I think the paper falls short by not going one step further and providing additional insight to try and explain some of its (many) experimental findings. This makes it harder to tell whether and how these results would transfer to other architectures, modalities or datasets considered here, which ultimately limits the impact of the paper. I think the paper is in a good shape, but it would need additional ablations and analysis to warrant acceptance at ICLR.\n\nStrengths:\n- While the overall conclusion that larger models pre-trained on more (and more diverse) are better is not surprising, I think it is useful to confirm this finding finding with respect to OOD generalization specifically.\n- Detailed analysis and experimental results, which should be useful for practitioners in the field\n- Experimental results are overall well presented\n\nWeaknesses:\n- Unnecessarily general title and abstract: the paper is very much vision specific and this should be made clear from the get go. As it stands I found the opening of the paper rather misleading. There is some work outside of vision which looked at this particular problem and bears mentioning, for instance in NLP Hendricks et al. \"Pretrained Transformers Improve Out-of-Distribution Robustness\" https://arxiv.org/pdf/2004.06100.pdf\n- While there are a lot of experimental results, I think the paper suffers from not going one step further and investigating the underlying causes behind a lot of the results. As it stands, the reader is left with rather vague and somewhat expected conclusions: \"larger and more diverse datasets are better\" and \"smaller learning rates are better\". \n   * What specifically in the larger datasets helps? Is it really diversity or size? For example, if IG-1B-Targeted is downsized to the size of Imagenet, do the improvements observed persist?\n   * What about the smaller learning rates make them more suitable? I have a suspicion that it has to do with the weights deviating less from their pre-trained starting point during fine-tuning. It would be interested to look at the interaction between \"catastrophic forgetting\" of the pre-training task and OOD generalization.", "summary_of_the_paper": "Within the now common pre-train/fine-tune training paradigm, this paper provide an empirical examination of the influence of pre-training on out-of-distribution accuracy *after* fine-tuning, specifically for computer vision. The authors showcase evidence suggesting that the choice of the pre-training dataset, the size of the model and the fine-tuning learning rate directly influence downstream OOD accuracy throughout a series of tasks. In addition, they provide some analysis of the effect that various training techniques (eg. label smoothing, augmentation etc..) have on OOD accuracy when they are used at fine-tuning time.", "summary_of_the_review": "I think that the paper places itself nicely within the existing literature by casting the focus on OOD generalization specifically. While the conclusions (that larger models and more diverse datasets are better) are not unexpected, such detailed empirical studies are worthwhile for practitioners and researchers alike. However I think the paper falls short by not going one step further and providing additional insight to try and explain some of its (many) experimental findings. This makes it harder to tell whether and how these results would transfer to other architectures, modalities or datasets considered here, which ultimately limits the impact of the paper. I think the paper is in a good shape, but it would need additional ablations and analysis to warrant acceptance at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WQt0cxkNl8W", "reviewer": "Reviewer_jtZq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides extensive experimental results on four datasets to show how pre-trained data, pre-training model, learning rate and some ID tricks affect the OOD accuracy. Overall, the paper is easy to read and the organization is also good. The considered topic, OOD generalization, is interesting and important to the deep learning community. However, I have found some of the results unclear or incorrect. \n\n1. The claim \"models trained with smaller learning rates achieve much better OOD generalization\" may be misleading/incorrect. It seems to me that the authors want to claim that the smaller lr, the better performance. This result is not well-supported. For example, Figure 2(h) shows that the performance with lr$=10^{-4}$ is worse than that with lr$=10^{-3}$; Figure 2 (e) and (f) show that the result for lr$=10^{-3}$ and the result for lr$=10^{-4}$ are almost same. Although the authors provide experiments with a range of learning rates between 0.05 and 0.0001, more experiments with smaller learning rates such as $10^{-5}$, $10^{-6}$ and $10^{-7}$ are needed. \n\n2. The claim \"methods listed in Table 2 does not significantly improve the OOD generalization across four datasets\" is incorrect. We can see from Table 2 that both Label Smoothing (0.2) and SAM (0.02) are better than ERM (in Table 1) on 3 (of 4) datasets, while PatchGaussian (0.5) is better on 2 (of 4) datasets. \n\nThus, if above two issues cannot be addressed, then the main claims about learning rate and ID techniques are not correct/well-supported, which makes the contribution of this paper limited. \n\nThe novelty of using pre-training model to improve OOD generalization is not significant and this idea is not new. For example, see the ICML 2021 paper \"Improved OOD Generalization via Adversarial Training and Pre-training\". Although the authors show that the larger models and larger datasets need to be used simultaneously, but the contribution is incremental since the idea is too straightforward given the ICML paper. Furthermore, the intuition of using larger models and larger datasets simultaneously is unclear. It would be better if the authors could add some sentences to describe the intuition. \n\nThe contribution about ID tricks in this paper is mazy. It seems the paper's topic is about pre-training and fine-tuning on OOD generalization, which is not directly related to ID tricks. I am confused about adding ID tricks in this paper. \n\nTwo related papers are not discussed:\n\na. In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness, ICLR 2021\n\nb. Improved OOD Generalization via Adversarial Training and Pre-training, ICML 2021", "summary_of_the_paper": "This paper empirically studies pre-trained model on OOD generalization. Specifically, the authors consider four datasets in this paper: PACS, Office-Home, VLCS and TerraIncognita. Based on the experimental results, the authors concludes three main observations . First,  larger models and larger datasets need to be simultaneously leveraged to improve OOD performance. Second, using smaller learning rates during fine-tuning is critical to achieving good results. Third, the strategies that improve in-distribution accuracy may lead to poor OOD performance. ", "summary_of_the_review": "1. The topic is important and interesting.\n\n2. Two main contributions about learning rate and ID tricks may be incorrect or not well-supported.\n\n3. The idea of using pre-training model is not new and the novelty of this paper is limited.\n\n4. Unnecessary ID tricks and missed related work.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E9eGzRxwh6Z", "reviewer": "Reviewer_1Vts", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this paper provides a good set of empirical studies on an important problem: what contributes to the better OOD generalization performance of pre-trained models (and relatedly, how are ID accuracies related to these OOD accuracies). \n\nStrengths:\n1. Extensive experiments. I especially like that the authors tried to tackle at this problem from multiple angles, such as LR, model size, model type, augmentations, etc.\n2. The paper is overall well-written and easy to understand.\n3. The underlying problem studied here is very important and while some of the insights have been brought up by prior works, this paper provides an in-depth analysis of how pre-trained models behave (under the influence of all these different factors) when there's a distribution shift. I have no doubt that the conclusion of this paper will be very useful for future work.\n4. Good results and on large scales. The authors are able to take advantage of all these findings to produce good OOD generalization outcomes.\n\nWeaknesses/problems:\n1. The usefulness of pre-training was already studied before (e.g., [1,2]), though in somewhat different directions. This paper puts more emphasis on training mechanisms, which provide useful, interesting, but still incremental insights.\n2. The conclusions are still empirical, domain-specific, and have some important questions (in my mind) unanswered. For example, since learning rate was a major factor studied in the paper, it makes sense to consider the effect of the learning rate annealing schedule as well, which is a technique very commonly used in training image classification models. As another example, the paper mentions multiple times how a **\"more diverse\"** dataset for pretraining would help (e.g., IG-1B-Targeted vs. ImageNet-1k). However, I feel that this is a more relative and subjective measure--- e.g., how do we know a'priori whether a dataset is \"diverse enough\"? \n3. I like the discussion on ID and OOD accuracy, so a related question: let's say we train 1) a model A on dataset C to reach 70% accuracy on C's test set; and 2) a model B (with the same **architecture** as A) on a larger pretraining dataset D first, and then fine-tune a bit (just a bit, not thoroughly; for example, you can early stop) on C to make it also reach 70% accuracy on the downstream dataset C's test set. Does the conclusion of this paper imply model B will always have better OOD performance than model A? This is different from the results presented in Table 8 because the models there have substantially worse ID accuracy as well.\n\n\n\n[1] http://proceedings.mlr.press/v139/yi21a/yi21a.pdf\n[2] https://aclanthology.org/2020.acl-main.244.pdf", "summary_of_the_paper": "This paper provides a thorough and relatively in-depth analysis of how pre-trained models affect OOD generalization performance. Specifically, the authors dissect this issue and perform extensive ablations on learning rates, model architecture, dataset size, model size, etc. By studying large-scale pretraining on vision tasks, the paper claims that 1) pre-trained models typically have better OOD generalization performance (which prior works have somewhat discussed as well); 2) small learning rates during the fine-tuning phase tend to create more robust models; and 3) some of the previous understandings (e.g., of how ID and OOD accuracies are related; or of the effect of training data size) might need to be revisited.", "summary_of_the_review": "I recommend (weak) acceptance of this paper because 1) the conclusions drawn in this paper, while somewhat incremental, are still very useful and would be a great addition to this line of work; and 2) the experiments are thorough and realistic. I don't identify any obvious flaw in the methodology or analysis.\n\n---------------------\n\nPost-rebuttal: See my comment below. I'm updating the score to 7 (which doesn't exist in the ICLR review form, but just to show that I'm advocating more for acceptance).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presented an empirical study of pre-trained models on the Out-of-distribution Generalization problem. 
Authors evaluated various factors (such as model sizes, datasets, learning rate, etc) and claim some major findings:  1) larger models have better OOD generalization, and combining both larger models and larger datasets is critical; 2) smaller learning rate during fine-tuning is critical; 3) strategies improving in-distribution accuracy may hurt OOD. Overall, this paper is a well-written empirical study with some useful insights, but the new findings from the empirical studies are generally not surprising and the overall contribution is not significant enough for acceptance.
Decision: Reject