Empirical Study on Optimizer Selection for Out-of-Distribution Generalization
Authors: Hiroki Naganuma
Kartik Ahuja
Shiro Takagi
Tetsuya Motokawa
Rio Yokota
Kohta Ishikawa
Ikuro Sato
Ioannis Mitliagkas
Venue: Accepted by TMLR ()

Abstract: Modern deep learning systems do not generalize well when the test data distribution is slightly different to the training data distribution. While much promising work has been accomplished to address this fragility, a systematic study of the role of optimizers and their out-of-distribution generalization performance has not been undertaken. In this study, we examine the performance of popular first-order optimizers for different classes of distributional shift under empirical risk minimization and invariant risk minimization. We address this question for image and text classification using DomainBed, WILDS, and Backgrounds Challenge as testbeds for studying different types of shifts---namely correlation and diversity shift. We search over a wide range of hyperparameters and examine classification accuracy (in-distribution and out-of-distribution) for over 20,000 models. We arrive at the following findings, which we expect to be helpful for practitioners: i) adaptive optimizers (e.g., Adam) perform worse than non-adaptive optimizers (e.g., SGD, momentum SGD) on out-of-distribution performance. In particular, even though there is no significant difference in in-distribution performance, we show a measurable difference in out-of-distribution performance. ii) in-distribution performance and out-of-distribution performance exhibit three types of behavior depending on the dataset---linear returns, increasing returns, and diminishing returns. For example, in the training of natural language data using Adam, fine-tuning the performance of in-distribution performance does not significantly contribute to the out-of-distribution generalization performance.

Published in Transactions on Machine Learning Research (06/2023)
Empirical Study on Optimizer Selection for Out-of-Distriuttion Generalization
Hiroki Naganuma?.2 Kartik Ahuja!,2, Shiro Takagi', Tetsuya Motokawa', { hanganumaahiiokk ahajj, @umäla.quebec, ftakagi4646, moto.t.03 t...tt..3.dd.... (...di.aa.aaallll.......mm...... com,
Mila Quebec Al Institute, Uniiersitté de Montréal, Independent Researcher, 4 University of Tsukuba, Rio Yokota Kohta Ishikawa", Ikuro Sato* Ioannis Mitliagkas 1,2,7
Sasyokktt@gsic...o fitech....cc........... kishikwwo@d-itlan wit.ddd..bb.......... wit.ola..tttchac....... ionnnis @emila. @eilla.qubee Tokyo Institute of Technology, Denso IT Laboratory Inc., Canada CIFAR AT Chair
Reviewed on OpenReview: https: // openreview. fort? id= iPF
Abstract
Modern deep learning systems do not generalize well when the test data distribution is slightly different to the training data distribution. While much promising work has been accomplished to address this fragility, systematic study of the role of optimizers and their out-of-distribution generalization performance has not been undertaken. In this study, we examine the performance of popular first-order optimizers for different classes of distributional shift under empirical risk minimization and invariant risk minimization. We address this question for image and text classification using DomainBed, WILDS, and Backgrounds Challenge as testbeds for studying different types of shifts namely correlation and diversity shift. We search over a wide range of hyperparameters and examine classification accuracy (in-distribution and out-of-distribution) for over 20,000 models. We arrive at the following findings, which we expect to be helpful for practitioners: i) adaptive optimizers (c.g., Adam) perform worse than non-adaptive optimizers (e.g., SGD, momentum SGD) on out-ol- distribution performance. In particular, even though there is no significant difference in in- distribution performance, we show a measurable difference in cuttt--distribution performance. ii) in-distribution performance and behavior depending on the dataset-lines returns, increasing returns, and diminishing returns. For example, in the training of natural language data using Adam, fine- tuninn the performance of in-distribution performance does not significantly contribute to the out-of-distribution generalization performance.
performance exhibit three types of
1 Introduction
The choice of numerical optimization method can make a big difference when it comes to successfully training deep neural networks. In particular, the choice of optimizer influences training speed, stability, and generalization performance. Several studies have compared variety of optimizers and investigated their influence on trainability and generalization (Wilson et al., 2017; Schneider et al., 2019; Choi al., 2019). Some concluded that non-adaplive optimizers yield better generalization (Wilson al., 2017; Balles & Hennig, 2018), while others countered that optimizer selection does not affect generalization performance (Schneider et al., 2019; Schmidt et al., 2021).
The conflicting nature of past reports can be explained by disparities in the methodology used for hyperpa- rameter search. For Adam, in particular, hyperparameter e controls the degree of adaptation. Low values, which are used by default, lead to highly adaptive method. Umusually high values lead to less adaptation. In the limit of large e, Adam turns into a non-adaptive momentum method. In other words, when arbitrary tuning is allowed, methods like Adam can be thought of as superset of gradient descent with momentum.

---

Published in Transactions on Machine Learning Research (06/2023)
The authors in Choi et al. (2019) take this approach, and also consider the less adaptive and non-adaptive regimes of Adam. Not surprisingly, they find that in this full generality Adam never performs worse than gradient descent with momentum in terms of in-distribution generalization, and in some cases, it can perform better.
Although these studies have made substantial progress to improve our understanding of optimizer char- acteristics, they are based on a common, foundational assumption in learning: training and test data are drawn from the same distribution. In applications, however, it is often the case that test data obey a distribution different from the one for training data. This distributional shift violates the typical assumption of independent and identically distributed (i.i.d.) data for learning (Nagarajan et al., 2021). Comparing the generalization performance of different optimizers under this distributional shift, known as out-of-distribution (OOD) generalization (Shen et al., 2021), is of great interest in theory and practice.
In our large suite of experiments, we focus on this OOD generalization question for Natural Language Processing (NLP) and image classification tasks. We train deep neural networks under the principle of Empirical Risk Minimization (ERM) Invariant Risk Minimization (IRM) (Arjovsky et al., 2019) using variety of optimizers. Because our objective is to investigate the impact of commonly used optimizers, we target five of the most popular optimizers that have been used and studied in recent years (Schmidt et al., 2021): stochastic gradient descent. (SGD), Momentum SGD (Polyak, 1964), and Nesterov accelerated gradient (NAG, also known as Nesterov's 's momentum) (Nesterov, 2003) in the family of non-adaptive methods as well as RMSProp (Tieleman Hinton, 2012) and Adam (Kingma & Ba, 2015) in the family of adaptive optimizers. We evaluate the OOD generalization performance of these optimizers on 10 different benchmarks: DomainBed (which includes seven image datasets) (Gulrajani & Lopez-Paz, 2021), the Backgrounds Challenge dataset. (Xiao et al., 2021), and CivilComments-WLL (Koh et al., 2021).
As discussed above, methods like RMSProp and Adam can be tuned not to be adaptive, in which case they would subsume methods like gradient descent with momentum, and we would trivially get that the more general method can never underperform. Instead, we focus on the more nuanced question of adaptive vs non-adaplive methods: we tune RMSProp and Adam using a range of values for the hyperparameter 6, which is strictly wider than ranges used in practice but still keeps the optimizers in their adaptive regime, In this context, we conduct exhaustive hyperparameter scarch to select configurations that give good in-distribution validation accuracy for each optimizer. We then test the selected models on the above OOD test sets. Importantly, we demonstrate that our experiments explore more hyperparameter configurations than many existing benchmarks, as betrayed by the fact that we find better-performing models on said benchmarks.
In summary, our contributions are as follows:
We design and perform comparison of the effect of optimizers on OOD generalization on a number of OOD benchmarks. To the best of our knowledge, we are the first to consider such a widc varicty and scale of datasets. Also, we conduct an empirical study using a wide range of hyperparameter configurations, examining over 20,000 models, evaluating their performance, and measuring the performance gap when moving from the in-distribution test set to the OOD test set.
We demonstrate on a large number of image classification and NLP tasks that different optimizer choices lead to differences in OOD generalization performance. Furthermore, we show that adaptive optimizers yield more in-distribution overfitting and degrade OOD performance more than non- adaptive optimizers.
We show that the observed correlation behaviors between in-distribution performance and OOD performance can be categorized into typical patterns: linear return, diminishing return, and increasing return This categorization should help practitioners better understand and select optimizers.
The following evidence supports the claim that non-adaptive methods outperform adaptive methods in OOD settings: i) There no significant difference in the in-distribution generalization performance between adaptive
These show how much performance can be expected in the out-of-distribution if increase the in-distribution performance. These terms are explained in detail in Section 4.3.
2

---

Published in Transactions on Machine Learning Research (06/2023)
and non-adaptive optimizers, ii) Adaptive optimizers perform worse in 10 out of 12 tasks regarding best out-of-distributioo generalization performance, iii) We match models trained by adaptive optimizers to models trained by non-adaptive optimizers that yield the same in-distribution performance, Using this matching scheme for our comparison, models trained by non-adaptive methods achieve better OOD performance on 11 out of 12 tasks. These results are based on comprehensive hyperparameter search and validated through soundness checks in the Appendix G.4.
Given the similarity of the results between ERM and IRM training principles, we have opted to focus our analysis on the former. Therefore, in the main section of this paper, primarily report the results of ERM. Our observations that adaptive optimizers perform worse than non-adaptive methods in OOD performance align with theoretical results (Zou et al., 2022) previously reported in the literature, highlighting the drawbacks of adaptive optimizers such as Adam under the i.dd. assumption and their tendency to fit noise in the data.
The paper is structured as follows. In Section 2, we discuss optimizer selection under the i.i.d. assumption and outline the problem of OOD generalization in the presence of distributional shifts, which can be encountered in real-world problems. In Section 3, outline the optimizers and the OOD datasets that are the subject of this study, as well as our model selection method. In Section 4, we present the experimental protocol and the experimental results of 10 different datasets and 5 optimizers in each experimental setting.
2
Related Work
2.1 Optimizer Selection
Understanding the characteristics the many optimizers proposed for deep neural network training (Schmidt et al., 2021) is of great importance to the machine learning research community. In terms of convergence, preconditioned optimizers, including Adam, are known to be superior non-adaptive optimizers (Kingma & Ba, 2015; Liu et al., 2020; Amari, 1998).
While preconditioned optimization methods seem to be better than non-preconditooee ones in terms of convergence, Zhang et al. (2019) argued that there is a tradc-off between generalization performance and convergence rate (Zhang et al., 2019). Wadia et al. (2020) also reported that preconditioned methods, especially second-order methods, do not provide great generalization performance either empirically or theoretically. With a simple theoretical and empirical analysis, Wilson et al. (2017) showed that adaptive optimizers are worse at generalization than simple SGD. Balles & Hennig (2018) also reported that Adam generalizes worse than Momentum SGD.
Contrary to these studies, Schneider et al. (2019) found that single optimizer is the "best" in general. Similarly, Schmidt et al. (2021) claimed that optimizer performance varies from task to task. Choi et al. (2019) have come to a different conclusion from all the studies cited above. As described in Section 1, Choi et al. (2019) tuned the hyperparameter e of adaptive methods that control the degree of adaptivity, which leads to non-adaptive methods. As a result, they found that well-tuned adaptive optimizers never underperform simple gradient methods.
These studies provided insights into how optimizer selection influences generalization. However, they focused the classical supervised learning setling, in which the test distribution is assumed to be the same as the training distribution. Our research differs from theirs in that we investigate optimizer selection's influence on OOD generalization.
2.2 Out-of- D--tsiiiibttto Generalization
Taming the distributional shift a big challenge in machine learning research (Sugiyama & Kawanabe, 2012; Ben-David et al., 2010; Pan & Yang, 2009; Szegody et al., 2014; Arjovsky et al., 2019). Geirhos al. (2020) argued that many modern deep neural network models sometimes learn shorteut features instead of intended features and overfit to a specific dataset.
3

---

Published in Transactions on Machine Learning Research (06/2023)
Some studies have focused generalization with adversarial noise to evaluate the impact of optimizer selection on OOD generalization. For instance, the theoretical and empirical analysis by Khoury (2019)
showed that SGD is more robust against adversarial noise than adaptive optimizers. Wang et al. (2019)
argued that adversarial examples to some mothods are not necessarily adversarial to others. Abdelzad et al.
(2020) found that the best optimizers for OOD detection vary by experimental selting. Metz et al. (2020) reported that a learned optimizer somehow unexpectedly outperformed human-designed optimizer in terms of the OOD generalization.
These studies provide valuable insights optimizer selection for the OOD problem. However, we emphasize that the previous work discussed above explores hyperparameters in a limited range. Khoury (2019) conducted the most exhaustive hyperparameter search but tuned only the learning rate for adaptive optimizers. As we briefly explain in Section 1, an exhaustive hyperparameter search is crucial for the empirical investigation of an optimizer's effect. Thus, we explored more hyperparameters than previous studies, including scarching for over 20,000 models.
Here we emphasize that only shifts such adversarial noise have been studied in previous studies of optimization selection for OOD generalization. Thus, including image classification and NLP tasks where the distributional shift is significant, covariate shift, correlation shift, subpopulation shift, and background shift, not only domain generalization. The set of datasets we explored is the most exhaustive for evaluating the optimizer's 's role in OOD generalization, as far as we know.
much more diverse set of real OOD datasets,
Finally, we mention to some relevant works Kumar et al. (2022a); Wortsman et al. (2022); Chen et al. (2023); Kumar et al. (2022b). Kumar et al. (2022a); Wortsman et al. (2022); Kumar et al. (2022b) have studied the intricate balance between ID and OOD performance when fine-tuning pre-trained models. Chen et al. (2023) theoretically showed that optimizing the ERM with the relaxed OOD penalty is not likely to have a good performance and proposed a better practical solution.
3 Preliminaries
3.1
Optimizers Subjected to Our Analysis
Similar to previous studies (Wilson et al., 2017; Schneider et al., 2019; Choi et al., 2019), we compare two types of optimizers. The first one is non-adaptive optimizers. The update equation at iteration t of model parameter O₂ is as follows:
U. +- 7-11 + T) ((0,11)) 0 V
(1)
where 7/e is the learning rate, {(0) the loss, and V o-_, is the stochastic gradient, the particular case of stochastic gradient descent, 7 = 0. Optimizers with momentum terms such as Momentum SGD (Polyak, 1964), and Nesterov momentum (Nesterov, 2003) are also classified as non-adaptive optimizers, and is the parameter for controlling the momentum term. For Nesterov momentum, E(0,-1) should be replaced - (10,-1770,-0
The second type of optimizer is adaptive methods. Adam and RMSprop are adaptive optimizers and they can be written the form of the generic adaptive optimization method. The generic adaptive optimization method can be written as Algorithm 1. This is based on what (Liu et al., 2020) and (Roddi et al., 2018) propose.
Our selection of optimizers aligns with prior research optimizer comparison (Choi et al., 2019) and recent studies on out-of-distribution (OOD) tasks, which have predominantly focused on Adam rather than alternatives such as Adamw. Therefore, we concentrate on the five most commonly employed optimizers in practice (Schmidt et al., 2021). Given their widespread usage, we chose to focus on studying Adam's performance under the adaptive regime, as this would provide more pertinent findings for present-day practices. Experimental results for AdamW (Loshchilov & Hutter, 2017) and SAM (Foret al., 2020), although not for all data sets, are also discussed in Appendix T.3.
4

---

Published in Transactions on Machine Learning Research (06/2023)
Algorithm 1 Generic adaptive optimization method setup.
Require: {no)z step size, fΦr, th di function to calculate momentum and adaptive rate, Oo: initial
parameter, {(0): objective function
1: for = 1-110 to T' do
2: gr - Vef((0r-1) (Calculate stochastic gradients w.r.t. objective at timestep t)
3: wi t @g 9t) (Calculate momentum)
4: 1 - Ve........ gr) (Calculate adaptive learning rate)
0. +- 0--1 - uard (Update parameters)
end for
3.2 Out-of-Distribution Generalization Datasets
We use the following datasets to evaluate the optimizer influence on OOD generalization: DomainBed (Gulra jani & Lopez-Paz, 2021), Backgrounds Challenge (Xiao et al., 2021), (Koh et al., 2021), and CivilComments- WILDS (Koh et al., 2021). These datasets include both artificially created and real-world data, and the applications include image classification and NLP tasks. Although we describe the details of these datasets in Appendix C, we sunmarize them below.
DomainBed consists of a set of benchmark datasets for domain general- ization, which includes PACS (Fang al., 2013), VLCS (Li et al., 2017), Offico-Home (Venkateswara et al., 2017), Terra Incognita (Beery et al., 2018) DomainNet (Peng al., 2019), Rotated MNIST (Ghifary et al., 2015), and Colored MNIST (Arjovsky et al., 2019). These datasets contain variety of distributional shifts. VLCS is a set of different image datasets, for example, images of birds from several datasets. Terra Incognita is a dataset consisting of images taken by cameras at different locations. The difference in the location of the camera corresponds to the difference in the domain. PACS, Office-Home, and DomainNet are image datasets whose style varies by domain. For example, an image of PACS in one domain is photography, while that in another domain is a sketch. Rotated MNIST an artificially generated dataset of domains that have been given different rotation angles.
Image Classification Datasets:
Colored MNIST is anomaly in the DomainBed dataset. P(Y remains the same for all datasets (covariate shift) except in Colored MNIST. This dataset is designed to make models fail by exploiting spurious correlations in training environments. In particular, the dataset is such that the model can only exploit the source of spurious correlation (color) and achieve a very high training accuracy without relying on the true invariant source of correlation (shape). In contrast, none of the other datasots have such strong spuriousness. The strength of the spurious correlation flips in the test domain and thus induces a negalive correlation between the validation and the test accuracy.
The Backgrounds Challenge dataset measures model's robustness against background shift (Xiao et al., 2021). A model is trained an image and evaluated on the same image with different background. If the model exploits the background features during training, it will be fooled during evaluation. Therefore, if the model strongly depends on the background, this dataset is a difficult OOD dataset, while if the model does not, this dataset is easy to model. To further strengthen our claim, we also performed experiments on CIFARIÓ-C and CIFAR10-P which can be casted image corruption and perturbation shift. The results are shown in Appendix I.1.
Natural Language Processing (NLP) Datasets: The CivilComments-WILn dataset is cast subpopulation shift problem. The shift problem tackles a binary classification problem that predicts the toxicity of comments on articles, and domain vectors are assigned according to whether the comment refers to one of eight demographic identities. In the subpopulation shift, the test distribution is a subpopulation of the training distribution.
The dataset has the characteristies of a hybrid shift of a subpopulation shift and domain shift. This dataset is cast the problem of estimating rating of 1-5 from the rating comments of each user. In this kind
5

---

Published in Transactions on Machine Learning Research (06/2023)
of dataset, each user corresponds to domain, and the goal is to produce high performance for all user comments.
3.3 Model Selection Method and Evaluation Metrics
We leverage two distinct model selection methodologies. Our first approach involves partitioning the data from the training domains into a training dataset and a validation dataset, subsequently selecting the model that demonstrates the highest average performance (accuracy) based the validation data from the training domain. This approach, referred to training-domain validation set model selection strategy, follows the rescarch methodology established by Guirajani & Lopez-Paz (2021). The second method employs an Oracle-based model selection strategy using lest domain data. The following shows the model selection methodology when using validation in the training domain.
In the training phase of DomainBed datascts, we do not access the data in the test domain but split data from the training domains into training dataset and validation dalaset. We choose the model with the highest average performance (accuracy) on the validation data in the training domain. As a metric for evaluation, we evaluated the gencralization performance in the test domain as the OOD accuracy.
The Backgrounds Challenge uses Imagenet-1k 1k as the training data set and selects models based on their accuracy on the validation data in the training domain. After that, we measure the in-distribution performance with IN9L (Xiao et al., 2021), which aggregates the test data into nine classes. As for the OOD performance, we measure the classification performance on the data where the background image of IN9L is replaced with the background image of other images.
In CivilComments- WILDS, we divide the data into training, validation, and test datasets and maximize worst-growe accuracy in the validation data (and by association, maximize the average accuracy over all domains). Then, we perform model selection and evaluate the OOD accuracy on the test data. We adopt the same hyperparameter selection strategy for datasets as that for CivilComments- WILLI. As a metric, we do not evaluate the worst-group performance but rather the 10th-percentile accuracy for the performance of each domain by following the standard federated learning literature (Caldas et al., 2018).
4 Experiments
4.1
Experimental Overview
Our study aims to elucidate the influence of optimizer selection under distributional shifts. To that end, we perform image classification and NLP tasks and evaluate the trained model accuracy on the benchmark datascts introduced Soction 3.2. By comparing the test accuracy for in-distribution samples with that for OOD samples, we can observe how the solution found by each optimizer is robust to the distributional shift. We investigate both ERM and IRMv1 (Arjovsky et al., 2019), problem set to solve IRM, to clarify the relationship between the problem formulation and optimizer selection in the OOD problem. For VREx (Krueger et al., 2021) and CORAL (Sun & Sacnko, 2016), small-scale experimental results are also provided in Appendix T.7.
We compare five optimizers for all datasets except for the Backgrounds Challenge dataset. For the Backgrounds Challenge dataset, we compare only Momentum SGD and Adam due to the computational cost. We describe the configurations of hyperparameters and protocol for the experiments in further detail in Appendix F. and Appendix D respectively. The remaining experimental settings of environment are explained in Appendix B.
Hyperparameter Tuning: The hyperparameters are tuned using Bayes optimization functionality of Boights&Biasess by evaluating in-distribution validation accuracy. Bayesian optimization sequentially explored the potential hyperparameter candidate points, and we evaluated all the trained models in the search process for comparison. As a confirmation of the soundness of our hyperparameter search, Appendix H.1 shows the histogram that the explored hyperparameters are drawn from the reasonably wide range, In addition, investigated the relationship between the number of trials to explore hyperparameters and the
2https://vands ai/site

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Comparison of the best OOD accuracy (%) ERM between five optimizers. We use Oracle (see definition at 3.1 in Gulrajani Lopez-Paz (2021)) as model selection method in this table. The model selection results in the training-domain validation set are shown in Table 13. Except for a small set of problems, non-adaptive optimizer outperforms adaptive optimizer in all but 10 of 12 tasks. As soundness check, we confirm that our Adam results outperform all existing benchmark results using Adam. Details are given in Appendix G.4.
Non-Adaptive Optimizer Momentum
Adaptive Optimizer
Model
OOD Dataset
SGD
Netsterov
RMSProp
Adam
ColoredMNIIT
34.01%
34.23%
40.56%
89.30%
73.92%
4-Layer CNN
RotatedMNIST
90.00%
95.41%
94.06%
96.27%
96.40%
VLCS PACS
99.43%
99.43%
99.29%
99.29%
99.29%
88.67%
89.55%
80.25%
88. 88%
89.30%
OfficeHomme
64.64%
65.01%
63. 82%
62. 22.91%
63.82%
ResNet50
TerraIncoguita
63.21 %
62.41%
62.85%
62.31%
61.35%
DomainNet
58.38%
61.91%
62.24%
55.74%
58.48%
BackgronndCChallennee
80.09%
77.90%
Amszon-WILDDS
52.00%
54.66%
54.66%
53.33%
52.00%
DistilBERT
WwilCouunettWILLDS
51.66%
37 69%
60.07%
45.30%
46.82%
ResNet-20
ColoredMNIST
33.50%
31.47%
VIT
PACS
91.80%
91.26%
best OOD performance and show results in Appendix G.2. The impact of initialization strategies during hyperparameter optimization is also confirmed in Appendix H.3. The data shown in the box plot (Figure 2, 3, and 4) are from the evaluation of several trained models.
The number of epochs (the steps budget) used in line with previous studies (Gulrajani & Lopez-Paz, 2021; Koh et al., 2021; Choi et al., 2019) to ensure the soundness our experimental design. Since we use the fixed epoch, it might seem to be unfair than when tuning the epoch as well for the optimizer that converges faster. Thus, we studied the effect of early stopping, which corresponds to the tuned epoch in Appendix G.4, and the result confirms that employing fixed epoch does not impair the fairnoss of our comparison experiments. We discuss the details in Appendix G.4.
Boxplot: We believe that sharing the whole distribution of tuning outcomes is important because; it gives an idca of how sensitive methods are to tuning and how much tuning effort required. We also share the raw data as scatter plots (Figure 15, 16, and 16) Appendix F.4.
4.2 Experimental Results
For training-domain validation set model selection, Figure shows the relationship between the in-distribution and OOD accuracy in the ERM setting. The x-axis of the plot is the in-distribution accuracy, and the y-axis is the OOD accuracy. To clarify the trend, the in-distribution accuracy corresponding to the x-axis is divided into ten bins, and the average performance of the OOD accuracy in each bin is shown on the y-axis. We compared Momentum SGD, the best non-adaptive optimizer, and Adam, the best adaptive optimizer. Our findings reveal that, in our field of study, where high in-distribution performance is achieved, Momentum SGD outperforms Adam on 11 out of 12 tasks (Table 13).
With respoct to the Oracle model selection strategy, best OOD performance, non-adaptive optimizers surpassed adaptive optimizer methods on 10 out of 12 tasks (Table 1). These results suggest that non- adaptive optimizers more advantageous than adaptive optimizers in OOD, despite their similar performance in the ID environment. For a more detailed explanation of the experimental results for each datasct, please refer to the following section.
Figure 2 shows a box plot of the difference between the average in-distribution and OOD accuracy for ERM. In the following discussion, we call this difference a gap for convenience. Due the limitation of the paper length, only the plots of PACS and Office- Home are shown here. All results, including IRM results, shown in Appendix F.2.1.
DomainBed:

---

Published in Transactions on Machine Learning Research (06/2023)
ColoredMNIST:ERM
RotatedMNISTEERM
VLCS:ERM
PACS: EAS.
Officeliome:ERM
1.0
2.0
1.0
out out-at-distin ait-attdisstbbbttion acc
mermerk.mm
0.8
at-of-jaribaatin 452
at-ofdiarrbaaoonn acc
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
04
0.4
0.2
0.2
0.2
0.2
0.0
0.0
1.0
0..00
0.25
0.50
0.75 acc
1.00
8.000
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
08.00
0.25
0.50
0.75
1.00
stribution
in-distribunnn tuu
in-distriuution
Terral ncognita EERM
DomainNet:ERM
Background Challenge
WILDS AMazon:ERM
divlicomments.ERM
1.0
df a0c
0.8
0.8
0.2
adtaf-tiarbb ace
0.
oat-d--aarrbulilen acc
0.8
0.6
Q.6
0.4
0.4
0.4
04
0.4
8.00
0.25
0.50
0.75
1.00
0..00
0.25
0.50
0.75
1.00
80.00
0.25
0.50
0.75
1.50
0.25
0,50
0.75
1.00
0.0
0.00
0.25
0.50
0.75 1.00
in distribbution scc
In in.distrbutionn. acc
In inddistriuttonn. aCc
in distdtiuuuoooonoo acc
Indisstbubionn acc
Figure 1. Comparison of the in-distribution accuracy and the OOD accuracy of ERM between Momentum SGD and Adam. We also show the training results in the exhaustive hyperparameter search range a scatter plot in Appendix F.3. In these len plots, for each dataset, the in-distribution performance is separated by every ten bins 0.1. The average OOD performance when evaluating the checkpoints in that bin is shown on the vertical axis. We compare which optimizer shows better OOD performance for each model that achieves equivalent in-distribution performance. This approach can be characterized as a model selection strategy, which follows the methodology of utilizing training-domain validation set. In most cases, momentum SGD outperforms Adam in OOD performance in the rightmost region of our interest (the region where high in-distribution performance is achieved).
We found two distinct optimizer effect patterus, depending on the dataset: i) PACS, Office-Home, VLCS, Terra Incognita, Rotated MNIST, and DomainNet, and ii) Colored MNIST. Because these patterns appear in both the results of ERM and IRM, we focus on ERM for the explanation.
For PACS, Office Home, VLCS, Terralncognita, RotatedMNIST, and DomainNet, the OOD accuracy is greater for non-adaptive optimizers. The gap between the mean OOD accuracy and the mean in-distribution accuracy was smaller for the non-adaptive optimizer except for TerraIncognita. This means the models trained with the non-adaptive optimizer are more robust on average. In TerraIncoguita, the non-adaptive optimizer significantly outperforms the adaptive optimizer on the average in-distribution performance and OOD performance. We note, however, that except in this problem, the adaptive optimizer achieves a smaller gap between the two. As shown in Figure 1, when comparing models with the same in-distribution performance, the non-adaptive optimizer showed higher OOD accuracy than the adaptive optimizer. The results in Figures and 2 confirm that the non-adaptive method achieves higher OOD accuracy than the adaptive method, both on average and for models with the same in-distribution performance.
Colored MNIST shows the opposite pattern of these results, where the adaptive optimizer is better than the non-adaptive optimizers. As outlined in Section 3.2, Colored MNIST differs from the other datasets. To comprehend why adaptive optimizers are more effective in OOD generalization for this dataset, we have plotted the average training, validation, and test accuracy over time during training, illustrated in Figure 33. Our explanation for this outcome can be found in Appendix G.3.
We note our considerations regarding the exceptional behavior of ColoredMNIST. ColoredMNIST is a binary classification lask dataset with random labels, where spurious features are positively correlated with invariant features in in-distribution and negatively correlated with OOD. Non-adaptive optimizers learn the spurious features, achieve the oracle performance for in-distribution and perform worse than a random guess for OOD due inverted correlations. Adaptive optimizers, in contrast, seem to overfit the training data, achieving 100% accuracy in the training set (more than oracle 3) in this dataset, and failing to learn the spurious
Sinne ColoredMNIST includes lahel flip noise, even the best model that correctly learns the data rules will only perform 85%.
8

---

Published in Transactions on Machine Learning Research (06/2023)
PACS: ERM
PACS: ERM
PACS: ERM
1.0
1.0
1.0
Nut-of-dlstribbur acc
0.8
In-distribution acc
0.8
0.0
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
500
Momentum
kMrSingp
optimizer name
optimizer name
optimizer name
OfficeHome: ERM
OfficeHome: ERM
OfficeHome: ERM
1.0
out Nuttaa-dlstribuutin acc
in-distribution acc
0.0
0.6
0.4
0.4
0.2
0.0
0.0
RMSPop
Monercam Nestorev
Momertum RMSPro
optimizer name
optimizer name
optimizer name
Figure 2. PACS and OfficeHome in DomainBed: Comparison of the in-distribution (validation) accuracy and (test) accuracy of ERM across five optimizers. Non-adaptive optimizers outperform the adaptive optimizers in terms OOD generalization, and the gap between in-distribution performance and OOD performance is small. The details and results of other dataset experiments are described in Appendix F.2.
the
Background Challenge
Background Challenge
Background Challenge
1.0
1.0
1.0
distiiiiiiibbi
0.8
in n-distribution ace
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
Momentum
Adam
Momentum
Adam
Momentum
Adam
optimizer_nee
_amel name
optimizer_name name
optimizer_nne jammizerrnameeeee _nmee
Figure 3. Backgrounds Challenge: Comparison of the in-distribution (validation) accuracy and the out-of- distribution (lest) accuracy of ERM across two optimizers. In terms of OOD generalization, Momentum SGD outperforms Adam in both average and best OOD performance. The highest value of in-distribution accuracy remains the same, but Momentum SGD shows higher performance on average.
features. Due the aforementioned (synthetic) inverted correlations in the dataset, this overfitting behaviour in-distribution happens to favor adaptive optimizers. This behavior happens exceptionally on ColoredMNIST due to these synthetic flips in correlation. This exploitation unexpectedly enables Adam to avert being trapped in the training domain and produces better OOD generalization.
Backgrounds Challenge: Backgrounds Challenge requires training of ImageNet-1k ResNet50 from scratch, and it takes 256 GPU hours to obtain single trained model, so only compared Momentum SGD with Adam. Because Momentum SGD achieved competitive performance among non-adaptive optimization methods in the DomainBed and WILDS experiments, we adopted Momentum SGD to represent non-adaptive optimization methods. In the same way, Adam outperformed RMSProp in almost all the benchmarks, so we adopted Adam as a representative the adaptive optimizers.
Figure compares the accuracy for ORIGINAL (in-distribution) and Mixed-Rand (out-of- disiroffdistrbbtiion).. The best. in-distribution performance the same for each optimizer, but concerning the best OOD performance,

---

Published in Transactions on Machine Learning Research (06/2023)
WILDS amazon: ERM
WILDS amazon: ERM
WILDS amazon: ERM
1.0
1.0
1.0
Nut-of-dlstrrbbrr acc
0.8
in-distribution acc
0.8
0.0
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
500
Momertum
RMSSimp
optimizer name
optimizer name
optimizer name
WILDS civilcomments: ERM
WILDS civilcomments: ERM
WILDS civilcomments: ERM
1.0
out fut--f-dlstrbbuutin acc
in-distribution.ae acc
0.0
0.6
0.4
0.4
0.2
0.0
0.0
RMSPop
Momercam Nestorer
Momertum
RVSPip
optimizer name
optimizer name
optimizer name
Figure 4. and CovilComeesss WILDS: Comparison of the in-distribution (validation) accuracy and the cirrof-distrbuttion (test) accuracy of ERM across five optimizers. Both non-adaptive and adaptive optimizers have similar in-distribution accuracy, but the adaptive method significantly degrades the performance of the OOD generalization.
the non-adaptive optimizer outperformed the adaptive optimizer. As can be seen from Figure (socond row, middle column), particularly in the region of high in-distribution performance the right, the OOD performance of Momentum SGD exceeds that of Adam.
WILDS: A comparison of in-distribution and OOD averages for WILDS is shown in Figure 4. It can be clearly seen that the adaptive optimizer is fit. too well to the in-distribution in the WILDS problem setting. For and CivilComment- SrrlCommenntWILDDS, as DomainBed and Backgrounds Challenge, the non-adaptive optimizer outperforms the adaptive optimizer in terms OOD generalization. The gap between in-distribution accuracy and OOD accuracy is also tiny for non-adaptive optimizers. In particular, the CivilComments- WILDS experimental result. is remarkable, as both non-adaptive and adaptive optimizers show similar high in-distribution performance, but in the OOD environment, adaptive methods significantly fail to make inferences.
4.3 Correlation Behaviors
Our results show that three typical types of behavior are observed in terms of the correlation between in-distribution performance and OOD performance for different datasets. Detailed results of the experiments on all datasets are shown in Appendix F.4. We follow Miller et al. (2021), who used probit transform to show the relationship. The three types are increasing return, linear return, and diminishing return. These show how much performance OOD can be expected we increase the in-distribution performance.
The increasing return is an example, as shown in the leftmost part of Figure 5. The increasing returns in large regions of the in-distribution generalization significantly affect the OOD generalization. This suggests that the last tuning is significant for the OOD generalization, seen in all domain generalization datasets except for DomainNet.
The linear return is as shown in the middle of Figure 5. The OOD accuracy increases linearly with the in-distribution accuracy. This is generally the same result for in-distribution validation and test.
Conversely, diminishing return behavior, illustrated in the rightmost part Figure 5, indicates that substantial in-distribution improvement leads to saturation of OOD generalization with only marginal effect. This
10

---

Published in Transactions on Machine Learning Research (06/2023)
observation implies that the effort invested in fine-tuning might not always yield significant enhancements in OOD generalization. We have observed similar trends in settings with subpopulation shifts, such CivilComments- WILDS.
In Appendix F.4, we present our experimental results without a probit transform, confirming that diminishing returns are not necessarily linear before probit transformation. This finding aligns with recent studies by Wenzel et al. (2022); Teney et al. (2022), stating that the accuracy of ID and OOD varies across datasets. Moreover, Back et al. (2022) suggests, the occurrence of linear return depends on the problem set. Our results support these claims, providing comprehensive analysis of datasets and problem sets that Miller et al. (2021) did not address and uncovering trends they did not reveal.
For practitioners, our findings offer valuable insights into adjusting their expectations and stralegies based on the observed behavior. For instance, they work with a dataset similar to CivilComments- WILDS, they can anticipate one of the identified types of behavior. Should their datasct exhibit saturating behavior, they can adjust their expectations accordingly; conversely, their dataset demonstrates a regime where every slight. improvement in in-distribution accuracy aids, they should pursue enhanced in-distribution optimization.
5 Discussion and Conclusion
We conduct an exhaustive empirical comparison of the generalization performance of various optimizers under different practical distributional shifts. Notably, ten state-of-the-art OOD datasets were used to study the environment of broad shifts in correlation and diversity shifts. The investigation elucidates how optimizer selection affects OOD generalization. As the main claim, the answer to our research objective is that the non-adaptive optimizer is superior to the adaptive optimizer in terms of OOD generalization.
The following evidence supports this: i) when comparing in-distribution accuracy is the same, OOD accuracy of non-adaptive optimizers is greater than that of adaptive optimizers (Figure 1); ii) on average and top performance, the OOD accuracy of the non-adaptive optimizer is higher (Figures 2, 3, and 4). All these points support our main claim that uon-adaptive optimizers are superior in OOD generalization within our exhaustive experiments. We have tuned Adam to the range that is used in practice and have updated Adam's scores on all OOD) datasets, which use Adam optimizer as default for benchmarks. This supports the soundness of our experiments.
Overall, we can conclude that optimizer selection influences OOD generalization in the cases we are interested in. Future research should consider the algorithm or loss function and optimizers in the OOD problem. The results of IRM show a trend similar to ERM's, but a more detailed analysis is needed to consider the differences in loss landscapes. All these points support our main claim that non-adaptive optimizers are superior in OOD gencralization within our exhanstive experiments.
PACS:ERM (probit scaling)
DomainNet:ERM (probit scaling)
WILDS MLLDS_CVVicommens::RM (probit scaling)
0.50
500
Humret
Hesorn Homeitur 5i0 Adam
out- of distributtion DEC
Hararoo Homa
fottoo-distribuuiion
0.70
5.25
distributton
0.35
0.50
nanamar 560
991
Pestersy Panwiter 550
0.25
0.30
0.30
0.25
0.32
0,70
0.50
distributiin acc.
In didddiibuutionnoomm acc
In-distribttion
Figure 5. Three types of correlation behavior: increasing return (PACS), linear return (DomainNet), and diminishing return (CivilComments- (ooommnss-WWLDS). The legend circles on the right side of each figure show, in order, the SGD, Momentum SGD, Nesterov Momentum, RMProp, and Adam. The details and results of other dataset experiments are described in Appendixes 15, 16, and 16
11

---

Published in Transactions on Machine Learning Research (06/2023)
Finally, we would like to mention the limitations of our work. One limitation is that we did not study recently proposed and less popular optimizers.
The choice of optimizers we study is in line with previous work (Choi et al., 2019), (Schmidt et al., 2021) on optimizer comparison and with most recent OOD work; those studies overwhelmingly focused on Adam rather than e.g,, AdamW (Loshchilov & Hutter, 2017). Similarly, other less popular optimizers such as SWA (lzmailov et al., 2018), SWAD (Cha et al., 2021), SAM (Foret al., 2020), have been omitted to allow for a more extensive study of the chosen methods* 4
Another limitation is that we employed total of six models used in the DomainBed (ConyNet, ResNet20, ResNet50 and Vision Transformer (Dosovitskiy et al., 2020)), Backgrounds Challenge (ResNet50), and WILDS (DistilBERT (Sanh et al., 2019)) benchmarks.
Acknowledgments
Our deepest gratitude goes out to the anonymous reviewers and the Action Editor whose invaluable insights substantially enhanced the quality of this manuscript. We express our heartfelt thanks to Kilian Fatras (Mila, McGill), Divyat Mahajan (Mila, UdeM), and Mehrnaz Mofakhami (Mila, UdeM) for their constructive feedback that significantly contributed to the research process. Our sincere appreciation extends to the Masason Foundation Fellowship, which generously supported this work and made award to Hiroki Naganuma.
The computational resources instrumental to this study were provided under the auspices of the 'ABCI Grand Challenge' Program, National Institute of Advanced Industrial Science and Technology (AIST), and the TSUBAME Grand Challenge Program, Tokyo Institute of Technology. Special thanks to the AI Bridging Cloud Infrastruuttre (ABCI) and the TSUBAME 3.0. Moreover, we acknowledge the generous allocation of computational resources from the TSUBAME3.0 supercomputer, facilitated by Tokyo Institute of Technology. This assistance came through the Exploratory Joint Research Project Support Program from JHPCN (EX23401) and the TSUBAME Encouragement Program for Young / Female Users, whose support was instrumental to this research.
References
Vahdat Abdelzad, Krzysztof Czarnecki, and Rick Salay. The effect of optimization methods on the robustness of out-of-distributio detection approaches. arXiv preprint DrXiv:2006.14530 2020.
Shunichi Amari. Natural gradient works efficiently in learning. Newral computation, 10(2):251-270) 1998.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprinl xrXio:1907.02800 2019.
Christina Baek, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-onn-th--llne:: Predicting the performance of neural networks under distribution shift. arXiv preprint arXiv:2206.1908) 2022.
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, ude and variance of stochastic gradients. In International Conference on Machine Learning, pp. 404-413. PMLR, 2018.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Fhteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018.
Sara Beery, Grant Van Horn, and Pictro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 456 473, 2018.
Sara Beery, Elijah Colc, and Arvi Gjoka. The iwildcam 2020 competition datasct. arXiv preprint Xiv: 10340, 2020.
In some of the limited tasks, SAM and AdamW were also evaluated, the results of which are presented in Appendix
12

---

Published in Transactions on Machine Learning Research (06/2023)
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, 2019.
Scbastian Caldas, Sai Mcher Karthik Duddu, Peter Wu, Tian Li, Jakub Koncčný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings, arXiv preprint 18 1810001977 2018.
Junbum Cha, Sanghyuk Chun, Kyungiae Lee, Han- Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by secking flat minima. Advances in Neural Information Processing Systems, 34, 2021.
Yongqiang Chen, Kaiwen Zhou, Yalao Bian, Binghui Xie, Bingzhe Wu, Yonggang Zhang, MA KAILI, Han Yang, Peilin Zhao, Bo Han, and James Cheng. Pareto invariant risk minimization: Towards mitigating the optimization dilemma in out-of-distribution generalization. In The Eleventh International Conference on Learning Represenlations, 2023. URL https://openreviee wtwppppee....eeeeooonneeeeeeeeee
OpSL.
Dami Choi, Christopher J Shallue, Zachary Nado, Jachoon Lee, Chris J Maddison, and George E Dahl. On empirical comparisons of optimizers for deep learning. arXiv preprint arXie:1910.05440 2019.
Myung Jin Choi, Joseph J Lim, Antonio Torralba, and Alan Willsky. Exploiting hierarchical context on a large database of object categories. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 129-136. IEEE, 2010.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
Etienne David, Simon Madec, Pouria Sadeghi- Tehran, Helge Aasen, Bangyou Zheng, Shouyang Liu, Norbert Kirchgessner, Goro Tshikawa, Koichi Nagasawa, Minhajul A Badhon, Curtis Pozniak, Benoit de Solan, Andreas Hund, Scott C. Chapman, Frederic Baret, lan Stavness, and Wei Guo. Global wheat head detection (gwhd) dataset: large and diverse dataset of high-resolution rgb-labelled images to develop and benchmark wheat head detection methods. Plant Phenomics, 2020, 2020.
Etienne David, Mario Serouart, Daniel Smith, Simon Madec, Kaaviya Velumani, Shouyang Liu, Xu Wang, Francisco Pinto Espinosa, Shahameh Shafiee, Izzat S. A. Tahir, Hisashi Tsujioot,, Shuhei Nasuda, Bangyou Zheng, Norbert Kichgessner, Helge Aasen, Andreas Hund, Pouria Sadhegi- Tehran, Koichi Nagasawa, Goro Tshikawa, Sebastien Dandrifosse, Alexis Carlier, Benoit Mercatoris, Ken Kuroki, Haozhou Wang, Masanori Ishii, Minhajul A. Badhon, Curtis Pozniak, David Shaner LeBauer, Morten Lilimo, Jesse Poland, Scott Chapman, Benoit de Solan, Frederic Baret, Ian Stavness, and Wei Guo. Global wheat head dataset 2021: an update to improve the benchmarking wheat head localization with more diversity, 2021.
Alexey Dosovilskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. Xiv preprint arXiu:200..0 11929, 2020.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303 338, 2010.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. Tn Proceedings of the IEFE International Conference on Computer Vision, pp. 1657 1664, 2013.
Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and poltern recognition workshop, pp. 178-178. IEFF, 2004.
13

---

Published in Transactions on Machine Learning Research (06/2023)
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXie:2010.014 01412, 2020.
Robert Geirhos, förn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11): 665 673, 2020.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2559, 2015.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Dan Hendrycks and Thomas Dielterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05.. 2018.
Mare Khoury, Adaptive versus standard descent methods and robustness against adversarial examples. arXiv preprint ar arXte:1911.0878, 2019.
Diederik Kingma and Jimmyy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pp. 5637-5664. PMLR, 2021.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning, pp. 5815 5826. PMLR, 2021.
Ananya Kurnar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform distof-distribution.. In International Conference on Learning Representations, 2022a. URL https:
https::/ http///ppnreeeeeeeeeeeeeee net seee//fooommidd--yeeeeSS..
Ananya Kumar, Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar. How to fine-tune vision models with sgd. arXiv preprint XrXi::2211.00 1.09359, 2022b.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference Learning Representations, 2020.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint orXiv:1711.05101, 2017.
Shuai Lu, Daya Guo, Shuo Ren, Junjic Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. or Xie preprint or arxiu:2102 0ei:2102.0464, 2021.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint 2.ii:2000.1122443 2020.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy the line: the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning, pp. 7721-7735. PMLR, 2021.
Norman Mu and Justin Gilmer. Mnist-c: A robustness benchmark for computer vision. Xiv preprinl urXi::106600553 2019.
Vaishnavh Nagarajan, Anders Andreassen, and Behnam Neyshabur. Understanding the failure modes of
out-of-distribution generalization. In International Conference on Learning Representations, 2021.
Yurii Nesterov. Introductory lectures on conver optimization: A basic course, volume 87. Springer Science & Business Media, 2003.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly-labcled reviews and fine-grate fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) 2019.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions knowledge and data engineering, 22(10):34451338 2009.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1406-145 1415, 2019.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr compulational mathematics and mathematical physics, 4(5):1-17, 1964.
Veselin Raychev, Pavol Bielik, and Martin Vechev. Probabilistic model for code with decision trees. ACM SIGPLAN Notices, 2016.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference Learning Representations, 2018.
Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1-3):157 173, 2008.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXi::1910.01108 2019.
Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through crowded valley-benchmarking deep learning optimizers. In International Conference Machine Learning, pp. 9367-9376. PMLR, 2021.
Frank Schneider, Lukas Balles, and Philipp Hennig. DeepOBS: A deep learning optimizer benchmark suite. In International Conference on Learning Representations, 2019.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of- distribution generalization: A survey. arXiv preprint ar Xiv:21108. 19624, 2021.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stationary environments: Introduction to covariate shift adaptation. MIT press, 2012.
Baochen Sun and Kate Sacnko. Deep coral: Correlation aligument for deep domain adaptation. In Computer Fision-ECCh 2016 Workshops: Amsterdam, The Netherlonds, October 10 and 15-16, 2016, Proceedings, Part III 14, pp. 443 450. Springer, 2016.
15

---

Published in Transactions on Machine Learning Research (06/2023)
Christian Szegedy, Wojeiech Zaremba, Ily Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. Tn International Conference on Learning Representations, 2014.
J. Taylor, B. Earnshaw, B. Mabey, M. Victors, and J. Yosinski. Rxrxl: An image set for cellular morphological variation across many experimental batches. In International Conference Learning Representations (ICLR), 2019.
Damien Teney, Yong Lin, Scong Joon Oh, and Ehsan Abbasnejad. Id and ood performance are sometimes inversely correlated on real-world datasels, 2022. URL
Tijmen Tieleman and Geolfrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26 31, 2012.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5018 5027, 2017.
Neha S Wadia, Daniel Duckworth, Samuel S Schocnholz, Ethan Dyer, and Jascha Sohl-Dickstein. Whitening and second order optimization both destroy information about the dalaset, and can make generalization impossible, arXiv preprint arXiv:2008.074: 07545, 2020.
Yixiang Wang, Jiqiang Liu, Jelena Mišić, Vojislav B Mišić, Shaohua Lv, and Xiaolin Chang. Assessing optimizer impact dnn model sensitivity to adversarial examples. IEEE Access, 1152766-1127771 2019.
Florian Wenzel, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, et al. Assaying out-of-distributiio generalization in transfer learning. arXiv preprint orXiv:2207.0 09239, 2022.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value adaptive gradient methods in machine learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2017.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongscok Namkoong, et al. Robust fine- tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 7959-7971, 2022.
Kai Yuanqing Xiao, Logan Engstrom, Andrew llyas, and Aleksander Madry, Noise or signal: The role of image backgrounds in object rocognition. In International Conference on Learning Representations, 2021.
Christopher Ych, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 2020.
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E Dahl, Christopher J Shallue, and Roger Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. arXiv preprint ar wxxi::990.004166 2019.
Difan Zou, Yuan Cao, Yuanzhi Lú, and Quanquan Gu. Understanding the generalization of adam learning neural networks with proper regularization, 2022. URL https://openrevier: wtt.eorrriieeeeeeeooiiiiei C7PfyLimZBp.
16

---

Published in Transactions on Machine Learning Research (06/2023)
Contents (Appendix)
A Optimizers
19
A.1 SGD
19
A.2 Momentum-SGD
19
A.3 Nesterov Momentum
19
A.4 RMSprop
19
A.5 Adam
19
B Implementation and Environment for Experiments
19
C Datasets
20
C.1 DomainBed
20
C,2 Backgrounds Challenge Dataset
20
C.3 Amazon-W WILDS and DivilComments-WIL.D Dataset
21
C.4 CIFAR10-C and CIFAR10-P Datasel
21
D Experimental Protocol
22
D.1 DomainBed
................................
22
D.2 Backgrounds Challenge Dataset
................................
22
D.3 WILDS
23
D.4
CIFAR10-C and CIFAR10-P
23
E Hyperparameters and Detailed Configurations
24
E.1 DomainBed
24
E.2 Backgrounds Challenge Dataset
................................
26
E.3 WILDS
26
E.4
CIEAR10-C and CIFAR10-P
27
F Full Results of Experiments
28
F.1
Full Results of Table
................................
28
F.2
Full Results of Boxplot
................................
29
F.2.1 Full Results of Filtered Boxplot (ERM)
................................
29
F.2.2 Full Results of Filtered Boxplot (IRM)
33
F.3
Full Results of Bin-Diagram Plots
35
F.4 Full Results of Scatter Plots
36
F.4.1
ERM
36
F.4.2
IRM
37
G Ablation Study
39
G.1 Probit Transformed Scatter Plot
39
G.1.1 ERM
................................
39
G.1.2 IRM
................................
42
G.2 Model Performance Transition throught Hyperparameter Search
................................
44
G.2.1 Hyperparameter Trial Budget vs OOD Accuracy
44
G.2.2 Hyperparameter Trial Budget vs OOD Error
46
G.3 Learning Curve of ColoredMNIST
50
G.4 Early Stopping
51
Soundness Check Our Experiments
61
H.1 Histgram of Hyperparameters
61
H.2 Hyperparameters and OOD Accuracy Box-Plot
61
H.3 Effect of Initial Configuration Hyperparameter Optimization
64
H.4 Best OOD Performance Comparison against with Existing Benckmark
66
17

---

Published in Transactions on Machine Learning Research (06/2023)
Additional Study
67
1.1
Corruption and Perturbation Shift.
67
1.2
Model Architecture..
67
1.2.1
ResNet-20 for ColSTedMNIST
67
T.2.2
Vision Transformer for PACS
I.3
Stato-of-the-Arts Optimizers.
1.3.1
Sharpness Aware Minimization (SAM)
68
I.3.2
Adam with Decoupled Weight Decay (AdamW)
68
1.4
Large for Adam
69
1.5
Learning Rate Schedule
69
1.6
The Effect of Random Seeds
70
1.7
Algorithms (ERM, IRM, VREx and CORAL)
71
18

---

Published in Transactions on Machine Learning Research (06/2023)
A
Optimizers
As we explained in Section 4.1, we compared SGD, momentum-SGD, Nesterov momentum, RMSprop, and Adam. We write the algorithm of these optimizers below.
A.1 SGD
0, t 01-1 -1) ((01-))
(2)
A.2 Momentum-SGI
vr 7/1-1 + 1/t Vo, (((--1)
(3)
0, 0:-1
(4)
A.3 Nesterov Momentum
11 - 70-1 + 13t 1
(((1--1 - - 0, 0:-
(5)
(6)
A.4 RMSprop
1t + + + (1 (1-a)V va)
(7)
17: V Vo,_ (()))
mi + 1--1 +
(8)
v0, +
0, + 0 - m2
(9)
A.5 Adam
m: < B-m--1 + (1 - (1) Vo. (0.)
(10)
v; + B2v1-1 +(1
(1- - B2) V v (0,)²
(11)
b, v1 1 3
(12)
$
0, - 0--1 7i Uz + b
(13)
B Implementation and Environment for Experiments
We perform our experiment with ABCI (AI Bridging Cloud Infrastructure), supercomputer owned by the National Institute of Advanced Industrial Science and Technology, and TSUBAME, supercomputer owned by the Tokyo Institute of Technology.
All codes for experiments are modifications of the codes provided by the authors who introduced the datasets Gulrajani Lopez-Paz (2021); Koh et al. (2021); Xiao et al. (2021). Licenses of the codes are MIT license for DomainBed Gulrajani & Lopez-Paz (2021) and WILDS Koh et al. (2021). The code of Backgrounds Challenge does not indicate the license. Our code can be found the link below. https://githunn com Hirkki wwwrrokiixxxx// 1tppptiiiiete.iei.eeeeeem. Caw.reeroommmarrion__ee 00D
19

---

Published in Transactions on Machine Learning Research (06/2023)
C Datasets
C.1 DomainBed
DomainBed consists of sub-datasets shown in Table 2, where exclude Terra Incognita and Rotated MNIST stated in Section 3.2. We summarize the dataset information in the Table by referring to (Gulrajani & Lopez-Paz, 2021).
Table 2: DomainBed: Dataset Information
domain
examples class
Colored MNIST Rotated MNIST
0.1, 0.2, 0.9
70000
0, 15, 30, 45, 60, 75]
70000
10
PACS
lart, cartoons, photos, sketches]
9991
VLCS
Caltech101, L art, clipart, product, real]
VOC2007]
10729
fffico-Home
15588
65
Terralncognita
L.100, T.38, I.43, I.46)
24788
10
DomainNet
clipart, infograph, painting, quickdraw,420ren sketch
586575
345
Colored MNIST is a dataset for binary classification of MNIST dataset (Arjovsky et al., 2019). The digits from to 4 are labeled 0, and those greater than 5 are labeled 1, and the task is to classify these classes successfully. However, each digit also colored by either red or green. This for having models to confuse the important fcature for classification. The domain indicates the correlation of the label with color. For example, if the domain is 0.1, the correlation between, say red, with the number smaller than is 0.1. Furthermore, the label is flipped at constant rate: in this paper, 15 % label is flipped. Therefore, the correlation between color and digit is d, while between label and digits is 0.85. That is, what models should learn is the correlation between label and noise, resulting in classification accuracy of 0.85. However, if the model exploits spurious correlation of the domain, it will learn the correlation between label and color, resulting in training accuracy being 0.9 but test accuracy being 0.1 in this case.
PACS and Office-Home are image datasets whose domain determines the style of the image. These are benchmark datasets for domain generalization.
VLCS is set of different photographic datasets, PASCAL VOC (Everingham et al., 2010), LabelMe (Russell el al., 2008), Caltech101 (Fei-Fei et al., 2004), and SUN09 (Choi et al., 2010). PASCAL VOC, LabelMe, and SUN09 are benchmark datasets for object detection. Caltech101 is 101 classes image datasets, where each class has 40 80 samples.
DomainNet is a large datasel proposed for the study of domain generalization. The number of classes, domains, and dataset size is the largest in the DomainBed dataset.
TerraIncognita is dataset consisting of images taken by cameras at different locations. The difference in the camera's 's location corresponds to the difference in the domain.
RotatedMNIST is a dataset that artificially rotates MNIST and divides the domain according to the rotation angle. The number in the domain corresponds to the rotation angle.
We leave one domain for the final evaluation and use the remaining domains for training. To evaluate the performance during training, we split the data of each domain into training data and validation data. The split ratio is 80 % for training and 20 % for validation. We take average of test accuracies and validation accuracies across domains, respectively, and use them to evaluate the OOD generalization.
C.2 Backgrounds Challenge Dataset
In Section D.2, we explained that we use the subset of ImageNet (ORIGINAL). ORIGINAL consists of nine classes displayed in table These classes are synthetically created from ImageNet classes based on WordNet. ID. This table is copy of table in the original paper (Xiao et al., 2021).
20

---

Published in Transactions on Machine Learning Research (06/2023)
Table 3: Backkrounds Challenge: Dataset information originally created (Xiao et al., 2021)
classes
WordNet ID
num sub-classes
Dog
n02084071
116
Bird
n01503061
52
Vehicle
n04576211
42
Reptile
n01661091
36
Carnivore
n02075296
35
Insect
n02159955
27
Instrument
n03800933
26
Primate
n02469914
20
Fish n02512053
16
This dataset is filtered balance samples across classes. We follow the same filtering procodure as the original paper. For further details, please refer to the original paper (Xiao el al., 2021).
C.3 Waazon-WiLDD and CivilComments- WWLDDD Dataset
WILDS set ob benchmark datasets with distributional shift and their variants: iWildCam (Beery el al., 2020), Camelyon17 (Bandi et al., 2018), RxRx1 (Taylor al., 2019), GlobalWheat (David et al., 2020; 2021); CivilComments (Borkan al., 2019), FMoW (Christic et al., 2018), PovertyMap (Yeh et al., 2020), Amazon (Ni et al., 2019), and Py150 (Lu et al., 2021; Raychev et al., 2016). We use CivilComments and Amazon for our experiment as NLP tasks.
(Hu al., 2020),
C.4 CIFAR10-C and CIFAR10-P Dataset
First, for the corrupted perturbed data generalization studies, use CIFAR-1O-C, CIFAR-10-P to evaluate the generalization performance Hendrycks & Dictterich (2019); Mu & Gilmer (2019). CIFAR-10-C consist of CIFAR image data corrupted by 19 noise patterns. 1IFAR-10-P is similar, but with ten perturbations, Similar corruptions and perturbations frequently occur in real-world imaging applications. Thus, evaluating the robustness against this noise is essential for improving practical applicability. Because noises changes the input samples, we can regard the corrupted and perturbed data being sampled from different distribution P(x,y) y)' than the original distribution: P(x,y) v) * P(x:, y). Tf classifier f X - is robust to corruption X X, Eece [P(f(((x)) v))] with corruption distribution C, we can say that the classifier generalize for the corrupted samples. The same is true for perturbation.
21

---

Published in Transactions on Machine Learning Research (06/2023)
D Experimental Protocol
We employ Bayesian hyperparameter search strategy in the sweep functionality of Weights&:Biases" In this strategy, the relationship between parameters and evaluation metrics is modcled as Gaussian process, and the parameters are selected in such a way as to optimize the improvement probability. Table 4 shows the number of hyperparameter optimizations performed for each task. The transition of the best ood accuracy in these hyperparameter optimizations shown in Appendix G.2.
In line with previous studies (Gulrajani & Lopez-Pazz 2021; Xiao al., 2021; Koh et al., 2021) different benchmarks use different evaluation metrics, each of which is outlined in the following sections.
Table 4: Number of Experiments for Each Datasct
SGD
Momentum
Nesterov
RMSprop
Adam
RotatedMNNIST
200
200
200
200
200
ColoredMNIST
200
342
200
200
200
PACS
200
200
200
200
412
VL.CS
200
200
200
200
324
OfficeHomee
399
200
200
200
699
Terralncoguita
200
200
200
451
202
DomainNet.
490
515
857
1160
1137
Amazon-WILDS
440
438
449
489
466
Amazon-Civiae CivilComments
594
543
588
575
554
Background Challenge
347
567
D.1 DomainBed
We follow the setting that is employed in the original paper (Gulrajani & Lopez-Paz, 2021). We train models with training domains and evaluate their performance on the test domain, which is the domain not used for training. We use ResNet-50 for PACS, VLCS, Office-Home, DomainNet, Terralncognita, and MNIST ConvNet (Gulrajani & Lopoz-Paz, 2021) for RotatedMNIST and Colored MNIST.
In our experiments, one domain is used the test domain (OOD: out-of-disttrbuution) and the other domain the training domain (ID: in in-distribution). More precisely, the test domain is "Art" for PACS, 'Caltech101 for VLSC, "Art" for Office-Home, 'Clipart" for DomainNet, "L100" for TerraIncoguita, '30" for RotatedMNIST, and *-90%* for ColoredMNIST. We explain the experimental configurations in Section E.I.
D.2 Backgrounds Challenge Dataset
We follow Xiao al. (2021) for using Backgrounds Challenge datasot for evaluation. Thus, we use the following data-generating procedure proposed by Xiao al. (2021). We train ResNet-50 ImageNet-Ik with two popular optimizers, Momentum SGD, and Adam. The test datasets to evaluate the trained model are derivations of ImageNet datasot. First, we construct subset of the ImageNet which has nine coarso-grained classes, c.g. insect (ORIGINAL). Especially, we refer to ORIGINAL with all images from ImageNet as IN9L. Then, we create a dataset. by changing the background of the images IN9L. In particular, we change the background call this dataset Mixed- Random, following Xiao al. (2021). By comparing the accuracy of Mixed-Same with that of ORIGINAL, can measure the dependence of the model on the spurious correlation of background information. Thus, investigate the relations between these two accuracies. The search range for hyperparameters is shown in Section E.2.
each image into a random background cropped from another image in ORIGINAL. We
hhttps://vandb. ai/site
22

---

Published in Transactions on Machine Learning Research (06/2023)
D.3 WILDS
We also follow the setling that is employed in the original paper (Koh et al., 2021). First, we divide the dataset into train, validation, and test and train a model using the train data. For model selection, we use the performance evaluation of validation data. the test dataset, considering the subpopulation shift, wo measure the performance of the OOD in the worst group for CivilComments- WWnnDD and in the domain of 10-percentile for Amazon-WII.DD In both Amazon-WTI.ID and CivilComments- WIL.DS, DistilBERT (Sanh al., 2019) is used the deep neural network model architecture. We explain the further details of the experimental configurations in Section E.3.
D.4 CIFAR1O-C and CIFAR10-P
First, we trained deep neural networks with CIFAR-10 Then we evaluated the trained models with CIFAR- 10-C, CIFAR-10-PT, For the corruption datasets (CIFAR-1O-C), compared the accuracy for the corruption perturbation test samples (samples from CIFAR-10-C) with that for the training domain test samples (samples from CIFAR-10). For the perturbation dataset, we measured the perturbation robustness introduced by Hendrycks & Dietterich (2019).
23

---

Published in Transactions on Machine Learning Research (06/2023)
E Hyperparameters and Detailed Configurations
We report the hyperparameter's search space. For vanilla. SGD, search learning rate v, learning rate decay rate p and the timing to decay learning rate 6, and regularization coefficient of weight decay A. When ō 0.7, it means that the learning rate decays when training passes 70 % of the total iterations. We do not search p and 8 for DomainBed because we do not employ a learning rate schedule.
For non-adaptive momentum methods, a parameter to control momentum 7 is added to the hyperparametens. For RMSprop, we further add parameters a and e, which control the second-order momentum and numerical stability, respectively. Although is originally introduced for numerical stability, this parameter is found play a crucial role in generalization performance (Choi al., 2019). Thus, we follow Choi et al. and vary this parameter as well.
For Adam, we add 6, 31, 82 to vanilla SGD's configuration. The parameter is the same as that for RMSprop and 31 and 82 control first and second-order momentum terms, respectively.
E.1
DomainBed
We conduct Bayesian optimization for hyperparameter search of DomainBed. First, we sampled hyperparam- eters from uniform distribution whose minimum and maximum values are shown in the table 6 and 7 Then, we conducted Bayesian optimization on these sampled candidate hyperparameters and selected some of the hyperparamctors among them. For reference, the scatter plot of Adam's learning rate for the OfficeHome dataset is shown in Appendix H.1. Note that the values for batch size B in the tables do not indicate minimum and maximum for Bayesian optimization but those for grid search. Unlike other datasets, we implement IRMv1 in addition to ERM. IRMv1 heuristic optimization problem solve IRM, introduced by Arjovsky et al. (2019). Thus, search the hyperparameters of IRMv1 as well.
IRMv1 is the following constrained optimization problem (Arjovsky et al., 2019):
min x 6gEsr
Σ (T) + ARRM |/ /f (w 4)
(14)
where & is representation function and w is the weight on top of the function of a model f = w o $. X' is the input domain and y is the output domain. The character e indicates an environment in training environment set Eur and R is the risk of the environment. Because this the optimization with regularization term, we search coefficient AIRM. In addition, we implement annealing of this coefficient and we try various penalty annealing iterations NIRM too. The basic workload is summarized in table 5.
We made two changes to the experimental setup in the original paper, as well as to the optimizer. We added regularization to the training of ColoredMNIST and RotatedMNIST to stabilize the learning. In addition, we increased the steps budget for RotatedMNIST because we observed that the training loss of RotALedMNIST was not sufficiently reduced.
We also experimented with two additional model architectures (ResNet-20 and Vision Transformer) for the benchmarks proposed in the existing Bed.
24

---

Published in Transactions on Machine Learning Research (06/2023)
Table 5: DomainBed: Workloads
Model
Dataset
Batch size
Step Budget
MNIST ConvNet (Gulrajani & Lopez-Paz, 2021)
Colored MNIST
[128, 512, 2048]
5000
MNIST ConvNet (Gulrajani & Lopez-Paze 2021)
Rotated MNIST
128, 512, 2048)
100K
ResNet-50
VL.CS
[64, 128]
5000
ResNet-50
PACS
[64, 128)
5000
ResNet-50
Office Home
[64, 128]
5000
ResNet-50
NemainNet
[64, 128)
5000
ResNet-50
Terralncognita.
64, 128)
5000
Vision Transformer (for Additional Study)
PACS
[64]
5000
ResNet- 20 (for Additional Study)
Colored MNIST
[64]
5000
Table 6: DomainBed: ResNet-50
B
A
AIRM
NIRM
[el
[le+0,
SGD
[64, 128]
[1e-5, 1e-2]
[1e-6, 6, 1e-2]
le45)
le+4
[l--1.
[le+0,
Momentum
[64, 128]
[1c-5, 1c-2]
[1e-6, 1e-2]
[0, 0.99999]
le+5]
le+4
[le-1.
[le+0,
Nesterov
[64, 128]
[le-5, 1e-2]
[1e-6, 6, 1e-2]
[0, 0.99999]
lc+5
lc+4
lel1,
[le+0,
RMSprop
[64, 128]
[1e-5, 1e-2]
[1e-6, 1e-2]
[0, 0.99999]
[0, 0.999]
[1e-8, 1e-3]
le+5]
le+4]
B
32
AIRM
NIRM
[e-11
[le+0,
Adam [64, 128)
[le-5, 1e-2]
[1e-6, 1e-2]
[0, 0.999)
[0, 0.999)
[1e-8, 1e-3)
le+5]
le+4
B
17
?
p
SAM
64
lle5, le-2
[0.9)
[1 le-4]
5e-2
Table 7: DomainBed: MNIST ConvNet (Gulrajani & Sopez-Pazz 2021)
B
AIRM
NIRM
SGD
[128, 521, 2048)
[le-5, 1e-2]
1e-6, 1e-2
[le-1,
[le+0,
Momentum
[128, 521, 2048]
[le-5, 1e-2]
[1e-6, le-2]
[0, 0.99999]
le+5)
lo+4
1e-1,
le+0,
Nestcrov
[128, 521, 2048)
[le-5, 1e-2]
[1c-6, 1e-2]
[0, 0.99999]
le+5
le+4]
[e-1,
[e++
RMSprop
1228, 521, 2048]
[1e-5, 1e-2]
[1e-6, 1e-2]
[0, 0.99999]
[0, 0.999]
[1e-8, 1e-3)
let5)
le+4
B
1
A
B
Bg
ARR NIRM
[1e-1,
[le+0,
Adam [128, 521, 2048)
[1e-5, 1e-2]
[0, 0.999)
[0, 0.999)
[1e-8, 1e-3]
1o+5
lot4
25

---

Published in Transactions on Machine Learning Research (06/2023)
E.2 Backgrounds Challenge Dataset
We conduct Bayesian optimization for the Backgrounds Challenge dataset well. We use 4096 the batch size. For all configurations other than batch size, we follow Choi et al. (2019).
We conduct the hyperparameter search and restart training from scratch. Of the trained models, we evaluate trained model performance in the OOD dataset.
Table 8: Backgrounds Challenge: ResNet-50
7
um
le-5, 1e-2
[e-6,
1e-2
0, 0.99999
A
81
B2
Adam [1o-5, 1c-2]
[1c-6, 1c-2]
[0, 0.999]
[0. 0.999]
[1c-8, 1c-3]
E.3 WILDS
We conduct Bayesian optimization for the hyperparameter search of WILDS. The hyperparameters are sampled by uniform distribution whose minimum and maximum values are shown in the table 9 and 10. Note that the values for batch size B is fixed in these experiments due to computational efficiency. We implement IRMvl as well as DomainBed experiments.
For training the Amazon- WILDS and CivilComments- WILDS datasets that we used as NLP datasets, we followed the deep neural network model and training configuration proposed the original paper. DistilERRT, distillation of BERT Base, is used as the deep neural network model.
Table 9: WILDS-Amazon DistilBERT
B
1
A
γ
AIRM
NIRM
[le-1,
[le+0,
SGD
[8]
[1e-5, 1c-2
[1c-6, 1e-2]
le+5
lc+4
[le-1,
[le+0.
Momentum
[8]
[le-5, 1e-2]
(1e-6, 1e-2]
[0, 0.99999]
le+5]
lc+4
[le-1,
[1e+0,
Nesterov
[1c-5, 1e-2]
[1c-6, 1e-2]
[0, 0.99999]
le+5
let4
[le-1,
[le+0,
RMSprop
[le-5, le-2]
[1e-6, le-2]
00, 0.99999
[0, 0.999]
[le-8, 1e-3]
let5
lc+4
1)
A
6
B₂
c
ARRM
NIRM
1c-1
[1c+0,
Adam [8]
[le-5, 1e-2]
[1e-6, le-2]
[0, 0.999]
[0, 0.999]
[le-8, 1e-3]
let5
le+4
B
81
82
AdamW [8]
10-5,
1c-2
0.9
(0.999
1c-88
B
n
p
SAM
8
1o-5,
1e-2
[0.9)
1e-4
56-2
26

---

Published in Transactions on Machine Learning Research (06/2023)
Table 10: WILDS-CivilCommnen DistilBERT
7
a
AIRM
NIRM
[le-1,
le+00
SGD
[16]
[1e-5, 1e-2]
[1e-6, 1e-2]
le+5
1e+4
[1e-1,
11++0,
Momentum
[16]
[1e-5, 1e-2]
[1e-6, 1e-2]
[0, 0.99999]
le+5)
le+4]
[Te-1,
1e++0
Nesterov
166
[1e-5, 1e-2]
1e-6, 1e-2]
[0, 0.99999]
le+5
le-4 4]
[le-1,
[1++0,
RMSprop
[66
[1e-5, le-2]
[1e-6, 1e-2]
[0, 0.99999]
[0, 0.999]
[le-8, 1e-3]
le45
1e-4
B
1
A
31
32
AIRM [1e-1,
NIRM
[1e-0,
Adam [16]
[1e-5, 1e-2]
[1e-6, 1e-2]
[0, 0.999]
[0, 0.999]
[le-8, 1e-3]
le-5) 5)
le+44
E.4 CIFAR10-C and CIFAR10-P
The corruption dataset we consider contains 19 corruption patterns. These are Gaussian Noise, Shot. Noise, Impulse Noise, Defocus Blur, Frosted Glass Blur, Motion Blur, Zoom Blue, Show, Frosi, Fog, Brightness, Contrast, Elastic, Pizelate, and JPEG. Sample corrupted images are shown in the original paper by Hendrycks & Dietterich (2019). Following Hendrycks & Dietterich (2019), error Escc for each corruption with five-point scale for the noise severity s. Averaging over five-point scale, we measure the classification accuracy for each corruption Acc. as follows:
Corruption:
compute the classification
Acce E E.Sc
(15)
This accuracy the Lest accuracy (corruption) we defined in the main body of this paper. We evaluate the generalization performance by comparing this quantity with the accuracy calculated using the original test. dataset Acc E. Note that É the classification error for the test (training domain) samples.
Perturbation:
The perturbation dataset has six corruption patterns that are the same as in the corruption dataset and four additional digital transformations: Gaussian Noise, Shot Noise, Motion Blur, Zoom Blur, Show, Brightness, Translate, Rotate, Till, and Scale. The perturbation dataset differs from the corruption dataset in that each corruption generates more than thirty perturbation sequences. Hence, we use not a simple measure of accuracy but the following metric to determine the perturbation robustness:
max
d (f(m), = Σ Σ
(16)
(6,
where T(æ) is the ranked prediction of the classifier on image and o(i) = 7<x'//(xx) This criterion measures how the top-5 prediction on two different images differs. By using this quantity, top-5 robustness perturbation can be written as follows:
1
uT5D, m(n 1) 12
(
(17)
27

---

Published in Transactions on Machine Learning Research (06/2023)
Table 11: Hyperparameter Search Range:
ResNet-8 / CIFAR10
B
A
Momentum 256
1e-6, le+1]
[l--5, le-4]
le-4 0.9999
À
B1
32
Adam 256 1e-4, le-1] [le-5, le-4]
[0.9, 0.999]
[0.99, 0.9999
le-4, le-2
F Full Results of Experiments
We conducted an evaluation of optimizers using two distinct model selection strategies. One approach is Oracle-based, while the other employs method of training-domain validation set (Gulrajani Lopez-Paz, 2021). In Section F.I, we compare the OOD accuracy of models obtained through each strategy, as displayed in tabular format. Particularly for the latter strategy, present both the average and variance of the Top-10 models, well as the average and variance for models surpassing certain threshold.
The rationale behind providing the average and variance of the Top-10 models lies in our adoption of Bayesian optimization for hyperparameters, with the aim of illustrating the error bars of high-performing models obtained through this method. The results for models exceeding a certain threshold (in the rightmost bin) are sharod to facilitate comparison of average OOD accuracy performance when non-adaptive and adaptive optimization methods are comparable in terms of ID accuracy.
In Section F.2, present the results for all hyperparameters random guess), along with their distributions in a box plot. Finally, in Scction F.4, we present the OOD accuracy and ID accuracy results for all the results from which these results are derived as scatter plot.
explored (except for the model inferior to
F.1 Full Results of Table
Oracle:
In this section, provide additional information Table in the paper. Table 12 shows the mean and standard deviation for the top-10 models, including those selected by the Oracle model selection method. In eight of the ten datasets, the non-adaptive method outperforms the adaptive method in diitt--diitiiittoonn accuracy.
Table 12: Top-10 Trials: Mean and Standard Deviation for Each Dataset (Model Selection by Oracle). The variance is relatively suppressed in many tasks because it is the mean and variance of the model that achieves Top-10 OOD accuracy.
Non-Adaptive Optimizer Momentum
Adaptive Optimizer
Model
OOD Datasct
SGD
Netsterov
RMSProp
Adam
MoIST
89.62 89.6244.18
93.23
92. 89.
95.81 =0.16
96.00 20.06
4-Layer CNN
ColoredMNIST
14.644n.07
29.80
28. 85
52 02.8226..21
52.30+6..
VLCS
98.98 98.98.01
98. 18
98. 49.
96.23
97.09
PACS
86.561u83
86.90
86.01 86.01
81.10
82.59 82.591.2.4
OfficeHome
63. 63.681u.12
63.52. 63.52...44
62. 40
55.341g30
62. 02.12.0.33
ResNet50
Terralncogaita
52, 11.02
56.05
52. 12.9811.61
53. 12.67712.62
48.77 18.78
DomainNet
55.31
36.12
58. 58.1912.2
52.7 72.71
55.74 55.74+1.60
BackgroundChallengg
78.99 78.9920.44
75.55
WILDS amezzn
52.00+0.0
54.67=0.0
54.67
48.67
52.4 52.44+1..
DistilBERT
WILDS civileommentss
51.81+0.20
56.40 $6.40-a..0
55. 55.98+0.47
38. 13344277.00
37.71+4.0
Net-22
Colore:MMNIST
28.93
29. 29.8441.64
VIT
PACS
90. 90.28-0.44
90.066+0.29
Training-Domall Validation Set: Table 13 shows the results of top-10 models with training- domain
validation set (Gulrajani & Lopez-Paz, 2021) as the model selection method. It shows that 11 out of tasks showed that the non-adaptive optimization method superior. As exhibited in Table 14, we present the out-of-distribution accuracy along with the mean and standard deviation of the subset exhibiting high
28

---

Published in Transactions on Machine Learning Research (06/2023)
validation performance within targeted training domain, graphically depicted in Figure 1. This data specifically corresponds to the apex of the rightmost bin in the bar chart encapsulated in Figure 1. As noted, the relatively elevated standard deviations observed can be primarily attributed to our calculation methodology. This approach calculates the mean and standard deviation values of the out-of-distribution accuracy, specifically locusing on the rightmost bins indicative of high validation accuracy as portrayed Figure It crucial to acknowledge that this unique approach could potentially yield a broader range of variances.
Table 13: Top-10 Trials: Mean and Standard Deviation for Each Dataset (Model Selection by Training- Domain Validation Sct). The variance is relatively suppressed in many tasks because it is the mean and variance of the model that achieves Top-10 OOD accuracy.
Non-Adaptive Optimizer
Adaptive Optimizer
Model
OOD Dataset
SGD
Momentum
Netsterov
RMSProp
Adam
MNIST
89.134a.56
93.09
91.36
95.51
95. 76.4.44
4-Layer CNN
ColoredMNIST
10.4+0.a
10.04+0.07
10. +0.09
9.84=0.11
10.08 10.08+0.4
VLCS
98. 30,3210a4
18.15
97. 07.931u.2
94.49
96.7612
PACS
85.01
85.60
84.91
81.10
82.36
OfficeHome
62.62 62.62
62.65_1102
60.98
55.0223..4
60.09 =1.00
ResNet50
TerraIncognita
46.84
44.67
45.21
44.-48
41.82
DomainNet
55. 15.25111.11
55.96
58.19
52.71 .:274
55.67
BackgroundChallenge
78.99 20.48
75.55
WILDS
52. +0.00
54. 13=00.99
63.77
50,93-a3.25
52.44
DistiBBERT
WILDS eivilcomments
49.17
=0.09
50.40 84
3L.65
33.13+5..0
ResNet-20
ColoredMNIST
11.00 =..52
10.09
VII
PACS
90.28=20.54
90.05
Table 14: Trials in Rightmost Bin: Mean and Standard Deviation for Each Dataset (Model Selection by Training- Domann Validation Set). The variance is relatively large because compute the mean and variance of models that exceed specific threshold (the rightmost bin in Figure 1) in ID accuracy.
Non-Adaptive Optimizer
Adaptive Optimizer
Model
OOD Dataset
SGD
Momentum
Netsterov
RMSProp
Adam
MNtS
82 22.83
87 25.477
87 43
94.29+1
94. 43.911
4-Loyer CNN
ColorodMNIST
10.95 10.93-1.61
12.86=4.66
10. 20.4442.55
22, 22.84.6.44
16. 16.12=7.98
VLCSS
97.95.1.0
96.81 12.00
96.54_22.3
94.07 13.4
94.491s.an
PACS
79.67
78.58
78. 78.11
74.98
70.8615.06
OfficeHome
61.11:2207
60.38
58 19.55
55 66.98
57. 37.7443.1
ResNet50
Terralnooguita
42.56=5.81
2.
41.4446.3
41.96
40.76
DomainNet
55.79=1.04
57.40
57.51+2.55
55.09
56.05
BackgromndChaallengee
74.98
69.1445.61
Amazon- WILDS
50.5821.07
52 22.20+1..0
52. 52.00=1.50
51.73+1
51.32+1.88
DistiBBERT
Civil DvillComments-oe WILDS
42.24
44.794n..0
44.15
21.79
23.55
Net-20
CoIOredMNIST
19.62.47.a
21. 21.70=7.00
VIT
PACS
37.077.5.52
85.17
F.2
Full Results of Boxplot
Since we could not include all the results in the main paper, report all the OOD accuracy, ID accuracy and their difference (we denote as gap) results including ERM and IRM results in box plot.
F.2.1 Full Results Filtered Boxplot (ERM)
What we want to find out the OOD generalization performance for models that perform better than random guess the ID set, i.c., models that have been trained. As Filtered Results, we define random guess 1/mum of classes in each problem setting, and show the results of climinating those models whose ID accuracy does not exceed this threshold.
29

---

Published in Transactions on Machine Learning Research (06/2023)
RotatedMNIST: ERM
RotatedMNIST: ERM
RotatedMNIST: ERM
1.0
1.0
1.0
citt-o-dlstribuuttt acc
0.8
in-distributionn acc
0.8
0.0
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
500
Momertum:
RMSSimpp
optimizer
optimizer name
optimizer name
ColoredMNIST: ERM
ColoredMNIST: ERM
ColoredMNIST: ERM
1.0
out Sut-aa-distrbbuttio acc
In-distribution.ac acc
0.0
0.6
0.4
0.4
0.2
0.0
0.0
RMSPop
Manerram Nestorer
Momertum
RMSPip
optimizer name
optimizer name
Bettmizer_namee
optimizer name
Figure 6. Filtered Results of RotaledMNIST and ColOredMNIST in DomainBed: Comparison of the in n-distributton (validation) accuracy and the out-of-distribution (test) accuracy of ERM across five optimizers.
30

---

Published in Transactions on Machine Learning Research (06/2023)
PACS: ERM
PACS: ERM
PACS: ERM
1.0
1.0
1.0
Nut-of-dlssrribuu acc
0.8
In-distribution acc
0.8
0.0
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
500
Momertum
optimizer name VLCS: ERM
optimizer name VLCS: ERM
optimizer name
1.0
1.0
VLCS: ERM
cut-of-distributtoon acc
0.8
n-distribution acc
0.8
0.75
0.6
0.6
0.4
0.4
0.25
0.2
0.2
0.0
0.0
-0.25
50D
Nomrmmum
Nembersv
RMiShhp
Adem
Momenttum Neetemy
SMSProp
Adbm
optimizer_nan _ame
optimizer
optimizer name
OfficeHome: ERM
OfficeHome: ERM
OfficeHome: ERM
1.0
Nutt-of-distrbburloo acc
0.8
i-ddssribution.. acc in-distribtion acc
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
optimizer_naa _ame
optimizer name
optimizer_nanee name
Terralncognita: ERM
Terralncognita: ERM
Terralncognita: ERM
1.0
fut-of-dlstributioo acc
0.8
0.6
0.4
Nomenum Nestrrs KVShop
Mamertam Nestaroy optimizer name DomainNet: ERM
Adam
Nomertum Nazidrox
hVStrop
optimizer name
optimizer name
DomainNet: ERM
DomainNet: ERM
1.0
cutt-of-dlstrbbutinn acc
0.8
(n-distrbution
0.6
0.0
500
Momenium Netera
NM5hap
Adam
5CD
Momentum
Nestarov
ANSPrpp
Adam
50D
Momartum
Nasidrow
Adum
optimizer_nanee _ame
optimizer_namee _naeee
optimizer_name _aame
Figure Filtered Results of PACS, VLCS, OfficeHome,, Terralncognita, and DomainNet in DomainBed: Comparison of the in-distribution (validation) accuracy and the out-of-distributtin (test) accuracy of ERM across five optimizers.
31

---

Published in Transactions on Machine Learning Research (06/2023)
Background Challenge
Background Challenge
Background Challenge
1.0
1.0
1.0
0.8
in in-distribution ace
Gut-of.ditributton
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
Momentum
Adam
Momentum
Adam
Momentum
Adam
optimizer, name
optimizer_nane name
optimizer name
Figure& 8. Filtered Results of Backgrounds Challenge: Comparison of the in-distribution (validation) accuracy and the dustr-init (test) accuracy of ERM across five optimizers.
WILDS amazon: ERM
WILDS amazon: ERM
WILDS am amazon: ERM
1.0
1.0
tit--o-dsstrbuttio acc
in-distiibution acc
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
500
Momenmum
Nesterxx
RMGhop
Adam
Momertum Nestoray
RMSPhsp
Adam
Nomentum
Nastarov
RMSPhop
Adam
optimizer_
optimizer
optimizer
WILDS civilcomments: ERM
WILD5 civilcomments:
ERM
WILDS civilcomments: ERM
1.0
1.0
1.0
out-of--lstributtin Cooluuttooo acc
0.8
T-ddstributonaac acc
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
Nomertum
optimizer_name _ame
optimizer_nann name
optimizer _aae
Figure 9. Filtered Results of Amazon-WILDD. and CivilComments- WILDS: Comparison of the in-distribution
(validation) accuracy and the out-of-distribution (test) accuracy ERM across five optimizers.
32

---

Published in Transactions on Machine Learning Research (06/2023)
F.2.2 Full Results of Filtered Boxplot (IRM)
PACS: IRM
PACS: IRM
PACS: IRM
1.0
cut-of-dlstributtooo acc
in-distribution acc
0.0
0.6
0.4
0.2
0.0
0.0
RMSPop optimizer name
Manieraam
Nomertum
RMSPro
optimizer name VLCS: IRM
optimizer name
VLCS: IRM
VLCS: IRM
out cot---f-lstrbbuutoo acc
1.0
0.8
0.8
in-distributin
0.5
gap
0.4
0.4
0.2
0.0
0.0
SGD
Momentumiiesterov RM5Prop Adam
optimizer_nee _aae
optimizer name
optimizer
OfficeHome: IRM
OfficeHome: IRM
OfficeHome: IRM
1.0
Nut-of-dlisribuutonn acc
n-distribution acc
0.8
0.6
0.4
0.0
5GD
Homemum
Nesteess
RMishop
Momentumm
Nestery
Admm
Nomentum
hVStrop
optimizer name optemizer_name
optimizer name
optimizer_name _nname
Terralncognita: IRM
Terralncognita: IRM
Terralncognita: IRM
Nut-aa--distibbutiin acc
1.0
n-dissribbutionaa acc
0.4
0.2
RMGFop
0.0
SGD
MomentumWesteroo RMSProp Adam
optimizer name
optimizer name DomainNet: IRM
optimizer
DomainNet: IRM
DomainNet: IRM
1.0
out aittaf-disrrbbttion acc
0.8
in-distributionnac
0.6
0.0
500
Momeruum Nestrrx
RVSPap
Man
56D
Mortertmm
Nesterw
AMSPIiO
Adom
500
Momerium
Nestorox
Asam
optimizer name
optimizer_nann name
optimizer_namee name
Figure 10. Filtered Results of PACS, VLCS, OfficeHome, Terralncognita and DomainNet in DomainBed: Comparison of the in-distribution (validation) accuracy and the out-of-distribution (test) accuracy of IRM across five optimizers.
33

---

Published in Transactions on Machine Learning Research (06/2023)
RotatedMNIST: IRM
RotatedMNIST: IRM
RotatedMNIST. IRM
1.0
1.0
1.0
Nut-of-dlstribbur acc
0.8
in-distribution acc
0.8
0.0
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
500
Momentum:
RMSSimpp
optimizer
optimizer name
optimizer name
ColoredMNIST: IRM
ColoredMNIST: IRM
ColoredMNIST: IRM
1.0
out: suttaa-distrrbuutin acc
In-distribution... acc
0.0
0.6
0.4
0.4
0.2
0.0
0.0
RMSPop
Mamerram Nestorer
Momertum
RMSPip
optimizer_ne name
optimizer name
optimizer name
Figure 11. Filtered Results of RolatedMNIST and ColoredMNIST in DomainBed: Comparison of the in in-istribbttonn (validation) accuracy and the out-of-distribution (test) accuracy of IRM across five optimizers.
WILDS amazon: IRM
WILDS amazon: IRM
WILDS amazon: IRM
1.0
but-of-dlstributto acc
Indistributionaa acc
0.0
0.6
0.4
0.2
0.0
RMGFrpp
optimizer name
optimizer name WILDS civilcomments: IRM
optimizer name
WILD5 civilcomments: IRM
WILDS civilcomments: IRM
1.0
out- out-of-de of-disstibbution acc
in-distributionaac
500
Homeruum
Nestrrx
RMSPop
Aan
SGD
Morrentum:
Nestorw
AMSPIID
Adom
500
Momerium
Nestorox
k0GPmp
Asam
optimizer name
optimizer_nann name
optimizer_nane __mmeeeemeee
Figure 12. Filtered Results of Amazon-WILDDS and CivilComments- WILDS: Comparison of the in-distribution (validation) accuracy and the out-of-distribution (test) accuracy of IRM across five optimizers.
34

---

Published in Transactions on Machine Learning Research (06/2023)
F.3
Full Results of Bin- Bin-Diage Diagrii Plots
CollrrdMMIST
RotatcdMNNSTERMM
PACS/ERM
VLCS:ERM
DfficeHome ERM
1.0
momeniam
0.6
0.66
0.6
0.5
0.55
0.4
0-44
0.4
0.4
0.2
0.0
0.00
6.25
0.50
0.75
1.00
9 6.25
0.5c
0.75
1.00
0
0.50
0.25
0.56
6.75
1.00
98
0.25
0.50
0.75
1.50
i-glstributton
aCc
Terf TerralinorritatE LarEM
DomainNet::S EMLM
Background Challenge
WILDS Amazon:ERM
1.0
WILDS
Ww.e--Brriiioon 805
mum-al--arttaa Mommtttttttemmmm acc
01 UF-01-30 sideroutton scc
OF Sptriuutoon acc
0.6
6.66
0.6
0.5
0.6
0.4
3.00
0.25
0,50
0.75
1.00
1.00
0.25
0.50
0.75
1.00
8..00
0.25
0.50
0.75
1.50
0.55
0.25
0.50
0.75
1.00
98.00
0.25
0.75 Indistribullon acc
0.50
1.00
in-diinibution acc
Bulitttbtttoonnnoo acc
aLc
aLc
Figure 13. DomainBed, Backgrounds Challenge, Amazon-Wiin WILDS, and CivilComments- WILDS: Comparison of the in-distribution accuracy and the out-of-distribution accuracy of ERM between Momentum SGD and Adam. Since Adam showed better OOD performance than RMSProp, Adam is presented as a representative of adaptive methods. Momentum SGD shows competitive performance in OOD with Vanilla SGD and Nesterov Momentum and representative of non-adaptive methods.
1.0 ColaredMNIST.RM
RotatedMNNSTT.Ie
PACS:IRM
VLCS-IRM
Officehlome:lkk
1.0
1.0
1.0
1.0
memeemmmgg adaim
0.8
0.0
0.8
0.6
2.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
1.0
80.00
8.00
88.00
In-distributton acc
indistrbultion
Indistributiee
Terrainrognita.io
DemainNet:IRM
WILDS Amazon:IAM
WILDS
civlicamments.iiee
1.0
1.0
1.0
1.0
act
0.4
0.4
0.4
0.4
8..00
0.25
1.00
00.00
0.25
3.50
0.75
00.00
0.75
0.000
0.25
0.75
Indistibuutonee
Figure 14. DomainBed, Amazon-WILDS, and CivilComments- WILDS: Comparison of the in-distribution accuracy and the out-of-distribution accuracy of IRM between Momentum SGD and Adam. Since Adam showed better OOD performance than RMSProp, Adam is presented as representative of adaptive methods. Momentum SGD shows competitive performance in OOD with Vanilla SGD and Nesterov Momentum and is representative of non-adaptive methods.
35

---

Published in Transactions on Machine Learning Research (06/2023)
F.4 Full Results of Scatter Plots
The results of the comparison of OOD accuracy and in-distribution accuracy shown in Section 4.3 are shown below for all benchmarks.
The box plots shown in Section F.2 and the Reliability Diagram Like Plot shown Section F.3 are based on the data shown in the Scatter Plot shown below.
F.4.1 ERM
ColoredMNIST:ERM
RotatedMNIST:ERM
uuttof-distribution acc
1.0
out of-isiiiiititoonnnnn acc
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc PACS:ERM
in-distribution acc
VLCS:ERM
cittof-distribuuiinn acc
1.0
nut-oo-distrribution acc
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc
in-distribution acc
OffIceHome:ERM
Terralncoogita:ERRM
iittof-distribution acc
1.0
distiiiiiititt acc
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc
in-distribution acc
Figure 15. DomainBed (ColoredMNISTT, RotatedMNIST, PACS, VLCS, OfficeHome and Terralncoguita:: Comparison of the in-distribution accuracy and the out-of-distribution accuracy of ERM across optimizers. The legend circles on the right side of each figure show, order, VanillaGGD, Momentum SGD, Nesterov Momentum, RMProp, and Adam. The difference in each data point indicates the difference in hyperparameter configuration.
36

---

Published in Transactions on Machine Learning Research (06/2023)
Background Challenge
DoMAinNet:ERM
1.0
offtoorriitiuuttion acc
1.0
out- -fstt-iiiiitttttion acc
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc
in-distribution acc
WILDS amazon: ERM
WILDS CILDD:Scvvic
1.0
1.0
tnttoffdistrbuttio acc
fut-of-distribution acc
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc
in-distribution acc
Comparison of the in-distribution accuracy and the out-of-distribution accuracy ERM across optimizors. The legend circles on the right side of each figure show, in order, VanillaSGD, Momentum SGD, Nesterov Momentum RMProp, and Adam. The difference each data point indicates the difference hyperparameter configuration.
Figure 16. DomainBed (DomainNet), Backgrounds Challenge and WILDS:
F.4.2 IRM
ColoredMNIST:IRM
RotatedMNIST.IRM
1.0
1.0
out- of-distribution acc
out -idistribbunnon acc
0.8
0.8
0.6
0.6
g0
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
in-distribution acc PACS:IRM
in-distribution acc
VLCS:IRM
out- out- of-distribbution acc
1.0
out- out- of-distributton acc
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc
in-distribution acc
Comparison of the in-distribution accuracy and the ont-of-distribution accuracy of IRM across optimizers. The legend circles on the right side of each figure show, in order, VanillaSGD, Momentum SGD, Nesterov Momentum, RMProp, and Adam. The difference in each data point indicates the difference in hyperparameter configuration.
Figure 17. DomainBed (ColorcdMNIST, RotatedMNIST. PACS and VLCS):
37

---

Published in Transactions on Machine Learning Research (06/2023)
IMM
Terralncognita.IRRM
of-toffriibbution acc off-offfiiirtbuttin acc
1.0
out-of-disste -iit----iitiitttionn acc
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc
in-distribution acc
DomainNet:IRM
WILDS amazon:IRM
1.0
1.0
lott----iistibuuiio acc
0.8
0.8
0.6
0.6
0.4
0.4
0.2
out
0.2
0.0
0.8.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc
in-distribution acc
WILDS
IR
1.0
out-of-distribution acc
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
in-distribution acc
Figure 18. Ded (OfficeHome, Terralncoguita, DomainNet) and WILDS: Comparison of the in- distribution accuracy and the out-of-distributiio accuracy of IRM across optimizers. The legend circles on the right side of each figure show, in order, VanillaSGD, Momentum SGD, Nesterov Momentum, RMProp, and Adam. The difference in each data point indicates the difference in hyperparameter configuration.
38

---

Published in Transactions on Machine Learning Research (06/2023)
G
Ablation Study
In addition to the experimental results presented in the main paper, we provide more in-depth analysis, which is shown in this chapter.
First of all, We show that the pattern of linear correlation of accuracy claimed by (Miller et al., 2021) does not necessarily oceur even when the correlation patterns we observed, such as diminishing returns, are probit transformed. The results of the probit transform of the scatter plots shown in Appendix F.4 are shown, following the method of (Miller al., 2021). This may be due to the fact that our experimental setup uses a larger number of data sets and takes IRM and other factors into account.
In the second section, present results comparing the performance improvement of OOD with larger trial budget in the search for hyperparameters by Bayesian optimization.
Thirdly, We investigale why Adam shows better OOD performance than SGD in the negatively correlated case seen in Figure for ColoredMNIST by considering the learning curve.
Finally, we examine the effectiveness of the carly stopping for cach optimizer study if adaptive optimizers overfit because of their speed of convergence.
G.1 Probit Transformed Scatter Plot
As shown in Section 4.3, the work of (Miller et al., 2021) compares the accuracy of OOD with the accuracy of in-distribution by showing a plot on a probit scale. Their comparison shows that the correlation of accuracy is linear.
In this section, convert the scatter plots identified in Scction F.4 to probit scale and confirm that they do not necessarily show linear correlation (Figure 19, 21, 22, 23, 23, and 24).
G.1.1 ERM
ColoredMNIST:ERM (probit scaling)
RotatedMNIST:ERN (probit scaling)
0.95
0.50
out- sott-f-ddsttibutiio acc
Nesters ce Vomentum 5GD
out- cifoddistrbuutionnnneo acc
0.90
hesterov Homentum SGD
RMSSProp
BHSShap
0.70
0.25
0.50
0.10
0.10
0.50
0.70
0.80
0.10
0.25
0.70 0.80
0.50 0.95
0.99
in-distribution
in-distribution acc
Figure 19. Probit Scale / DomainBed (CoIoredMNIST and RotatedMNIST)): Comparison of the in-distribution accuracy and the out-of-distribution accuracy of F.RM across optimizers. The difference in each data point indicates the difference in hyperparameter configuration.
39

---

Published in Transactions on Machine Learning Research (06/2023)
PACS:ERM (probit scaling)
ELC (probit scaling)
0.90
0.90
SG3
out-of-distribbll crsof-ddstributton acc
0.80
Neate Homentum
out-of-distributoen acc
Nastarov
SGD
1015htop
0.95
IMMSh
Adam
0.90
Adam
0.80
0.50
0.70
0.25
0.10
0.10
0.10
0.25
0.50
0.70
0.80
0.90
0.95
0.10
0.25
0.50
0.70
0.80
in-distribution
in-distribution acc
OfficeHome: ERM (probit scaling)
DomainNet: ERM (probit scaling)
0.70
SG0
Murrotan SGD
Mari turn SGD
out-of-distrlbution acc
0.50
Noste uo Homentu 560
out-offdistribbttion acc
Nesterw Homentum 560
0.25
0.25
0.30
0.50
in-distribution acc
in-distribution acc
TerraIncognita:ERM (probit scaling)
SGD
0.50
out-of-distributtio acc
Nestorce Homentam
Reespop
Adam
0.10
0.10
0.25
0.50
0.70
0.50
in-distribution
Figure 20. Probit Scale / DomainBed (PACS, VLCS, OfficeHome, DomainNet and Terralncoguita): Com- parison of the in-distribution accuracy and the ont-of-distribution accuracy of ERM across optimizers. The difference each data point indicates the difference in hyperparameter configuration.
40

---

Published in Transactions on Machine Learning Research (06/2023)
Background Challenge (probit scaling)
0.80
Homentum SGD
out-of-distribuui acc
0.70
0.50
0.70
0.80
0.90
0.95
in-distribution
Figure 21. Probit Scale / Backgrounds Challenge: Comparison of the in-distribution accuracy and the out-of-distribution accuracy of ERM across optimizers. The difference in each data point indicates the difference hyperparamoter configuration.
HILDS_amazon:ERM (probit scaling)
WILDS Civilcomments:ERM (probit scaling)
0.50
Mome turn 5GD
out-of-distributte acc
en iMSthop Adam
out-of-distribunn didddistributiion acc
Nosterty Manoniant SLD
0.25
sil
0.25
0.50
0.70
0.50
0.70
0.80
0.90
In-distribution
In-distribution acc
Figure 22. Probit Scale / WILDS: Comparison of the in-distribution accuracy and the out-of-distribution accuracy of ERM across optimizers. The difference in cach data point indicates the difference in hyperparameter configuration.
41

---

Published in Transactions on Machine Learning Research (06/2023)
G.1.2 IRM
ColoredMNIST:IRN (probit scaling)
RotatedMNIST:IRM (probit scaling)
SGD
Vemertum SGD
out-of-distribution acc
0.50
out-of-distributin acc
Adam
0.80
0.25
0.10
0.10
0.50
0.70
0.80
0.10
0.25
0.50
0.70
0.50
0.90
0.95
0.99
in-distribution acc PACS:IRM (probit scaling)
In-distribution acc
VLCS:IRM (probit scaling)
0.99
0.80
Mumoonan SGD
Mari turn SGD
out-of-distrlaution acc
Noste to Homentu 560
out-of-distributioo acc
0.95
Nesteriw Momentum 560
0.70
0.90
0.50
0.50
0.10
0.95
0.25
0.70
in-distribution acc
in-distribution acc
OfficeHome:IRI (probit scaling)
DomainNet: IRM (probit scaling)
0.70
SG3
0.50
Put-of-diistribution acc
0.50
es
out-of-distriin dif-iiiiiiibuttion acc
RMSiip op
0.25
Adum
Adam
0.25
0.10
0.10
0.25
0.50
0.70
0.90
0.10
0.25
0.50
in-distribution acc
in-distribution acc
Terralncognita:1RM (probit scaling)
0.50
Pomentam 5GD
out-of-distren cott---distribttion acc.
Nestarce Mamontan sCo
0.25
0.10
0.70 0.00
In-distribution acc
Figure 23. Probit Scale / DomainBed (ColoredMNIST., RotatedMNIST, PACS, VLCS, OfficeHome, Do- mainNed and TerraIncognita): Comparison of the in-distribution accuracy and the ofits--distiinttiin accuracy of IRM across optimizers. The difference in each data point indicates the difference in hyperparameter configuration.
42

---

Published in Transactions on Machine Learning Research (06/2023)
WILDS_ WILDSSammazon:RN (probit scaling)
MLLDS_ClVlcommeets::RMM (probit scaling)
put-of-distribution acc
cn Nomentum 1915hrop Adam
out- distributtion acc
SCD
0.10
Nateroy
Iuthhpp
Adam
0.10
0.25
0.50
0.10
0.25
0.50
0.70
0.80
0.90
In-distribution
in-distribution acc
Figure 24. Probit Scale / WILDS: Comparison of the in-distribution accuracy and the out-of-distribution accuracy of IRM across optimizers. The difference in each data point indicates the difference in hyperparameter configuration.
43

---

Published in Transactions on Machine Learning Research (06/2023)
G.2 Model Performance Transition throught Hyperparameter Search
For practitioners, we show how much the trial budget for hyperparameter optimization affects OOD general- ization.
G.2.1
Hyperparameter Trial Budget vs OOD Accuracy
DomainNet ERM Momentum
DomainNet ERM Adam
1.0
out-distribution acc
0.8
out-distribution acc
0.8
0.6
0.4
0.4
0.2
top
0.2
top
top 10 average
top 10 average
all average
all average
trials OfficeHome ERM Momentum
trials
OfficeHome ERM Adam
1.0
1.0
out-distribution acc
Aut-distributionn.. acc
0.6
0.6
0.4
0.4
top-10 average
top-10
all average
all average
0.0
0.0
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
trials
trials
Terralncognita ERM Momentum
Terraincognita ERM Adam
1.0
1.0
out- sittdisstrbbtion acc
0.8
fut-distribution acc
0.8
0.6
0.6
0.2
top-1
0.2
top-1
average
top-10 average
0.0
0.0
25
75
100 trials
125
150
175
200
25
50
100
125
150
175
200
trials
Figure 25. ERM DomainNet, OfficeHome and TerraIncognita / OOD accuracy when horizontal axis is the trial budget
44

---

Published in Transactions on Machine Learning Research (06/2023)
ColoredMNIST ERM Momentum
ColoredMNIST ERM Adam
1.0
1.0
out-distribution acc
0.8
Out-distribution acc
0.8
0.6
0.6
0.4
0.4
tap-1
teg-10 average
top-10 average
all average
all average
0.0
0.0
0
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
trials
trials
RotatedMNIST ERM Momentum
RotatedMNIST ERM Adam
1.0
1.0
tut-distribution acc
aut-distribution acc
0.8
0.6
0.6
0.4
0.2
top-1
0.2
top-1
average
top-10 average
all average
all
0.0
0.0
25
75
100
125
150
175
200
25
50
100
125
150
175
200
trials
trials
ERM Momentum
VLCS Adam
1.0
1.0
out-distrlbution acc
0.8
out-distribution acc
0.8
0.4
0.4
0.2
top-1
0.2
top-1
top 10 average average
top-10 average
average
125
150
175
100
125
150
175
200
trials
trials
PACS ERM Momentum
PACS ERM Adam
1.0
1.0
tut-distribution acc
0.8
aut-distribution acc
0.8
0.6
0.6
0.4
0.4
1 top-10 average all average
top-10 average all average
0.0
0.0
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
trials
trials
Figure 26. ERM ColoredMNISTT, RotatedMNISTT, VLCS and PACS / OOD accuracy when horizontal axis is the trial budget
45

---

Published in Transactions on Machine Learning Research (06/2023)
Background Challenge Momentum
Background Challenge Adam
1.0
1.0
out-distribution acc
Nut-distributton acc
0.8
0.8
0.4
0.2
top 1
0.2
top-1
top-10 average
top-10 average
average
average
0.0
125
200
trials
trials
Figure 27. ERM Background Challenge / OOD accuracy when horizontal axis is the trial budget
WILDS Amazon ERM Momentum
WILDS Amazon ERM Adam
1.0
1.0
out-distribution acc
cit-distribution acc
0.8
0.4
0.4
0.2
tap-1
top-1
top-10 average average
top-10 average
all average
trials
trials
WWLLDS_ciillcommennn ERM Momentum
WILDS ciniicmmivls ERM Adam
1.0
nut-distribution acc
out- distributiion acc
0.8
0.6
0.4
0.2
top average average
top-10 average
all average
0.0
0.0
25
50
75
100
125
150
175
200
75
100
125
150
175
200
trials
trials
Figure 28.
ERM WILDS Amazon and WILDS CivilComment / OOD accuracy when horizontal axis is the
trial budget
G.2.2 Hyperparameter Trial Budget vs 0OD Error
To visualize the slight increase the end of OOD accuracy more clearly, we visualized in log scale as OOD
46

---

Published in Transactions on Machine Learning Research (06/2023)
DomainNet ERM Momentum
DomainNet ERM Adam
out-distribution error
dut-distribbttion error
10-1
10-1
top
top -1
top average
average
all average
all average
10
10
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
trials
trials
OfficeHome ERM Momentum
OfficeHome ERM Adam
out-distribution error
out dit-iistri error
10-1
10-1
top 1
tap
top average
top average
average
average
10
10
25
75
100
125
150
175
200
25
50
75
100
125
150
175
200
trials
trials
Terralncognita Momentum
Terralncognita Adam
out-distribution error
out out-distribute sit-distribution error
10-1
10-2
top-1
top-1
top average
top 10 average
average
average
175
125
trials
trials
Figure 29, ERM DomainNet, OfficeHome and Terralncognita / 00D) error when horizontal axis the trial budget
47

---

Published in Transactions on Machine Learning Research (06/2023)
ColoredMNIST ERM Momentum
ColoredMNIST ERM Adam
out-distribution error
outtdistributton error
10-1
10-1
top
top -1
top average
average
all average
all average
10
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
trials
trials
RotatedMNIST ERM Momentum
RotatedMNIST ERM Adam
out-distribution error
out- ditrristti error
10-1
10-1
top
tap-1
top average
top average
average
average
10
10
25
75
100 trials
125
150
175
200
25
50
75
100
125
150
1.75
200
trials
Momentum
VLCS ERM Adam
out-distribution error
out out-distribution error
10-1
10-2
top-1
top average
top 10 average
average
average
175
200
125
trials
trials PACS ERM Adam
PACS ERM Momentum
out-distribution error
out out-distribution error
10-1
top-1
top-10 average
top-10 average all average
average
25
100
125
150
175
200
25
50
100
125
150
175
200
trials
trials
Figure 30. ERM ColoredMNISTT RotatedMNIST, VLCS and PACS / OOD error when horizontal axis the trial budget
48

---

Published in Transactions on Machine Learning Research (06/2023)
Background Challenge Momentum
Background Challenge Adam
out-distribution error
dut-distribbttion error
10-1
10-1-
top
top
top average
average all average
all average
10
0
25
50
75
100 trials
125
150
175
200
25
50
75
100 trials
125
150
175
200
Figure 31. ERM Background Challenge / OOD accuracy when horizontal axis is the trial budget
WILDS Amazon ERM Momentum
WILDS Amazon ERM Adam
but-distribution error
out distributio error
10-1
10-1
top-1
top-1
top average all average
top-10 average
all average
10-2
10:2
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
trials
trials
WILDS civilcommeesss ERM Momentum
WILDS WWL.DS_civllcoommenn ERM Adam
out-distribution error
out dittdiiiiibttonn error
10-1
Qut-distha
10-1
top-1
top-1
top- average average
top-10 average
average
10-2
10:2
25
75
100
125
150
175
200
25
50
75
100 trials
125
150
175
200
trials
Figure 32. ERM WILDS Amazon and WILDS CivilComment OOD accuracy when horizontal axis is the trial budget
49

---

Published in Transactions on Machine Learning Research (06/2023)
G.3 Learning Curve of ColoredMNIST
In Section 4.2, we conjecture that the better performance of Adam in Colored MNIST classification may come from overfitting to training data. To confirm this hypothesis, we plot the averaged training accuracy, the averaged validation accuracy, and the averaged test accuracy throughout the training. We pick the top-14 results in terms of test accuracy and use them for the plot. We show the result in Figure 33.
As is evident from Figure 33 (a) and Figure 33 (b), observe that training accuracy increases while the validation accuracy keeps unchanged. This indicates that overfitting occurs in the training. However, Figure 33 (c) indicates that test accuracy gradually improves as the model is overfitting, On the other hand, SGD shows no such overfitting and OOD generalization improvement, as shown in Figure 34. Thus, we can empirically support our conjecture that Adam produces the better OOD generalization performance by overfitting training data.
ERM ColoredMNIST Adam
ERM ColoredMNIST Adam
1.0
trial-1
average validation accuracy
1.0
trial-1
average train accuracy
0.9
trial-2
0.9
trial-2
trial.3
0.8
trial-4
0.8
trial-4
trial-5
triak-5
trial-7
trial-7
trial-8
0.5
trial-8
trial-9
0.4
trial-10
0.4
trial-10
trial-11
trial-11
0.3
trial-12
0.3
trial-13
0.2
triel-13
trial-14
trial-14
1000
2000
3000
4000
5000
steps
steps
(a) ERM: Averaged training accuracy Adam on Colored
(b) ERM: Averaged validation accuracy Adam on Col-
MNIST throughout the training.
ored MNIST throughout the training.
ERM ColoredMNIST Adam
1.0
average test accuracy
trial-2
trial:3
0.7
trial-5
trial-6
0.6
trial.8
0.5
trial-9
0.4
trial-11
trial-12
trial-13
trial-14
0.1
1000
2000
3000
4000
5000
steps
(c) ERM: Averaged test accuracy Adam on Colored
MNIST throughout the training.
Comparison of averaged training accuracy, averaged validation accuracy, and averaged test. accuracy throughout the training, We plot these values of check points whose train accuracy is in the top-14.
Figure 33. Adam:
50

---

Published in Transactions on Machine Learning Research (06/2023)
ERM ColOredMNIST vanilla SGD
ERM ColoredMNIST vanilla SGD
1.0
trial-1
average jalidation accuracy
1.0
triai-1
average train accuracy
0.9
trial-2
0.9
trial.2
0.8
trial-4
0.8
trial.4
trial-5
0.7
trial5
trial-6
trial-7
trial-7
0.5
triat-8
trial-9
trial-9
0.4
trial-10
0.4
triel-10
0.3
trial-12
0.3
trial-12
0.2
trial-13
0.2
trial-13
trial-14
0.1
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
steps
steps
(a) ERM: Averaged training accuracy of SGD on Colored MNIST throughout the training-
(b) ERM: Averaged validation accuracy of SGD Col-
ored MNIST thronghout the training.
ERM ColoredMNIST vanilla SGD
1.0
average test accuracy
0.9
trial-2
trial-3
trial.5
0.6
trial-7
0.5
trial-8
trial-9
0.4
trial-10
trial-11
trial-12
trial-13
trial-14
1000
2000
5080
steps
(c) ERM: Averaged test accuracy SGD on Colored MNIST throughout the training.
Figure 34. SGD: Comparison of averaged training accuracy, averaged validation accuracy, and averaged test accuracy of SGD throughout the training. We plot these values of checkpoints whose train accuracy is in the top-14.
G.4 Early Stopping
In Section 4.2, figures shows that adaptive optimizers tend to overfit to training domain. A possible reason is that the training speed of adaptive methods is faster than non-adaptive ones. That is, in the same steps budget, adaptive optimizers converge faster in effect. To validate if this is the case, we investigate whether early stopping improves the OOD generalization of adaptive optimizers.
In particular, compute the difference between averaged accuracy at carly stopping and at last epoch for test accuracy and validation accuracy, respectively:
No.
Ni.
1
Accaiff
Acces-i
Acciei
(18)
Nos
Nic
1=1
where Accait represents the difference in accuracy between early stopping and the last epoch, Nes is the number of trials at early stopping, Nie the number of trials at the last epoch, Accest denotes the accuracy at the carly stopping for the th trial, Acqe- denotes the accuracy the last epoch for the i th trial.
If the average accuracy at early stopping larger, the difference is positive and vice
51

---

Published in Transactions on Machine Learning Research (06/2023)
Figures 35 to 40 are the results of this comparison for each dataset and algorithm. The y-axis is the difference of out-of-distribution (test) accuracy and the x-axis is the difference of in-distribution (validation) accuracy. The color indicates the epoch of early stopping, The darker color indicates that early stopping is conducted in relatively carlier epochs.
For PACS, both differences are positive and lighter colors are concentrated at points small difference of the validation accuracy, as indicated in Figures 35 and 36. Therefore, can conclude that when the validation accuracy deteriorates by further training, the test accuracy also gets worse. However, the effect of further training is less evident for Adam. In other words, early stopping does not influence Adam much but keeps test accuracy from decrcasing for SGD.
The result of VLCS shows a similar pattern as PACS (Figures 37 and 38). If anything, Further training after early stopping makes adaptive optimizes be likely to result in better test accuracy than SGD, though they have a large variance. With respect SGD, additional training degrades the test accuracy much as it degrades validation accuracy.
Office-Horee shows similar results PACS, as presented in Figures 39 and 40.
In summary, we find that early stopping does not influence adaptive optimizers. Therefore, we can conclude that adaptive optimizer overfits not because it trains faster than non-adaptive methods.
The fact that early stopping does not affect the adaptive optimizer means that adjusting the number of epochs instead of fixed number of epochs will not change the result. We have followed previous studies and experimented with fixed number of epochs in the present study, and results suggest that this does not have serious impact the comparison of adaptive and non-adaptive optimizers. Thus, we using a fixed epoch number does not undermine the validity of our optimizer comparison experiment in this sense.
that
52

---

Published in Transactions on Machine Learning Research (06/2023)
vanilla_sgo sorted by percentage of early stopping momentum_sger sorted by percentage of early stopping
difference of test accuracy difference of test accuracy
difference of test accuracy difference of test accuracy
ERM PACS |Acc @Early Stopping Acc @last Epoch)
ERM PACS (Acc @Early Stopping Acc @ast Epoch)
1.0
0.8
0.8
0.8
0.6
0.4
0.2
0.4
0.0
0.2
0.2
-0.2
-0.2
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
difference of validation accuracy
nesterov nesterov_momentune www._/mmmmmemssseee sgd sorted by percentage of early stopping
mmsprop sorted by percentage of early stopping
ERM PACS (Ace @Early Stopping Acc @aast Epoch)
@Early Stopping
@Last Epoch)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
-0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
difference of validation accuracy
adam sorted by percentage of early stopping
difference of test accuracy
ERM PACS (Acc @Early Stopping Acc @Last Epoch)
0.8
0.4
0.2
-0.2
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
Figure 35. ERM/PACS: Difference between accuracy at early stopping and at last epoch. The y-axis is the difference out-of-distribution (test) accuracy and the x-axis the difference of in-distribution (validation) accuracy. The color indicates the epoch of carly stopping. The darker color indicates that early stopping is conducted in relatively earlier epochs.
53

---

Published in Transactions on Machine Learning Research (06/2023)
vanilla_sgo sorted by percentage of early stopping momentum_sger sorted by percentage of early stopping
difference of test accuracy difference of test accuracy
difference of test accuracy difference of test accuracy
IRM PACS (Acc @Early Stopping - Acc @last Epech)
IRM PACS (Acc @Early Stopping Acc @Last Epoch)
1.0
0.8
0.8
0.6
0.4
0.2
0.0
0.2
0.2
-0.2
0.0
-0.2
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
difference of validation accuracy
nesterov, nesterov_mamentumg wwt.mmmmmmtmssste sgd sorted by percentage of early stopping
mmsprop sorted by percentage of early stopping
IRM PACS (Acc Early Stopping Acc @ Last Epech)
@Early Stopping Acc Epoch)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
-0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
difference of validation accuracy
adam sorted by percentage of early stopping
difference of test accuracy
IRM PACS (Ace @ Early Stopping Acc Last Epoch)
0.8
0.4
0.2
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
Figure 36. IRM/PACS: Difference between accuracy at early stopping and at last epoch. The y-axis is the difference out-of-distribution (test) accuracy and the x-axis the difference of in-distribution (validation) accuracy. The color indicates the epoch of carly stopping. The darker color indicates that early stopping is conducted in relatively earlier epochs.
54

---

Published in Transactions on Machine Learning Research (06/2023)
vanilla_sgo sorted by percentage of early stopping momentum_sger sorted by percentage of early stopping
difference of test accuracy difference of test accuracy
difference test accuracy difference of test accuracy
ERM VLCS |Acc @earlyy Stopping Acc @aat Epoch)
ERM VLCS (Acc @Early Stopping Acc @Last Epoch)
1.0
0.8
0.8
0.6
0.6
0.4
0.2
0.4
0.0
-0.2
-0.2
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
difference of validation accuracy
nesterov, nesterov_mamentinn wwt.m/mmmmmmsssseee sgd sorted by percentage of early stopping
mmsrrop sorted by percentage of early stopping
ERM VLCS @ Stopping Acc @Last Epoch)
@Early Stopping @ast Epoch)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0..2
0.0
-0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
difference of validation accuracy
adam sorted by percentage of early stopping
difference of test accuracy
ERM VLCS (Acc @Early Stapping Acc @Last Epoch)
0.8
0.4
0.2
-0.2
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
Figure 37. ERM/VLCS: Difference between accuracy at early stopping and at last epoch. The y-axis is the difference out-of-distribution (test) accuracy and the x-axis the difference of in-distribution (validation) accuracy. The color indicates the epoch of carly stopping. The darker color indicates that early stopping is conducted in relatively earlier epochs.
55

---

Published in Transactions on Machine Learning Research (06/2023)
vanilla_sgo sorted by percentage of early stopping momentum_sger sorted by percentage of early stopping
difference of test accuracy difference of test accuracy
difference of test accuracy difference of test accuracy
IRM VLCS (Acc @Early Stopping - Acc @last Epoch)
IRM VLCS (Acc @Early Stopping @Last Epoch)
1.0
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
-0.2
0.0
-0.2
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
difference of validation accuracy
nesterov, nesterov_momenteee htt.mmmmmmmmssstem sgd sorted by percentage of early stopping
mmsprop sorted by percentage of early stopping
IRM VLCS (Acc @Early Stapping Act @Lat Epoch)
@Early Stopping Epoch)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
-0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
difference of validation accuracy
adam sorted by percentage of early stopping
difference of test accuracy
IRM VLCS (Acc @arlyy Stopping Acc @Last Epach)
0.8
0.4
0.2
-0.2
0.0
0.4
0.6
0.8
1.0
difference of validation accuracy
Figure 38. IRM/VLCS: Difference between accuracy at early stopping and at last epoch. The y-axis is the difference aatoof-distribution (test) accuracy and the x-axis the difference of in-distribution (validation) accuracy. The color indicates the epoch of carly stopping. The darker color indicates that early stopping is conducted in relatively earlier epochs.
56

---

Published in Transactions on Machine Learning Research (06/2023)
vanilla_sgo sorted by percentage of early stopping momentum_sge sorted by percentage of early stopping
difference of test accuracy difference of test accuracy
difference of test accuracy difference of test accuracy
ERM OfficeHiome (Acc @Early Stopping - Acc @aast Epoch)
ERM OfficeHome (Acc @Early Stopping @Last Epoch)
1.0
0.8
0.8
0.8
0.6
0.1
0.4
0.2
0.0
0.2
0.2
-0.2
-0.2
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
difference of validation accuracy
nesterov nesterov_mamentune www._mmmmmmmmmmmmmme sgd sorted by percentage of early stopping
mmsrrpp sorted by percentage of early stopping
ERM OfficeHome (Acc @Early Stopping Act Last Epoch)
(Acc @aally Stopping
@Last Epoch)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
-0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
difference of validation accuracy
adam sorted by percentage of early stopping
difference of test accuracy
ERM OfficeHame (Ace @Early Stopping Acc @Last Epoch)
0.8
0.4
0.2
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
Figure 39. ERM/Office-Hone Difference between accuracy at early stopping and at last epoch. The y-axis is the difference of out-of-distribation (test) accuracy and the x-axis is the difference of in-distribution (validation) accuracy. The color indicates the epoch of carly stopping. The darker color indicates that carly stopping is conducted in relatively earlier epochs.
57

---

Published in Transactions on Machine Learning Research (06/2023)
vanilla_sgo sorted by percentage of early stopping momentum_ssee sorted by percentage of early stopping
difference of test accuracy difference of test accuracy
difference of test accuracy difference of test accuracy
IRM Officehome (Acc @Early Stopp Ing Acc @last Epoch)
IRM Officettome (Acc @Early Stopping Acc @Last Epoch)
1.0
0.8
0.8
0.6
0.4
0.2
0.0
0.2
0.2
-0.2
0.0
-0.2
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
difference of validation accuracy
nesterov nesterov_mamnntumg wwt.mmmmmmmtmmmmmmm sgd sorted by percentage of early stopping
mmsrrop sorted by percentage of early stopping
IRM Officeltome (Acc Early Stopping Acc @laat Epoch)
(Ace @Early Stopping
@Last Epoch)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
-0.2
0.0
-0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
difference of validation accuracy
adam sorted by percentage of early stopping
difference of test accuracy
IRM Officellome (Acc @arlyy Stopping Acc @last Epach)
0.8
0.4
0.2
-0.2
0.0
0.4
0.6
0.8
1.0
difference of validation accuracy
Figure 40. IRM/ OfM/OfficeHomm:: Difference between accuracy at early stopping and at last epoch. The y-axis is the difference of out-of-distribation (test) accuracy and the x-axis is the difference of in-distribution (validation) accuracy. The color indicates the epoch of carly stopping. The darker color indicates that carly stopping is conducted in relatively earlier epochs.
However, we find different results from Colored MNIST. As shown in Figures and 42, we can observe that the difference of validation accuracy positive and that of test accuracy nogative. That further training after early stopping decreases validation accuracy but increases accuracy on Colored MNIST, while there is difference for SGD. This is consistent. with the result of Appendix G.3.
58

---

Published in Transactions on Machine Learning Research (06/2023)
vanilla_sso sgd sorted by percentage of early stopping momentum __d sorted by percentage of early stopping
difference of test accuracy difference of test accuracy difference of test accuracy
difference of test accuracy
ERM ColoredMNIST (Acc @Errly Stopping Acc @aat Epoch)
ERM ColoredMNIST (Acc Early Stopping Acc Last Epoch)
1.0
0.8
0.6
0.4
0.2
0.2
0.0
0.2
0.0
-0.2
-0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy
difference of validation accuracy rmsprop sorted by percentage of early stopping
nesterov, httpo/ummmmmmmmmmmmmm sgd sorted by percentage early stopping
difference test accuracy
(Acc @Early Stopping Acc Last Epach)
ERM ColoredMNIST (Acc @Early Stopping Acc @ast Epoch)
1.0
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.0
0.2
0.0
-0.2
-0.2
0.0
0.0
0,2
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
difference of validation accuracy adam sorted by percentage of early stopping
difference of validation accuracy
ERM ColoredMNIST (Acc @Early Stopping Acc Laat Epoch)
1.0
0.8
0.6
0.4
0.2
0.0
0.6
difference of validation accuracy
Figure 41. ERM/Colored MNIST: Difference between accuracy at early stopping and at last epoch. The y-axis is the difference of dfitsiibbitttiionn (test) accuracy and the x-axis the difference of in-distribution (validation) accuracy. The color indicates the epoch of early stopping. The darker color indicates that early stopping is conducted in relatively earlier epochs.
59

---

Published in Transactions on Machine Learning Research (06/2023)
vanilla_sgo sorted by percentage of early stopping momentum_sspe sorted by percentage of early stopping
difference of test accuracy difference of test accuracy difference of test accuracy
difference of test accuracy difference of test accuracy
IRM ColoredMNIST (Acc @Early Stopping Acc @last Epoch)
IRM Colored/MNIST (Acc @Barly Stopping Acc @Last Epoch)
1.0
0.8
0.8
0.3
0.6
0.2
0.4
0.2
0.0
0.1
0.2
-0.2
0.0
-0.2
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
difference of validation accuracy
difference of validation accuracy
nesterov Eittrro_mammentmmmo sgd sorted by percentage of early stopping
msprop sorted by percentage of early stopping
IRM CaloredMNIST (Acc @ Early Stapping Acc @Last Epoch)
ColoredMNIST @Early Stopping
Epoch)
1.0
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.0
0.2
0.2
0.2
0.0
-0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
difference of validation accuracy adam sorted by percentage early stopping
difference of validation accuracy
ColoredMNIST (Acc @Early Stopping Acc @last Epoch)
1.0
0.8
0.6
0.4
1.0
difference of validation accuracy
Difference between accuracy at early stopping and at last epoch. The y-axis the difference out-of-distribution (test) accuracy and x-axis is the difference of in-distribution (validation) accuracy. The color indicates the epoch of early stopping. The darker color indicates that early stopping is conducted in relatively earlier epochs.
Figure 42. IRM/Colored MNIST:
60

---

Published in Transactions on Machine Learning Research (06/2023)
H Soundness Check of Our Experiments
H.1 Histgram of Hyperparameters
We have shown the histgram of the hyperparamcters to be used for training just for reference. In particular, we display & result. for learning rate Momentum SGD and Adam for the each dataset. We observe that we could sample hyperparameters from a reasonably wide range.
ERM ColoreMNIIST
ERM RotatedMNSST
ERN PACS
300
netrentumspp 5gd
numcrtuu
sdam
330
sdlm
acam
230
230
150
150
400
100
130
16-
Learing
Leam ng Rate ERM, Officellome
Learting
ERM, VLCS
Terrabicaghhi
momerbum sot
namantur not
2200
200
3.00
2300
400
3300
100
3300
53
10""
10
10m4
16
1.99
1244
12-4
10-4
12-3
Launing tabe BRM_ Domainett
Pate ERM WiLDS_amaeon
Learinn tane
ERM_ WWt.:WWLDSSC. chlicomments
mementumm no1 NOWT
momeenum spe kdem
nomerbur ro
730
200
250
500
330
100
230
50
Figure 43. Histogram of explored hyperparameters (learning rate) in training of DomainBed, WILDS dataset in FRM. Momentum SGD and Adam results are included. Although uniform distribution is used the prior distribution for hyperparameter optimization, the histogram results do not match the uniform distribution because Bayesian optimization is used.
H.2 Hyperparameters and OOD Accuracy Box-Plot
The previous section provided information on the hyperparameter search range. In this section, we share the results of the out-of-distribution performance for a specific hyperparameter range box plot with the hyperparameters separating the bin shown in Figure 44, 45 and 46.
From those results, we observe that lr, which achieves high out-of-distribution accuracy, is within the search range of learning rate and thus has sufficient range to perform the search.
61

---

Published in Transactions on Machine Learning Research (06/2023)
momentum
out-of-distriiue stristutttoonn.oomm aCc
out of di strituttoon acc
6.30
0448 2,te-t
zanent)
ERM RotatedMNIST momentum sqd
ERM RotatedMNIST adam
sitt---dditribudtion acc
outt www.diiiiutiooo Instribbttionom apc
Letea))
AUROOA Aalber))
1640444 (01)
domen col1
gol zamm
AGAAEE 2.01
ERM, PACS momentum sgd
ERM.
PACS ERM_PACS
adam
calt-of-diserbbniii aCc
aut of diccni acc
the et, zedeet
isghoat saee
Gl40444 ml)
the Te-8..2 Bo
(4 apoust
ERM VLCS momentum sqd
ERM
cuf-of-distribuuuee aCC
out- of distribuuui acc
Best, 1100. 215ea
Figure 44. Pox-Plot of RotatedMNIST, PACS and VLCS
Accuracy per Log-Scale of Learning Rate: ColoredMNNIST,
62

---

Published in Transactions on Machine Learning Research (06/2023)
ERM OfficeHome, mamentum sgd
ERM OfficeHome, adam
out-of-dlstrrbunne arc
cut of distributtionn acc
Gile (akett)
ssees Teaotay
2011
dite 444-09
aMedd 20001)
Terralncognita momentum sgd
Terralncognita
Swttoofdlsttrbbuuiin &CC
out wel.-distrbbutiinnommm acc
MaNNI sambeti
UMM0E 280)
1G0 canzio
zamoo
0 (obaa))
ERM DomainNet momentum sgd
ERM DomainNet adam
O4l--o-distrib wesooooitibbbttoonn BCC
outb distribution aDC
JERRCIA candees)
call
Plaut zeles)
Coll
Figure 45. Box-Plot of Distr--Dittiitiion Accuracy per Lcg-Scale of Learning Rate: OfficeHome, Ter- raIncognita, and DomainNet
63

---

Published in Transactions on Machine Learning Research (06/2023)
ERM WILDS amazon, mamentum, sgd
ERM WILDS amazon adarn
Wat-of-distribuutio a00
out of distribbuttion apc.
620
(14tett Jut0n
camt
dett 2e0tes
deatt GUMAA
ERM WILDS, civil comments momentum sgd
ERM WILDS civil comments adam
Nuft-of-distrbuuiig acc
out of di-ddiittbu button acc
02
Heb 2ibeet
the is, 244eed
GUR944 2007
Camrid
230
Figure 46. Box-Plot of Out-of-Distributinn Accuracy per Log-Scale of Learning Rate: Amazon-WILDS. and GivilComments-Wln WILDS
H.3 Effect of Initial Configuration Hyperparameter Optimization
In this section, we investigate how the first hyperparameter combination affected the search in our Bayesian optimization of hyperparameters. We compared Momentum SGD to Adam and chose PACS as our dataset. The experimental results showed that random initialization which we used, given a sufficient number of trials (c.g., more than 200 trials within experimental protocol), did not differ from the final performance obtained when searching from the default hyperparameters of pytorch.
PACS ERM Momentum
PACS ERM Momentum
1.0
out- tut-distribution acc
0.8
fut-distribution acc
0.8
0.6
0.6
0.2
top-1
0.2
top-1
average average
top-10 average
0.0
0.0
25
75
100
125
150
175
200
25
50
100
125
150
175
200
trials
trials
Figure 47. ERM PACS MomentumSGD Comparison of Initialization (Left: Random initialization, Right: Pytorch default hyperparameter initialization)
64

---

Published in Transactions on Machine Learning Research (06/2023)
PACS ERM Adam
PACS ERM Adam
1.0
1.0
out-distribution acc
0.8
Out-distribution acc
0.8
0.6
0.6
0.4
0.4
top-1
top-10 average all average
top-10 average
all average
0.0
0.0
25
50
75
100 trials
125
150
175
200
25
50
75
100 trials
125
150
175
200
Figure 48. ERM PACS Adam / Comparison of Initialization (Left: Random initialization, Right: Pytorch
default hyperparameter initialization)
65

---

Published in Transactions on Machine Learning Research (06/2023)
H.4 Best OOD Performance Comparison against with Existing Benckmark
In order to confirm the soundness of our experiments, we compared our results with existing benchmarks to see how well they actually performed, in addition to the hyperparameter search ranges in the previous section. In particular, we compared our experimental results with those of DomainBed (Gulrajani & Lopez-Paz, 2021), an existing oracle benchmark that uses Adam.
Table 15: OOD accuracy (%) comparison of our experimental results with the benchmark results reported in DomainBed (Gulrajani & Lopez-Paze Paz, 2021)
Dataset
OOD Domain
Existing Benchmark Results Results(Adam)
Our Results( Results(Adan)
ColoredMNIST
0.9
30.040.3
73.92
RotAtedMNIST
0
96.010.2
96.40
VLCS
97.740.3
99.36
PACS
87.810.4
89.30
OfficeHome
61.2+1.4
63.12
TerraIncognita
L100
59.941.0
61.35
NomainNet
clipart
58.440.3
58.48
66

---

Published in Transactions on Machine Learning Research (06/2023)
Additional Study
Corruption and Perturbation Shift
In the main body of our paper, we presented experimental results for the seven types of domain shifts included in DomainBed (Gulrajani & Lopez-Paz, 2021), as well as the Background Challonge (Xiao et al., 2021), which deals with background shifts, and WILDS datasets (Koh et al., 2021) which deals with the population shift dataset. this section, we report the results our investigation of the corruption and perturbation datasets to investigate broader range of out-of-distribution generalization. Details of the experiments are described in Appendix C.4, D.4 and E.4. CIFAR1O-C and CIFAR10-P (Hendrycks & Dietterich, 2019) were used as the datasets. Momentum SGD and Adam were used optimization methods for comparison.
The CIFAR1O-C results are averaged performance results for 19 different noise types of corruption and are based on Hendrycks Dietterich (2019) experimental protocol. The higher the performance, the better. The CIFAR1O-P experiment shows the variability of inference for noise perturbations, with lower values indicating better performance. Experimental results show that Momentum SGD outperforms Adam in both CIFAR10-C and CIFARIO-P (Table 16).
Table 16: CIFAR-1O-C (averaging corruption classification accuracy %) and CIFAR-10-P (top-5 robustness perturbation): Performance comparison between Momentum SGD and Adam with mean and standard deviation.
Dataset
un
Adam 42.0616.79
CIFARTO-C (1)
42.89 16.66
CIFARIO-P (L)
1.38 10.33
1.62 10.26
1.2 Model Architecture
In DomainBed experiments, we followed Gulrajani & Lopez-Paz (2021) and used only ConvNet for the MNIST-based dataset and ResNet-50 for the ImageNet-basod dataset. In this scction, investigate the impact of changing the model architecture.
1.2.1 20 for ColoredMNIST
Here are the results of ResNet-20 on the ColoT Task (Figure 17). In ConvNet case, Adam outperformed Momentum SGD, but the results were reversed in ResNet-20.
Table 17: ColoredMNIST: OOD accuracy (%) comparison between Momentum SGD and Adam
Model Architecture
Momentum
Adam
ConvNet
12.86 14.66
16. 16.1227.99
ResNet-20
11.00
10.09+0.15
1.2.2 Vision Transformer for PACS
Vision Transformer (Dsovitoy et al., 2020) is neural network using the recent attention structure. We evaluated the out-of-distribution performance when using Vision Transformer as well as ResNet-50 used in DomainBed. However, due to computational resource constraints, we compared Adam and Momentum SGD only for the task on the PACS dataset. The experimental results are shown in Table 18.
As result, the experimental results with Vision Transformer show a significant performance improvement over the ResNet-50 case. Furthermore, the result that Momentum SGD outperforms Adam is consistent with the ResNet-50 case.
67

---

Published in Transactions on Machine Learning Research (06/2023)
Table 18: PACS: OOD accuracy(%) comparison between Momentum SGD and Adam
Model Architecture
Momentum
Adam
ResNet-50
87.03=0
83.94 40.84
ViT
90.28 :0.
90.05+0.29
1.3 Atrs Optimizers
1.3.1 Sharpness Aware Minimization (SAM)
Sharpness-Aware Minimization (SAM) (Forct et al., 2020) prevents convergence to high curvature local minima. Its convergence lowards smaller curvature solutions results high validation and test performance on in-distribution (TD) environment. SAM searches for points where the loss maximized within neighborhood of p and uses the gradient that point for iterative optimization. The larger the p, the higher the effect of preventing convergence to high curvature local minima.
We carried out experiments with SAM on PACS and Amazon-WILDS datasets tasks, comparing it to both Momentum SGD and Adamn over a range of hyperparameters outlined in Appendices E.3 and E.3. The experimental results indicated competitive performance by SAM, equaling Momentum SGD. In the case of the Ama.On-WT..SS dataset, SAM proved superior for both in-distribution and out-of-distribbttion accuracy.
Table 19: PACS: Accuracy (%)
comparison of SAM with Momentum SGD and Adam
Accuracy
Momentum
Adam
SAM
ID accuracy
96.79 96.79.0.9
96. 96.7840.42
97.5410.07 86.65
OOD accuracy
87.03
83.94
Table 20: Amazon-WILDS, Accuracy (%)
comparison of SAM with Momentum SGD and Adam
Accuracy
Momentum
Adam
SAM
ID accuracy
72.51
72. 72.02+0.07
73.42=0.07 54.0.1.94
OOD accuracy
53.33
52.67
1.3.2 Adam with Decoupled Weight Decay (AdamW)
The AdamW optimizer, proposed by Loshchilov & Hutter (2017) is an extension of the popular Adam optimization algorithm. AdamW addresses the shortcomings of the original Adam optimizer concerning weight decay regularization. In the original Adam algorithm, weight decay is directly applied to the adaptive learning rates, causing a discrepancy between the intended effect of weight decay and the actual effect in practice, The AdamW optimizer decouples weight decay from the adaptive learning rates, resulting in a more effective regularization method that is better suited for various deep learning tasks. By incorporating weight decay separately from the update step, the AdamW optimizer exhibits improved convergence properties and gencralization performance compared to the original Adam algorithm.
We show the results of our AdamW experiment in Table 21. After tuning learning rate and selecting the the experiment with three different seeds and computed the mean
model with the best learning rate, and variance. We found that AdamW performs better than Adam, but not as well as Momentum SGD.
Table 21: WILDS: Comparison of AdamW with Momentum SGD and Adam
Accuracy
Momentum
Adam
AdamW
ID accuracy
72.51+0.00 53.33 10.0
72.02+0.07 52.67 10.77
72.27
OOD accuracy
53.33 0033
68

---

Published in Transactions on Machine Learning Research (06/2023)
1.4 Large for Adam
The study Choi al. (2019) shows that high in-distribution performance similar to Momentum SGD can be achieved with large However, hyperparameter that has been introduced to prevent zero-percentage, 1e-8 in pytorch's default implementation". It is known that increases, Adam
and is specified approximate Momentum SGD (Choi et al., 2019). While this section of our paper investigated with e to the extent that it behaves as Adam, in this section we provide experimental results at large where Adam is expected to behave more like um SGD.
The results the experiment are shown Figure 49. The x marker the lower left corner shows the results for the default hyperparameters in Adam. The other circle markers are for different in Adam. Red stars indicate some of the Momentum results. It can be that the larger achieved performance closer to Momentum SGD than the default in Adam.
0.88
Momentum
0.87
Momentum
uuttof-distributiio acc
Momentum
Momentum
0.86
0.85
E ((Adam)
1e-02
1e-04
0.84
le+ 02
14+04
0.83
1e+06
1e+08
1e+14
Adam (s=1e-8)
0.82
0.940
0.945 0.950 0.955 0.960 0.965 0.970 0.975 0.980 in-distribution acc
Figure 49.
Demonstrating the varying performance out-of-distribution accuracy according to the value of Adam. As becomes larger, approaches the performance of Momentum.
1.5 Learning Rate Schedule
Amazon-WII.DS task uses linear scheduling without warmups 9. However, learning rate scheduling can affect performance. To consider this impact, we compared and verified the following four learning rate schedules and eight patterns with and without warmup.
Table 22 shows the results the Amazon-WILDS experiments. However, the results for the largest out-of- distribution accuracy are shown since no significant differences were found for all experiments. No significant differences were found with or without warmup. It was clear that introducing warmup is always effective. The Cosine LR Schedule was found to be the most effective in the problem setting of this study.
optin. Adam. html
69

---

Published in Transactions on Machine Learning Research (06/2023)
Table 22: Amazon- Amazon-WILDS: OOD Accuracy (%) comparison LR Scheduler and Warmup
Learning Rate Schedule
Momentum
Adam
Adamn
Constant LR
53.33
53.33
52.00
Constant LR + Warmup
53.33
52.00
53.33
Cosine LR
54.67
54.67
54.67
Cosine LR + Warmup
54.67
54.67
54.67
Linear LR (Default)
54.67
52.00
52.00
Linear LR. Warmup
54.67
54.67
53.33
MultiStep LR
52.00
53.33
52.00
MultiStep LR Warmup
50.67
53.33
52.00
The CivilComments-Wiie WILDS task, akin to Amazon WILDS, employs linear scheduling without warmups, as elucidated in the official repository We proceeded to examine the CivilComnents-WTI. dataset to dataset
identify any analogous trends that may emerge. Our analyses from the CivilComments-Winn corroborated the findings from our Amazon- WILDS study, thereby reinforcing our preliminary conclusion that non-adaptive optimizers typically exhibit superior performance over their adaptive counterparts. Notably, we observed a significant deviation the OOD performance when modifying the learning rate schoduler in the CivilComments WILDS datasel. This observation starkly contrasts with our experiences in the setting. Moreover, despite these modifications to the learning rate scheduler, could not surpass the performance outcomes achieved with the default learning rate schedule. This reiterates the efficacy of the default setting, and further validates our overall findings.
Amazon-Wllll
Table 23: MiI.Comments-WILDDD OOD Accuracy (%) comparison of LR Scheduler and Warmup
Learning Rate Schodule
Momentum
Adam
Constant LR
47.82
44.29
Constant LR Warmup
47.54
44.44
Cosine LR
56.98
45.40
Cosine LR + Warmup
56.83
45.48
Lincar LR (Default)
57.69
46.82
Linear LR + Warmup
57.14
45.00
MultiStep LR
47.82
44.29
MultiStep LR - Warmup
51.35
44.04
1.6 The Effect of Random Seeds
We conducted experiments on the effect of seed, which controls randomness, such as model initialization, on learning, using the PACS and AmazOn-WTLDS datasets.
In the PACS experiment shown in Table 24, performance difference of around 6% was observed due to the effect of seed, especially for Adam. In contrast, in Momentum SGD, the effect of seed not significant. In the Amazon-WILDS experiment shown in Table 25, seed had no significant effect either Adam Nesterov Momentum SGD.
Table 24: PACS: OOD Accuracy (%)
Table 25:
AmaZOn-WILDS: OOD Accuracy (%)
Different Seed Comparison
Different Seed Comparison
Seed
Momentum
Adam
Seed
Nesterov
Adam
2021
86.47
81.20
2021
53.33
52.00
2022
86.47
87.06
2022
53. 73
53.33
2023
87.74
85.69
2023
54.67
53.33
10nttps://github. watpt/hcooopppaaaaa
70

---

Published in Transactions on Machine Learning Research (06/2023)
1.7 Algorithms (ERM, IRM, VREx and CORAL)
In this study, we focused on ERM and IRM and obtained consistent results that the Non-Adaplive op- timizer outperforms the Adaptive optimizer in out-of-distribution performance. We also verified the use of VREx(Krueger et al., 2021) and CORAL(Sun & Sacnko, 2016) as the other algorithms. The results of these experiments are shown in Table 26. However, due to limited computing resources, experiments were conducted only for Momentum SGD and Adam for PACS.
Table 26: PACS: OOD Accuracy (%) comparison of algorithms (ERM, IRM, VREx and CORAL)
Algorithm
Momentum
Adam
ERM
87.03_0.65
83.94+0.88
IRM
83.06_0.32
83.05.10.44
VREx 85.70_0.24
84.85+1.88
CORAL
84.25 1.35
84.10
71


Reviews:
[{"review_id": "xHFQtHJdyU", "reviewer": "Reviewer_VQ4e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- Well-tuned benchmarks: As the authors point out, their hyperparameter tuning subsumes prior work, in turn leading to more competitive baselines.\n- Well-motivated idea: although the importance of benchmarking ML models on dataset shifts is well-recognized in the field, this is (to the extent of my knowledge) the first benchmark of optimizers within this context, providing a valuable contribution. \n- The analysis of Colored MNIST (including in the appendix) was really interesting! In general, most of the experimental analysis is very detailed.\n- Clarity: Overall, the paper is well written.\n\n# Weaknesses\nTo me, the major weakness of this work is in the set of benchmarks and in how the results are presented (as such, any author response that addresses or rebuts this weakness would increase my score). My chief concerns are the following:\n- Benchmark suite: common benchmarks for dataset shifts include several imagenet variants (Imagenet-C, Imagenet-A, Imagenet-R, Imagenet-V2, etc.), Cifar-10 shifts (Hendrycks & Dietterich, ICLR 2019), or SVHN after training on Cifar-10 (see, e.g., Ovadia et al, NeurIPS 2019). Some of the benchmarks considered in this work seem weak given the type of models that are being benchmarked (e.g., rotated MNIST). I strongly recommend authors consider additional datasets to facilitate comparisons with existing literature on dataset shift benchmarking. However, I do realize that many of this datasets can be expensive to validate against; if others more familiar with optimizer benchmarking support the current set, I will take this into consideration.\n\n- Reported results: some important experimental details were not easily obvious to me. In particular, how many different runs did you use (with the best hyperparameters) for the evaluation? (The text just states \"the evaluation of several trained models\", and no standard deviations are provided in Table 1.) \n- Similarly, what distribution of data was used for the histograms in Figure 1? Are the points from all training runs (including other hyperparameters? or simply from different model initializations with the best hyperparameters?\n- I found the barplots of Figure 1 somewhat hard to read; is there a reason not to use a scatterplot?\n\n- Finally, although I understand that this is possibly due to compute limitations, including Transformer-based architectures would strengthen the benchmark and the conclusions that are drawn.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors benchmark different optimizers (3 non adaptive optimizers: SGD, Nesterov accelerated gradient, and SGD+Momentum, and 2 adaptive optimizers: Adam and RMSProp) on image classification and NLP datasets, with the goal of understanding which optimizer(s) perform best when evaluated on downstream performance under dataset shift. The authors benchmark CNNs, ResNet 50s, and DistilBERT.\n\nCompared to previous benchmarks of this type, the key differences are the author's type of evaluation metric (performance under dataset shift) and the increased budget allocated to tuning the hyperparameters of each method.\n\nBased on their analysis, the authors draw the following conclusions:\n1. Overall, non-adaptive optimizers produce models that are more robust to dataset shift.\n2. That the relationship between in-distribution (IND) and out-of-distribution (OOD) performance is either sub-linear, super-linear, or essentially linear, reproducing + expanding upon previous known results (cited in the paper).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical to acceptance is \n- Clarifying the experimental setup (# of experimental replicas, whether results are computed over optimal hyperparameters or over the entire set of evaluated hyperparameters), adding standard deviations to the tables.\n\nStrongly suggested, but not strictly necessary for acceptance:\n- Including other datasets commonly used to evaluate robustness under dataset shift. This should be possibly doable for evaluation on the Imagenet corrupted versions described above, assuming the checkpointed models on Imagenet-1K from the Backgrounds challenge are available.\n", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AVapE70j2l", "reviewer": "Reviewer_JSuw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\ni) This work presents a comprehensive empirical study. The authors make great efforts to carefully control the influence of other factors in order to elucidate the inﬂuence of optimizer selection under distributional shifts.\n\nii) The experiments cover a broad scope of datasets, neural network architectures, and optimizers.\n\niii) The authors provide plentiful analysis and discussion on the empirical results such as different IID and OOD performance trends.\n\niv) The empirical trends are aligned with the existing literature.\n\n**Weakness**\n\ni) The evaluation setting may not consider all factors that can have a major influence on the OOD performance, such as model selection criteria.\n\nii) The experiments regarding the optimization with OOD objectives may not be adequate since only IRMv1 is considered.\n\niii) Missing discussion with some related works.\n\niv) It’s unclear how the categorization of IID and OOD performance can help practitioners better understand and select optimizers.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive study on optimizer selection for OOD generalization. The authors compare the IID and OOD performances of various models on multiple datasets from DomainBed, Wilds benchmark, and Backgrounds Challenge, when optimized with both adaptive optimizers (e.g., Adam) and non-adaptive optimizers (e.g., Momentum SGD). The empirical results suggest that, under the same hyperparameter tunning scope and evaluation protocol, the choice of optimizers can have a non-trivial influence on OOD performance, and non-adaptive optimizers achieve better OOD performance than adaptive optimizers in most of the datasets. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Although I appreciate the great efforts in the empirical comparison, I find the work can still be improved as follows (corresponding to each weakness):\n\ni) One major conclusion from DomainBed is that, the model selection can also have a large influence to the OOD performance. Throughout the paper, all of the models are selected according to the performance in a validation set that has the same distribution as the training set, which may significantly weaken the OOD performance of the selected models, and fail to uncover the full empirical trends. In fact, in practice, an OOD validation set is often available for model selection. Hence, I suggest the authors also present more results that use different validation sets as suggested by DomainBed, to examine whether the same empirical trends can be observed as using the current validation set.\n\nBesides, the model architecture can also have a influence on the optimization. For example, in ColoredMNIST, DomainBed uses a CNN network, which may be more suitable for Adam to coverge and even overfit. It would be further strengthen the reliability of the claims and to differentiate the influence of model architectures to the OOD performance, if the authors would provide additional results using other model architectures for some datasets such as ColoredMNIST.\n\nii) The experiments regarding the optimization with OOD objectives only consider IRMv1, while there is a rich literature on the OOD objectives. Although the authors find IRMv1 exhibits similar empirical behaviors as ERM, it is unknown whether other OOD objectives have a similar behavior. Since practitioners can use various OOD objectives to train the model, more discussions or empirical results are expected about the influence of OOD objectives on the discoveries of this work.\n\niii) The related work discussion misses some closely relevant works [1,2,3,4]. [1,2,4] discussed the trade-offs of IID and OOD performance when finetuning pre-trained models. [3] uncovered the optimization dilemma in OOD generalization and the trade-offs in the optimization of the ERM and OOD objectives. [3] proposed to consider both ERM and OOD losses in optimization and model selection, and also conducted experiments on a wide variety and scale of datasets.\n\niv) From the discussion on Sec. 4.3, i.e., different correlation behaviors between IID and OOD performance, it is unclear how different correlation behaviors can help practitioners better understand and select optimizers. I am also confused about the description of the rightmost part of Figure 5, that “the increase in a large region of the in-distribution saturates the OOD generalization with a small percentage of its e!ect.”, for which I cannot see from the figure. Besides, it seems the key takeaway of the paper is that practitioners should use non-adaptive optimizers. The authors could make the discussion of Figure 5 more informative.\n\n\n**References**\n\n[1] Kumar et al., Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, ICLR 2022.\n\n[2] Wortsman et al., Robust ﬁne-tuning of zero-shot models, CVPR 2022.\n\n[3] Chen et al., Pareto Invariant Risk Minimization: Towards Mitigating The Optimization Dilemma in Out-of-Distribution Generalization, ICLR 2023.\n\n[4] Kumar et al., How to Fine-Tune Vision Models with SGD, arXiv 2022.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QGX0bO7JMM", "reviewer": "Reviewer_Fje8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n* Good addition to [7] for empirical insights into optimizer performance with special focus on OOD generalization.\n* Tuning runs are shared as box-plots, showing how sensitive some of the methods are to tuning.\n* Interesting take-away that non-adaptive optimizers are superior in terms of OOD generalisation compared to adaptive methods.\n* Good experiments in H.2 to verify that, in the limit, random initialization for hyper parameter search does not differ from initialization with the PyTorch defaults.\n\n### Author-aware Weaknesses:\n\nHere I want to list a few weaknesses that the authors are already aware of but I believe they are still weaknesses that should be addressed to improve the presented work:\n\n* [**Large**]: While I do agree that selecting optimizers to benchmark is a difficult procedure since different readers might have varying expectations given the avalanche of methods (cf. Table 2 in [7]), the fact that no published optimizer after 2015 has been included doesn’t quite sit right with me. For NLP training (a focus of this work according to p.2), the community commonly sticks to an AdamW [1] variant (see e.g. [2], [3], [4]) while in the broader optimization community, sharpness-aware minimization (SAM) [5] has also attracted a lot of recent interest for improved generalization performance. Other works in this field (e.g. [7]), do at least include some more recently proposed optimizers in their studies, albeit the focus on OOD generalization here is slightly different.\n* [**Large**]: Building on top of the previous point, the presented work focuses only on first-order methods. Given the recent advances in second-order methods like Distributed Shampoo (see e.g. [6]), it would be interesting for the community to assess OOD generalization for this class of optimization algorithms as well.\n\n### Additional Weaknesses:\n\n* [**Large**]: Given the amount of randomness that comes with benchmarking it would be good to test for the impact of different seeds. For that, we can have a study on one or two of the datasets e.g. WILDSAmazon / WILDSCivilComment where the difference between Adam <> Nesterov is particularly large and re-run the tuning / evaluation on a few more random seeds to verify how large the impact is.\n* [**Large**]: Even though H.1 shows the histogram of learning rate, it is hard to verify from these if the tuning range is chosen to be sensible since the OOD generalization performance is not plotted against the corresponding hyperparameter value. Instead of showing a histogram, a box plot for the corresponding OOD performance would be much more useful. Assuming that the bayesian optimization works as expected, there are quite a few Adam learning rates that are sampled frequently towards the lower end of the tuning range (e.g. RotatedMNIST, VLCS, OfficeHome, WILDSAmazon) suggesting that the optimal learning rate falls outside of the search space. I'm expecting similar problems with some of the other hyperparameters that are not shown here.\n* [**Minor**]: As discussed, with large $\\epsilon$ Adam turns into a non-adaptive method and, as found by Choi et al. (2019), never performs worse than Momentum for in-distribution generalization. In the presented work, the tuning range for $\\epsilon$ is too small $[1e-8, 1e-3]$ (compared to Choi et al.) and does not consider this limit of large $\\epsilon$ Adam. It would be interesting to see if  large $\\epsilon$ Adam also never performs worse than Momentum for OOD generalization. \n* [**Minor**]: Only one type of learning rate schedule (linear decay) is considered, albeit with sampling when to decay from and the decay amount. However, there are many more schedules (cf. Table 3 in [7]) where especially warmup + step number decay is popular in NLP. I'm assuming that this will have an impact on the results.\n\n### Minor Comments & Typos:\n\n* Presentation of Figure 1 can be improved, the colour scheme is shared so the legend can be displayed once below all the subfigures. Also, what does the red color indicate? Spacing between top and bottom row should be increased to make it look less cramped.\n* Figure 2 Caption: OficeHome → OfficeHome, I suggest creating a macro to avoid typos and unify notation e.g. *\\newcommand{\\officehome}{\\textsc{OfficeHome}\\@\\xspace}*\n* The x-axis label spacing for some of the subfigures can be improved (e.g. Figure 2 or 4). The label for “Momentum” and “Nesterov” basically overlap.\n* Liu et al. and [7] have two entries in the bibliography, please consolidate and use the published version.\n* I believe the Appendix should be condensed, even though I appreciate the amount of data that is incorporated, I believe some of the plots are redundant and/or could be visualised better to condense them into a few key figures, might be worth considering plots similar to Figure 4 in [7] with shaded areas for tuning runs or another visualization style. \n* Figure 37 top row can probably be scaled better, there is nothing to see.\n* Eq. 15 is formatted weirdly, there is a lot of space to the main text\n\n---\n\n* [1]: [Decoupled Weight Decay Regularization](https://openreview.net/forum?id=Bkg6RiCqY7) , Loshchilov & Hutter, ICLR 2019 \n* [2]: [AdamW is default in fairseq as Adam](https://fairseq.readthedocs.io/en/latest/optim.html?highlight=AdamW#fairseq.optim.adam.FairseqAdam) so most papers that use fairseq are using it.\n* [3]: [No Language Left Behind: Scaling Human-Centered Machine Translation](https://arxiv.org/abs/2207.04672), NLLB-Team, 2022\n* [4]: [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971), Touvron et al., 2023\n* [5]: [Sharpness-aware Minimization for Efficiently Improving Generalization](https://iclr.cc/virtual/2021/spotlight/3497), Foret et al., ICLR 2021\n* [6]: [Scalable Second Order Optimization for Deep Learning](https://arxiv.org/abs/2002.09018), Anil et al., 2020\n* [7]: [Descending through a Crowded Valley -- Benchmarking Deep Learning Optimizers](https://proceedings.mlr.press/v139/schmidt21a.html), Schmidt et al., ICML 2021\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The presented work conducts an empirical comparison of 5 different optimizers on several domain generalization benchmarks to assess whether non-adaptive or adaptive optimizers lead to better out-of-distribution generalization performance. For that, they include SGD, Momentum, and Nesterov Momentum as non-adaptive optimizers and Adam, RMSProp as adaptive variants, and tune each optimizer with 200 trials for each benchmarking problem. From their experiments and analysis, they conclude that non-adaptive optimizers perform better than adaptive optimizers for OOD performance.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am slightly in favour of publishing, simply because such empirical insights are quite interesting for the broader community. However, adding experiments for the mentioned weaknesses would significantly improve the work and I urge the authors to include some of them, at least in the form of control experiments on a limited set of problems.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an extensive benchmark of our different optimization methods (adaptive and non-adaptive) performs with respect to Out-of-Distribution Generalization (OOD). The experiments were already extensive, and after requests by all reviewers, the authors have included yet more experimental setups and datasets, making this a useful benchmark for OOD results.
Decision: Accept (poster)