✳︎Empirical Study on Optimizer Selection
for Out-of-Distribution Generalization

Hiroki Naganuma 1,2, Kartik Ahuja1,2, Shiro Takagi†, Tetsuya Motokawa4,Rio Yokota5, Kohta Ishikawa6, Ikuro Sato5,6 ,Ioannis Mitliagkas1,2,3
1: naganuma.hiroki@mila.quebec2†: Independent Researcher

3

4

5

6

Introduction

Optimizers Subjected in Our Analysis

Optimizer Comparison in OOD Accuracy【Motivation】

We target five of the most popular and standard optimizers that have been used andstudied in recent years

We compared Momentum SGD as the best non-adaptive optimizer, with Adam as thebest adaptive optimizer• Optimizer selection• Crucial for the successful training of DNNs.
• Influences training speed, stability, and generalization performance.
• Previous studies of are based on a IID assumption• Out-of-distribution (OOD) generalization• In real-world applications, it is often the case that the test data obey a distributiondifferent from the training data
• Distributional shift violates the typical IID assumption for training• Comparing the OOD generalization performance among different optimizers is ofgreat interest in theory and in practice【Contribution】【Non-Adaptive Optimizers】
In addtition to SGD, optimizers with momentum terms such as Momentum SGD, andNesterov momentum are also classified as non-adaptive optimizers
˜ θ is stochastic gradient and γ is momentum.
where θt is model parameter, ηt is learning rate, ℓ(θ) is loss ∇t−1

【Adaptive Optimizers】
Adam and RMSprop are adaptive optimizers and they can be written in the form of thegeneric adaptive optimization method• Design and perform a comparison of the effect of optimizers onOOD generalization on OOD benchmarks

Figure: Relationship between the ID accuracy and the OOD accuracy in the ERM setting.
The x-axis of the plot is the in-distribution accuracy and the y-axis is the OOD accuracy. To make the trend more clear,
the in-distribution accuracy corresponding to the x-axis is divided into 10 bins, and the average performance of theOOD accuracy in each bin is shown on the y-axis.accuracy in each bin is shown on the y-axis.

• Evaluate 10 out-of-distribution generalization datasets(including image classification and NLP)
• Wide range of hyperparameter configurations(examining over 20,000 models)• Demonstrate optimizer characteristic under distributional shift• The adaptive optimizers provide more in-distribution (ID) overfitting and degrade
OOD performance more than the non-adaptive optimizers• Non-adaptive optimizer outperformed adaptive optimizer in terms of best
OOD accuracy (8 out of 10 datasets)【Experimental results and implication】

Experimental Protocol【Datasets】• In our area of interest, where a high in-distribution performance is achieved,Momentum SGD outperforms Adam on 9 of the 10 datasets in the sense ofaverage OOD accuracy (Figure)• This indicates that non adaptive optimizer is more advantageous than adaptive
optimizer in OOD, even though the performance is similar in the IID environment• Observed correlation behaviors: ID vs OOD performance• It can be categorized into typical patterns:linear return, diminishing return, andincreasing return

Limitation of IID Assumption

Table: Comparison of the best OOD accuracy of ERM between five optimizers.
Except for a small set of problems, momentum SGD outperforms Adam. As a soundness check, we confirm that our Adam
results outperform all existing benchmark results using Adam.

Empirical risk minimization (ERM) as known for standard training method could
achieve high ID performance by learning spurious correlations.

• When comparing the performance of the best OOD accuracy, the non-adaptive
optimisers outperformed the adaptive optimizers in 8 out of 10 data sets (Table)

Correlation Behaviour (IID vs OOD)

Our results show that three typical types of behavior are observed in terms of thecorrelation between in-distribution performance and OOD performance for different
datasets. These show how much performance in OOD can be expected if we increase thein-distribution performance.
Figure: OOD Datasets we evaluate in our study (Image taken from [Gulrajani21](Domainbed / left), [Xiao21](BackgroundChellenge / right), and [Koh2021] (WILDS / bottom))【 Model Selection Method and Evaluation Metrics】

Figure: Examples of invariant and spurious features.

Why Optimizer Selection?

✳︎• Learning method to mitigate the mentioned above is also studied• Invariant risk minimization (IRM) [Arjovsky19] is also conducted in our study• However, these methods have not provided sufficient OOD performance, and theinfluence of the optimizer has not been taken into account so far• Adam, due to its update formula, is likely to capture noise that is not an invariant
feature, although it converges quickly• We follow the benchmark respectively [Gulrajani21],[Xiao21] and [Koh2021]• For the image classification tasks• The training domain is split into training and validation data
• OOD performance is evaluated in the test domain• For the NLP tasks, the worst group is evaluated as the OOD performance【 Hyperparameter Tuning】• The exhaustiveness of the hyperparameter search is crucial for empiricalinvestigation of an optimizer's effect• We basically follow [Choi19], which most exhaustively searched hyperparameters for
optimizer comparison and explored more hyperparameters than did previousstudies

Figure: Three-types of correlation Behaviour:
increasing return (PACS), linear return (DomainNet), and diminishing return (Amazon-WILDS).

References[Arjovsky19] Martin Arjovsky et al. “Invariant risk minimization”. In: arXiv preprint arXiv:1907.02893 (2019)[Gulrajani21] Ishaan Gulrajani and David Lopez-Paz. “In Search of Lost Domain Generalization”. In:
International Conference on Learning Representations. 2021
[Xiao21] Kai Yuanqing Xiao et al. “Noise or Signal: The Role of Image Backgrounds in Object Recognition”. In:
International Conference on Learning Representations. 2021.
[Koh2021] Pang Wei Koh et al. “Wilds: A benchmark of in-the-wild distribution shifts”. In: Interna- tional
Conference on Machine Learning. PMLR. 2021, pp. 5637–5664.