Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
Authors: Devansh Arpit
Huan Wang
Yingbo Zhou
Caiming Xiong
Venue: NeurIPS 2022 Accept (2022)
Keywords: ensemble
model averaging
simple moving average
domain generalization
model selection
bias-variance trade-off

Abstract: In Domain Generalization (DG) settings, models trained independently on a given set of training domains have notoriously chaotic performance on distribution shifted test domains, and stochasticity in optimization (e.g. seed) plays a big role. This makes deep learning models unreliable in real world settings. We first show that this chaotic behavior exists even along the training optimization trajectory of a single model, and propose a simple model averaging protocol that both significantly boosts domain generalization and diminishes the impact of stochasticity by improving the rank correlation between the in-domain validation accuracy and out-domain test accuracy, which is crucial for reliable early stopping. Taking advantage of our observation, we show that instead of ensembling unaveraged models (that is typical in practice), ensembling moving average models (EoA) from independent runs further boosts performance. We theoretically explain the boost in performance of ensembling and model averaging by adapting the well known Bias-Variance trade-off to the domain generalization setting. On the DomainBed benchmark, when using a pre-trained ResNet-50, this ensemble of averages achieves an average of $68.0\%$, beating vanilla ERM (w/o averaging/ensembling) by $\sim 4\%$, and when using a pre-trained RegNetY-16GF, achieves an average of $76.6\%$, beating vanilla ERM by $\sim 6\%$.

Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
Devansh Arpit, Huan Wang, Yingbo Zhou, Caiming Xiong
Salesforce Research, USA devansharpit@gmail. com
Abstract
In Domain Generalization (DG) settings, models trained independently given set of training domains have notoriously chaotic performance on distribution shifted test domains, and stochasticity in optimization (e.g. seed) plays big role. This makes deep learning models unreliable in real world settings. We first show that this chaotic behavior exists even along the training optimization trajectory of single model, and propose a simple model averaging protocol that both significantly boosts domain generalization and diminishes the impact of stochasticity by improving the rank correlation between the in in-domain validation accuracy and out-domain accuracy. which is crucial for reliable early stopping. Taking advantage of our observation, we show that instead of ensembling unaveraged models (that is typical in practice), ensembling moving average models (EoA) from independent runs further boosts performance. We theoretically explain the boost in performance ensembling and model averaging by adapting the well known Bias-Vine Variance trade-off to the domain generalization setting. On the DomainBed benchmark, when using a pre-trained ResNet-50, this ensemble of averages achieves an average of 68.0%. beating vanilla ERM (w/o averaging/ensembling) by 4%, and when using pre-trained RegNetY- 16GF, achieves an average of 76.6%, beating vanilla ERM by 6%. Our code is available at hhttps://githeei com/ salesforeeccc ensemble-on -verages
Introduction
Domain generalization (DG, (5]) aims at learning predictors that generalize well on data sampled from test distributions that are different from the training distribution. Currently, deep learning models have been shown to be poor at this form of generalization [00] and excel primarily in the III) setting (51).
While a number of algorithms have been proposed to mitigate this problem (cf [5] for a survey), [8] demonstrate that models trained using empirical risk minimization (ERM, 43 along with proper model selection (i.e. early stopping using validation set), using subset of data from all the training domains, largely match generalization algorithms. This suggests that model selection plays an important role in domain generalization. Despite its importance, there has not been much investigation into the reliability of model selection. As we demonstrate in Figure [ the out-domain performance varies greatly along the optimization trajectory of model during training, not. This instability therefore hurts the reliability of model selection, and can become a problem in realistic settings where test domain data unavailable, because it causes the rank correlation between in-domain validation accuracy and out-domain test accuracy to be weak.
outperform the performance of most existing domain
though the in-domain performance does
In this paper, we first investigate simple protocol for model averaging that both boosts DG within the ERM framework, and mitigates performance instability of deep models on out-domain data,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Validation Acc 190
whhoot
averoging
Validation acc
Aenurery Best
10*
50-
Trainig
Trainig
1000
2000
3000
4000
5000
1000
2000
5000
4000
5000
Iberations
Meeraiion
weteot avanaging
Test Domain Aox PS1
with
Text Ace
50-
10
Trainia Loss
40
Accursty B:st
25
nag
20-
Trainig
1000
Iterations
Figure 1: Model averaging improves out-domain performance stability. Left: In-domain validation accuracy and out-domain test accuracy during training of models using ERM. Right: Same as left, except validation and test predictions are made using a simple moving average of the model being optimized, along its optimization path. Details: The plots are for the Terralncognita dataset with domain L38 used as the test domain, and others as training/validde validatioiinnnnno data, and ResNet-55. Solid lines denote accuracy, dashed lines denote training loss, and dash-dot lines denote best accuracy achieved during training and all runs (for reference). Each color denotes different run with different random split. Gist: Model averaging reduces out-domain performance instability, and makes the test curves correlate better with the validation curves, making model selection using in-domain validation set more reliable during optimization. We see similar pattern when using ensemble of models, with and without model averaging, in Figure
seed and raining/validation
specifically with respect to in-domain validation data. This makes model selection more reliable. Next, taking advantage of our observation, we show that ensembling moving average models further boosts performance, making it better choice for practical scenarios. Note that we do not claim that model averaging or ensembling can fully solve the problem of DG. The observation that model averaging can boost domain generalization performance is not new, and was exposed by SWAD X which inspired our work. Our contribution in this respect are as follows:
1. Hy plppepaaraeter- free:In Ine contrast to SWAD, which introduces three additional eyper-parametent for its model averaging algorithm that need tuning, we show that the simple strategy of maintaining simple moving average (SMA) of the model parameters throughout the optimization trajectory, starting near initialization (Appendix Figure 5). works just as well (when a pre-trained model is used as initialization). Although model averaging technically requires two typer-parametery averaging frequency and starting iteration, through empirical analysis, we show that setting the frequency to I and setting the start iteration close to 0 works well multiple datasets and architectures, making our proposal hyperparameter-frre in practice.
2. Computationally efficient:SWAD requires computing validation performance more frequently than is typically done (2x-6x on the DomainBed datasets), which is needed because it needs to find the start and end iteration between which model averaging is done. This increases compute requirements. This segment is selected based on the validation performance computed using the model being trained. Our proposal to instead use the SMA model to perform early stopping and inference, side-steps this need and does not require frequent validation performance check. We show that the root cause for this difference is that the model being trained has unstable performance on OOD data, while the SMA model has a more stable OOD performance (see Figure [and Table Z Thus this observation results in our hyperparameter--fre and more efficient model averaging strategy.
3. EoA: Taking advantage of our efficient model averaging protocol (section 2.2 we find that an ensemble of moving average models (EoA) outperforms a traditional ensemble of unaveraged models (Table D. We also show ablation analysis that the rank correlation between in-domain validation performance and out-domain test performance is better for the ensemble of average models (Table 3).
Theoretical explanation: To explain why both model averaging and ensembling improve OOD performance under unified theoretical framework, we adapt the well known Bias- Varianca decom- position to the domain generalization setting, and argue that the expected OOD loss for individual models comprises of both the bias and the variance term, while the expected OOD loss for ensembles and averaged models comprises mainly of the bias term only, and is thus strictly lower (section 3.2). Our explanation is in contrast with SWAD, which uses flat minima explain the improved OOD

---

generalization, which applies to model averaging, but is less straight forward for explaining the boost by ensembles.
5. Benchmarking: For benchmarking, we experiment with three different pre-trained models as initializations for DG training, with increasing pre-training dataset size and model size. In these experiments we find that EoA provides a larger gain over the corresponding ERM baseline with increasing dataset and model size. These gains range from 4% 6% (Table 4 Notice that this claim is different from existing work [20], which states that the baseline ERM performance improves with larger pre-traue pre-training data and model size.
2
Model Averaging
2.1 Terminology
For given supervised learning objective function, let fa(.) denote the deep network being optimized using gradient based optimizer, where 9 denotes the parameters of this model. We refer to fo the online model, or unaveraged model. The output of fa(.) is vector of K logits corresponding to the K classes in the supervised task.
Online Model:
While the online model is being trained, we maintain a moving average of the online model's parameters. This process sometime referred to as iterate averaging in existing literature. The deep network whose parameters are set to be this moving average is referred to as the moving average model, or more specifically simple moving average (SMA) model because of its use in our work. We denote the parameters of this model by Ó.
Moving Average (MA) Model:
2.2 Model Averaging Protocol
We use a simple moving average (SMA) of the online model. Instead of calculating the moving average starting from initialization (as done in Ruppert averaging), we instead start after certain number of iterations to during training (tail averaging), and maintain the moving average until the end of training. As we discuss in the next section, to is chosen to be close, but not equal the initialization when a pre-trained model is used as initialization. At any iteration t, we denote:
ift S to 0-1 + 1-1o+I O,, otherwise
0.
=
t-to r-fo
where 0, is the online model's state iteration f. Note that effectively, 0, == I-1o+I :===--+++
Ei-=t Or.
Further, at iteration f, if we need to calculate validation performance, we use 0, to do so, and not Or. As we show in the next section, the benefit of doing so is that the rank correlation between in-domain validation accuracy and out-domain test accuracy is significantly better when predictions using 8. This makes model selection more reliable for domain generalization. Finally, for given run, model selection selects 0,- for making test set predictions, such that Ô,- achieves the best validation performance. We discuss some theoretical perspectives on why model averaging can help domain generalization in section 5.1
made
2.3 Ablation Analysis
Here we perform four ablation studies: 1) impact of the start iteration to used in our SMA protocol in Eq. [ 2) the frequency of model averaging: 3) instability reduction of SMA model compared to the online mode along the optimization trajectory on out-domain data; 4) correlation between in-domain and out-domain accuracy across independently trained models.
Due to space limitation, we show experiments for 1,2 and 4 in Appendix section C In summary, we find that: 1) starting averaging close to initialization results in improved out-domain performance (Figure in Appendix) when the parameters are initialized used pre-trained model; 2) the frequency of SMA does not have a significant impact on performance, unless sampling is done at too large intervals (Figure Appendix): 4) the rank correlation poor between validation and test accuracy of independently trained models (Figure in Appendix). An implication of this is that it is difficult to discover the best model (for out-domain performance) from pool of independently trained models, based only their in mn-ommiin validation performance (echoing the findings of [10).
3

---

Table 1: Spearman correlation (closer 1 is better) between within-run in-domain validation accuracy and out-domain test accuracy on multiple datasets. Model averaging improves rank correlation for both individual models (left) and ensemble of averages (right).
Table 2: Individual Models
Table 3: Ensembles
Terralncognita
w/o avg
w/ avg
TerraIncognita
w/o avg
w/ avg
L100
0.21 = 0.07
0.90 + 0.05
L100
0,48
L38
0.12 = 0.13
0.83 + 0.05
L38
0.17
0.95
L43
0.30 = 0.06
0.67 + 0.18
L43
0.59
0.38
L46
0.03 = 0.11
0.52 + 0.14
L46
0.08
0.61
2.3.1
Instability Reduction: Rank Correlation
We study the reliability of model selection for domain generalization when using online models vs moving average models, using rank correlation (see Appendix C.4 for definition). To do so, we train models on a dataset, both with and without model averaging. and compute Spearman correlation between the in-domain validation accuracy and out-domain during the training process. Since there are multiple runs where a given domain acts as the test domain, calculate the mean and standard error of these values over these runs.
accuracy sampled at regular intervals
The rank correlations are shown in Table 2 (and Table in Appendix) for the PACS, VLCS, Office- Home, Terralncognita and DomainNet datasets. We find that majority of the cases, using model averaging results in a significantly better rank correlation compared to using the online model. These experiments therefore suggest that the reliability of model selection is significantly higher within run when using model averaging.
3 Ensemble of Averages (EoA)
18 propose a rigorous framework for evaluation in the domain generalization setting which accounts for randomness due to seed and hyper-pane hyper-parameter values, and recommend reporting the average test accuracy over all the runs computed using a model selection criteria. However, in practice, it is desirable have single predictor that has high accuracy. An ensemble combines predictions from multiple models, and is a well known approach for achieving this goal [I]] by exploiting function diversity [14]. However, we show, even ensembles suffer from instability in the domain generalization setting. Building on the observations of the previous section. we investigate the behavior of ensemble of moving average models and find that mitigates this issue. We begin by describing the EoA protocol below.
We perform experiments with ensemble of multiple independently trained models (i.e., with different aiper-parameters and seeds). When each these models are moving average models from their corresponding runs, refer this ensemble in short the ensemble of averages (EoA). Identical to how we make predictions for traditional ensembles (specifically the bagging method (6), the class ý predicted by an EoA for an input given by the formula:
EoA Protocol:
E Σ
① = arg max So
f(x;0))) 0.))&
E
where is the total number models in the ensemble, Ó, denotes the parameters of the ith moving average model, and the sub-script (.)x denotes the t element of the vector argument. Finally, the state 0, of the 5th moving average model used in the ensemble selected from its corresponding run using its in-domain validation set performance (described in section 2.2}. We now investigate the behavior of EoA compared with ensembles of online models on domain generalization tasks.
3.1 Analysis
Qualitative visualization: For the purpose of contrasting the behavior of traditional ensembles vs ensemble of averages, we begin by qualitatively studying the stability of out-domain performance of these two ensembling techniques during the training process. To do so, use the Terralncognita

---

without
averaging
Accuracy (%)
Accuracy (%)
50-
50
Validation Accuracy
Validation Acturacy
Accurayy
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
tteraiion
Iterations
Figure 2: Ensemble of moving averages (EoA) (right) has better out-domain test performance stability compared with ensemble of online models (left), w.r.t in-domain validation accuracy. Details: The plots are for the Terralncognita dataset with domain L338 used as the test domain, and others as training/validatio domain, and ResNet-50. Each ensemble has different models from independent runs with different random seeds, hyper--arameters, and training/validattnn split.
dataset, and fix one of its domains as the test domain while using the others as training/validattnn data. We then train 6 different models independently for 5, 000 iterations with different seeds, syper-parameters and moving average models corresponding to each of these models. At every 300 iterations, we form an ensemble of the 6 online models from their corresponding runs and compute the out-domain test accuracy. Since, each run has a different ciiiiing-vlidatiinnn split, we calculate the mean validation accuracy of each of these online models at that iteration. We follow an identical procedure for the moving average models and plot these performances in Figure 2
splits identical to the [18] protocol. We also maintain
We find that the ensemble of averages has better stability out-domain test set compared to the ensemble of online models.
For clarity, note that this procedure for calculating test accuracy at regular intervals different from what we proposed carlier for EoA for practical purposes. This experiment is only meant to highlight the fact that making predictions on out-domain data using an ensemble of online models suffers from instability along the optimization trajectory. while an ensemble of averages mitigates this issue. For plots on other domains of Terralncognita, see Figure 10 in the Appendix.
Rank correlation: We now measure the rank correlation between in-domain validation accuracy and out-domain test accuracy for a quantitative evaluation. The details of the metric and motivations behind this experiment are same as those described in section 2.3. Here we use the same experimen- tal setup described in the qualitative analysis above. But in addition, on VLCS, OfficeHome and DomainNet datasets. The results are shown in Table 3 (and Table 9 in Appendix). We find that in majority of the cases, using EoA results in significantly better rank correlation compared to using the online model ensemble. These results show more concretely the fact that predictions by an ensemble of online models on out-domain data suffers from instability along the optimization trajectory, and EoA mitigates this problem.
also conduct experiments
3.2
Why does Ensembling and Model Averaging Improve Performance?
We explain the performance boost achieved by ensemble of averages (see next section) by adapting the with one-hot labels, the Bias- Variance decomposition is given as (49,
decomposition [17 to the domain generalization setting. For classification tasks
Ex,yETCCE( f(x;
Ex.y (C,(y, + Ex.7 (F)()) T))
Blas?
Variance
where C. denotes the cross entropy loss, KT. denotes KI. divergence, T (xxf" ) are N IID samples drawn from the in-domain training distribution pin, f(x; T) denotes the prediction of the model ff sample x such that the model is trained on the dataset T, and f(x) Ex(f(x T) Finally (x. y) ~ pout where pout is the out-domain distribution. Notice how T and (x, y) come from different distributions. For instance, in PACS dataset, pir could be the union of art, cartoon and photo domains, and ppout could be the sketch domain.
The L.H.S. of the above equation is the expected cross entropy loss on the out-domain distribution achieved by individual models, i.e., when we train an individual model on particular instance the training dataset T. the expected out-domain test loss is denoted by L.H.S. Importantly, the Bias term on the R.H.S. denotes the expected cross entropy loss on the out-domain distribution achieved by the function f(), which is essentially ensemble. Finally, the variance term captures how much the
5

---

Test Domain Accuracy (80
52
02
Accuracy
0.9
Ensemble Size
Ensemble Size
(W/ PACS (w/o MA)
IN/ MA)
OfficeHomm MA)
Terralncognita (w/
MA] DomainNet (W/o MA)
-4- VLCS IW/O MA)
OfficeHomme (wio MA)
Terraincognita (w/o MA)
Figure 3: Left: Effect of ensemble size (number of models in ensemble) out-domain perfor- mance (mean and standard error) for models with and without moving average (MA) parameters for ResNet-50 pre-trained ImageNet. Right: Using the performance of ensemble of size 1 (shown in the left plot) as reference, right plot shows the percentage point improvement for ensembles of size 1. The plots show that i) ensemble of averages (solid lines in left plot) are consistently better than ensemble of models without averaging (dashed lines in left plot); ii) ensemble of averages consistently improves performance over averaged models (ensemble of size in right plot).
prediction of individual models differs in expectation from the ensemble prediction, which makes this term strictly greater than zero.
Therefore, the above decomposition tells that the expected test domain error of ensemble is strictly less than that an individual model. This interpretation directly explains why traditional ensemble of unaveraged models can be expected to perform better than individual unaveraged models. However, it is still not clear why EoA performs better that traditional ensemble in practice. To establish this connection, models to form a traditional ensemble due to computational constraints. Thus such ensembles do not behave identically to the expected ensemble f(.) described above. Model averaging on the other hand has been shown to approximate an ensemble [3 To see this, consider without any loss of generality that the ensemble contains models with parameters {01, f2...0r), and denote 0r := 7 > Oc. Then note that the second order Taylor's expansion around 0r of each model's hth dimension's prediction is given by,
note that in practice, we typically train small number of independent
>(0r (0r 0,) 0f(rr) aθT
+ 0.5( -0) ((r()r) (0r - 02) 00?
100)
T
T
Notice that f(-) is the model output and there- fore the first and second order terms the derivatives of the model output and not the loss gradient and Hessian. The first order term is zero due to Òr = + Et= O., A crucial dif- ference of our analysis compared to [23] is that they average model states that near different loss minima, while perform tail averaging.
10a
Logit Second Term
Frequency
10:
10a
-10.0
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
Therefore, the term (0, O.) may not behave similar to that their case. To shed light on its behavior, plot the histogram of the second order term and the moving average model's logit f0r)) in Eq. 3 for the first dimension (A: = 1) for test domain data figure (details and ad- ditional experiments provided in Appendix D).
Output (dimension 1)
Figure 4: The scale of terms- moving average model's logit and the second order term in Eq. 3 The latter concentrates around O, suggesting our model averaging protocol approximates en- sembles.
The histogram shows that the second order term concentrates near zeros while the logit values span wider range, which implies that under the second order approximation, the model averaging protocol used our work behaves like an ensemble. Finally, to study the impact of ensemble size on out-domain performance, we plot the test domain accuracy as a function of ensemble size figure The plots show that i. EoA outperforms tradi- tional ensembles for all ensemble sizes (left); and ensembles of larger size typically have better

---

Table 4: Performance benchmarking 5 datasets of the DomainBed benchmark using two different pre-trained models. SWAD and MIRO are the previous SOTA. See Table 10 in Appendix for comparison with more methods. Note that ensembles do not have confidence interval because an ensemble uses all the models to make a prediction. Gray background shows our proposal. Our runs implies we ran experiments, but we did not propose it. Experiments use the training-domen
validation protocol from [8].
Algorithm
PACS
VLCS
OfficeHome
Terralncognita
DomainNet
Avg.
ResNet-50 (25M Parameters, Pre-trained ImageNet)
ERM (our runs)
84.4 + 0.8
77.1 = 0.5
66.6 + 0.2
48.3 + 0.2
43.6 + 0.1
64.0
Ensemble (our runs)
87.6
78.5
70.8
49.2
47.7
66.8
ERM
[8
85.7 0.5
77.4 + 0.3
67.5 + 0.5
47.2 0.4
41.2 + 0.2
63.8
SWAD
88.1 0.4
79.1
70.6 0.3
50.0 + 0.4
46.5 = 0.2
66.9
MIRO [9]
85.4 - 0.4
79.0 - 0.
70.5 1.4
50.4 1.1
44.3 = 0.2
65.9
SMA (ours)
87.5 + 0.2
78.2 = 0.2
70.6 + 0.1
50.3 0.5
46 + 0.1
66.5
EoA (ours)
88.6
79.1
72.5
52.3
47.4
68.0
ResNeXt-50 32x4d [48] (25M Parameters, trained IB Images)
ERM (our runs)
88.9 + 0.3
79.0 = 0.1
70.9 + 0.5
51.4 +: 1.2
48.1 18.1 + 0.2
67.7
Ensemble (our runs)
91.2
80.3
77.8
53.5
52.8
71.1
SMA (ours)
92.7 0.3
79.7 0.3
78.6 + 0.1
53.3 0.1
53.5 + 0.1
71.6
EoA (ours)
93.2
80.4
80.2
55.2
54.6
72.7
RegNetY-16GF [40] (81M Parameters, Pre-trained on 3.6B Images)
ERM (our runs)
92 1 0.4
78,6 1 0.6
73.8 1 0.5
55.6 1 0.9
53.1 +: 0.2
70.6
Ensemble (our runs)
95.1
80.6
80.5
59.5
57.8
74.7
ERM
89.6 0.4
78.6 0.3
71.9 0.6
51.4 + 1.8
48.5 = 0.6
68.0
SWAD
9
94.7
79.7 0.2
0.1
57.9 =.7
53.6 0.6
73.2
MIRO
[9]
97.4 0.2
79,9 0.6
80.4
58.9 1.3
53.8 = 0.1
74.1
SMA (ours)
95.5 + 0.0
80.7 0.1
8.0
59.7 0.0
60.0 = 0.0
75.6
EoA (ours)
95.8
81.1
83.9
61.1
60.9
76.6
out-domain performance. See discussion functional diversity of ensembles vs model averaging in Appendix
4 Empirical Results
4.1
DomainBed Benchmarking
We now benchmark our model averaging protocol (SMA) and ensemble of averages against online models (ERM, without MA) and ensemble of online models (ensembles). Note that all these models are trained using the ERM objective as before. We evaluate on PACS (27). VLCS [13, OfficeHome
45 TerraIncognita
3 and DomainNet 35 datasets in DomainBed. The training-evaluation
protocols are the same as described in section 2.3 for moving average and online models, and in section 3 for ensembles. Full details can be found in section B in the Appendix.
Comparison with existing results using ResNet-50 pre-trained on ImageNet: Here we compare existing methods with our runs. All methods use ResNet-50 (25M parameters) [19] pre-trained on ImageNet as initialization. Comparing ERM [8] and ERM (our runs), we find that they perform similarly, especially considering we have used a smaller hyper-paramete space (further discussion in Appendix ED. comparison between SWAD and SMA shows that SWAD is slightly better (by 0.4% on average). However, recall that our protocol retains the advantage of not tuning any hyper- parameters while SWAD has 3 additional ones that they tune separately in addition to the optimization hyper-parameters. Interestingly, traditional ensembles and SMA achieve similar performance (66.8% and 66.5% respectively). Finally, EoA outperforms all the existing results: ERM by 4% and SWAD (previous SOTA) by 1.1%. Importantly, note that while all non-ensemble models report the average test accuracy of multiple models following the protocol of [18), EoA test accuracy is achieved by a single predictor that combines the output of multiple models.
Experiments with larger pre-training datasets and larger models: addition to ResNet-50 pre- rre-trained on ImageNet, we now also experiment with ResNeXt-50 32x4d (25M parameters), that is pre-trained using semil supervised objective on Instagram
images and ImageNet labeled data (88, and RegNetY-16GF (81M parameters) pre-trained using Instagram 3.6B images. Note that both 5e and ResNeXt-50 32x4d have similar number of parameters, while RegNerY-16GF has more than 3x the number of parameters. On the other hand, also notice that the three architectures

---

are respectively pre-trained on an increasing size of datasets. The rationale behind this choice is that recent trends in deep learning has shown that models pre- re-ttrinnd on larger datasets and architectures achieve better downstream transfer performance 12 32 20] Therefore, we expect the latter models to improve the ERM baseline, and our goal is to investigate the out-domain performance gain by model averaging and EoA relative to the corresponding ERM baseline with increasing pre-training dataset size and model size.
The experimental results are shown in Table 4
To investigate models with the same size, but one pre-trained on larger dataset, we compare the results of Re and ResNeXt-550 32x4d. On average across all five datasets, the gain of SMA over ERM (our runs) is 2.5% for ResNet-50 and 3.9% for ResNeXt-50 32x4d. The gain of EoA over ERM is larger: 4% vs 5% respectively. This suggests that pre- vre-training the model on larger dataset increases the gain of model averaging and EoA over the corresponding ERM baseline, while the ERM performance itself improves.
Next, to investigate the impact of both larger model size and larger pre-training dataset, we compare the results of ResNeXt-50 32x4d and RegNetY-I6GF. On average across all five datasets, the gain of SMA over ERM (our runs) is 3.9% for ResNeXt-50 32x4d and 5% for RegNerY- The gain of EoA over ERM is again larger: 5% 6% respectively. This suggests that increasing both model size and pre-training dataset size allow model averaging and EoA to provide larger out out-domain gains over the corresponding ERM baseline. Notice that these claims are different from existing work (20]. which states that the baseline ERM performance improves with larger pre- trainingn data and model size.
4.2 In-domain Performance Improvement using Model Averaging
We study the in-domain test accuracy on PACS and OfficeHome datasets using ImageNet pre- trained ResNer-50 with and without our SMA protocol. In this experiment, we combine all the domains of PACS and split it into training/- validation/test splits 10.8//.1/0.1,, We run 10
Table 5: SMA outperforms ERM without model averaging in the IID setting.
Algorithm
PACS
OtficeHome
ERM (no averaging)
94.39
0.46
77.09 1 0.57
SMA (ours)
96.77
0.20
83.56 1 0.21
different runs with different seeds and randomly chosen splits for each dataset. The best model for
each run is chosen using the validation set. The remaining optimization details are identical those used in the previous section. The test accuracy mean and standard error using these best models are shown in Table 5] As expected, SMA outperforms models without averaging.
Related Work
5.1 Model Averaging
theoretical perspective: In our model averaging protocol, we compute a simple moving average of the model parameters starting early during training. This is known as tail-averaging [24]. which is slightly different from Polyak- Ruppeet averaging (36) in that the latter starts averaging from the very beginning of training. In the context of least square regression in the IID setting, [24] theoretically study the behavior of tail averaging and show that the excess risk of the moving average model is upper bounded by a bias and a variance term. This bias term depends on the initialization state of the parameter, but interestingly, it decays exponentially with ty, where to is the iteration at which model averaging is started. The variance term on the other hand depends on the covariance of the noise inherent in the data w.r.t. the optimal parameter, and is shown to decay at a faster rate when using model averaging, as opposed to a slower rate without averaging. This motivated them to propose tail-averaging.
Model averaging has also been shown to have a regularization effect (34] similar to that of Tikhonov regularization (4). This regularization has been classically used in ill-posed optimization problems (typically least squared regression), which are under-speciieed This property provides an interesting connection between model averaging and the under-specite seeeifsspciiiittionn problem discussed in [10], where the authors perform large scale experiments showing that the performance of multiple over werr-prameterized deep models, trained independently with different hyper-parameters and seeds, have high variance on out-domain data, even though their in-domain performances are very close together. Based on this connection, a simple intuition why one can expect model averaging to help in domain generalization is its Tikhonov regularization effect. However, this intuition requires more thorough investigation.
8

---

SWAD 8
SWAD propose flat minima as means for improving domain generalization. Following the intuition of stochastic weight averaging (SWA, (23). they use model averaging to find flat minima. However, their proposal is different from sampling model states at regular intervals and towards the end of training (as done in SWA). SWAD selects contiguous model states along the optimization path for averaging, based on their validation loss. This is done to prevent including an under-performe state (determined using the in-domain validation set) in the moving average model. SWAD however adds additional hyper-parameters of its own: the validation loss threshold below which the the model states are selected, and patience parameters (number of iterations that determine the start and end of the averaging process). Note that this also requires computing validation loss more frequently during training. In this context, we show that instead of finding the start and end period for model averaging meticulously, we can simply start model averaging early during training and continue till the end. This difference arises from the fact that SWAD uses the online network to calculate validation performance while we use the SMA model in our protocol. This is explained further in section 2.2 The benefit our observations provide over SWAD is that they allow us to take advantage of model averaging without the additional hyper-parameters and compute required by SWAD.
5.2 Domain Generalization
Existing methods aimed at domain generalization can be broadly categorized into techniques that perform domain alignment, regularization, data augmentation, and meta-leamnngg Domain alignment is perhaps the most intuitive direction, in which methods aim to learn latent representations which have similar distributions across different domains 41 30 39] 377. There different variants of this idea, such as minimizing some divergence metric between the latent representation of different domains (E.g. DANN (16)). ). less strictly, minimizing the difference between the latent statisties of different domains (E.g. DICA 33 CORAL ). In the meta learning category, source domains are typically split into 2 subsets to be used as the training and test domains in episodes simulate the domain generalization setting 28 29 Data augmentation is also a popular tool used for improving domain generalization. ranges from introducing various types of augmentations to simulate unseen test domain conditions (E.g. style transfer [50] (2)) to self-supervised learning involving matching the representations of an image with different augmentations (E.g. 7 Finally, different ways of regularizing models (implicit and explicit) have also been developed with the goal of encouraging propose regularization such that the classifier is optimal in all the environments. Representation Self- Challenging (1 propose to suppress the dominant features that get activated on the training data, which forces the network to use other features that correlate with labels. Risk extrapolation [26) propose a regularization that minimizes the variance between domain-wise loss, in the hope that is representative of the variance including unseen test domains. See [5] for survey on DG methods.
feature learning 33
47 46). For instance, invariant risk minimization
Our investigation in this work is complementary to all these domain generalization methods. Addi- tionally, one of our main focus is to also study and improve performance instability on out-domain data during training. which results in more reliable model selection. This aspect has not received much attention.
6 Conclusion
We investigated hyperparameter-ffre and efficient protocol for model averaging in the ERM framework, and showed that it provides a significant boost to out-domain performance compared to un-averaged models. Building on this observation, we showed that an ensemble of moving average models performs better compared to an ensemble of un-averaged models. Importantly, we showed that in both cases, model averaging significantly improves the rank correlation between in-domain validation accuracy and out-domain test accuracy, which is crucial for reliable model selection using in-domain validation data. We experimented with three pre- traiiriied models with increasing pre- -reiirinn dataset and model size, and found that EoA provides a proportionally larger gain compared to the corresponding ERM baseline, and lies in the range of 4% 6%. Finally, we explain the performance boost of EoA by adapting the Variance trade-off perspective to the domain generalization setting. Further discussions along with limitations of our work are provided in Appendix E

---

References
[1] Isabela Albuquerque, Nikhil Naik, Junnan Li, Nitish Keskar, and Richard Socher. Improving out-of-distribution generalization via multi-task self-supervised pretraining. arXiv preprint Stii::0003.135255 2020.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv: 1907.0299, 2019.
[3] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita, In Proceedings of the European conference on computer vision (ECCV), pages 456 3556-73,2 2018.
[4] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
[5] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifi- cation tasks to new unlabeled sample. Advances in neural information processing systems, 24:2178-2186, 2011.
[6] Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
[7] Silvia Bucei, Antonio D'Inncentte, Yujun Liao, Fabio Maria Carlucci, Barbara Caputo, and Tatiana Tommasi. Self-supervised learning across domains. IEEE Transactions on Pultern Analysis and Machine Intelligence, 2021.
[8] Junbum Cha, Sanghyuk Chun. Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. arXiv preprint arXiv:2 arXiv:2102.08604 2021.
[9] Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual-informatiin regularization with pre-trained models. arXiv preprint 10X:::2003.10789, 2022.
[10] Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, al. Under- specification presents challenges for credibility modern machine learning. arXiv preprint arXiv:2011.03322 2020.
[11] Thomas G Dietterich.
Ensemble methods in machine learning. International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[13] Chen Fang, Ye Xu, and Daniel Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pages 1657-1664, 2013.
[14] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv: arXiv:191..02757/ 2019.
[15] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pages 1050-1059. PMLR, 2016.
[16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):209-2008 2030, 2016.
[17] Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
10

---

[18] Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of-distribtiinn generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8340-8349, 2021.
[21] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross- domain generalization. In Computer AisionEECY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 124-140. Springer, 2020.
[22] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[23] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:180.004407, 2018.
[24] Prateek Jain, Sham Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Paral- lelizing stochastic gradient descent for least squares regression: muni-batching averaging, and model misspecification. Journal of Machine Learning Research, 18, 2018.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6988, 2014.
[26] David Krueger, Ethan Caballero, Joer--Herik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrap- olation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
[27] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
(28) Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
[29] Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1446-1455, 2019.
[30] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018.
[31] Ya Li, Xinmei Tian, Mingming Gong. Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 624.639, 2018.
[32] Dhruv Mahajan, Ross Girshick. Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018
[33] Krikamol Muander, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pages 10-18. PMLR, 2013.
11

---

[34] Gergely Neu and Lorenzo Rosasco. Iterate averaging as regularization for stochastic gradient descent. In Conference On Learning Theory, pages 2222-3242. PMLR, 2018.
[35] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1415-145,, 2019.
[36] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838-855, 1992.
[37] Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distribution generalization. arXiv preprint arXiv:2109.02934 2021.
[38] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case 2019.
generalization. arXiv preprint 19Iiv:991..80
[39] Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint urXiv:2104009933 2021.
[40] Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. arXiv preprint arXiv:2201.08377 2022.
[41] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision. pages 443-450. Springer, 2016.
[42] Andrey Nikolayevich Tikhonov. On the stability of inverse problems. In Dokl. Akad. Nauk SSSR, volume 39, pages 195-198, 1943,
[43] Vladimir Vapnik and Vlamimir Vapnik. Statistical learning theory wiley, New York, 1(624):2. 1998.
[44] Ramakrishna Vedantam, David Lopez-Paz, and David J Schwab. An empirical investigation of domain generalization with empirical risk minimizers. Advances in Neural Information Processing Systems, 34, 2021.
[45] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5027, 2017.
[46] Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3622-3626. IEEE, 2020.
[47] Minghao Xu, Jian Zhang, Binghing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6502-6509, 2020.
[48] I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1955.0055: 2019.
[49] Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias- variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pages 10767-10777 PMLR, 2020.
[50] Xiangyu Yue, Yang Zhang. Sicheng Zhao, Alberto Sangiovanni-VVnnentllli Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to to-rall general- ization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2100-2110, 2019.
[51] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, arXiv preprint arX1v:2103.02503, 2021.
12

---

[52] Kaiyang Zhou, Chen Change Loy, and Ziwei Liu, Semi-supervised domain generalization with stochastic stylematch. arXiv preprint (rXi::210000592,, 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Appendix D. we have discussed various aspects including the limitations of our and existing methods in addressing
domain generalization, functional diversity of ensembles model averaging strategy, and more.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See section
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you's 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "s0nrVNAEeF6", "reviewer": "Reviewer_ao9C", "summary": "Authors point out the chaotic behavior exists even along the training optimization trajectory of a single  model, and propose a simple model averaging protocol that both significantly boosts domain generalization performance. Authors also show that instead of simple moving average, ensembling moving average models from multiple runs can boost the performance even further. Authors show results on various popular datasets and show the effectiveness of proposed method. ", "questions": "1) SMA - Line # 54 SMA abbreviation is used before actually defining it. \n\n2) Results on datasets: Take an example of PACS. Authors show average results on PACS dataset. But in appendix results on per domain show substantially worse results on Photos. Is it surprising or expected? Some discussion on this will be really helpful. \n\n3) What is the time complexity difference between SMA and EoA? It looks like EoA may have much higher complexity but performance gains are not much over SMA. Can authors comment? \n\n4) Would using SMA or EoA on top of some existing methods (e.g. MIRO) improve the performance further? Is there anything that stops us from combining SMA with other methods too? \n\n5) How would one scale the current approach when number of domains are really large? Do authors see any issue in this? \n\n6) Right y-axis of Figure 1: \"trainig\" → training \n\n", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n\n1) Figure 1 explains the motivation and intuition of what authors want to do. \n2) Paper is clearly written and even though idea is simple, authors show the effectiveness of the proposed method. \n\n\nWeaknesses\n1) Complexity and run times compared to existing methods should be discussed \n2) Summary of limitations should be discussed in the paper \n \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kf8U-1wx0zk", "reviewer": "Reviewer_UuXX", "summary": "This paper shows that simple model averaging over the course of training improves the rank correlation between in-domain validation accuracy and out-of-domain test accuracy. They also show that ensembling these averaged models further improves this rank correlation, more so than ensembling unaveraged models. Finally, they illustrate that this improved rank correlation translates to improved performance on DomainBed, given how crucial it is for model selection or early stopping in domain generalization.", "questions": "- Does SMA, or learning rate decay, reliably improve other DG methods? E.g. IRM, VREx, CORAL, etc.\n- If the model is trained from scratch without pre-training, would you expect a larger t_0 to be needed? How would you select this hyperparameter?\n- Possible to move Table 7/8/9 into the main paper? In my opinion, these are the most insightful results.\n- [Minor] Why do the authors choose the notation $\\hat{\\theta}_t$ over say $\\bar{\\theta}_t$ for the averaged parameters?\n- [Minor, comment] Table 4: clarify in caption that training-domain validation set was used for model selection.\n", "limitations": "Yes, in Appendix E.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- *Clear, well-written paper.* \n- *Comprehensive and insightful experiments:* There has been a recent trend of cherry-picking DG datasets, so I appreciated the wideness of the evaluation here, with only CMNIST and RMNIST left out of DomainBed (understandably so).\n- *Simple and effective:* Averaging models, or simply decaying the learning rate (see below), is a very simple idea. Despite being well-explored outside of DG, its effectiveness for model selection in DG is quite surprising, and should perhaps become the standard in DG benchmarks where reliable model selection hinders fair comparison.\n\n**Weaknesses:**\n- *The simple moving average (SMA) is just learning rate decay and this connection is not made.* \n    - Let $\\theta_t = \\hat{\\theta}\\_{t-1} - \\eta \\nabla_{\\theta} L$, with $\\eta$ the learning rate, and $t_0=0$. Then we can rewrite Eq. 1 of the paper as:\n\\\\begin{align}\n\\\\hat{\\\\theta}\\_t &=  \\\\frac{t}{t +1} \\\\cdot \\\\hat{\\\\theta}\\_{t-1} + \\\\frac{1}{t + 1} \\\\cdot \\\\theta_t \\\\\\\\\n&= \\\\frac{t}{t +1} \\\\cdot \\\\hat{\\\\theta}\\_{t-1} + \\\\frac{1}{t + 1} \\\\cdot (\\\\hat{\\\\theta}\\_{t-1} - \\\\eta \\\\nabla_{\\\\theta} L ) \\\\\\\\\n&= \\\\hat{\\\\theta}\\_{t-1} - \\\\frac{\\\\eta}{t + 1} \\\\nabla_{\\\\theta} L \\\\\\\\\n\\\\end{align}\n    - Thus, a simple moving average of model weights is equivalent to a linear learning rate decay.\n    - Adding back in t_0 allows a standard warm-up period before decaying the learning rate, and corresponds to tail averaging. \n    - Indeed, the stochastic weight averaging paper (SWA, [1]) discussed the importance of learning rate scheduling, and in fact they decay the learning rate for the first 75% of training, then set the learning rate to a high constant value for the remaining 25% of the time in order to increase the diversity of the solutions / optima found. This paper essentially removes this diversity-seeking trick, which is also employed in the DG follow-up paper SWAD [2], and reverts to decaying the learning rate until the end of training. The diversity for model ensembling then comes from different initializations.\n    - *Main issue:* Model averaging, and “ensembles of averages”, takes center stage in this paper. However, it just translates to using linear learning rate decay when ensembling, and this connection is never made.\n    - *Potential remedies:* \n      - Make this connection clear throughout the paper, particularly in Section 2 and in the related work section. This could include a discussion of how SWA and SWAD deliberately avoid learning rate decay in the tails in order to arrive at more diverse solutions, which is perhaps undesirable for DG model selection. \n      - An experiment where linear learning rate decay is employed instead of model averaging, as this should be equivalent. \n      - Discuss works which explored the benefits of learning rate decay.\n    - I will increase my score if this can be adequately addressed.\n- *Not exactly hyperparameter-free, as claimed:*\n    - Throughout the paper, it is stressed that the proposed approach is hyperparameter free, and that this is a major advantage over SWAD.\n    - However, the method still requires specification of a start iteration t_0 and an averaging frequency. For t_0, the authors “arbitrarily choose 100 as the start iteration”. While Appendix C demonstrates that the proposed method is not too sensitive to these values for the datasets and pretrained models explored in the paper, they likely still need to be selected for new datasets and models, particularly if the models are trained from scratch.\n    - *Remedy:* tone down the hyperparameter free claim/statement, particularly in the contribution list of the introduction. Can still say it is much less sensitive than SWAD, which it appears to be.\n\n[1] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018). Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence (pp. 876-885). \n\n[2] Cha, J., Chun, S., Lee, K., Cho, H. C., Park, S., Lee, Y., & Park, S. (2021). SWAD: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems, 34, (pp. 22405-22418).\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N6BlG6P6e5q", "reviewer": "Reviewer_vuPK", "summary": "This paper proposes an approach to improve the test accuracy of the model when evaluated on a test dataset from a different domain. The methods used for this purpose include the moving average of model weights and the ensemble of moving averaged models. Empirical experiments are conducted to show the effectiveness of the proposed method.", "questions": "* Ablation study 1) on the impact of the start iteration t0. The authors claim that “starting averaging close to initialization results in improved out-domain performance XXX”. The reviewer is not convinced by this statement based on the empirical results. It is true that  there is an increase of test accuracy at iteration 100, but this does not imply the statement the authors have. If the statement is correct, does the method gives the best performance at iteration 10 or even 1? Finer grids and more solid experiment results are needed for the statement.\n* In the analysis on the ensemble in Section 3.2, in the Taylor expansion term, it is shown that the output of EoA is roughly the same as the model evaluated at $\\hat \\theta_T,$ which is the average of the model parameters over time. Can you explain why $f(\\hat \\theta_T)$ leads to a better performance?\n* The experiment in Appendix C.6 on cross-run rank correlation claims that “in-domain validation performance based model selection is not a reliable approach for selecting a model from a pool of multiple independently trained models.” The statement seems interesting. If the statement is true, then how do you choose your model in practice since conceptually we have a pool of models trained with different learning rates, batch size, etc. In this case, how do you perform model selection?\n* In Table 8 for the spearman correlation,  in for example Photo dataset, with average leads to negative correlation. You have claimed that closer to 1 is better. However, -0.38 implies a stronger correlation between two outputs then 0.09. Can we claim that with average still make the early stopping reliable given that they are negatively correlated?\n* The paper needs to be better proofread:\n\t* L76, \\cdot not \\ldot\n\t* L87 reference needed for Polyak-Ruppert averaging \n\t* L201, should not include “not”?", "limitations": "I really like that the authors have a full section discussing the limitations of their work. The limitations and questions of the paper the reviewer had is given in Questions section.\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Originality: The techniques used in the paper are not new and are well studied in other areas. The use of these techniques in domain generalization is however new. Related work is cited and discussed.\n* Quality: The methods are used appropriate, the work is complete and I really like the fact that the authors are not overselling their work. However, I am not fully convinced by some of the arguments in the paper based on the experiment results. See Questions for details.\n* Clarity: The presentation needs some improvement. \n\t* There are a lot of cross references in the paper that makes the paper a bit difficult to follow. For example, in section 2.3 of the ablation study, the authors refer the readers to Appendix C for the experiment result where Appendix C refer the readers to Appendix B for experiment details. The reviewers find a lot of such references in the paper. \n\t* The paper cited a lot of reference for experimental details (see for example L150 for the protocol), it is therefore unclear to judge purely from this paper whether an expert is able to reproduce the paper. The code for the paper is submitted in the supplementary material, which is helpful for reproducing purposes. \n\t* The figures in the paper need to be presented in a different way. It is often the case that for the test accuracy the y-axis have only two ticks 0 and 50 and it is very hard to tell the information from the figure.\n* Significance: Based on the experiment result, the proposed ensemble method can beat the SOTA while the SMA has comparable performance with the SOTA. While the performance of SMA is not as good as the SOTA, the authors claim that the proposed method has computational efficiency. The results, while not earthshaking, may be of practical use in real applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ByLBr90TY42", "reviewer": "Reviewer_ibW9", "summary": "This paper proposes two different models for OOD generalization, evaluated on the DomainBed benchmark across pretrained encoders and compared to representative baselines. First, it introduces a new hyperparameter free model averaging (MA) strategy during training called SMA, inspired by tail-averaging in optimization. SMA averages all parameters from a pre-defined starting point. Second, it proposes to ensemble several SMAs (EoA); EoA achieves SOTA results on DomainBed and mitigates instability in OOD performance across training. \n\nIn addition to these two models, it proposes a new model selection procedure for MA that performs early stopping and inference with SMA instead of the online model. This model selection procedure is justified by an extensive study of OOD performance stability throughout training and rank correlation between validation and test performance. \n\nFinally, it explains the success of ensembling and MA using the Bias-Variance decomposition, adapted to a domain generalization setting.", "questions": "* How realistic is the theoretical analysis w.r.t. experimental setup?\n    * the proposed bias-variance decomposition does not account for different seeds and hyperparameters used in the experiments.\n    * l188 “the expected test domain error of an ensemble is strictly less than that of an individual model” + “that of ensembles is dominated by the bias term alone, and is thus strictly lower” l661: this analysis is valid only when the ensemble has infinite number of members. In practice, an ensemble has finite number of members, here 6 members for EoA, such that this analysis is unrealistic.\n    * Authors mention that MA has less diversity than ensembles which consider different datasets, hyperparameters and seeds l678-679. This puts into question the validity of l195 “model averaging on the other hand has been shown to approximate an ensemble” and of the subsequent analysis.\n\n* Unjustified statements in the theoretical section: \n    * why does EoA outperform traditional ensembling theoretically? \n    * “ensembles of larger size typically have better out-domain performance.\" l219-220; not explained by the analysis.\n    * “$\\hat{\\theta}_T-\\theta_t$ may not behave similar to that in their case” l207-208. Could you clarify this statement? How valid is the second order approximation of Izmailov2018 for tail-averaging if the behavior is not the same? \n    * “the model averaging protocol used in our work behaves like an ensemble” l215-216: the outputs of the ensemble should be also plotted along the first and second terms to show that (or a distance between predictions). Here, only the first and second terms are plotted.\n\n* Additional experiments:\n    * How does SMA compare to SWA or SWAD in-domain? Section 4.2 compares only SMA to ERM. Cha2021 already showed that SWAD improved ERM in-domain so these conclusions are not new.\n    * EoA should be evaluated on a same training / validation split to be fairer (cf limitation section).\n\n* MIRO with RegNetY-16GF is reported at 74.1 and not. 70.1 in the original paper.\n\nCha2021: SWAD: Domain Generalization by Seeking Flat Minima\n", "limitations": "The limitations highlited by the authors are interesting. Yet, several important limitations are not enough discussed / missing:\n\n* How realistic is the theoretical analysis w.r.t. experimental setup? C.f. question above. \n\n* The SoTA results are obtained with two major limitations which are not discussed. This puts into question the fairness of the comparison.\n    * high inference cost of EoA and ensembling: 6 times higher than the baselines; this important question is not discussed by the authors.\n    * the ensembling and EoA models see more data than baselines as there are trained on different training/validation splits. It seems normal that the results are better than using a single training/validation split. A fairer evaluation would only consider a same training/validation split.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Ensembling MAs is novel to my knowledge and this model achieves SoTA results on the challenging DomainBed benchmark (yet, open question of fairness of comparison c.f. below). Authors performs thorough empirical validation (across datasets, architectures) and ablation studies. The stability of MA models w.r.t. online models OOD is interesting and well evaluated. \n* SMA and the corresponding model selection procedure are not groundbreaking but have practical use: SMA removes hyperparameters in SWAD, the model selection procedure is more robust than in SWAD. It is thus more applicable than the original work SWAD that motivated this paper.\n* It also proposes some theoretical results for explaining the success of ensembling and MA OOD by adapting the bias-variance decomposition to domain generalization (yet with several limitations detailed below). \n* The paper is well written and easy to follow.\n\nWeaknesses:\n\n* Limited novelty of some contributions:\n    * SMA is mostly based on related models (SWA and SWAD); it “simply” removes unnecessary hyperparameters but the principle of averaging models is unchanged. Ensembling SMAs is novel but this contribution is mostly empirical as a theoretical explanation of its success over standard ensembling is missing.\n    * The change in the model selection procedure of SMA vs SWAD is also minor although useful in practice; it would be interesting to see if better stability with MA applies to other learning objectives (e.g. domain-invariance criterias) to better highlight this contribution. \n    * The theoretical section simply applies the well-known bias-variance trade-off to domain generalization with minor consideration of the experimental setting (c.f. questions below). A big part of the theoretical results (Taylor expansion l198) tries to validate empirically an existing result (Izmailov2018). The novelty of this section would be improved if an explanation why EoA outperforms standard ensembling is provided, which is currently missing. \n\nTwo other weaknesses that justify my score:\n* Question on applicability of theoretical results to practise and unjustified statements in the theoretical section (see questions below). Missing explanation of the sucess of EoA.\n\n* Fairness of the comparison to baseline models (see limitations below). In particular, EoA and the ensemble baseline see more training data than other baselines. \n\nIzmailov2018: Averaging wights leads to wider optima and better generalization.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept