Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Authors point out the chaotic behavior exists even along the training optimization trajectory of a single  model, and propose a simple model averaging protocol that both significantly boosts domain generalization performance. Authors also show that instead of simple moving average, ensembling moving average models from multiple runs can boost the performance even further. Authors show results on various popular datasets and show the effectiveness of proposed method. 
strengths_and_weaknesses: Strengths 

1) Figure 1 explains the motivation and intuition of what authors want to do. 
2) Paper is clearly written and even though idea is simple, authors show the effectiveness of the proposed method. 


Weaknesses
1) Complexity and run times compared to existing methods should be discussed 
2) Summary of limitations should be discussed in the paper 
 

questions: 1) SMA - Line # 54 SMA abbreviation is used before actually defining it. 

2) Results on datasets: Take an example of PACS. Authors show average results on PACS dataset. But in appendix results on per domain show substantially worse results on Photos. Is it surprising or expected? Some discussion on this will be really helpful. 

3) What is the time complexity difference between SMA and EoA? It looks like EoA may have much higher complexity but performance gains are not much over SMA. Can authors comment? 

4) Would using SMA or EoA on top of some existing methods (e.g. MIRO) improve the performance further? Is there anything that stops us from combining SMA with other methods too? 

5) How would one scale the current approach when number of domains are really large? Do authors see any issue in this? 

6) Right y-axis of Figure 1: "trainig" → training 


limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper shows that simple model averaging over the course of training improves the rank correlation between in-domain validation accuracy and out-of-domain test accuracy. They also show that ensembling these averaged models further improves this rank correlation, more so than ensembling unaveraged models. Finally, they illustrate that this improved rank correlation translates to improved performance on DomainBed, given how crucial it is for model selection or early stopping in domain generalization.
strengths_and_weaknesses: **Strengths:**
- *Clear, well-written paper.* 
- *Comprehensive and insightful experiments:* There has been a recent trend of cherry-picking DG datasets, so I appreciated the wideness of the evaluation here, with only CMNIST and RMNIST left out of DomainBed (understandably so).
- *Simple and effective:* Averaging models, or simply decaying the learning rate (see below), is a very simple idea. Despite being well-explored outside of DG, its effectiveness for model selection in DG is quite surprising, and should perhaps become the standard in DG benchmarks where reliable model selection hinders fair comparison.

**Weaknesses:**
- *The simple moving average (SMA) is just learning rate decay and this connection is not made.* 
    - Let $\theta_t = \hat{\theta}\_{t-1} - \eta \nabla_{\theta} L$, with $\eta$ the learning rate, and $t_0=0$. Then we can rewrite Eq. 1 of the paper as:
\\begin{align}
\\hat{\\theta}\_t &=  \\frac{t}{t +1} \\cdot \\hat{\\theta}\_{t-1} + \\frac{1}{t + 1} \\cdot \\theta_t \\\\
&= \\frac{t}{t +1} \\cdot \\hat{\\theta}\_{t-1} + \\frac{1}{t + 1} \\cdot (\\hat{\\theta}\_{t-1} - \\eta \\nabla_{\\theta} L ) \\\\
&= \\hat{\\theta}\_{t-1} - \\frac{\\eta}{t + 1} \\nabla_{\\theta} L \\\\
\\end{align}
    - Thus, a simple moving average of model weights is equivalent to a linear learning rate decay.
    - Adding back in t_0 allows a standard warm-up period before decaying the learning rate, and corresponds to tail averaging. 
    - Indeed, the stochastic weight averaging paper (SWA, [1]) discussed the importance of learning rate scheduling, and in fact they decay the learning rate for the first 75% of training, then set the learning rate to a high constant value for the remaining 25% of the time in order to increase the diversity of the solutions / optima found. This paper essentially removes this diversity-seeking trick, which is also employed in the DG follow-up paper SWAD [2], and reverts to decaying the learning rate until the end of training. The diversity for model ensembling then comes from different initializations.
    - *Main issue:* Model averaging, and “ensembles of averages”, takes center stage in this paper. However, it just translates to using linear learning rate decay when ensembling, and this connection is never made.
    - *Potential remedies:* 
      - Make this connection clear throughout the paper, particularly in Section 2 and in the related work section. This could include a discussion of how SWA and SWAD deliberately avoid learning rate decay in the tails in order to arrive at more diverse solutions, which is perhaps undesirable for DG model selection. 
      - An experiment where linear learning rate decay is employed instead of model averaging, as this should be equivalent. 
      - Discuss works which explored the benefits of learning rate decay.
    - I will increase my score if this can be adequately addressed.
- *Not exactly hyperparameter-free, as claimed:*
    - Throughout the paper, it is stressed that the proposed approach is hyperparameter free, and that this is a major advantage over SWAD.
    - However, the method still requires specification of a start iteration t_0 and an averaging frequency. For t_0, the authors “arbitrarily choose 100 as the start iteration”. While Appendix C demonstrates that the proposed method is not too sensitive to these values for the datasets and pretrained models explored in the paper, they likely still need to be selected for new datasets and models, particularly if the models are trained from scratch.
    - *Remedy:* tone down the hyperparameter free claim/statement, particularly in the contribution list of the introduction. Can still say it is much less sensitive than SWAD, which it appears to be.

[1] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018). Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence (pp. 876-885). 

[2] Cha, J., Chun, S., Lee, K., Cho, H. C., Park, S., Lee, Y., & Park, S. (2021). SWAD: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems, 34, (pp. 22405-22418).


questions: - Does SMA, or learning rate decay, reliably improve other DG methods? E.g. IRM, VREx, CORAL, etc.
- If the model is trained from scratch without pre-training, would you expect a larger t_0 to be needed? How would you select this hyperparameter?
- Possible to move Table 7/8/9 into the main paper? In my opinion, these are the most insightful results.
- [Minor] Why do the authors choose the notation $\hat{\theta}_t$ over say $\bar{\theta}_t$ for the averaged parameters?
- [Minor, comment] Table 4: clarify in caption that training-domain validation set was used for model selection.

limitations: Yes, in Appendix E.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes two different models for OOD generalization, evaluated on the DomainBed benchmark across pretrained encoders and compared to representative baselines. First, it introduces a new hyperparameter free model averaging (MA) strategy during training called SMA, inspired by tail-averaging in optimization. SMA averages all parameters from a pre-defined starting point. Second, it proposes to ensemble several SMAs (EoA); EoA achieves SOTA results on DomainBed and mitigates instability in OOD performance across training. 

In addition to these two models, it proposes a new model selection procedure for MA that performs early stopping and inference with SMA instead of the online model. This model selection procedure is justified by an extensive study of OOD performance stability throughout training and rank correlation between validation and test performance. 

Finally, it explains the success of ensembling and MA using the Bias-Variance decomposition, adapted to a domain generalization setting.
strengths_and_weaknesses: Strengths:

* Ensembling MAs is novel to my knowledge and this model achieves SoTA results on the challenging DomainBed benchmark (yet, open question of fairness of comparison c.f. below). Authors performs thorough empirical validation (across datasets, architectures) and ablation studies. The stability of MA models w.r.t. online models OOD is interesting and well evaluated. 
* SMA and the corresponding model selection procedure are not groundbreaking but have practical use: SMA removes hyperparameters in SWAD, the model selection procedure is more robust than in SWAD. It is thus more applicable than the original work SWAD that motivated this paper.
* It also proposes some theoretical results for explaining the success of ensembling and MA OOD by adapting the bias-variance decomposition to domain generalization (yet with several limitations detailed below). 
* The paper is well written and easy to follow.

Weaknesses:

* Limited novelty of some contributions:
    * SMA is mostly based on related models (SWA and SWAD); it “simply” removes unnecessary hyperparameters but the principle of averaging models is unchanged. Ensembling SMAs is novel but this contribution is mostly empirical as a theoretical explanation of its success over standard ensembling is missing.
    * The change in the model selection procedure of SMA vs SWAD is also minor although useful in practice; it would be interesting to see if better stability with MA applies to other learning objectives (e.g. domain-invariance criterias) to better highlight this contribution. 
    * The theoretical section simply applies the well-known bias-variance trade-off to domain generalization with minor consideration of the experimental setting (c.f. questions below). A big part of the theoretical results (Taylor expansion l198) tries to validate empirically an existing result (Izmailov2018). The novelty of this section would be improved if an explanation why EoA outperforms standard ensembling is provided, which is currently missing. 

Two other weaknesses that justify my score:
* Question on applicability of theoretical results to practise and unjustified statements in the theoretical section (see questions below). Missing explanation of the sucess of EoA.

* Fairness of the comparison to baseline models (see limitations below). In particular, EoA and the ensemble baseline see more training data than other baselines. 

Izmailov2018: Averaging wights leads to wider optima and better generalization.

questions: * How realistic is the theoretical analysis w.r.t. experimental setup?
    * the proposed bias-variance decomposition does not account for different seeds and hyperparameters used in the experiments.
    * l188 “the expected test domain error of an ensemble is strictly less than that of an individual model” + “that of ensembles is dominated by the bias term alone, and is thus strictly lower” l661: this analysis is valid only when the ensemble has infinite number of members. In practice, an ensemble has finite number of members, here 6 members for EoA, such that this analysis is unrealistic.
    * Authors mention that MA has less diversity than ensembles which consider different datasets, hyperparameters and seeds l678-679. This puts into question the validity of l195 “model averaging on the other hand has been shown to approximate an ensemble” and of the subsequent analysis.

* Unjustified statements in the theoretical section: 
    * why does EoA outperform traditional ensembling theoretically? 
    * “ensembles of larger size typically have better out-domain performance." l219-220; not explained by the analysis.
    * “$\hat{\theta}_T-\theta_t$ may not behave similar to that in their case” l207-208. Could you clarify this statement? How valid is the second order approximation of Izmailov2018 for tail-averaging if the behavior is not the same? 
    * “the model averaging protocol used in our work behaves like an ensemble” l215-216: the outputs of the ensemble should be also plotted along the first and second terms to show that (or a distance between predictions). Here, only the first and second terms are plotted.

* Additional experiments:
    * How does SMA compare to SWA or SWAD in-domain? Section 4.2 compares only SMA to ERM. Cha2021 already showed that SWAD improved ERM in-domain so these conclusions are not new.
    * EoA should be evaluated on a same training / validation split to be fairer (cf limitation section).

* MIRO with RegNetY-16GF is reported at 74.1 and not. 70.1 in the original paper.

Cha2021: SWAD: Domain Generalization by Seeking Flat Minima

limitations: The limitations highlited by the authors are interesting. Yet, several important limitations are not enough discussed / missing:

* How realistic is the theoretical analysis w.r.t. experimental setup? C.f. question above. 

* The SoTA results are obtained with two major limitations which are not discussed. This puts into question the fairness of the comparison.
    * high inference cost of EoA and ensembling: 6 times higher than the baselines; this important question is not discussed by the authors.
    * the ensembling and EoA models see more data than baselines as there are trained on different training/validation splits. It seems normal that the results are better than using a single training/validation split. A fairer evaluation would only consider a same training/validation split.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an approach to improve the test accuracy of the model when evaluated on a test dataset from a different domain. The methods used for this purpose include the moving average of model weights and the ensemble of moving averaged models. Empirical experiments are conducted to show the effectiveness of the proposed method.
strengths_and_weaknesses: * Originality: The techniques used in the paper are not new and are well studied in other areas. The use of these techniques in domain generalization is however new. Related work is cited and discussed.
* Quality: The methods are used appropriate, the work is complete and I really like the fact that the authors are not overselling their work. However, I am not fully convinced by some of the arguments in the paper based on the experiment results. See Questions for details.
* Clarity: The presentation needs some improvement. 
	* There are a lot of cross references in the paper that makes the paper a bit difficult to follow. For example, in section 2.3 of the ablation study, the authors refer the readers to Appendix C for the experiment result where Appendix C refer the readers to Appendix B for experiment details. The reviewers find a lot of such references in the paper. 
	* The paper cited a lot of reference for experimental details (see for example L150 for the protocol), it is therefore unclear to judge purely from this paper whether an expert is able to reproduce the paper. The code for the paper is submitted in the supplementary material, which is helpful for reproducing purposes. 
	* The figures in the paper need to be presented in a different way. It is often the case that for the test accuracy the y-axis have only two ticks 0 and 50 and it is very hard to tell the information from the figure.
* Significance: Based on the experiment result, the proposed ensemble method can beat the SOTA while the SMA has comparable performance with the SOTA. While the performance of SMA is not as good as the SOTA, the authors claim that the proposed method has computational efficiency. The results, while not earthshaking, may be of practical use in real applications.
questions: * Ablation study 1) on the impact of the start iteration t0. The authors claim that “starting averaging close to initialization results in improved out-domain performance XXX”. The reviewer is not convinced by this statement based on the empirical results. It is true that  there is an increase of test accuracy at iteration 100, but this does not imply the statement the authors have. If the statement is correct, does the method gives the best performance at iteration 10 or even 1? Finer grids and more solid experiment results are needed for the statement.
* In the analysis on the ensemble in Section 3.2, in the Taylor expansion term, it is shown that the output of EoA is roughly the same as the model evaluated at $\hat \theta_T,$ which is the average of the model parameters over time. Can you explain why $f(\hat \theta_T)$ leads to a better performance?
* The experiment in Appendix C.6 on cross-run rank correlation claims that “in-domain validation performance based model selection is not a reliable approach for selecting a model from a pool of multiple independently trained models.” The statement seems interesting. If the statement is true, then how do you choose your model in practice since conceptually we have a pool of models trained with different learning rates, batch size, etc. In this case, how do you perform model selection?
* In Table 8 for the spearman correlation,  in for example Photo dataset, with average leads to negative correlation. You have claimed that closer to 1 is better. However, -0.38 implies a stronger correlation between two outputs then 0.09. Can we claim that with average still make the early stopping reliable given that they are negatively correlated?
* The paper needs to be better proofread:
	* L76, \cdot not \ldot
	* L87 reference needed for Polyak-Ruppert averaging 
	* L201, should not include “not”?
limitations: I really like that the authors have a full section discussing the limitations of their work. The limitations and questions of the paper the reviewer had is given in Questions section.
review_title: null

Meta-review: metareview: This work introduces a hyperparameter-free model averaging and ensembling scheme for the domain generalization setting that achieves state-of-the-art results. Though the proposed method is simple and is related to existing techniques, the strong and comprehensive evaluation demonstrates that it is highly effective and therefore is of interest to the community. In addition, the paper is clear and well written, with careful analyses and useful theoretical intuitions. In summary, this paper will be a useful addition to the NeurIPS program.
confidence: Less certain
Decision: Accept