Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper provides some theoretical justification that so-called flat solutions exhibit better domain generalization along with a strongly performing training methodology, SWAD, for improving domain generalization. They first prove theorems bounding the domain generalization performance in terms of robust risk minimization. Then they propose an approach to finding flat solutions related to previous work on in distribution generalization, and they thoroughly explore this method with regard to the flatness of solutions found, its domain generalization performance, and also perform ablations of the method.


main_review: The connection between the theoretical work in the first part of the paper and the method developed and tested is fairly tenuous. I appreciate the motivation for finding flatter minima but the paper would be tighter if there were a more direct connection between the flatness term in eq (1) and the algorithm. In particular, does SWAD encourage flatness w.r.t. the first term in eq. (1) more than other measures of flatness? My apologies if I missed this.

How novel/different is Theorem 1 compared to classic results from e.g. Ben-David (2010)? Of course, flatness doesn’t enter into these results but the rest seems somewhat similar.

The performance of the SWAD algorithm seems quite impressive, and I think this makes the paper a solid submission.
limitations_and_societal_impact: A thoughtful limitations section was included but societal impacts was not. However, I don’t see particular societal impact concerns specific to this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies how to improve the model performance on domain generalization problems from the perspective of loss landscape. Specifically, the authors propose to apply weight averaging (WA) to find flat minima and demonstrate this could lead to improve model generalization on unseen domains. Theoretically, the authors provide generalization bounds for the proposed method. Empirically, the proposed method achieves better performance on benchmark datasets compared with previous methods.
main_review: This paper tries to tackle the domain generalization problem by finding flat minima. It is interesting to investigate the loss landscape of the DNNs on domain generalization problems. 

**Originality**: The method proposed in this paper (i.e., SWAD) is mainly built on previous approaches such as SWA, and the authors mention this in the submission.

**Quality**: This submission is a complete piece of work and the claims are well supported by the empirical results. The evaluation is reasonable. 

**Clarity**: This submission is well-written and easy to follow.

**Significance**: This paper provides interesting empirical observations and improves the state-of-the-art by applying their proposed new approach.

Pros:
1. The proposed method consistently improves the model performance on domain generalization problems, and the proposed method is flexible and can be easily integrated with other training approaches for better performance.

Cons:
1. The generalization bound does not provide intuition for robust risk minimization (RRM), e.g. the theoretical results do not suggest performing RRM is better than standard ERM.
2. The proposed method is heavily built on the previous approach SWA, which only differs in terms of how frequently the averaging is performed and which iteration is being selected for averaging. 

Suggestions:
1. It would be better to put the algorithm (Algorithm 1 in appendix B) in the main body for describing the proposed method.

Missing reference:
1. Reference on generalization bound, [1].

Overall, I think this paper provides a simple approach that can improve model (out-of-domain) performance on domain generalization problems. I would suggest `Marginally above the acceptance threshold'.

[1]. A theory of learning from different domains. Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, Jennifer Wortman Vaughan. Machine learning, 2010.


======================================[updates]======================================

Thanks for the authors' response. I have read the authors' responses as well as the other reviews. I will stick with my score.
limitations_and_societal_impact: N.A.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the domain generalization (DG) problem setting and proposes a modification to the stochastic weight averaging (SWA) method that leads to better performance on DG testbeds. The proposed method, named SWAD, uses the same basic idea as SWA but samples parameters more densely and uses additional techniques for preventing the incorporation of overfit parameters. Some theoretical analysis supports the general thrust of seeking flat minima for generalization. The focus of the paper, however, is empirical, as SWAD demonstrates improved performance compared to many prior methods on several DG testbeds. Furthermore, SWAD can be combined with these prior methods, such as CORAL, for even better performance.
main_review: Originality
---
The paper is, perhaps by design, not particularly original. Rather, it makes the observation that an existing set of techniques with a particular goal (seeking flat minima for improved generalization) also has applications to a different goal (DG). This is still, to the best of my knowledge, a novel observation and contribution, and furthermore the modifications made to SWA to arrive at SWAD also appear novel. Nevertheless, these modifications are relatively small, so the method can still be viewed as a version of the general idea of SWA, and thus originality is not the paper's strong suit.


Quality
---
The paper is of relatively high quality, primarily due to the empirical results. I did not carefully check the theory for correctness, but I am unsure as to the significance of the theorems. They appear rather similar to prior theoretical results on how flat minima can lead to better generalization, but with some additional divergence terms thrown in as we are dealing with the DG setting. So the high level conclusion seems to be: standard (in distribution) generalization also helps with DG, so long as the domain divergence is not too large. This is not an interesting result. It would be nice to have stronger theory that more directly connects flat minima to DG, but the current theory does not do this.

Two more minor comments about the theory. First, the reliance on VC dimension is not desirable, since this could be in the billions for the models used and, in particular for DG testbeds, many orders of magnitude larger than the number of training points. Second, I am unsure about the argument in L114-120. "It implies that..." seems to ignore several terms on the RHS of Eq (2) that could be rather large. The final sentence ("Hence, Theorem 2 and...") also seems to require quite a bit more support for such a general statement.

Nevertheless, as mentioned, I am appreciative of the empirical results that the paper provides, including a detailed study on several testbeds and an ablation study that verifies all of the proposed components of the method. I am inclined to view this paper as an empirical contribution, and a fairly nice one that further supports the notion that strong results can be achieved on existing DG testbeds without paying much attention to the domain labels themselves (instead focusing on standard generalization).


Clarity
---
The paper is overall well written and easy to follow. See my comments above for some suggestions regarding the clarity of the theory. My other comments are more minor. First, though the technical description of the overfit-aware sampling strategy in L151-160 is nice, it would be made more clear with additional intuition and descriptive text, e.g., what is each parameter doing specifically and how did this particular instantiation come about? Second, there is a minor typo on L157 (remove the "\mathcal{E}_{val}^{(t_s)} ="). Third, it is not immediately obvious what the significance is of Figure 5.


Significance
---
As noted, I view this paper as a nice (significant) empirical contribution. Perhaps the significance could be further extended by exploring problems beyond DG. Indeed, there seems to be nothing central to the proposed method that ties it to DG, so could it perhaps be applied to other problems? A discussion of this point or, even better, some experiments would further improve the paper.

---
Edit after reading the author response
---
Thanks for your additional comments. My scores remains unchanged and I continue to recommend acceptance.
limitations_and_societal_impact: Yes, the authors have adequately addressed limitations and societal impacts in Section 5 and the Appendix.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper extends the idea of Stochastic Weighting Averaging (SWA) to the context of Domain Generalization (DG). Theoretically, the authors argues that finding a flat minima improves domain generalization. Algorithmically, the authors modify SWA by increasing the number of samples being averaged (e.g. per iteration) and by selecting the interval from where the samples are taken. The proposed algorithm  is evaluated in  the DomainBed benchmark and shows compelling results.
main_review: Pros:

- The paper tackles an important and relevant problem to the ML community.
- The proposed algorithm is relatively simple to implement in practice and perform well experimentally.

Cons:

- My major concern is wrt to the novelty of the work and the connection/motivation between the proposed algorithm SWAD, domain generalization and the derived theoretical insights.

- __Theory__.  While I appreciate the  provided theoretical insights, they directly follow from standard results in domain adaptation [1,2,3]  and Norton and Royset [29] . Most importantly, I believe some details  have been overseen in the technical proof.  Specifically, I believe the proof misses a term  (line 86 eq. 20 in suppl material).  Lemma 1 is between any two h, but the risk is between h and the labeling function. If I understand correct, this is overseen in line 86.  The author should rewrite lemma 1 before directly using it in the proof. Please refer to [1] and [2] for details. The second part of the proof and Lemma 2 follow from Norton and Royset [5].  This should be clearly mentioned and if they are different the authors should highlight main differences.  I would appreciate if the authors can clarify on this.

    Moreover, in practice,  the provided upper bound can be easily dominated by the divergence between  source and target.  Theorem 1, does not capture either the dissimilarity of the labeling functions (as a result of the missing term), which can also simply dominate the performance on target and being small is necessary for adaptation [2,3,4].  Even if these were all negligible and the result correct, the provided bound only shows that a flat minimal can  improve generalization but this is not necessarily novel.   It is hard to see a clear connection between these results and the proposed algorithm.  In other words, the theoretical results do not necessarily motivate the use of SWAD vs SWA. I would appreciate if the authors can provide further explanation on this aspect. 

- __Algorithm__. The proposed algorithm is based on a heuristic on top of SWA  that does not necessarily exploit the current scenario (DG). I believe this heuristic improves SWA in general and it might be algorithmically incremental vs vanilla SWA. Moreover, it introduces 3 new hyperparameters (well tuned in the experiments).  A better motivation than the one provided in 134-147 can improve the presentation/motivation of the algorithm.

- __Experiments and Analysis__.  This is one of the strongest sections of the paper.  Just a few questions:
    - Why SWA is not shown in Table 2?
    - Table 4 mentioned HP search was not applied but ERM+SWAD includes HP search (This is the same result from Table 2).

__Suppl__

- Lemma 1 is a well know result also from [1,2,3] and should be cited appropriately.
- Lemma 2 follows [5] and show be cited appropriately.






**Originality**: The use of SWA in the context of DG is new.  However, algorithmically the proposed method is an extension (based on a heuristic) of a well known technique.  Theoretically, the presented results also follow from existing work and it is not clear how they motivate the need of SWA vs SWAD (see theory and algorithm above for more details).  

**Quality**:  I believe the proposed algorithm should be better motivated as it is not clear the connection/motivation between  SWAD, domain generalization and the derived theoretical insights.  Moreover,  I believe several  small details where overseen in the technical proofs and those should be corrected to make them technically sound. (theory above for more details)

**Clarity**:  The submission is clearly written and well organized. The main issue in this aspect is with respect to the motivation and connection among the two different parts (theoretical results and algorithms).  (see algorithm above for more details)

**Significance**:  The empirical section is one of the strongest part of the paper and the results advance SoTA.  The method is relatively simple to implement in practice. However, it introduces a few extra hyperparams. 

As expressed above, my major concern is wrt to the novelty of the work and the connection between the proposed algorithm SWAD, domain generalization and the derived theoretical insights. I believe the paper could significantly benefit  from a major revision.


__References__

[1] Ben-David et al.  A theory of learning from different domains Springer 2010

[2] Impossibility Theorems for Domain Adaptation PMLR,2010.

[3] Acuna et al. f-Domain-Adversarial Learning: Theory and Algorithms. ICML 2021

[4] Zhao et al On Learning Invariant Representations for Domain Adaptation, ICML 2019

[5] Norton et al Diametrical Risk Minimization: Theory and Computations, 2019 (cited)
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper proposes Stochastic Weight Averaging Densely (SWAD) to improve domain generalization. As the name suggests, it is a modification of Stochastic Weight Averaging (SWA) by averaging more densely (every iteration).

Reviewers mostly agree on strength and weaknesses of this work. They all believe that this work is a solid empirical contribution. However, there are two major shortcomings: 1- The novelty is very limited given the minimal changes applied to SWA to get SWAD. 2- The theoretical results seem very unrelated to reasons for the success of SWAD (in particular, as opposed to SWA). I highly recommend rewriting the paper with less emphasize on the theoretical results (maybe moving it to appendix?).

Overall, given the empirical contributions, I'm slightly inclined to accept the paper. I hope authors would take reviewers' suggestion into account to improve the paper.
Decision: Accept (Poster)