Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper argues that domain generalization methods that jointly minimize the empirical risk with a penalty term fail to decrease the empirical risk and hurt the performance both on the source and unseen target domains. Therefore, it proposes an optimization algorithm that minimizes the penalty under the constraint of optimality of the empirical risk.
strengths_and_weaknesses: Strengths:
- The proposed algorithm in this paper provides a new idea for manipulating the gradients during the training for penalty-based domain generalization methods.
- Overall, the paper is well written and organized.

Weaknesses:
- As the authors point out, the proposed method can only solve domain generalization problems where a universal model solving all domains (seen and unseen) exists, which could be a weakness when applied to real-world applications.
- The proposed simplification for the space of gradients for updating looks over-simplified. It would be better to provide some justifications for the choice of gradient space in the paper or consider more sophisticated choices.
questions: - Should be \hat{L} in Eq. (4), (5), and (6) be L since L is already the empirical risk defined in Line 87?
- Why is the penalty minimized at \theta^t + G^t in Eq. (5) and (6)? Why is it not \theta^t - \eta * G^t?
- It would be better to point out in the paper instead of the appendix that the constraint is replaced with a stronger constraint of uniform bounds over domains (Line 602, Appendix).
limitations: See weaknesses.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work argues that penalty-based DG methods often perform poorly in practice as, due to the hardness of joint optimization, they fail to minimize the empirical risk (i.e. in-distribution performance). To address this, the authors: (1) assume that excess risk is not required for generalization, i.e. that in-domain performance need not be sacrificed for out-of-domain performance; and (2) eliminate excess empirical risk by reformulating penalty-based DG such that the penalty is minimized under the constraint of optimal empirical risk. Then, drawing on connections to rate-distortion theory, a method is proposed to solve the proposed optimization problem. Experiments on the WILDS and DomainBed benchmark datasets illustrate the effectiveness of this method.
strengths_and_weaknesses: **Strengths**
- *Clarity of writing*
    - This paper was a joy to read. Lucid writing, easy to follow, well organised. 
    - Sections 2.2--2.4 were particularly impressive.
- *Novel perspective and interesting solution*
    - Blaming excess risk for the failures of penalty-based DG methods seems new, as does formulating a DG problem in which the constraint pertains to the empirical risk rather than the penalty (e.g. as in IRM).
    - While I disagree with the assumptions (see below), I think the perspective, solution and results are valuable to the community.
- *Nice theory and connections to rate-distortion*

**Weaknesses**
- *The reported experimental results seem hand-picked*
    - CORAL and ERM are reported on 6 WILDS datasets, along with their generalization gaps.
    - FISH is reported on 3 WILDS datasets.
    - VREx is reported on DomainBed.
    - The ad-hoc reporting makes me question the results -- why not report all methods on all datasets? In particular, I would question why:
      - ERM is left out of DomainBed
      - CORAL and FISH are left out of DomainBed
      - FISH is only reported on a 3 of 6 WILDS datasets -- this seems to be due to the hyperparameter ranges of FISH being designed/reported for only 3 or 4 datasets, but the results could still be reported to see if SDG improves FISH.
    - Happy to increase my score if this is addressed.
- *The central assumption---in-domain performance need not be sacrificed for out-of-domain performance---is often violated*
    - This essentially assumes no spurious correlations in the (possibly few) training domains---a central motivation for DG methods in the first place, with many works reporting real-world failure modes due to spurious correlations (e.g. [1]).
    - The authors justify this by claiming that:
      - "for the benchmarks we are interested in, which involve supervised learning from multiple datasets, this is not the case as a universal model solving all datasets (seen and unseen) exists" -- here I would question both the existence of such a universal model and its inference from a small subset of the domains.
      - "Since the models we use are significantly overparameterized, the space of solutions to empirical risk minimization is quite large, likely including many domain-invariant and domain-sensitive solutions." -- while I agree that the set of minimal-risk solutions is large, I would question how often domain-invariant solutions lie in this set.


[1] Geirhos, R., Jacobsen, J. H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. *Nature Machine Intelligence*, 2(11), 665-673.

questions: - See above.
- In Eq. 6, adding in another separate penalty/regularization seemed strange to me -- can't this just be absorbed into the abstract $\texttt{Penalty}(\\cdot)$ term? Was this addition purely for the theory/derivations? Does it make a difference empirically (e.g. via an ablation study)?
limitations: Yes, the authors discuss the computational complexity of their method as a limitation in 3.2.2 and in the conclusion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper argues that the joint optimization of the empirical risk and the penalty in penalty-based DG methods can increase empirical risk, which hurts performance on both training and unseen distributions. Based on this hypothesis, this paper proposes to minimize the penalty under the constraint of optimality of the empirical risk and presents solutions by connecting to rate-distortion theory. Experiments show that the proposed optimizer (SDG) effectively improves the OOD test performance.
strengths_and_weaknesses: Strengths (originality, quality, clarity and significance):
1. As far as I know, the originality is good.
2. The proposed method is theoretically and technically sound.
3. This paper is well organized.
4. The obversion of the generalization gap and in-distribution performance in the WILDS benchmarks is interesting. The proposed optimizer (SDG) significantly improves the performances of CORAL, FISH, and VREx on several datasets.

Weaknesses:
1. The evaluation of the proposed optimizer (SDG) is a little weak. Experiments with more penalty-based DG methods will make the effectiveness of SDG stronger.
2. The statement that the proposed optimizer improves empirical risk without hurting the generalization ability as desired in lines 184 – 287 is unconvincing. In Table 2, SDG increases the generalization gap on four datasets (IWILDCAM., OGBMOLPCBA, AMAZON, and PY150) and decreases the generalization gap on only three datasets (CAMELYON17, FMoW, and POVERTYMAP).
3. In lines 309 – 311, the explanation that the performance of the FISH for FMoW is already strong is far-fetched with the worst accuracy of 34.6.
questions: My main concerns are about the evaluation of experiments. Please see weaknesses for more details.
limitations: The authors addressed the limitations of their work. I didn't see potential negative societal impact of this work.
review_title: null

Meta-review: metareview: The paper considers the class of penalty-based methods methods for domain generalization which aim at minimizing a combination of the empirical risk and a penalty term used as proxy of the generalization error in the unseen domains (examples include IRM, CORAL, etc). The paper argues that such methods often perform poorly in practice either due to an erroneous penalty term, or due to the hardness of joint optimization which results in the failure to minimize the empirical risk. To address this issue,  the authors propose a method which, instead of jointly minimizing empirical risk with the penalty, it minimizes the penalty under the constraint of optimality of the empirical risk. The success of this methods is based on the assumption that assume that excess risk is not required for generalization. In other words,  in-domain performance need not be sacrificed for out-of-domain performance (no trade-off between the two). As a result, the proposed method essentially eliminates excess empirical risk by reformulating penalty-based DG such that the penalty is minimized under the constraint of optimal empirical risk. 

All the three reviewers found the results novel an interesting. The reviewers had some concerns (about the framework and the experiments) which were addressed during the discussion period. I recommend that the authors also apply all the reviewers' comments in the revised manuscript. Also, for the experiments, I think it'd useful to compare the numbers with SOTA numbers for the datasets considered in the experiments (they are all available in the WILDS leaderboard). 


confidence: Certain
Decision: Accept