Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work studied a variance-based regularization method based on weighting correction for domain generalization. It also theoretically analyzed the potential benefits of the proposed method over ERM, and the connection between the proposed variance-based regularization method and the group DRO problem from an optimization perspective.
strengths_and_weaknesses: Strengths:
(1) This paper proposed a weighting correction scheme for variance-regularized domain generalization methods inspired by the generalization error bound.
(2) It theoretically compared the proposed method with ERM.
(3) It demonstrated that the objective function of the proposed method is equivalent to a group DRO problem from an optimization perspective.

Weaknesses:
(1) More experiments need to be provided to validate the observations of this paper.
(2) It is unclear how the generalization gap between the training and test data in Line 101-102 is derived.
(3) The computational complexity of variance-regularized domain generalization method with weighting correction scheme is not analyzed.
questions: (1) In section 4, how would the selection of the anchor distribution affect the generalization results?
(2) More empirical comparison between the proposed method and ERM can be better for supporting the observations in section 3.1.
limitations: The authors showed that no potential negative societal impact was observed in this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper investigated the variance-based regularization method for domain generalization. In particular, the authors originally study how the empirical efficient estimator ($\hat{\mathbf{q}}$) effects the generalization performance for a convex hull of domains. The authors first give an generalization bound for in-distribution generalization based on the in-domain variance and out-of-domain variance (Thm. 1). Then, they study the relationship of empirical efficient estimator and the group DRO objectives, and show the equivalence between group DRO and the variance-based regularization (Thm. 3).

Overall, this paper gives some new insights for the variance-based domain generalization, such as the effects of empirical efficient estimator and the in-distribution generalization bound based on variance. However, I still think the theorical results in this paper is insufficient for answer the question: "Can Variance-Based Regularization Improve Domain Generalization?".  Besides, the main contribution of studying the effect of empirical efficient estimator seems a bit trivial, since there exists work on variance-based domain generalization [1][2].

[1] Krueger et al.. Out-of-distribution generalization via risk extrapolation (rex). ICML 2021.  
[2] Xie el tal., Risk variance penalization. arXiv preprint arXiv:2006.07544, 2020.

strengths_and_weaknesses: **Strengths**
1. some new insights for the variance-based domain generalization, such as the effects of empirical efficient estimator and the in-distribution generalization bound based on variance.  
2. Although there exists work on variance-based generalization [3]. the study of variance-based domain generalization is interesting to me.
3. The theoretical derivations are complete.

[3] Duchi et al., Variance-based regularization with convex objectives 2017

**Weaknesses**
1. The main concern to me is the significance of the theoretical results. Thm 1 gives an in-domain generalization result (bound the generalization risk of training domains), which is far away from the setting of domain generalization, where the target domain is unseen during training and the discrepancy between the target domain and training domains can be large. Although the authors have discussed this problem in Eq. (9), I still think they should give a generalization bound on the standard setting of domain generalization.

2. Another major concern is the novelty of this paper. To my knowledge, Xie et al. [2] have proposed the relationship between group DRO objective and the variances. I think the novelty of this paper is the study of empirical efficient estimator for variance-based domain generalization, which seems a bit minor.

3. For experimental results in Tab. 1, I notice the results on PACS are amazingly strong (96.9%). The details of experimental setting and analysis for the results is missing, but I think the experiment is important for this work.
questions: I hope the authors to clarify my concerns, but if I am wrong, please correct me.
limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper addresses the connection between domain generalization and group distributionally robust optimization. Variance-regualarized ERM is approximated by Group DRO with $\phi$-divergence ball. Unlike previous works, they take empirical domain distribution as an anchor of uncertainty sets of DRO instead of the uniform distribution, and provides guarantees of in-distribution generalization. They argue that with some conditions, their method can outperform ERM for domain generalization.
strengths_and_weaknesses: Strengths
* This paper did a focused job of generalizing and analysizing the connection between varizance-based regualarization and domain generalization. Their theoretical result for generalization bound for in-distribution seems solid.

Weaknesses
* Their work seems similar to [1]. The work [1] showed that group DRO formulation with unifrom distribution as a anchor is bounded with risk variance and the equality can be established with some condition. In this paper, the study is extended into general anchor including empirical distribution. However, theorem 3 and 4 can be straightforwardly extended from the results of [1]. 

* They argue that a weighting correction scheme can improve generalization in some cases. It would be better to show some experiments to make their arguments valid.

1. Xie, et al. Risk variance penelization, 2020.
questions: I cannot fully understand the benefits of using empirical domain distribution as a anchor distribution of $phi$-divergence ball, instead of the uniform distribution. can it be  shown through some experiments?
limitations: The author have addressed the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the author studied the problem of whether the variance-based regularization approach can improve domain generalization. With a mild assumption of the out-of-distribution data is generated by the shift of the domain distribution, the paper proposed a weighting correction scheme based on the previous variance-regularized domain generalization methods. They prove the guarantees for in-distribution generalization and show the potential advantages of the proposed method when compared to the Empirical Risk Minimization. Experimental results show improvements in some cases.
strengths_and_weaknesses: Strength

1) The paper is well-written with strong mathematical support.

2) The paper studies the question of whether variance-based regularization can improve domain generalization. Based on strong theoretical support, they proposed a simple weighting correction scheme and provide guarantees of in-distribution generalization. 

Weakness

1) It could be better to include the experiments in the main paper. The main paper did not mention that there are experiments in the Appendix. More experiments can be done to support the proposed idea.

2) Based on the description from Line 194 to 204, the sample size m is critical to the proposed approach. In Line 199, it looks like a larger m could benefit the proposed method while Line 203 shows that the m should not be too large.  An empirical experiment might be required to demonstrate the impact of the sample size m.

questions: 
1) Although the proposed method has a different assumption compared to invariant learning methods, is there any connections between these two approaches?

2) In Line 18-20, do you assum that the OOD test data is generated by domain shift from one single domain or a mixture of multiple domains?

limitations: NA
review_title: null

Meta-review: metareview: This paper has been widely discussed between reviewers and authors. Unfortunately, even after the reviewers updated their scores, the paper was still judged to be below the acceptance threshold. I encourage the authors in taking into account the reviewers' comments while preparing the next iteration of their work.
confidence: Certain
Decision: Reject