Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a probable domain generalisation framework for finding predictors that generalise with high probability instead of generalisation in the worst or average case. It proposes two algorithms: QRM, SQRM, which are based on minimising $\alpha$ quantile of training risks with the interpretation that the predictor so trained would generalise well to at least $\alpha$ fraction of test domains. The last statement holds because the train and test domains are sampled from the same (latent) underlying distribution of domains.

----------
**Post rebuttal**.   
I thank the authors for clarifying most of my questions.     
The merits of the paper are better understood when there is risk-robustness tradeoff such as in their synthetic setting or the two real-world settings: iWildCam, OGB-MolPCBA. Their synthetic experiment has varying noise levels per environment, and iWildCam likely has varying noise or difficulty level per domain, but I do not see any reason for risk-robustness tradeoff in DomainBed experiments. Therefore, their results on DomainBed are arguable. On CMNIST, they claim to recover causal predictor, as further elaborated in Appendix A, but when the number of training domains is so small as two, $T_F$ estimate is noisy, which is not accounted for in their theorem proofs of Appendix. Besides, in the case of CMNIST, it is unclear why any of the existing methods cannot recover a function with invariant loss across domains. Some of the reviewers (including me) are requesting for a better explanation/intuition on their DomainBed results perhaps because some gains could also arise from implementation differences.    
Also, the contribution of SQRM to this paper is unclear. Authors note that it could lead to interesting future work, but I feel it does not contribute much to this work (this is a relatively minor issue though).    
This paper has solid contributions, but I believe they could have been expressed more crisply with better experiment design and crisper writing/presentation. For this reason, I am changing my earlier assessment to a weak (but not enthusiastic) accept. 
strengths_and_weaknesses: ## Strengths
Probable DG is a very sensible and a practical idea. Training data could include domains with varying difficulty levels and imposing invariance like constraints on them could be detrimental. 

## Weaknesses/comments 
**Writing**   
The writing is unclear and some important information should have been in the main content rather than in the appendix. Let me explain, 

* L36-37: (Q) what is “standard DG formulation”, need citations. From the description, it resembles (DG) of L76-77, which inturn resembles DRO. 
* L101-110: I do not see why talking about “esssup” is important. It clearly is just DG as remarked in L113: (3.1) is equivalent to DG.  
Would help if Preposition 3.1 and eq 3.3 are described more, I could not follow.
* L152-154: Not a good idea to describe in main content a figure in Appendix. Should have been here if it is important. 
* L163-165: I do not see why the note on finance is interesting here. Could have been in related work.
* L266-268: “A: each $\alpha$ is optimal at the corresponding quantile” needs further description. I believe the authors are using “optimal” in a different sense? 
* L261: “Ex. A.3” which shows SCM of linear setting should have been in the main content. 

**Evaluation metric**   
The paper advocates to go beyond average or worst accuracy measures for DG and suggest looking at quantiles of test risk/accuracy. Tables 2-5 shows 7 quantiles and average risk. From what I understand, authors judge superior method to be the one that minimises risk on all quantiles, but such a method would also have lower average risk on test domains. So, it seems the authors are going for a more nuanced comparison of quantiles. (Q) They should mention explicitly how quantiles are contrasted to reveal the superior method. In my opinion, evaluation should go hand-in-hand with a human specified quantile level or some such. 

**Missing comparisons**   
(Q) I expect to see comparisons with other DG methods in Tables 2-5.
(Q) And also std. dev. for all numbers.   

**QRM vs SQRM**   
The way SQRM is formulated, SQRM is not the same as QRM. Moreover, QRM is never compared with SQRM. In tables 2-5, QRM has consistently lower risk than SQRM. If the contribution of SQRM is efficiency or some such, that should have been discussed. I fail to see the contribution of SQRM.

**ColorMNIST**    
(Q) What’s the value of $\alpha$ and how did you pick it?
(Q) As $\alpha\rightarrow 1$, QRM (eq 4.1) is the same as (3.1), which is the same as DRO, correct? (Q) Why does QRM perform so much better than DRO/GrpDRO in Table 1?   

**DG formulation**    
The paper misrepresents existing DG work. The related work did some justice by having a section on “Robust DG” and “other DG”, but the main content completely disregarded “other DG” and went on to call min-max formulation of risk objective as the standard. This should be fixed. 
questions: Please respond to questions marked with (Q) in the section above.
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors proposed the Quantile Risk Minimization to have the *probable* DG, where the authors argue that DG should both seek and evaluate predictors based on their ability to perform well with high probability, rather than in the worst-case or on-average, which are widely adopted in concurrent works. The authors also showed that the empirical QRM can recover the causal predictor for the desired generalization probability. The paper is technically sound and generally well-written.
strengths_and_weaknesses: Originality: the originality of the paper lies in the proposal of a new objective for the domain generalization problems. I found the contribution fair and somewhat limited.

Strengths:

1. The theoretical analysis was complete, I checked most parts of the appendix, and it seems technically sound.

2. The empirical evaluations were conducted on diverse scenarios including regression, digits and some real-world applications. 

Weaknesses:

1. The motivation for adopting quantile risk minimization is not quite clear. The benefits of why opt for Quantile Risk Minimization rather than the 'worst-case' or 'on-average' are not clear.

2. The authors' claim on "Recovering the causal predictor" was not formally elaborated.

3. The empirical comparison was only conducted on a very limited number of baselines. For example, the experiments on ColorMNIST missed many recent baselines, and for the experiments on the *real-world* datasets, only the ERM was compared. This makes the effectiveness of the proposed method hardly to be evaluated. 

4. The organization of the presentation needs improvements. Many key components of the paper are listed in the appendix rather than in the main text. The paper itself should be self-contented.

questions: 
1. Theorem 4.1 indicates that the number of samples and domains should be sufficiently large. If so, no matter adopting what criteria ('worst case' or 'on average'), the generalization can succeed since we have enough domains and training samples. Thus, I'm wondering what are the exact benefits opt for quantile risk minimization? 

2. Following the first question: If the number of samples and domains should be sufficiently large, will the generalization property rely on the model complexity?

3. I would expect more experimental results comparing recent baselines (e.g. the ones provided in DomainBed) on more standard DG datasets (e.g. the PACS, Office-Home, DomainNet etc.).

4. a minor question:  In line 101, it seems that the difference between the assumption of *a distribution over environments* and the general assumption in many concurrent DG works that there exists a meta distribution between domains is not clear. Some explanations or discussions can help.
limitations: The motivation is not quite clear, and empirical evaluations are not sufficient.
====
After rebuttal comments: I would like to thank the authors for further providing the explanations and experimental results in the rebuttal phase.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a quantile risk minimization (QRM) and its empirical version (EQRM) that aim to learn predictors that can provide minimal risk in a domain generalization with probability $\alpha$. Interestingly, it has been shown in the paper that EQRM turns to the causal predictor when $\alpha \to 1$. The experimental results in the paper show the importance of quantile performance in evaluating domain generalization methods. These results also show that the proposed QRM can produce better results on both real and synthetic data sets compared to some strong baselines. 
strengths_and_weaknesses: Originality: The idea of introducing a probable domain generalization (by using quantile risk minimization) is novel and interesting due to the lack of a theoretical guarantee for a generalization bound or learnability of current domain generalization algorithms.

Quality: It seems that the main theoretical contribution of the paper is Proposition 3.1. However, could the authors provide a detailed definition of the distribution $\mathbb T_f$ and a representation of each element $R$ of $\mathbb T_f$?  In the formulation of (QRM), does the condition $Pr_{R \sim \mathbb T_f} \{ R \leq t \} \geq \alpha$ need to be satisfied $\forall R$? Otherwise, it would be very hard for me to verify that statement. More importantly, does QRM or EQRM learn an invariant predictor/representation across domains? If so, how does that work, and are there any theoretical guarantees for that?

Clarity: The paper is well-written and easy to follow.

Significance: As mentioned in the paper, the theoretical result in Proposition 4.2 is only true for linear predictors -- which really restricts the application of the proposed method in domain generalization. Note that the result for an invariant optimal predictor across domains has been introduced in a very well-known paper [1].  

[1] @article{arjovsky2019invariant,
  title={Invariant risk minimization},
  author={Arjovsky, Martin and Bottou, L{\'e}on and Gulrajani, Ishaan and Lopez-Paz, David},
  journal={arXiv preprint arXiv:1907.02893},
  year={2019}
}  
questions: Please see my comments on the weaknesses part of the paper.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work advocates for achieving high-probability generalization across domains, which is in between of the average-case (domain) generaliaztion for which ERM empirically works well, and the worst-case problem which motivates most of previous work in this area.  It assumes the training and test domains are i.i.d. samples from a shared distribution, and proposes to minimize the $\alpha$-quantile and superquantile of the risk distribution of predictors. The authors establish consistency guarantees for the proposed objectives, and connect them to other distributionally robust optimization and invariant prediction formulations.  Experiments show that the proposed method outperforms ERM in terms of the (super)quantile risk.

**Post-rebuttal update**: I have read the authors' response to my and other reviewers' questions and find them satisfactory.  I'm happy to see the new DomainBed experiment, which shows the promise of the proposed method in settings with a smaller number of domain, but the reason for such improvement isn't fully clear.  While I don't believe such discussions are in any sense necessary for the present work (and will not change my score in any case), I think this claim would be better supported if there are more experiments in the small-number-of-domain setting, or some theoretical discussion on the probability of improvement over ERM in such cases.
strengths_and_weaknesses: The main strength of this work lies in its new formulation; while various relaxiations of worst-case generalization have been proposed before, the new formulation appears easier to implement and analyze, especially for nonlinear models, while still being intuitive and interpretable when its assumptions are adequate.  It thus brings new promise for achieving domain generalization in nonlinear models, as is also evidenced by the experiments.

It should be noted that the method relies on the assumption of i.i.d. domains, and requires that we have a relatively large number of training domains so that quantiles can be estimated to a suitable precision.  The limitations are understandable, given the challenging nature of the problem, and the authors have been very transparent about them, but they will put the contributions better in context.

And while the real-world experiment shows the promise of the method in scenarios where a large number of domains are available, it would strengthen the paper if evaluations are also provided in scenarios where only a handful of domains are available, such as some of the DomainBed benchmarks.

questions: - Section 7 claims that we can deploy the trained model to "all [test domains] where it will be successful with probability ≈ α".  How can we identify such domains?  Are you assuming we have a labeled validation set in all test domains?

- It may also be helpful to move some of the IRM / GroupDRO results in appendix to the text.
limitations: Yes.
review_title: null

Meta-review: metareview: The authors have convinced the reviewers of the merits of the paper after extensive and detailed rebuttal and discussion. The reviewers have clearly converged towards acceptance.
confidence: Certain
Decision: Accept