Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This manuscript studies the problem of domain generalization and proposes a method, dubbed Rex, for this purpose. The main movitation of this work over the invariant risk minimization (IRM) [1] paradigm is that IRM is not robust to covariate shift, while the authors claim that Rex can deal with both covariate shift and concept shift together. Although the argument that IRM is not robust to covariate shift in the feature space is true, out-of-domain generalization is not the original goal of IRM either. My main concern is that it is not clear to me why Rex could deal with both covariate and concept shift together, as from the optimization formulation in Eq. (6), neither invariant predictor nor invariant representation is enforced. In particular, no theoretical analysis is given to justify this claim. 

Perhaps what adds more confusion to me is that, what's the meaning of negative probability since the authors allow the combination weight \lambda of different domains to be negative? As the authors have already pointed out on page 5 (Probabilities vs Risks), the risk is a linear functional of the joint distribution over X and Y, hence using affine combination in the risk functions from different domains directly translate to allowing the use of negative coefficient for probabilities. This is quite strange, since the mixture distribution is only a convex combination, not affine combination.

Furthermore, I also found some of the discussions in the related work section misleading: 
-   "The first method for invariant prediction ... is IRM". This is not correct. As the authors have already realized, ICP [2] has been proposed in 2015, and the definition of IRM is essentially the same as ICP. 

-   The discussions about invariant representations on page 4 are not accurate. Only P_e(\phi) (but not P_e(\phi | Y)) is called invariant representations, and only this one can fail domain adaption if the marginal label distributions differ. The C-ADA does not try to find invariant P_e(\phi | Y). Instead, it tries to find invariant P_e(\phi x \hat{Y}), where \hat{Y} is the classifier output, and this is precisely the reason why C-ADA also fails under different label distributions. See more discussions in [4]. In fact, it has recently been shown in [4] that matching P_e(\phi | Y) provably works for domain adaptation, see Theorem 3.1 of [4]. 

-   "Also, unlike Rex, IRM seeks to match E[Y | \phi], not the full P(Y | \phi)". Again, this discussion is misleading. For the purpose of out of domain generalization, the learner does not need to match the full distributions P(Y | \phi). Only the conditional mean matters. See Theorem 4.1 of [5] as well as Theorem 3.3 of [6] in the context of fairness.

-   About the discussion on fairness of equalizing risk across groups. In fact a sufficient condition for this goal has been proved in Theorem 3.3 of [6]. Given the close relationship between these two problems, I feel it's necessary to cite and have a discussion of this result here as well. 


More detailed comments:
-   The reference of David et al., 2010 should be Ben-David et al. 2010 on page 2

-   I think the naming of invariant prediction on page 2 is not very accurate. To be precise, as long as the same hypothesis (classifier) is used over different domains, this amounts to be an invariant prediction rule. Instead, what IRM enforces is the invariant OPTIMAL predictor, i.e., invariant conditional means.  

-   Eq. (3) is not correct: the second term is a weighted combination of samples from different domains, where the larger the sample size from a domain the more weight it has in the combination. However, the right most term has a wrong weight for a domain. The correct one should be |D_e| / \sum_e |D_e|. 

-   On page 4, the citation of Pan et al. 2010 is not accurate. Invariant representations for domain adaptation is first proposed by [3].



[1]     Invariant Risk Minimization
[2]     Causal inference using invariant prediction: identification and confidence intervals
[3]     Unsupervised Domain Adaptation by Backpropagation
[4]     Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
[5]     On Learning Invariant Representation for Domain Adaptation
[6]     Inherent Tradeoffs in Learning Fair Representations

review_title: Not convinced by the main claim of the paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview:

The authors propose Risk Extrapolation (Rex) which is an invariance-based approach to domain generalization. The main idea is to go from worst-casing over a convex combination of domains to an affine set of domains (MM-Rex) or to penalize the variance (V-Rex). Evaluations compare to IRM and ERM and show uniform improvements.

Positives:

The paper has a well written and motivated introduction, and there's substantial added expository material in the supplement. I would maybe tone down on the amount of bolded text. 

The arguments are pretty well-thought-out, including some discussion of the differences between causal recovery, invariant prediction, and domain generalization. 

The method itself seems simple (in a good way), though I have some questions below. I also wonder if the variance version of the objective can be tied to DRO based approaches via the fact that DRO on a chi-squared perturbation ball is equivalent to variance regularization of the risk. 

The method seems to work well overall compared to IRM, and although it only matches ERM in the domain generalization benchmark, I think this is still a decent result given that most methods underperformed ERM.

Negatives:

This is possibly a comment that refers to a paper that's too recent, so not addressing this comment won't affect my rating of the paper, but it seems worthwhile from a scientific perspective to address how recent negative theory results (Rosenfeld, Ravikumar, Risteski 2020) about IRM (and REx) affect the intro framing. In particular, I'd like to see the claim in the intro about how REx can extrapolate can be reconciled with the claims in the other paper that IRM and REx succeed under the same conditions as ERM. 

As a note: I also don't think Williamson and Menon suggest the use of variance of risks. They explicitly state that the R_{sd} risk aggregator is not a fairness risk measure. The risks fulfilling the axioms in that paper are coherent risk measures, and I believe they will all fall under the category of RI methods through duality arguments.

I can believe that minimizing the max risk is a reasonable thing to do, but it seems like an added leap of faith to want to actively increase the risk of the other domains since minimax risk would naturally equalize the different domains (if it's possible to achieve equal risk). It would be nice to see a clearer justification for this behavior.

Having looked at the supplement, there's substantial and good material there, and I would maybe suggest that the authors cut something like figure 3 and algorithm 1 to bring back some more intuition and motivation about REx, maybe some result from section E.

Having looked at the Thm 1 result, wouldn't 3 interventions on every variable also recover the true beta with ERM? This result also depends very heavily on fixed noise across environments. I do think this is a neat result though, and it might be useful to use this to give additional intuition about REx in the main text.

Does E.1.2 require pointwise homoskedasticity ? that seems wildly strong for a general result... In classification, I think this means that the map from x → y has to be deterministic and in general, for any log-probably loss, this says that the entropies have to be identical everywhere. 

Minor: 

Might be worth re-stating what the methods are in the experiment section (RI, for example, is defined pretty early on).

DRO is usually expanded to distributionally robust optimization, not domain robust optimization.

Figure 3 seems unnecessary.

Is Epsilon_j in the statement of Theorem 1 a typo? is there a typo in the subscript of beta_j = beta_{0,j} for all j?
review_title: An interesting approach, some comments clarifying the motivation and conditions for success

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the out-of-distribution (OOD) generalisation problem via risk extrapolation (REx). The authors propose two methods, MM-REx and V-REx, and empirically show that REx can recover the causal mechanisms on Colored MNIST, while also providing some robustness to covariate shift. The authors deal with the relationship between robustness, invariance and causality carefully, and provide experimental evidence beyond Colored MNIST. 

I vote for weakly rejecting. This paper reviews various previous work but does not provide a clear comparison from them.

I have some comments and questions as follows:
1. Contribution 1 and Table 1 state that REx is suitable for invariant prediction. In the experiments, the authors also take IRM as the competitor. So I think the main objective of REx is the invariant prediction, rather than covariate shift. However, the authors emphasize that REx can deal with covariate shift. Please explain more on how REx discovers the invariant prediction.
2. When considering the invariant prediction, we aim to discover a stable conditional distribution $P(Y|X)$ or an invariant conditional mean $E[Y|X].$ However, the covariate shift refers to the changes in the distribution of X. What is the novelty of assuming covariate shift here? Please also figure out the importance of considering covariate shift under the invariant prediction.
3. What is the expression (1) in Page 1? If (1) is the risk function of the OOD generalization problem, $\mathcal{F}$ is unseen and should be the same throughout this paper. At the end of Page 1, you say: "Our method minimax Risk Extrapolation (MM-REx) is an extension of DRO where $\mathcal{F}$ instead contains affine combinations of training risks, see Figure 1." Does MM-REx solve a different OOD problem? Please figure out the definitions of $\mathcal{F}$ of  DRO, MM-REx and IRM respectively.
4. Please explain Contribution 3. In general, the equality of risks is not a sufficient condition of the causality. 
5. In the VLCS and PACS experiments, the evaluation is incorrect. In Section 4.3, the task is to train on three domains and generalize to the fourth one at test time. However, this test accuracy is not worst-case performance.  According to (1), the problem is to generalize to all four domains at test time and to find out the worst domain. Then Table 3 should report the average of the worst-domain accuracy.

review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper addresses the problem of distributional shift in transfer learning from multiple training domains. The authors propose Risk Extrapolation (REx), which is a novel approach for out-of-distribution generalization when the new test domain for which we do not even have the covariate matrix. Thorough empirical experiments show that REx significantly outperforms state-of-the-art.

Pros:
- This is a highly quality paper with strong theoretical and empirical results. 
- The paper is clearly written and easy to understand.
- Based on the thorough literature review, this idea of this work is original.
- The results of this work are highly significant and of interest to the domain adaptation and transfer learning community.

Cons:
- Although I understand the page limit, most of the major parts of the paper (especially the theoretical aspects) can be found in the appendix. As a person who is not very familiar with the literature on distributional shift from multiple domains, I did appreciate having this thorough overview; however, the detailed discussions of the contributions of the paper might be overlooked if (when) located in the appendix. Specifically, there is  only half a page on introducing the proposed methods for REx  in Sec. 3.1.

Minor comment(s):
- Reference “Peter Bühlmann. Invariance, causality and robustness, 2018a.” is duplicated.

############################################################

Post-Rebuttal:

After reviewing the concerns raised by the other reviewers, and the responses provided by the authors, I have decided to adjust my scores.

Moreover, I was disappointed that the authors did not use the extra one page to move some material from the appendix to the main text in order to elaborate on the proposed method.

review_title: A significant contribution to the field of domain adaptation and transfer learning.

Meta-review: The paper is proposing Risk Extrapolation (REX) as a domain generalization algorithm. Authors extends the distributionally robust learning to affine mixture of distributions from convex mixture. Authors later uses variances instead of this extension and demonstrate various empirical and theoretical properties. The paper is reviewed by four expert reviewers and the reviewers did not reach to a consensus. Hence, I also read the paper in detailed and reviewed it. In summary, reviewers argue the following:

- R#2: Main argument is the lack of justification of the claim "Rex could deal with both covariate and concept shift together". Authors try to address this in their response. Moreover, reviewer also argues in the private discussion that manuscript is not updated and authors did not address any of the issues during the discussion period.
- R#3: Argues that (similar to R#2), dealing with covariate shift is not explained properly. Reviewer is not persuaded that REX results in invariant prediction.
- R#1 and R#4: Largely positive about the paper. In the mean time, argue that organization of the paper is lacking and some of the material in the supplement is relevant and should be moved to the main text. R#1 decreases their score due to the lack of re-organization during the discussion.

The value of the paper is clear to me, the joint treatment of minimax perspective, domain generalization and invariances is definitely interesting and valuable. Hence, the paper has merit to be published. However, the presentation is lacking significantly.  The main contribution of the paper lies in Table 1 but the invariant prediction property is not justified at all in the main text. Hence, Table 1 is not justified properly. Authors discuss Thm 1&2 in their response but they both are in the supplement. From reading only the main text, confusion of the reviewers are well justified. ICLR guidelines clearly states that "...Note that reviewers are encouraged, but not required to review supplementary material during the review process..." It is authors' responsibility to make the main paper self contained. Even more worrisome is the fact that authors dismiss this concern in their response to R#1 which eventually leads to R#1 decreasing their score. Hence, I decided to reject the paper since the presentation is subpar and authors did not persuaded reviewers that they can fix this presentation issue by the camera-ready deadline. On the other hand, I think the paper can be really influential if it was written clearly. I suggest authors to revise the claims more precisely, extended the discussion on the claims and move the theorems to the main paper.
Decision: Reject