Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a new gradient-based framework for learning invariant mechanisms (often called "relations" in the paper) from data drawn for multiple environments (data generating processes).  Overall, the writing is excellent, and the central ideas are interesting and valuable.

A key idea of the paper is that training data drawn from different environments can be exploited to learn mechanisms that remain invariant across those environments.  While true, this is unsurprising and well-established.  Fundamental principles of causal inference, known for decades at this point, directly imply that different environments (data generating processes with different interventions) will allow identification of different sets of causal dependencies.  Practical methods for such identification have been demonstrated using graphical models and relatively simple methods for parameterization of those models.  The paper could be improved by spending less time on the known results (or at least making clearer connections to prior work) and spending more time clarifying what is genuinely novel about the proposed ideas.  In addition, the authors should make a greater effort to distinguish between central ideas and implementation details.

Multiple times in the paper, basic results from the causal inference literature are attributed to relatively recent papers (e.g., Peters et al. (2017)), including the special properties of the causal factorization and the idea of invariance of mechanisms in response to intervention.  These ideas can be traced back much further.  For example, the basic idea of invariance to intervention (so called "autonomy" or "modularity") has been known since at least the 1930s.  Heckman and Pinto (2015) note that: "In the language of Frisch (1938), these structural equations are autonomous mechanisms represented by deterministic functions mapping inputs to outputs. By autonomy we mean, as did Frisch, that these relationships remain invariant under external manipulations of their arguments." The paper would be improved by making clearer when concepts were first identified and by who.

The empirical evidence provided for the claims in the paper is relatively modest.  The simulated results provided in Table 1 shows only very small differences in L2 errors among variants of the authors' proposed methods, and more substantial improvements over ICP and ERM (in three of four cases).  The discussion of these results is excellent.  The results on the "Colored MNIST" data show the expected results.  However, good performance on simulated data and only a single real data set is still relatively weak evidence for the claims made in the paper.  The paper would be improved by increasing the number of real data sets used for evaluation.

References

Heckman, J., & Pinto, R. (2015). Causal analysis after Haavelmo. Econometric Theory, 31(1):115-151.
review_title: An effective exposition of key ideas in causal inference, but novelty should be clearer

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors propose a gradient-based learning framework, with a two part objective function in which one part improves the informativeness about the target variable, and the other part enforces the invariance of the relation. The second part is based on the ICM principle and increases the stability and renders domain generalization possible. 

The paper is well written and, for the most part, is easy to follow.

We should note that the ICM principle is only usable if we have no hidden confounders, i.e., causal sufficiency, in the system. The authors should clarify that causal sufficiency is an important assumption early in the manuscript and should clarify what will happen to the results if it is violated.

In general, the assumptions in this work are very strong and I do not believe they will hold in reality. Specially, regarding Assumption 2, if we are assuming some of the causal mechanisms are changing across environments, why the one corresponding to the target should not change?

Although the assumptions are strong, same assumptions were considered in few other works such as (Peters et al., 2016). Compares to existing work with the same assumptions, this paper provides a good implementation method that is an improvement over past work and would be of interest to the ICLR community.

The authors also discuss the conditions under which the recovered stable relations correspond to the true causal mechanisms. The use of ICM for causal discovery is also extensively studied in the non-parametric case in [Huang et al., Causal Discovery from Heterogeneous/Nonstationary Data], and in the linear case in [Ghassami et al., Multi-domain Causal Structure Learning in Linear Systems].

The definition of do-intervention in page 3 is not standard. What is referred to as do intervention in this paper is usually referred to as hard intervention in the literature, and what is referred to as hard intervention in this paper is usually referred to as atomic intervention in the literature.
review_title: The assumptions are very strong, yet a good implementation method is proposed

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is well-motivated and studies and important topic, but unfortunately it is let down by the presentation of their contributions which is confusing and at times misleading. 

First - a more minor complaint (which I put here because its a source of confusion for the rest of the review) - The normalizing flow section is confusing because the mapping between the base distribution and y isn't clear. I normally think of a normalizing flow as a map from some base distribution u to some target y such that y = T(u) and p(y) = p(u)|det J_T (u) | where u=T^{-1}(y) (adding conditioning as required to make a conditional flow). This paper uses T(Y; h(X)) everywhere - which I think is referring to T^{-1}(Y; h(X)) because we normally think of T as acting on the base distribution U and T^{-1} as acting on the target variable. My review assumes that I should read T(.) as a map from Y -> U... but that's a little weird and should be explained explicitly. 

More seriously, I don't understand why Lemma 1 isn't trivial? By the data processing inequality, any transformation of X can only lose information about X. So if the identity function is among the set of feature extractors, then h^* includes it,  because it maximizes I(h(X), Y). The fact that h^* is independent of the flow's latent variable trivially follows from the fact that choosing the identity is always optimal. Of course, things get more complex if there is some constraint on H such that the identity isn't included,  but this isn't discussed. On a second reading, I think that this constraint is meant to come from the Y \perp E | h(X) condition in section 4, but how this condition interacts with Lemma 1 needs to be clearer.

The presentation of the method in section 4 also needs work: the domain generalization problem is presented as the problem of finding h that maximizes the mutual information between Y and h(X) in the worst case environment under the constraint that Y\indep E | h(X). As far as I can tell, the independence constraint is the important part of that objective: under that constraint, it is not clear why I wouldn't want to maximize the average mutual information, or some other objective? 

Similarly - it's not clear why theorem 1 is useful until we get to equation (5) (and it took me a couple of reads to realize that this is actually the important step) - on its own, it just essentially says that if we have conditional independence, then applying a 1:1 function maintains that conditional independence.

Having gotten to this point in this review, I think that many of my issues would be resolved if the presentation order was reversed. The key condition you need is Y \perp E | h(X); The paper would be far easier to follow by making it clear that is is the condition you need, explaining both why we can't optimize for it directly, and why this particular normalizing flow approach gives an indirect approach to achieving the condition. In the current order of presentation which leads with a discussion of normalizing flows, we are presented with theoretical results about flows which, in isolation, seem trivial. 

The experiments show the method shows promise (though they should report both IRM & REX [Kruger et al 2020]'s performance for coloured MNIST to make it clear that there are better methods on that dataset)...

[Kruger et al 2020] Out-of-Distribution Generalization via Risk Extrapolation
review_title: Well-motivated paper, but more detail needed

Meta-review: This paper proposes a new framework for improving supervised learning via invariant mechanisms. The reviewers agree that overall, this paper is well-written and contributes to a growing body of work on invariant prediction and causality in supervised learning. At the same time, there are some concerns regarding novelty and significance in light of previous work, as well as the overall organization of the paper, which could be improved to highlight the main contributions more clearly. Ultimately, this was a borderline decision, but it is clear that the paper needs a major revision before acceptance. Although the authors have already incorporated some of the minor comments which is appreciated, the authors are urged to consider the major comments (e.g. see R2's comments regarding presentation) when revising the paper.
Decision: Reject