Learning Robust Models using the Principle of Independent Causal Mechanisms
Authors: Jens Müller
Robert Schmier
Lynton Ardizzone
Carsten Rother
Ullrich Koethe
Venue:  (2021)
Keywords: Causal Discovery
Principle of Independent Causal Mechanisms
Normalizing Flows
Domain Generalization

Abstract: Standard supervised learning breaks down under data distribution shift. However, the principle of independent causal mechanisms (ICM, Peters et al. (2017)) can turn this weakness into an opportunity: one can take advantage of distribution shift between different environments during training in order to obtain more robust models. We propose a new gradient-based learning framework whose objective function is derived from the ICM principle. We show theoretically and experimentally that neural networks trained in this framework focus on relations remaining invariant across environments and ignore unstable ones. Moreover, we prove that the recovered stable relations correspond to the true causal mechanisms under certain conditions. In both regression and classification, the resulting models generalize well to unseen scenarios where traditionally trained models fail.

LEARNING ROBUST MODELS USING THE PRINCIPLE OF INDEPENDENT CAUSAL MECHANISMS
Anonymous authors Paper under double-blind review
ABSTRACT
Standard supervised learning breaks down under data distribution shift. However, the principle of independent causal mechanisms (ICM, Peters et al. (2017)) can turn this weakness into an opportunity: one can take advantage of distribution shift between different environments during training in order to obtain more ro- bust models. We propose new gradient-based learning framework whose ob- jective function is derived from the ICM principle. We show theoretically and experimentally that neural networks trained in this framework focus on relations remaining invariant across environments and ignore unstable ones. Moreover, we prove that the recovered stable relations correspond to the true causal mechanisms under certain conditions. In both regression and classification, the resulting mod- els generalize well to unseen scenarios where traditionally trained models fail.
INTRODUCTION
Standard supervised learning has shown impressive results when training and test samples follow the same distribution. However, many real world applications do not conform to this setting, that research successes do not readily translate into practice (Lake et al., 2017). The task of Domain Generalization (DG) addresses this problem: it aims at training models that generalize well under domain shift. In contrast to domain adaption, where a few labeled and/or many unlabeled examples are provided for each target test domain, DG absolutely no data is available from the test domains' distributions making the problem unsolvable in general.
In this work, we view the problem of DG specifically using ideas from causal discovery. To make the problem of DG well-posed from this viewpoint, we assume that there exists feature vector h" (X) whose relation the target variable Y is invariant across all environments. Consequently, the conditional probability p(Y he (X)) has predictive power in each environment. From a causal perspective, changes between domains or environments can be described as interventions; and causal relationships unlike purely statistical ones remain invariant across environments unless explicitly changed under intervention. This is due to the fundamental principle of "Independent Causal Mech- anisms" which will be discussed in Section 3. From causal standpoint, finding robust models is therefore a causal discovery task (Bareinboim & Pearl, 2016; Meinshausen, 2018). Taking a causal perspective DG, we aim at identifying features which (i) have invariant relationship to the target variable Y and (ii) are maximally informative about Y.
This problem has already been addressed with some simplifying assumptions and discrete com- binatorial search in Magliacane et al. (2018); Rojas-Carulla et al. (2018), but we make weaker as- sumptions and use gradient based optimization. Gradient based optimization is attractive because it readily scales to high dimensions and offers the possibility to learn very informative features, in- stead of merely selecting among predefined ones. Approaches to invariant relations similar ours were taken in Ghassami et al. (2017), who restrict themselves to linear relations, and Arjovsky et al. (2019); Krueger et al. (2020), who minimize an invariant empirical risk objective.
Problems (i) and (ii) are quite intricate because the search space has combinatorial complexity and testing for conditional independence high dimensions notoriously difficult. Our main contribu- tions to this problem are the following:
By connecting invariant (causal) relations with normalizing flows, we propose a differen- tiable two-part objective of the form T(Y; Lr, where / is the mutual information

---

Under review as a conference paper at ICLR 2021
and L₁ enforces the invariance of the relation between h(X) and Y across all environments. This objective operationalizes the ICM principle with a trade-off between feature informa- tiveness and invariance controlled by parameter Ay. Our formulation generalizes existing work because our objective is not restricted to linear models.
We take advantage of the continuous objective in three important ways: (1) We can learn invariant new features, whereas graph-based methods can only select features from pre- defined set. (2) Our approach does not suffer from the scalability problems of combinato- rial optimization methods as proposed in e.g. Peters et al. (2016) and Rojas-Carulla et al. (2018). (3) Our optimization via normalizing flows, i.c. in the form of a density estimation task, facilitates accurate maximization of the mutual information.
We show how our objective simplifies in important special cases and under which con- ditions its optimal solution identifies the true causal parents of the target variable Y. We empirically demonstrate that the new method achieves good results on two datasets pro- posed in the literature.
2 RELATED WORK
Different types of invariances have been considered in the field of DG. One type is defined on the feature level, i.c features h(X) are invariant across environments if they follow the same distri- bution in all environments (e.g. Pan et al. (2010); Ganin et al. (2016); Ben-David et al. (2007)). However, this form of invariance problematic since for instance the distribution of the target vari- able might change between environments. In this case we might expect that the distribution h(X) changes as well. A more plausible and theoretically justified type of invariance is the invariance of relations (Peters et al., 2016; Magliacane et al., 2018; Rojas-Carulla et al., 2018). A relation be- tween a target Y and some features is invariant across environments, the conditional distribution of Y given the features is the same for all environments. Existing approaches model conditional distribution for each feature selection and check for the invariance property (Peters et al., 2016; Rojas-Carulla al., 2018; Magliacane al., 2018). However, this does not scale well. We provide a theoretical result connecting normalizing flows and invariant relations which in turn allows for gradient-based learning of the problem. In order to exploit our formulation, we also use the Hilbert- Schmidt-Indeeendeer Criterion that has been used for robust learning by Greenfeld & Shalit (2019) in the one environment setting. Arjovsky et al. (2019) propose a gradient- based learning framework which exploits weaker notion of invariance. Their definition is only a necessary condition, but does not guarantee the more causal definition of invariance we treat in this work. The connection between DG, invariances and causality has been pointed out for instance by Meinshausen (201 8); Rojas-Carulla et al. (2018); 1);; Zhang et al. (2015). From causal perspective, DG is a causal discovery task (Meinshausen, 2018).
For studies on causal discovery in the purely observational setting see e.g. Spirtes & Glymour (1991); Pearl (2009); Chickering (2002), but they cannot take advantage of variations across en- vironments. The case of different environments has been studied by Hoover (1990); Tian & Pearl (2001); Mooij et al. (2016); Peters et al. (2016); Bareinboim & Pearl (2016); Magliacane et al. (2018); Ghassami et al. (2018); Huang et al. (2020). Most of these approaches rely on combinato- rial methods based on graphical models defines a continuous objective for very general non-linear models. The distinctive property of causal relations to remain invariant across environments in the absence of direct interventions has been known since at least the 1930s (Frisch, 1938; Heckman & Pinto, 2013). However, its crucial role as a tool for causal discovery was to the best of our knowledge- only recently recognized by Peters et al. (2016). Their estimator - Invariant Causal Prediction (ICP) returns the intersection of all subsets of variables that have an invariant relation w.r.t. Y. The output is shown to be the set of the direct causes of Y under suitable conditions. However, their method assumes an underlying linear model and must perform an exhaustive search over all possible variable sets Xy, which does not scale. Extensions to time series and non-linear additive noise models were studied in Heinze- Deml et al. (2018); Pfister et al. (2019). Our treatment of invariance is inspired by these papers and also discusses identifiability results, i.e. conditions when the identified variables are indeed the direct causes. Key differences between ICP and our approach are the following: Firstly, we propose a formulation that allows for gradient-based learning without strong assumptions on the underlying causal model such as linearity. Second, while ICP tends to exclude features from the parent set when in doubt, our algorithm prefers to err in the direction of best prediction performance in this case.
restricted linear mechanisms, whereas our model
2

---

Under review as a conference paper at ICLR 2021
3 PRELIMINARIES
In the following we introduce the basics of this article as well as the connection between DG and causality. Basics on causality are presented in Appendix A. We first define our notation as follows: We denote the set of all variables describing the system under study as X (Xi,...; }n). One of these variables will be singled out as our prediction target, whereas the remaining and may serve as predictors. To clarify notation, we call the target variable = X, for some i E {I...., DJ, and the remaining observations are X - X JY Realizations of a random variable denoted with lower case letters, e.g. 24., We assume that observations can be obtained in different environments E E. Symbols with superscript, e.g. Y refer to a specific environment, whereas symbols without refer data pooled over all environments. We distinguish known environments &xeen where training data are available, from unknown ones E &unseen. where we wish our models to generalize to. The set of all environments & = Eveen U Eunseen- We assume that all random variables have density PA with probability distribution PA (for some variable or set A). We consider the environment to be random variable and therefore a system variable similar to Mooij et al. (2016). This gives an additional view casual discovery and the DG problem.
are observed
Independence and dependence of two variables A and B is written as L. and A Y B respec- tively. Two random variables A, B are conditionally independent given C if P(A, B C) P(A C)p(BB C). This is denoted with A - B C. Intuitively, it means A does not contain any infor- mation about B if C is known (for details e.g. Peters al., 2017). Similarly, one can define independence and conditional independence for sets of random variables.
3.1 INVARIANCE AND THE PRINCIPLE OF ICM
DG in general unsolvable because distributions between seen and unseen environments could differ arbitrarily. In order to transfer knowledge from Eveen to Eunseen: we have to make assumptions on how seen and unseen environments relate. These assumptions have a close link to causality.
We assume certain relations between variables remain invariant across all environments. A subset Xs C X of variables elicits an invariant relation or satisfies the invariance property w.r.t. Y over subset W C & of environments if
Ve, E W: P(Y X u) P(Y X
for all u where both conditional distributions are well-defined. Equivalently, we can define the in- variance property by Y 1. E. Xs and T(Y; F Xs) = 0 for E restricted to W. The invariance property for computed features lxX) is defined analogously by the relation Y 1 E' ((X).
Although can only test for (equation 1) in Exeen taking a causal perspective allows plausible conditions expressed by Assumptions 1 and 2 - for an invariance to remain valid in all environments &. In brief, we assume that environments correspond to interventions the system and invariance arises from the principle of independent causal mechanisms (Peters et al., 2017, ICM).
derive
At first, consider the joint density px (X). The chain rule offers a combinatorial number of ways to decompose this distribution into a product of conditionals. Among those, the causal factorization
P& (x (2 rD) II [1pp(2 Xpa(s))
is singled out by conditioning each X; onto its causal parents or direct causes X Xpo()), where pa(()) denotes the appropriate index set. The special properties this factorization are discussed in Peters al. (2017). The conditionals p, of the causal factorization are called causal mecha- nisms. An intervention onto the system is defined by replacing one or several factors in the de- composition with different (conditional) densities p. Here, we distinguish sofi-interventionss where (ay)) ( is
(,( Xpa(z)) * P5(;; Xpa(j)) and hard-interventionn where Pyy (xy Xpa()) = T density which does not depend on Tpa(j) (e.g. an atomic intervention where Ty is forced take specific value F). The resulting joint distribution for single intervention is
D
II D& (II,..., . ID) = (x xpa()) i=1,i
pe(as xpa(i))
(3)
3

---

Under review as a conference paper at ICLR 2021
and extends multiple simultaneous interventions in the obvious way. The principle of independent causal mechanisms (ICM) states that every mechanism acts independently of the others (Peters et al., 2017). Consequently, intervention replacing Pj with Pf has no effect the other factors Piyj+ as indicated by equation 3. This is a crucial property the causal decomposition - alternative factorizations do not exhibit this behavior. Instead, a coordinated modification of several factors is generally required to model the effect of an intervention in a non-causal decomposition.
We utilize this principle as a tool to train robust models. To do so, we make two additional assump- tions, similar to Peters et al. (2016) and Heinze-Deml et al. (2018):
Assumption 1. Any differences in the joint distributions Pxx from one environment to the other are fully explainable as interventions: replacing factors (x:; Xpa(s)) in environment e with factors of' (z; Xpa(t)) in environment e' (for some subset of the variables) is the only admissible change. Assumption 2. The mechanism p(y Xpa(Y)) for the target variable is invariant under changes of environment. In other words, we require conditional independence Y 1 E Xpa(Y)-
Assumption 2 implies that Y must not directly depend E. In addition, it has important con- sequences when there exist omitted variables H, which influence Y but have not been measured. Specifically, if the omitted variables depend hidden confounder of X pa(Y)) and Y (the system is not causally sufficient and X pa(Y) becomes a
the environment (hence H K E) or H contains
"collider", hence H 1 E X))())), then Y and E are no longer d-separated by X Xpa(Y) and As- sumption 2 is unsatisfiable. Then our method will be unable find invariant mechanism (see Appendix B for more details).
If we knew the causal decomposition, could use these assumptions directly to train robust model for Y - we would simply regress Y on its parents X po(()) However, we only require that a causal decomposition with these properties exists, but do not assume that it is known. Instead, our method uses the assumptions indirectly - by simultancously considering data from different environments - to identify stable regressor for Y.
We call a regressor stable if it solely relies on predictors whose relationship to Y remains invariant across environments, i.e. is not influenced by any intervention. By assumption 2, such a regressor always exists. However, predictor variables beyond X X may be used as well, e.g. children of Y or parents of children, provided their relationships Y do not depend on the environment. The case of children is especially interesting: Suppose X; is a noisy measurement of Y, described by the causal mechanism Px Y). As long as the measurement device works identically in all environments, including X2 and illustrate Assumption 2 in Appendix B. In general, prediction accuracy will be maximized when all suitable predictor variables are included into the model. Accordingly, our algorithm will asymptotically identify the full set of stable predictors for Y. In addition, we will prove under which conditions this set contains exactly the parents of Y. Note that there are different ideas whether most supervised learning tasks conform to this setting (Schölkopf et al., 2012; Arjovsky et al., 2019).
predictor of Y is desirable, despite it being child. We discuss
3.2 DOMAIN GENERALIZATION
In order to exploit the principle of ICM for DG, we formulate the DG problem as follows
h arg max
mim (X(; s.t. Y L E h(X)
(4)
(eee
KEH
where h E H. denotes a learnable feature extraction function h: Rd + RM where M is hyper- parameter. This optimization problem defines maximin objective: The features h(X) should be as informative as possible about the response Y even in the most difficult environment, while conform- ing to the ICM constraint that the relationship between features and response must remain invariant across all environments. In principle, our approach can also optimize related objectives like the av- crage mutual information over environments. However, very good performance in a majority of the environments could then mask failure in single (outlier) environment. We opted for the maximin formulation to avoid this.
As it stands, equation 4 is hard to optimize, because traditional independence tests for the constraint Y L E (X) cannot cope with conditioning variables selected from an infinitely large space H. A re-formulation of the DG problem solve this problem is our main theoretical contribution.
4

---

Under review as a conference paper at ICLR 2021
3.3 NORMALIZING FLOWS
Normalizing flows form a class of probabilistic models that has recently received considerable at- tention, see Papamakarios al. (2019) for an in-depth review. They model complex distributions by means of invertible functions 7 (chosen from some model space 7) which map the densities of interest to latent normal distributions. The inverses F = T then act as generative models for the target distributions. Normalizing flows are typically built with specialized neural networks that are invertible by construction and have tractable Jacobian determinants.
In our case, we represent the conditional distribution P(Y | h(X)) using a conditional normalizing flow. To this end, we seek a mapping = T())(X)) that is diffeomorphic in Y such that R ~ (0,01) L (xX) when Y P(Y ((X)). This is generalization of the well-studied additive Gaussian noise model R Y - ((((X))) see section 4.2. The inverse Y F(R: (x))) assumes the role of structural equation for the mechanism p(Y h(X)) with R being the corresponding noise variable.' However, in our context it is most natural to learn T (rather than F) by minimizing
the negative log-likelihood (NLL) of Y under I (Papamakarios et al., 2019), which takes the form
LNLL (T, h) = E [IT(()
log det V (Y((Y)((X)) + CC
(5)
where det V is the Jacobian determinant and C = dim(() log(v/2) is a constant that can be dropped. If we consider the NLL on particular environment E E, we denote this with LLLL- Lemma 1 shows that normalizing flows optimized by NLL are indeed applicable to our problem:
Lemma 1. (proof in Appendix C) Let h T arg H,TTT (NLL(T, h) be the solution of the NLL minimization problem on sufficiently rich function space T. Then the following properties are guaranteed for arbitrary sets H of feature extractors:
(a) ht also maximizes the mutual information, i.e. h² = 9" with g" = arg maxgEN [g((X;;
(b) hf is independent of the flow's latent variable: he (x) 1 R with R = (X))))((X))
Statement (a) guarantees that h+ extracts as much information about Y as possible. Hence, the objective (4) becomes equivalent optimizing (5) when restrict the space H of admissi- ble feature extractors to the subspace H.g satisfying the invariance constraint Y 1 E h(X):
arg min max minreT LL (T; h)
arg m minces T(Ye; h(X*)) (Appendix C). Statement (b) ensures that the flow indeed implements a valid structural equation, which re- quires that R can be sampled independently of the features f(X). More details about normalizing flows can be found in Appendix C.
4 METHOD
In the following we propose a way of indirectly expressing the constraint in equation 4 via normal- izing flows. Thereafter, we combine this result with Lemma to obtain differentiable objective for solving the DG problem. We also present important simplifications for least squares regression and softmax classification and discuss relations of our approach with causal discovery.
4.1 LEARNING THE INVARIANCE PROPERTY
The following theorem establishes a connection between invariant relations, prediction residuals and normalizing flows. The key consequence that a suitably trained normalizing flow translates the statistical independence of the latent variable R from the features and environment (h(X), ) E) into the desired invariance of the mechanism P(Y I(X)) under changes of F. We will exploit this for an elegant reformulation of the DG problem (4) into the objective (7) below.
Theorem 1. Let h be a differentiable function and Y, X, E be random variables. Furthermore, let R T(Y: be continuous. differentiable function that is diffeomorphism in Y. Suppose that R 1. (I(X),
), F.). Then, it holds that Y FE] ((()..
Proof. The decomposition rule for the assumption R. 1. (h(X), E)) (1) implics R 1 h(X) (2). To simplify notation, we define Z == h(X). Because T is invertible in Y and due to the change of
F is the concatenation of the normal CDF with the inverse CDF of P(Y h(X)), see Peters al. (2014).
5

---

Under review as a conference paper at ICLR 2021
variables (c.o.v.) formula, obtain
Priz,E(U 2,0) (c.o...)
PRIZ,E 2, 2) det
87
dy (u.z) =)
@ pn(r) det
0T dy (U.2)
OT
(c.v.)
Priz(r [2) det By (y) (2)
priz(y
This implies Y L E Z.
The theorem states in particular that if there exists suitable diffeomorphism T such that R 1 (h(X), E), then ((X) satisfies the invariance property w.r.t. Y. Note that Assumption 2 violated, the condition R I. (h( (h(X),E) E) is unachievable in general and therefore the theorem is not applicable (see Appendix B). We use Theorem I in order to learn features h that meet this requirement. In the following, we denote a conditional normalizing flow parameterized via 0 with Tg- Furthermore, hs denotes feature extractor implemented as a neural network parameterized via Φ. We can relax condition R. 1. (X), E)) by means of the Hilbert Schmidt Independence Criterion (HSIC), kernel-based independence measure (see Appendix D for the mathematical definition and Gretton al. (2005) for details). This loss, denoted as Lı, penalizes dependence between the distributions of R and (X)( E). The HSIC guarantees that
L1 (PRR PRo(X),E) = = R L (X,,
where R= = (Y;; (X)) and PR. Ph.P),) are the distributions implied by the parameter choices and 0. Due to Theorem 1, minimization of Lr(Pre (n,(X),E) w.r.t. ø and 0 will thus approximate the desired invariance property y I. E (X), with exact validity upon perfect convergence.
When R 1 (X,(X), E) is fulfilled, the decomposition rule implies R IL E as well. However, if the differences between environments are small, empirical convergence accelerated by adding a Wasserstein loss which explicitly enforces the latter, see Appendix D and section 5.2 for details.
4.2 EXPLOITING INVARIANCES FOR PREDICTION
Equation can be re-formulated as a differentiable loss using Lagrange multiplier λ on the HSIC loss. 'r acts hyperparameter adjust the trade-off between the invariance property of ha (X) w.r.t. Y and the mutual information between hx(X) (X) and Y. See Appendix E for algorithm details.
Using Lemma I(a), we maximize min.gs [(Y hss (X*)) by minimizing maxese (ToLL(To, he]] w.r.t. 0. To achieve the described trade-off between goodness-of and invariance, we therefore optimize
Normalizing Flows
arg min
max eEc'
(TNLL + Ar Pro()),E) (n,,x)))
where R = To(Y (X*)) and A > O. The first term maximizes the mutual information between ha (X) and Y in the environment where the features are least informative about Y and the second term aims to ensure an invariant relation.
Additive Noise
Let fo be regression function. Solving for the noise term gives R Y (x( (X()). we make two
which corresponds to a diffeomorphism in Y, namely TY X) simplified assumptions: (i) the noise is gaussian with zero mean and (ii) R 1 (X),, then obtain
T(:: (X)) HYY H( (x)) = HYY - (R So (X))
H(Y) (R) H(Y) 1/2 1/2log(2meoo))
where 0² = (Y In this case maximizing the mutual information ((Y: fe fe())) amounts minimizing E[ ( w.r.t. 0, i.e. the standard L2-loss. From this, we obtain a simplified
version of equation 4 via
arg min
max eeE
{ [Y -orr-fo(x))) fo(x)) (PR,
where Rd - fa(X)) and A > O. Under the conditions stated above, the objective achieves the mentioned trade-off between information and invariance.

---

Under review as a conference paper at ICLR 2021
Alternatively we view the problem to find features ho: RR - R such that [ (X), Y
gets maximized under the assumption that there exists a model fo(h((X)) + R = Y where R independent of h h(X) and R is gaussian. In this case obtain similarly as above the learning
objective
arg min 0.4
max eEEun
E[ (hoho())))) + (rRI Pho(x),e)
Classification
The expected cross-entropy loss is given through
-Ex,y [r(x)r log Σ exp(S(X):)]
where f: X - Th returns the logits. Minimizing the expected cross-entropy loss amounts to maximizing the mutual information between f(X) and Y (Qin & Kim, 2019; Barber & Agakov, 2003, eq. 3). Let f(Y;(())) = Y softmax(f(XX) with component-wise multiplication, then T is invertible in Y conditioned on the softmax output. Now we can apply the same invariance loss as above in order to obtain solution equation 4.
4.3 RELATION TO CAUSAL DISCOVERY
Under certain conditions, solving equation leads to features which correspond to the direct causes of Y (identifiability). In this case, we obtain the causal mechanism by computing the conditional distribution of Y given the direct causes. Therefore equation can also be seen as approximation of the causal mechanism when the identifiability conditions are met. The following Proposition states under which assumptions the direct causes Y can be recovered by exploiting Theorem 1.
We assume that the underlying causal graph G is faithful with respect to Px We further assume that every child of in G is also child of E in G. A variable selection h(X) XT corresponds to the direct causes if the following conditions are met: (i) : E, h()) is satisfied for diffeomorphism T-; h(X)), (fi) h(X) is maximally informative about Y and (iii) l(X) contains only variables from the Markov blanket o
Proposition 1.
The Markov blanket of Y is the only set of vertices which are necessary predict Y (see Appendix A). We give proof of Proposition as well as a discussion in Appendix F.
For reasons of explainability and for the task of causal discovery, we employ a gating function he in order to obtain a variable selection. We use the same gating function as in Kalainathan et al. (2018). The gating function h represents a 0-1 mask of the input. A complexity loss C(Φ) represents how many variables are selected and therefore penalizes to include variables. Intuitively speaking, if we search for a variable selection that conforms to the conditions in Proposition 1, the complexity loss would exclude all non-task relevant variables. Therefore, if H is the set of gating functions, then h* in equation would correspond to the direct causes of Y under the conditions listed in Proposition 1. The complexity loss as well as the gating function can be optimized by gradient descent.
EXPERIMENTS
5.1 SYNTHETIC CAUSAL GRAPIIS
To evaluate our methods for the regression case, we follow the experimental design of Heinze- Deml et al. (2018). It rests on the causal graph in Figure 1. Each variable XI,.., X6 is cho- sen as the regression target Y in turn, so that a rich variety of local configurations around is tested. The corresponding structural equations f(ppa()) = Σ (sp((s) mech(a,.), + NN, where 'mech' is either the identity (hence we get a linear SCM), Tanhshrink, Softplus or ReLU, and one multiplicative noise mechanism of the form (X N₁) (Ejerata) X,X,)) (1 + (1/4)N,) + N₁, resulting in 1365 different settings. For each setting, define observational environment (using exactly the selected mechanisms) and three interventional ones, where soft or do-interventions are applied to non-target variables accord- ing to Assumptions and (full details Appendix G). Each inference model is trained on 1024 realizations of three environments, whereas the fourth one is held back for DG testing. The tasks are to identify the parents of the current target variable Y, and to train transferable regression model based on this parent hypothesis. We measure performance by the accuracy of the detected parent sets and by the L2 regression errors relative to the regression function using the ground- truth parents.
selected among four model types of the form
7

---

Under review as a conference paper at ICLR 2021
Accuracy
0.6
0.4
301
0.2
10"4
X1
X1
X5
X5
X6
CERM
ERN
Flawe
Flow
AMHG
AN8
Figure 2: Detection accuracy of the direct causes for baselines and our gating architec- tures, broken down for different target vari- ables (left) and mechanisms (right: Linear, Tanhshrink, Softplus, ReLU, Multipl. Noise).
Figure 3: Logarithmic plot of 1.2 errors, nor- malized by CERM test error. For each method (ours in bold) from left to right: training er- ror, error on seen environments, domain generalization error on unseen environments.
We evaluate four models derived from our theory: two normalizing flows as in equation 4 with and without gating mechanisms (FlowG, Flow) and two additive noise models, again with and without gating mecha- nism (ANMG, ANM), using feed-forwart network with the objective in equation 9 (ANMG) and equation (ANM). For comparison, we train three baselines: ICP (a causal discovery algorithm also exploiting ICM, but restricted to linear regression, Peters et al. (2016)), a variant of the PC-Algorithm (PC-Alg, see Appendix G.4) and standard empirical-risk- minimization ERM, a feed-forward network minimizing the L2-loss, which ignores the causal structure by regressing on all other variables. We normalize our results with ground truth model (CERM), which is identical ERM, but restricted the true causal parents of the respec- tive Y.
X
X
X₂
X6
Figure 1: Directed graph of our SCM. Target vari- able Y is chosen among XI,.... X6 in turn.
The accuracy of parent detection shown in Figure 2. The score indi- cates the fraction of the experiments where the exact set of all causal parents was found and all non-parents were excluded. We see that the PC algorithm performs unsatisfactorily. whereas ICP exhibits the expected behavior: it works well for variables without parents and for linear SCMs, i.c. exactly within its specification. Among our models, only the gating ones explicitly identify the par- ents. They clearly outperform the baselines, with slight edge for ANMG, as long its assumption of additive noise is fulfilled.
Figure and Table 1 report regression errors for seen and unseen environments, with CERM indi- cating the theoretical lower bound. The PC algorithm is excluded from this experiment due to its poor detection of the direct causes. ICP wins for linear SCMs, but otherwise has largest errors, since it cannot accurately account for non-linear mechanisms. ERM gives reasonable test errors (it even overfits the training data), but generalizes poorly unseen environments, as expected. Our models perform quite similarly to CERM. We again find slight edge for ANMG, except under multiplica- tive noise, where ANMG's additive noise assumption violated and Flow superior. All methods (including CERM) occasionally fail in the domain generalization task, indicating that some DG problems are more difficult than others, e.g. when the differences between seen environments are too small to reliably identify the invariant mechanism the unseen environment requires extrapo- lation beyond the training data boundaries. Models without gating (Flow, ANM) seem to slightly more robust in this respect. A detailed analysis of our experiments can be found in Appendix G.
5.2 COLORED MNIST
To demonstrate that our model is able to perform DG in the classification case, we use the same data generating process as in the colored variant of the MNIST-dataset established by Arjovsky et al. (2019), but create training instances online rather than upfront. The response reduced to two labels 0 for all images with digit (0.....4) and for digits (5....9) with deliberate label noise
8

---

Under review as a conference paper at ICLR 2021
Models
Linear
Tanhshrink
Softplus
ReLU
Mult. Noise
FlowG (ours)
1.05....2
1.08..4.8
1.09...5.2
1.08.....7
1.55...8.64
ANMG (ours)
1.02
1.03...2.23
1.04.....6
1.03..4.32
1.46...4.2
Flow (ours)
1.08.....61
1.14...1.57
1.14.....55
1.14...154
1.35...4.07
ANM (ours)
1.05...1.52
1.15...1.47
1.14...1.47
1.15...1.54
1.48...4.19
ICP (Peters et al., 2016)
0.9
1.44...20.39
3.9...23.77
4.37...23.99
8.94...33.40
ERM
3.79
1.89....89
1.99..3.71
2.01..3.62
2.08...5.80
CERM (true parents)
1.06...1.88
1.06...1.84
1.06...2111
1.07...2115
1.37...5.1
Table I: Medians and upper 95% quantiles for domain generalization L2 errors (i.e. unseen environments) for different model types and tata-generating mechanisms (lower better).
classifier Co
Co +
56
70
60-
50
5.
Em..
End
Fov.
(Unsennn)
Enn
Emi.
Envy (Unee (
10
(Unseen)
10
10
10th
107
10-
Env. 90.3
Env.
Env.
Loss weight
Only Cu Cu + A1L1
79.9
10.2
73.7
74.9
67.1
Figure 5: Performance of the model in the three environments, depending the hyper- parameter A1.
Figure 4: Accuracy of a standard classifier (only
Lo, left), and our model (Co + >1 right)
that limits the achievable shape-based classification accuracy to 75%. To confuse the classifier, digits additionally colored such that colors spuriously associated with the true labels at accuracies of 90% resp. 80% in the first two environments, whereas the association is only 10% correct in the third environment. A classifier naively trained on the first two environments will identify color the best predictor, but will perform terribly when tested on the third environment. In contrast, a robust model will ignore the unstable relation between colors and labels and use the invariant relation, namely the one between digit shapes and labels, for prediction. We supplement the HSIC loss with Wasserstein term to explicitly enforce R E, i.e. L, HSIC + (sorst((R*)) sort (or"( Appendix D). This gives better training signal as the HSIC alone, since the difference in label- color association between environments and (90% vs. 80%) deliberately chosen very small to make the task hard to learn. Experimental details can be found in Appendix H. Figure shows the results for our model: Naive training (A1 = 0, i.c. invariance of residuals is not enforced) gives accuracies corresponding to the association between colors and labels and thus completely fails in test environment 3. contrast, our model performs close to the best possible rate for invariant classifiers in environments and and still achieves 67% in environment The recently proposed REx method from (Krueger et al., 2020) even got 68.7% accuracy on the unseen environment. will be interesting investigate whether this is deeper consequence of their objective just an insignificant effect of better network tuning and minor differences in experimental protocols. Figure demonstrates the trade-off between goodness of fit in the training environments 1 and 2 and the robustness of the resulting classifier: the model's ability to perform DG to the unseen environment improves Ry increases. If X, too large, it dominates the classification training signal and performance breaks down in all environments. However, the choice of X is not critical, good results are obtained over wide range of settings.
(R*))) (see
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1 0008990093 2019.

---

Under review as a conference paper at ICLR 2021
David Barber and Felix V Agakov. The im algorithm: variational approach information maxi- mization. In Advances in neural information processing systems, pp. None, 2003.
Elias Bareinboim and Judea Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy Sciences, 113(27):7345-755) 2016.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pp. 137-144, 2007.
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-554 2002.
R Frisch. Statistical versus theoretical relations in economic macrodynamics.pap given at league of nations, reprinted d.f. hendry and m.s. morgan (1995). The Foundations of Econometric Analysis, 1938.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. Romain-adversoee training of neural networks. The Journal of Machine Learning Research, 17(1):2096 2--09922030,, 2016.
AmirEmad Ghassami, Saber Salchkaleybar, Negar Kiyavash, and Kun Zhang. Learning causal struc- tures using regression invariance. In Advances in Neural Information Processing Systems, pp.
3011-3021, 2017.
AmirEmad Ghassami, Negar Kiyavash, Biwei Huang, and Kun Zhang. Multi-domand causal struc- ture learning in linear systems. In Advances in neural information processing systems, pp. 6266- 6276, 2018.
Daniel Greenfeld and Uri Shalit. Robust learning with the hilbert-schmidt independence criterion. arXiv preprint arXiv:1910.00270 2019.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
James J Heckman and Rodrigo Pinto. Causal analysis after haavelmo. Technical report, National Bureau of Economic Research, 2013.
Christina Heinze- Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6(2), 2018.
Kevin D Hoover. The logic of causal inference: Econometrics and the conditional analysis of cau- sation. Economics & Philosophy, 6(2):207-234, 1990.
Biwei Huang, Kun Zhang, Jiji Zhang, Joseph Ramsey, Ruben Sanchez-Romero, Clark Glymour, and Bernhard Schölkopf. Causal discovery from hacroogancous/noostttionnay data. Journal of Machine Learning Research, (19)):1-553, 2020.
Diviyan Kalainathan, Olivier Goudet, Isabelle Guyon, David Lopez-Paz, and Michèle Sebag. Sam: Structural agnostic model, causal discovery and penalized adversarial learning. arXiv preprint 2018.
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Sliced-wasseneee autoen- coder: An embarrassingly simple generative model. arXiv preprint arXiv: 2018.
David Krueger, Ethan Caballero, Joern- Doern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville. Out-of distriititiittoonn generalization via risk extrapolation (rex). arXiv preprint ar arXiv:2003.0066 2020.
Brenden M Lake, Tomer Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building ma- chines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Sara Magliacane, Thijs Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In Advances in Neural Information Processing Systems, pp. 10846-10855 2018.
10

---

Under review as a conference paper at ICLR 2021
Youssef Marzouk, Tarek Moselhy, Matthew Parno, and Alessio Spantini. Sampling via measure transport: An introduction. In R. Ghanem, D. Higdon, and H. Owhadi (eds.), Handbook of Un- certainty Quantification, pp. 1-41. Springer, 2016.
Nicolai Meinshausen. Causality from distributional robustness point of view. In 2018 IEEE Data Science Workshop (DSW), pp. 6-10. IEEE, 2018.
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Represen- tation learning via invariant causal mechanisms. arXiv preprint arXi::2010.07922 2020.
Joris M Mooij. Sara Magliacane, and Tom Claassen. Joint causal inference from multiple contexts. arXiv preprint arXiv: /6/ 10351, 2016.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-210, 2010.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv: 1:Xi::/912.02762 2019.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters, Joris M Mooij, Dominik Janzing, and Berhard Schölkopf. Causal discovery with continuous additive noise models. The Journal of Machine Learning Research, 15(1):2009-2053, 2014.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):947-1012, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithmms. MIT press, 2017.
Niklas Pfister, Peter Bühlmann, and Jonas Peters. Invariant causal prediction for sequential data. Journal of the American Statistical Association, 114(527):1244-1277 2019.
Zhenyue Qin and Dongwoo Kim. Rethinking softmax with cross-entropy: Neural network classifier as mutual information estimator. arXiv preprint 19X1v:911..106: 2019.
Mateo Rojas-Carulla Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309-134) 2018.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv: 6471,,66111116 2012.
Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review, 91):622- 1991.
Jin Tian and Judea Pearl. Causal discovery from changes. Uncertainty in Artificial Intelligence (UAI), pp. 512-521, 2001.
Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: A causal view. In Twenty-inihh AAAI conference on artificial intelligence, 2015.


Reviews:
[{"review_id": "Xaqeqf8IPJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well-motivated and studies and important topic, but unfortunately it is let down by the presentation of their contributions which is confusing and at times misleading. \n\nFirst - a more minor complaint (which I put here because its a source of confusion for the rest of the review) - The normalizing flow section is confusing because the mapping between the base distribution and y isn't clear. I normally think of a normalizing flow as a map from some base distribution u to some target y such that y = T(u) and p(y) = p(u)|det J_T (u) | where u=T^{-1}(y) (adding conditioning as required to make a conditional flow). This paper uses T(Y; h(X)) everywhere - which I think is referring to T^{-1}(Y; h(X)) because we normally think of T as acting on the base distribution U and T^{-1} as acting on the target variable. My review assumes that I should read T(.) as a map from Y -> U... but that's a little weird and should be explained explicitly. \n\nMore seriously, I don't understand why Lemma 1 isn't trivial? By the data processing inequality, any transformation of X can only lose information about X. So if the identity function is among the set of feature extractors, then h^* includes it,  because it maximizes I(h(X), Y). The fact that h^* is independent of the flow's latent variable trivially follows from the fact that choosing the identity is always optimal. Of course, things get more complex if there is some constraint on H such that the identity isn't included,  but this isn't discussed. On a second reading, I think that this constraint is meant to come from the Y \\perp E | h(X) condition in section 4, but how this condition interacts with Lemma 1 needs to be clearer.\n\nThe presentation of the method in section 4 also needs work: the domain generalization problem is presented as the problem of finding h that maximizes the mutual information between Y and h(X) in the worst case environment under the constraint that Y\\indep E | h(X). As far as I can tell, the independence constraint is the important part of that objective: under that constraint, it is not clear why I wouldn't want to maximize the average mutual information, or some other objective? \n\nSimilarly - it's not clear why theorem 1 is useful until we get to equation (5) (and it took me a couple of reads to realize that this is actually the important step) - on its own, it just essentially says that if we have conditional independence, then applying a 1:1 function maintains that conditional independence.\n\nHaving gotten to this point in this review, I think that many of my issues would be resolved if the presentation order was reversed. The key condition you need is Y \\perp E | h(X); The paper would be far easier to follow by making it clear that is is the condition you need, explaining both why we can't optimize for it directly, and why this particular normalizing flow approach gives an indirect approach to achieving the condition. In the current order of presentation which leads with a discussion of normalizing flows, we are presented with theoretical results about flows which, in isolation, seem trivial. \n\nThe experiments show the method shows promise (though they should report both IRM & REX [Kruger et al 2020]'s performance for coloured MNIST to make it clear that there are better methods on that dataset)...\n\n[Kruger et al 2020] Out-of-Distribution Generalization via Risk Extrapolation", "extra_scores": {}, "extra_text": {}}, {"review_id": "G_ODmP1b1Dp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new gradient-based framework for learning invariant mechanisms (often called \"relations\" in the paper) from data drawn for multiple environments (data generating processes).  Overall, the writing is excellent, and the central ideas are interesting and valuable.\n\nA key idea of the paper is that training data drawn from different environments can be exploited to learn mechanisms that remain invariant across those environments.  While true, this is unsurprising and well-established.  Fundamental principles of causal inference, known for decades at this point, directly imply that different environments (data generating processes with different interventions) will allow identification of different sets of causal dependencies.  Practical methods for such identification have been demonstrated using graphical models and relatively simple methods for parameterization of those models.  The paper could be improved by spending less time on the known results (or at least making clearer connections to prior work) and spending more time clarifying what is genuinely novel about the proposed ideas.  In addition, the authors should make a greater effort to distinguish between central ideas and implementation details.\n\nMultiple times in the paper, basic results from the causal inference literature are attributed to relatively recent papers (e.g., Peters et al. (2017)), including the special properties of the causal factorization and the idea of invariance of mechanisms in response to intervention.  These ideas can be traced back much further.  For example, the basic idea of invariance to intervention (so called \"autonomy\" or \"modularity\") has been known since at least the 1930s.  Heckman and Pinto (2015) note that: \"In the language of Frisch (1938), these structural equations are autonomous mechanisms represented by deterministic functions mapping inputs to outputs. By autonomy we mean, as did Frisch, that these relationships remain invariant under external manipulations of their arguments.\" The paper would be improved by making clearer when concepts were first identified and by who.\n\nThe empirical evidence provided for the claims in the paper is relatively modest.  The simulated results provided in Table 1 shows only very small differences in L2 errors among variants of the authors' proposed methods, and more substantial improvements over ICP and ERM (in three of four cases).  The discussion of these results is excellent.  The results on the \"Colored MNIST\" data show the expected results.  However, good performance on simulated data and only a single real data set is still relatively weak evidence for the claims made in the paper.  The paper would be improved by increasing the number of real data sets used for evaluation.\n\nReferences\n\nHeckman, J., & Pinto, R. (2015). Causal analysis after Haavelmo. Econometric Theory, 31(1):115-151.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6ZBqystrj1I", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a gradient-based learning framework, with a two part objective function in which one part improves the informativeness about the target variable, and the other part enforces the invariance of the relation. The second part is based on the ICM principle and increases the stability and renders domain generalization possible. \n\nThe paper is well written and, for the most part, is easy to follow.\n\nWe should note that the ICM principle is only usable if we have no hidden confounders, i.e., causal sufficiency, in the system. The authors should clarify that causal sufficiency is an important assumption early in the manuscript and should clarify what will happen to the results if it is violated.\n\nIn general, the assumptions in this work are very strong and I do not believe they will hold in reality. Specially, regarding Assumption 2, if we are assuming some of the causal mechanisms are changing across environments, why the one corresponding to the target should not change?\n\nAlthough the assumptions are strong, same assumptions were considered in few other works such as (Peters et al., 2016). Compares to existing work with the same assumptions, this paper provides a good implementation method that is an improvement over past work and would be of interest to the ICLR community.\n\nThe authors also discuss the conditions under which the recovered stable relations correspond to the true causal mechanisms. The use of ICM for causal discovery is also extensively studied in the non-parametric case in [Huang et al., Causal Discovery from Heterogeneous/Nonstationary Data], and in the linear case in [Ghassami et al., Multi-domain Causal Structure Learning in Linear Systems].\n\nThe definition of do-intervention in page 3 is not standard. What is referred to as do intervention in this paper is usually referred to as hard intervention in the literature, and what is referred to as hard intervention in this paper is usually referred to as atomic intervention in the literature.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new framework for improving supervised learning via invariant mechanisms. The reviewers agree that overall, this paper is well-written and contributes to a growing body of work on invariant prediction and causality in supervised learning. At the same time, there are some concerns regarding novelty and significance in light of previous work, as well as the overall organization of the paper, which could be improved to highlight the main contributions more clearly. Ultimately, this was a borderline decision, but it is clear that the paper needs a major revision before acceptance. Although the authors have already incorporated some of the minor comments which is appreciated, the authors are urged to consider the major comments (e.g. see R2's comments regarding presentation) when revising the paper.
Decision: Reject