Recovering Latent Causal Factor for Generalization to Distributional Shifts
Authors: Xinwei Sun
Botong Wu
Xiangyu Zheng
Chang Liu
Wei Chen
Tao Qin
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: distributional shifts
domain generalization
invariance
causal factor
latent causal invariant model

Abstract: Distributional shifts between training and target domains may degrade the prediction accuracy of learned models, mainly because these models often learn features that possess only correlation rather than causal relation with the output. Such a correlation, which is known as ``spurious correlation'' statistically, is domain-dependent hence may fail to generalize to unseen domains. To avoid such a spurious correlation, we propose \textbf{La}tent \textbf{C}ausal \textbf{I}nvariance \textbf{M}odels (LaCIM) that specifies the underlying causal structure of the data and the source of distributional shifts, guiding us to pursue only causal factor for prediction. Specifically, the LaCIM introduces a pair of correlated latent factors: (a) causal factor and (b) others, while the extent of this correlation is governed by a domain variable that characterizes the distributional shifts. On the basis of this, we prove that the distribution of observed variables conditioning on latent variables is shift-invariant. Equipped with such an invariance, we prove that the causal factor can be recovered without mixing information from others, which induces the ground-truth predicting mechanism. We propose a Variational-Bayesian-based method to learn this invariance for prediction. The utility of our approach is verified by improved generalization to distributional shifts on various real-world data. Our code is freely available at \url{https://github.com/wubotong/LaCIM}.

Recovering Latent Causal Factor for Generalization to Distributional Shifts
Xinwei Sun"1, Botong Wu Xiangyu Zheng??, Chang Liu', Wei Chen', Tao Qin', Tie-Yan Liu'
Micrrssott Research Asia, Beijing, 100080
Peking University, Beijing, 100871
(xinsun, changliu, wche, taoqin, tyliu] (eriiiummmmiiim crosoft. com, botongwu@pku. edu.
Abstract
Distributional shifts between training and target domains may degrade the predic- tion accuracy of learned models, mainly because these models often learn features that possess only correlation rather than causal relation with the output. Such a correlation, which is known as "spurious correlation" statistically, is domain- dependent hence may fail to generalize unseen domains. To avoid such spurious correlation, we propose Latent Causal Invariance Models (LaCIM) that specifies the underlying causal structure of the data and the source of distributional shifts, guiding us to pursue only causal factor for prediction. Specifically, the LaCIM introduces pair of correlated latent factors: (a) causal factor and (b) others, while the extent of this correlation is governed by domain variable that characterizes the distributional shifts. On the basis of this, prove that the distribution of observed variables conditioning on latent variables is shift-invariant. Equipped with such an invariance, we prove that the causal factor can be recovered without mixing information from others, which induces the ground-truth predicting mecha- rriiational-Baesiann based method to learn this invariance
nism. We propose for prediction. The utility of our approach is verified by improved generalization distributional shifts on various real- wallwwrldd data. Our code freely available https: gi thub. com wwbbtttog//////////o//ooomoo LaCIM.
Introduction
Current data-driven deep learning models, revolutionary in various tasks though, often exploit all types of correlations to fit data well. Among such correlations, there can be spurious ones corresponding to biases (e.g., confounding bias due to the presence of third unseen factor) inherited from the data provided. Such data-dependent spurious correlations can erode the prediction power on unseen domains with distributional shifts, which can cause serious consequences especially in safety-critical tasks such as healthcare.
Recently, there is a Renaissance of causality in machine learning, expected to pursue causal rela- tionships [59] to achieve stable generalization across domains. The so-called area of "causality" is pioneered by Structural Causal Models [51], as a mathematical formulation of this metaphysical concept grasped in the human mind. The incorporation of these human priors about cause and effect endows the model with the ability to identify the causal structure [51] 1] which entails not only the data but also the underlying process how they are generated. To achieve causal modeling, the old-school methods [52, 10] directly causally related the output label Y to a subset of covariates X, which is however not conceptually reasonable in applications with sensory-level data (e.g. model pixels causal factors of the output does not make sense in image classification [11]).
Corresponding author
Work done during an internship at Microsoft Research Asia.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

For such applications, we rather adopt the manner of human visual perception [8, 9, 80] to causally relate the label Y to unobserved abstractions denoted by S, i.e. Y + S. We further assume the existence of another non-causal latent factor (of Y) denoted as Z, that together with S generate the input X: X < (S, 7). Such an assumption is similarly adopted in the literature [25, 27, 35, 75, 71]. To model shifts across domains, we allow Z to be spuriously correlated with S (hence also the output), as marked by the bidirected arrow in Fig. (a). Taking image classification as an example, the S and Z respectively refer to object-related abstractions (c.g., contour, texture) and contextual information (e.g,, s., background, view). Due to this correlation, the model can learn contextual information into prediction, which may fail to generalize to the domain such that this correlation is broken.
We encapsulate above assumptions into the skeleton illustrated in Fig. (a), which the spurious correlation between S and Z varies across domains, as marked by the red bi-directed arrow in Fig. (b). Taking a closer inspection, such a domain-dependent spurious correlation is governed by an auxiliary domain variable D in Fig. (c), which causes the domain shifts. We call the set of causal models augmented with as Latent Causal Invariance Models (LaCIM). Here, the "Causal Invariance" refers to PS))S), which together with P(X|S, Z), can be proved to be stable to the shifts across domains, under the assumptions embedded in the causal structure of LaCIM. Equipped with such an invariance, we prove that the S and the ground-truth predictor: P(Y (s)) for generated from (s", )² are identifiable up to transformations that do not mix the non-causal information.
Under such an identifiability guarantee, we propose to learn the P(Y and P(X|S, Z) by reformu- lating the Variational Auto-encoder (VAE) [37] to fit the joint distribution of the input and output variables from training domains. During the test stage, we first infer the value of S by optimizing the estimated P(X S, Z) over latent space, followed by the learned P(Y for prediction. We first use simulated data to verify the correctness of the identifiability claim. Then, to demonstrate the utility, we test our approach on real-world data, consistently achieving better generalization to the new distribution; besides, we find that our inferred causal factor can be concentrated in highly explainable semantic regions for the task of image classification.
We summarize our contribution as follows: Methodologically (in sec. 4.1), we propose LaCIM in which the causal assumptions of two latent factors and the distributional shifts are incorporated; Theoretically (in theorem 4.4). we prove the identifiability of the causal factor and the ground-truth predicting mechanism; Algorithmically (in sec. 4.3), guided by the identifiability, we reformu- late Variational Bayesian method to learn P(X|S, 2), P(Y|S) for prediction; Experimentally (in sec. 5.2), our approach generalizes better to distributional shifts, compared with others.
2 Related Work
Causality for Domain Generalization. Due to its stable transferability, the concept of causality has been introduced in many recent works for domain generalization [39, 59, 52, 10, 40, 21, 68). Most of these works learned the assumed (causal) invariance for generalizing to unseen domains. However, they suffer from either i) lacking explicit causal modeling; or ii) inappropriate causal relations made for the output. Specifically, for i), the [39, 59] are still data-driven methods to learn stable correlation (i.e., invariance) without incorporating causal assumptions [51] beyond data, which may impede its generalization to a broader set of domains; for ii), the [52, 10, 40, 21, 68) causally relate the output with covariates, which is inappropriate for sensory-level data.
Our Specification. We explicitly incorporate the causal assumptions. Specifically, we introduce i) latent factors and separate them into the causal and the non-causal factor; ii) the domain variable D, as a selecting mechanism to generate the varied S-Z correlation across domains. Such a causal modeling makes it possible to recover the causal factor S for generalization. In independent and concurrent works, [75] and [28] also explore latent variables in causal relation. As comparisons, [75] did not differentiate S from Z. The spurious correlation in [28] is limited in the correlation between domains and the output; while it is allowed in our setting to exist in a single domain, which is more aligned with real scenarios, e.g., the dog is more associated with grass than snow in domain when most samples are collected in sunny morning.
Other Conceptually Related Works: i) transfer learning that leverages invariance in the context of domain adaptation [60, 81, 17] or domain generalization [43, 63]; (ii) causal inference [51, 53] which builds structural causal models and define intervention (a.k., "do-calculus") for cause-effect reasoning and counterfactual learning: and (iii) latent generative model that assumes generation from latent space to observed data [37, 71] but aims at learning generator in the unsupervised scenario.

---

C
D
Z
S
Z
S
X
Y
X
Y
X
Y
(b)
Figure 1: The introduction of Latent Causal Invariant Models, set of SCMs augmented with domain variable D in (c), starting from the skeleton in (a). Specifically, from (a) to (b): the spurious corrllation between S and Z is due to change of P(C) and P(S, Z(C) across domains; from (b) to (c): we ascribe such shifts to the domain variable D, as the source of distributional shifts. Here, the observed and unobserved variables are respectively marked by solid and dot circles; the directed arrow ("-"" or "+") represents direct causal relation; while the bidirected arrows ("<+") represent the spurious correlation that can vary across domains; the red and blue respectively means the invariant and varied values/distributions.
unobserved confounder C, with distributional shifts referring
3 Preliminaries
Problem Setting, Let X, respectively denote the input and output variables. The training data JeEErain collected from multiple environments E Etrain, where each is associated with distribution P (X(X, Y) over X x y and De := {-f, vf hie(ne]
pe with [k] := (1....,k} for any k E Z+. Our goal is to learn a robust predictor f X y that only exploit the causal factor for prediction and generalize well to all domains & 3 Errain and Cursive letter to denote the random variable, the instance and the space, e.g., a is an instance in the space A random variable A. For : f(x) n with B := R x R1 x ... x RP[ig), the [f(I]]A denotes the f(x) restricted dimensions of A. i.e., [f(x)] [f,, (I), fir (z)). The Sobolev space (*rp((A) contains all such that S^ [aaf
ii,d ~
use respectively upper, lower case letter
A== [A=a dp(((a < oo, Va S k.
Structural Causal Model. The structural causal model (SCM) is defined as triplet M F,,F, P(æ)), in which i) the causal structure G == (V, E)) (V, E respectively denote the node and edge set) described by a directed acyclic graph (DAG); ii) the structural equations F == (ʃ& Vv are autonomous, i.e., intervening on V/2 does not affect others, based on which we can define the do- operator and calculate the causal effect; iii) the P(=) are probability measure for exogenous variables {=*JR- By assuming independence among {R, we obtain according to Causal Markov Condition that each P that compatible with G has P([Vk = Up vvvv P(V, = Pa(Pa(k) = pa(k)). An acyclic directed mixed graph (ADMG) further allow the existence of bidirectional arrows ++, meaning the spurious correlation between two variables connected.
Methodology
We first incorporate the causal assumptions into LaCIM in sec. 4.1. Under such assumptions, we identify the invariant distributions P(X|S, Z) and P(Y[S), which are repectively dubbed as generative invariance and causal invariance that are robust to domain shifts. Equipped with these invariances, we in sec. 4.2 show that the causal factor can be identified without mixing information from non-causal one during prediction. Finally, we introduce our learning method sec. 4.3 to estimate the PS Z) and SSY which are respectively resorted in the inference and prediction that constitute a robust predictor during test stage.
4.1 Latent Causal Invariance Models
In this section, we introduce a set of structural causal models dubbed as Latent Causal Invariance Model (LaCIM), which incorporates the causal assumptions mentioned above and also the source of distributional shifts. The corresponding causal structure of LaCIM is illustrated in Fig. 1 (c), which will introduce step-by-step from the skeleton in Fig. (a).
Fig. (a). Specifically, the ADMG in Fig. 1 (a) introduces latent factors == (S, Z} to model the abstractionsscconcppss that generate the observed variables (X,Y), similarly assumed in
3

---

unsupervised latent generative models [37] for image generation. Further, we explicitly separate the V into S and Z, with only S causally related to the label Y. In image classification, such a causal factor refers to the (shape,contour) of the object need to be classified; while the image X is additionally affected by contextual factor such as light, view.
Fig. 1 (a) - Fig. 1 (b), In addition, we assume that S is spuriously correlated with Z, as marked by the red "++" in Fig. (a). Such a spurious correlation corresponds the bias inherited from data, e.g. the contextual information in image classification. Therefore, the magnitude of this correlation distribution -eeeeddenn and thus can vary across domains. Statistically, the "spurious correlation" implicates the presence of a third unobserved (we use dot circle to represent unobserved variables) confounder, which is denoted as C' in Fig. 1 (b). The unblocked path from Z Y induced by C can lead to learning the non-causal factor during data- fitta-iiiiing, which can degrade the performance on unseen domains if the correlation between this non-causal factor and the output broken.
Fig. (b) + Fig. (c). Taking a further inspection in Fig. 1 (b), the varying degree of correlation can be either due to the distributional shift of S, Z|C or of the C itself across domains (we use red color to mean varied distributions). As both shifts are domain-dependent, we in Fig. 1 (c) ascribe them domain variable D, which causes the mutation of its children nodes' distribution, i.e. S, Z and C. Such a domain variable has been similarly introduced in [69, 68] to generate mutable variables. In our scenario, we do not require D be observed; rather, we only need the domain index d' (one-hot encoded vector with length m == |Eerain)) D. The set of SCMs augmented with D, with the SCM Markovian compatible to the DAG of C, Z, X,X,Y Y in Fig. I (c), is dubbed as Latent Causal Invariance Models (LaCIM) that is formally defined as follows:
Definition 4.1 (LaCIM), The LaCIM denotes a set of SCMs augmented with the domain variable D. i.e. {(M deEE, in which d' denotes the value of D and M == (c.Fe,((E)) for e. The G denotes the DAG restricted on C. Z, X.Y, Y. For each environment/domain e, the Fe == JnySer Je, Je fee correspond to generating mechanism of X,Y, Y, Z, C, with fe((s) == ge(Ee, d²)) [i(c, Es) == gs(e, dx) and (c, == 92 Ez, d') from some ges 9s: 92.
Remark 1. Different from scenarios in which X generates [28] nor generated from Y [1], we con- sider the scenario when the X and Y are generated concurrently, which can widely exist but ignored in the literature. For example, the clinicians are recording the disease status while implementing the ultrasound test at the same time, during medical diagnosis.
from
As an illustration, we consider the following example, in which the distributional shifts caused by domain variable D) can refer to sampling bias in data.
Example 4.1 (Sampling Bias). Consider the cat/dog classification, in which the animal in each image is either associated with the snow or grass. The D refers to the sampler, which generates the C' that denotes time and weather to collect each sample. The S, Z respectively refer to the features of animals and context. Since each sampler may have a fixed sampling pattern (e.g. gets used to going out in the sunny morning (or in the snowy evening)), the data one collects may have sampling bias: dogs (cats) more associated with grass (sRow) in the sunny morning (or snowy evening).
The Def. 4.1 specifies the generating mechanisms across environments and how they differ. Equipped with such specification, we can identify the invariant mechanisms that are stable to domain shifts:
Proposition 4.2 (Causal Invariance & Generative Invariance). For LaCIM in Def. 4.1, the P(Y\S) and P(X|S. SS 7) are invariant to shifts across E, and are respectively denoted as Causal Invariance (CI) and Generative Invariance (GI).
Remark 2.
The generating process from latent variables to observed variables follows from physical law, c.g., the shape, contour, color, view, light should satisfy physical constraints to generate reasonable image. Therefore, it is naturally hold that such generating processes are invariant.
The P(X|S, Z) and PSY|S) can induce an invariant predicting mechanism. Specifically, for new sample x + fs",, 22) ). y + fy(s", (o,,,,,) Ey)), we can first infer the causal factor s* from pf. (a) s, 2) by maximizing log- likelihood of pss (xrss, 2) over S x Z and then feed the estimated s into Ps, (y) )*) for prediction. To ensure the robustness of such a two-step invariant prediction, we need to answer two following identifibiilty questions:
1. Can the inferred causal factor S not mix the information of (disentangled from) others?
2. Can such invariant predictor recover the ground- ground-truth predictor P(Y\s*))

---

We will answer these questions in the subsequent section, followed by our learning methods to identify the causal factor and the causal/generative invariance for prediction.
4.2 Identifiability Analysis
We present the identifiability results regarding (i) the disentanglement of inferred causal factor S from non-causal Z, and (ii) the induced true predicting mechanism P(Y|s" [s*) for x: < f=(s", Ex). which respectively echo the two questions imposed in the last section.
Our main results are presented in theorem 4.4, To distinguish the causal factor S from others, our results require that the degree of diversity of 5-Z correlation across environments is large enough, which has been similarly assumed in the literature of identifiability [52, 1]. Such diversity condition implies the dramatical change of correlation between Z and Y. thus providing a clue to disentangle the S. Such disentanglement analysis, is crucial to causal prediction but is ignored in existing literature about identifiability, such as those identifying the discrete latent confounders [32, 62], or those relying on Additive Noise Model (ANM) assumption [31], or linear Independent Component Analysis (ICA) [14, 35, 36, 75] (Please refer to supplement D.1 for more exhaustive reviews). More importantly, will later in theorem 4.5 show the extension of above analysis from exponential family of P(S, Z(C) to Sobelev space; and from ANM for Y to categorical distribution for Y.
We assume the ANM for fz(s, 2, f has been widely adopted identify the causal factor [30, 54, 35]. We assume the fz to be bijective and invertible (we will discuss it later). We first narrow our interest to a subset of LaCIM denoted as Poxp in which any model in P Pexp satisfies that (i) the S, Z belong to the exponential family; and (ii) the Y is generated from the ANM:
&x (we replace fz with fa for simplicity). which
Pexp-
|LCIM with any > 0ly fs(() + Sy,
= a.r,, ona (t/e), Ve). with
T:
Pr c. (t) = II
exp
(t) + Bt(t.) Ac,dr:): rkk, q:
(1)
for f. = 8,2 and E E, with qt, hi respectively denoting the dimension of t 8, and the number of natural parameters in each dimension. The (TTr)(t))): (T² (Te.r.s)) denote the sufficient statistics and natural parameters, { } and (AL,dr.) denote the base measures and normalizing constants to ensure the integral of distribution equals to I. Let (') = [tt),,,... To (tyx)] Rhexee (T((t))== (t.), .., & (t.)), Vi E [qc)). I d := [Té f..d moy r .. E Rhexee (Ta := T c,d" s, Vi E [9:]). We further assume that the (C) serves to discrete distributions on the set {ci,.., cn.) with which the (*) = J p(ssc))(=le)ddP'() = Σ, 1' (s. (x can be regarded as the mixture of exponential family distributions. Rather than uniquely inference. we target on disentangling the S from Z and also recovering the ground-trutt -tuth predictor, which is formally defined as eep -exp--ideniiee -ideniifaabiiity as follows:
[T'
,.1....., (.,
fa(S x Z). We define a binary relation O exxp Ô on the parameter space of X x y: there exist two sets of permutation matrices and vectors, (M,, a.) and (M2, for and respectively, such that for any (z, y) E X x xy, the following hold:
Definition 4.3 (Nexp identifiiiilll. Suppose the X' 2
I (l))))s()) M,T (s(x))s())))))))))())))))))))) + as, T (T (x)) M. TM ((=)))(()) + azi
(2)
Pj, (ullf = Ps, (vI (s))))
(3)
We then say that 0 is Nexp -ddettffiie,,,, ifr any 0, p5 v) Pf (z, v) Ve E Erriin implies O eexp Ø.
This definition is inspired by but beyond the scope of unsupervised scenario considered in nonlinear ICA [27, 35] in that, the former further disentangle S from Z (in Eq. (2)) and identify the true predicting mechanism (in Eq (3)). To see disentanglement, note that for any clean (noise-free) sample + fs" 2²),, the Eq. (2) ensures that the inferred causal factor T (If="1]s( does nor mix the information of others, unless the extreme case that there a deterministic function between S and Z, in which it is impossible for S to be identified. With such an identification of s, the Eq. (3) further guarantees that the learned P7, ()(])=)[(())) can recover the ground-truth prediction probability density, i.e., Pj, ()(r))"))(())) = Py, (yls*). With noise, the s* can be inferred with some indeterminacy. The formal result is presented in theorem 4.4.
5

---

Theorem 4,4 (~exp- identifiability). For O Pexp in Def. 4.1 with m == |Ereatub we have that the O ~ exp identifiable under following assumptions:
1. The characteristic functions of EasEy are almost everywhere nonzero.
2. fu- Ima Jh are continuous and Ja, Jy are bijective;
3. The (iihh are linearly independent in S or Z for each i E [gr] for any = 5, and
T are twice differentiable for any t = 8, 8,2, E [ E [ka]]
4. The ([f=((())) (1(((")22())) :B(r)
0} contains a non-empty open set in
19, IR 19.. xK=, with ((x) == Ii, Bi, (I/ (..(0-(1,,,))) (L) Bi,
(x)) [T cg.ddm
5. The L = [pe Pa "C) E RR and [[T [[r -
r q
E (R(Rxm) xks) huve full column rank.
The assumptions 1-3 are trivial and easy to satisfy. The characteristics functions of Ex; Ey can be almost everywhere non-zero for most continuous variables, such as Gaussian, exponential, beta, gamma distribution. This assumption can ensure the identifiability of pffppp(fffpppp the appendix. The bijectivity of fo and fy have been widely assumed in [30, 54, 53, 35, 75] as basic condition for identifiability. It naturally holds for fz to be bijective since it has been empirically proven in auto-encoder [38] that the wiw-emnnsion embeddings (i.e., 9s + 9= 9z) can recover the original input well and also that the variational auto-encoder can extract meaningful representations from x. For the 9 with categorical Y such that pt k|s) = [f (s)/ (E(ffle(s)) the fy may not satisfy the bijectivity condition. We will shown identifiability for such a categorical case later in theorem 4.5. The assumption 3 can be uniformly satisfied for all distributions in the strongly exponential family. The containment of an open set in assumption (4) for ([T=([])()))) T (( (a))) B(xx) > 0} implies that space expanded by sufficient statisties are dense in some open set, as a sufficient condition for the mixture distribution Pd (C) and also P" (X)) to be identified. The diversity assumption (5) implies that i) m 2 R and m * R 2 max(k; * qz h2 * 9.) + 1; and that ii) different environments are diverse enough in terms of S-Z correlation, as an almost a necessary for the invariant one to be identified (a different version is assumed in [1].. In supplement B.2, (ere-((eeeeggee) measure
as will be shown in
will show that the ii) hold unless the space of I belong set. As indicated by the formulation, a larger m. would be easier to satisfy the condition, which agrees with the intuition that more environments can provide more complementary information. Besides, our result can be extended to non-independent case among {$1,..., Sq (or {Z1,... Rq. }), i.e. a Btt) A (t 8, 2), which will shown in supplement B.2. PT (t) = exp((TT) (t), I
Extension to the general forms of LaCIM. We extend to general forms of LaCIM in theorem 4.5 as long as its P(S, Z|C c) E ( 2) (for some r 2 2) and categorical Y. in the following theorem. This is accomplished by proving that any model in LaCIM can be approximated by a sequence of distributions with parameterization in Pexp. motivated by [3] that the exponential family dense in the set of distributions with bounded support, and in [44] that the continuous variable with multinomial logit model can be approximated by series of distributions with i..d Gumbel noise as the temperature converges to infinity. The proof is left in the supplement.
Theorem 4.5 (Asymptotic eex resspiidentiiiabiitty) Suppose the LaCIM satisfy that p(x|s, 2) and p(y)s) are smooth w.E. s, z and respectively. For each e and E C, suppose P (S, Z|c) E W (S x Z) for
some 2 2, we have that the LaCIM is asymptotically ~exp -exp-identiffable: Ve > 0, 3 eepp -repp-identffablee
Po E Peepp. s.t. dPok (p (X, Y). i% [X, Y)) < Ve E Errain
Our proof is built on [3] that any probability in Sobolev space can be approximated by a sequence of distribution with the number of natural paramters going to infinity, i.e., ke + co.
4.3 Learning and Inference
Guided by the identifiability result, we propose to learn P(X 2) and P(Y via generative modeling following from Fig. (c). Then to predict the label for new sample generated from
we first leverage the learned p(æ]s, 2) to infer s* that is ensured to be able to not mix the non-causal information, followed by learned P(ylš*) for prediction.
'The drok(Hi, μ2n denotes the Pokorov distance between and μ2, with lim dpok dpok 11) - 0
pn S μ.

---

4.3.1 Learning Method
To learn the P(X [s, 2), PSY for invariant prediction, we reformulate the objective function of Variational Auto-Encoder (VAE) the supervised scenario, in order to {x"(x, y) JEE As latent generative model, the VAE was originally proposed for unsupervised generation from latent variables V to high-dimensional input variable X. To make such a generation tractable, the VAE introduced variational distribution qú parameterized by z approximate the intractable posterior by
maximizing the following Evidence Lower Bound (ELBO): Ca, Bp(x) (v)x)) log pe(2.") S (o(u)z))
[o Ep(x) Po (z)), where the equality is achieved when (v(r)) = (v)].)) Therefore, maximizing the ELBO over and qu will drive (i) qu (1/2) to approximate pa (vla);; (ii) pe to estimate the ground-truth model p.
To adapt the above surrogate loss to our DAG in Fig. 1 (c), we introduce the variational distribution qe (s, 2/2,, y) for each environment e. The corresponding ELBO for e is
[ (s.elaa))) log (2 U. U.S,2) ),,,, where Pπ( 8, = 9 (s, 2 21
- -5 - B (ars)
( 2)po()(s)) 2).
Similarly, minimizing Sg can drive pa [rrss, =), Po to approximate the p(z] p(yl (s)s (). and
also (s, 2|2, v) to estimate (5 2|2,, v). Therefore, the ge can inherit the properties of pe. As ps(s, 2 y) = (((e)))))po(v1s) for our DAG in Fig. (c), we can similarly reparameterize gf 2/2, v) (5(u/z)
(pll))) with (vls)) replaced by po(y]s) (since the goal of tu is to mimic the behavior of as o (M
Po). Then, the 2 can be rewritten as:
log =))8 2)1 g (2)2z) (3 2|xr)
po(l)s)
-g. (x,)) log Le (glax) Re
(4)
where (v12)) - Is q (s)x)po(yls)de We correspondingly parameterize the prior model p5 (s, and inference model L.f (s, =|x) as ps, =ld²) and (s, d²), in which d (of environment e) denotes the domain index that can be represented by the one-hot encoded vector with length m = |. The overall loss function is:
CB
C
(5)
eEEtroin
The training datasets (D*) JeEEuni are applied to optimize the prior models {p(s, =)d") inference models (s, 2/2r, )e. generative model pa(z)s, 2) and predictive model po s). Particularly, the parameters of ( s, 2) and (y(ss) are shared among all environments, motivated by the the invariance property of P(X (S, Z) and P(Y|S) across all domains.
4.3.2 Inference & Prediction.
We leverage the learned SS 2), PL (8) for prediction. According to Prop. 4.2 and Eq. (3) in theorem 4.4, the induced predictor via P(X[S, [s, Z), P() can recover the true predicting mech- anism for any distributional shifts from E. Specifically, for any generated by (s*,2)), we first optimize the following log- likelhoodoood of po 2) over S x to infer s", 2*,
max log (x
(x|s, 2)) 1 2 + 1=11=
(6)
with hyperparameters A 0 and X2 > 0 in order to control the learned s, a in reasonable scale. Note that Eq. Eq. (6) is different from the maximum a posterior estimation since the posterior (s, is parameterized differently for different e while the pe(r|s,2) is invariantly parameterized for E. (this is because p(r]s, 2) is invariant). For optimization, we adopt the strategy in [61] that first sample some candidate points from \'0 7) and select the optimal one in terms Eq. (6) as initial point; then use Adam to optimize for another 7 iterations. The implementation details and optimization effect are shown in supplement F.2. Finally, with estimated 8* E*, we implement the leamed pe(y)á*) for prediction: ÿ arg max,, Po pe())3").
7

---

Experiments
We first verify the identifiability claims of theorem 4.4 in sec. 5.1. Then we evaluate LaCIM on real-world data in sec. 5.2: Non-I.I.D. Image dataset with Contexts (NICO); Colored MNIST (CMNIST); Alzheimer's s Disease Neuroimaging Initiative (ADNI www 1oni ucla. edu/ADN for early prediction of Alzheimer's s Disease), to verify the generalization ability of our method on the target domain with distributional shifts.
5.1 Simulation
To verify the identifiability claims, we implement LaCIM on synthetic data. We generate C, S, Z, X,Y following Fig. (with details left in supplementary). We choose m = 3, with the same total number of samples. To verify the advantage of learning (m > 1), we compare with pool-LaCIM: minimizing the loss Eq. (4) on the pooled data from all m domains. We compute the mean correlation coefficient (MCC) adopted in [35], which measures the goodness of identifiability under permutation by introducing cost optimization to assign each learned component to the source component. We run all methods for 100 times, with the average recorded in Fig. 2a. The superiority of LaCIM over pool-LaCIM, together with the fact that LaICM with m = performs better than = 3, verify the benefit of more domains satisfy the diversity condition. To illustrate the learning effect, we visualize the learned (with S left in supplement E. in Fig. 2b.
multiple diverse domains
Method
MCC for
MCC
Pool-LaCTM
0.41
0,71
LaCIM (Ours, m = 3) 0.63
0.84
LaCIM (Ours, m 5) 0,71
0.84
pool-LaCIM
(ii) LaCIM
(iii) pa- (z|D)
(a) Mean Correlation Coefficient for S and Z. (b) Estimated posteriors (i),(i) and the ground-trun ((ii)).
Figure 2: Identification of Latent Variables. As shown qualitatively (b) and quantitatively (a), the
LaCIM can identify the latent variables (up to permutation and point-wise transformation).
5.2 Real-world Data
We verify the generalization ability of LaCIM on three data: NICO, CMNIST and ADNI.
Dataset. We describe the datasets follows (X, Y denotes the input and output: D is unobserved):
NICO. We consider the cat/dog classification in "Animal" dataset in NICO, a benchmark for non-i.i.d problem in [20]. Each animal is associated with "grass", "snoww contexts. The "rras""""noow" D denotes the attributes of the sampler. The C' denotes the time and weather of sampling, which generates the S, Z that respectively denote the semantic and contextual features. We split the dataset into m training domains and the test domain, in which each domain has different proportions of contexts associated with each animal, i.e., (%cat in grass, Keat in snow, %dog in grass, %ddg in snow), due to different sampling strategies determined by D. The proportion vectors of all domains are introduced in Tab. The distributional shift refers to the spurious correlation between the context and the label.
CMNIST: We relabel the digits 0-4 and 5-9 as 0 and y 1. based on MNIST. Then we color pe (1 p²) of images with y 0 (y 1) as green and others as red. We set = 2 with p+ = 0.95, p²a 0.99; while the p for the test domain is set to 0.1. The denotes the attributes of the painter. The Z, S respectively represent the features related to the color and the digit. Their confounder C' denotes the time and weather for which the painter D draws the number and color, e.g., the painter tends to draw red more often than green I in the sunny morning. In this regard, the distributional shift refers to the spurious correlation between the color and the label.
ADNI. The y == (0, 1, 2). with 0,1,2 respectively denoting Normal Control, Mild Cognitive Impairment and AD. The X is structural Magnetic resonance imaging. We split the data into m = 2 training domains and the test domain, with different values of /) that denotes Age. TAU (a biomarker [24]). The C. S (Z) respectively denote the hormone level that
8

---

affects the brain structure development and the dissase-related (-unrelated) brain regions. The distributional shifts among all domains are due to different values of D.
Compared Baselines & Implementation Details. We compare with (i) Empirical Risk Mnimization from X + Y (ERM), (ii) cammain-adversariaa neural network (DANN) [15], (iii) Maximum Mean Discrepancy with Adversarial auto-Encoder (MMD-AAE) [43], (iv) Domain Invariant Variational Autoencoders (DIVA) [29], (v) Invariant Risk Mnimization (IRM) [1], (vi) Supervised VAE (sVAE);
our LaCIM implemented by VAE without disentangling S, Z. For all methods, the network structures of q. (s, 2|z), Po [a| =) and po (1) 8) for CMNIST, NICO and ADNI are shared (details introduced in supplement E.4, E.5, E.6, Tab. 7, 8). We implement SGD as optimizer. with learning rate (Ir) 0.5 and weight decay (wd) Le-5 for CMNIST; Ir 0.01 with decaying 0.2x every 60 epochs, wd 5e-5 for NICO and ADNI (wd is 2e-4). The batch-size are set to 256, 30 and for CMNIST, NICO, ADNI.
Table 1: Accuracy (%) on test domain. Average over 10 runs.
Dataser
NICO
CMNIST
ADNI (m 2)
14
m 2
D: Age
D: TAU
Params
Method
ACC
$ Params
ACC
Purnas
ACC
Params
ACC
ACC
FRM
60.3 + 2.8
18.08M
50.3 + 2.1
1808M
91.9 = 0.9
1.1MM
62.1 # 3.2
64.3 # 1.0
26.27M
DANN
58.9 + 1.7
19.13M
60.1
2.2 2.6
26.40M
84.8 0.7
1.MM
61.0
11.0:10 1.5
65.2
1.1
50.21M
MMD-AAE
60.8 + 3.4
19.70M
154.8 +: 7.7
19.00M
92.5 4: 0.8
1.23
G0.3
+ 2.2
60.2
:: 1.6
3068M
DIVA
58.8 3.4
14.66M
58.1 +
14.7M
86.1
100
61.8 11.8:11
64.8
1:4.8:000 0.8
33.22M
61.4 3.8
18.08M
62.8
18,0MM
92.9
1.12M
62.2
65.2
28.27M
60.4
18.25M
154.3
19.00M
93.66
092M
62.7
06.6
37,70M
LaCIM (Ours)
63.2 + 1.7
18.25M
66.4 + 2.2
19.70M
96.6 +: 0.3
092M
63.8 4:
1.1
67.8 :: 0.9
57.78M
Main Results & Discussions. We report accuracy over 10 runs for each method. As shown in Tab. 1, our LaCIM consistently outperforms others on all data. Specifically, the advantage over IRM and ERM may due to the incorporation of causal assumptions embedded in Fig. I (c). Further, the improvement over sVAE is benefited from the separation of S from others to avoid spurious correlation. Besides, a larger m (with the total sample size fixed) can bring further benefit on NICO, which may due to the easier satisfaction of the diversity condition in theorem 4.4.
Interpretability. We visualize the learned and on CMNIST and NICO. Specifically, for CMNIST, visualize the generated image (with only digit "0" among all classes that belong to Y = 0 and digit "7" among all classes that belong to Y = 1) by interpolating S (and Z) with fixed 2 (and S); for NICO, adopt the gradient method [67], which visualizes the derivatives of the S* (i.e., dimension of S that has the highest correlation with Y) with respect to each image.
As shown in Fig. 3a, the generated sequential images in the 1st and 2nd row look more like "7" from "0" as 8 increases; while the sequential images in the 2nd-row change from red to green as increases. Besides, different dimensions of S can learn different differentiating semantic information. For example, the first dimension can learn to add the dash in the hand-writing "7"; while the second dimension can learn to remove the left part of "0" to "7" as interpolated. For the dimension of Z, it learned other non-differentiating factors such as width, color. This result reflects that the learned S and Z correspond to the digit (causal factor of YY) and color-related features. For NICO, Fig. 3b shows the ability of identifying more explainable semantic features of LaCIM than ERM in which the learned features can mix the background information. Supplement E.5 provides more results.
6 Conclusions & Discussions
We propose recovering latent causal factor that is robust to distributional shifts caused by domain variable. We introduce the causal and non-causal latent factors that are spuriously correlated with each other, and generate the input and the output via invariant mechanisms. Under this invariance, the causal factor is guaranteed to be disentangled from the non-causal one, which induces the ground- truth predictor that holds on all domains. A reformulated generative model is proposed for inferring the causal factor and prediction. A possible drawback of our model lies in our requirement of the number of environments for identifiability, the relaxation of which is left in future work.

---

0
0
0
2
9
9
7
7
Generation with interpolated S with Z fixed
Cat on grass
Cat snow
0
0
Dog on grass
Dog on snow
Generation with interpolated Z with S fixed
(b) Visualization on NICO
(a) Visualization on CMNIST
Figure 3: (a) Visualization CMNIST via interpolation. (b) Visualization NICO via gradient
[67]. From the left to right: original image, ERM and LaCIM.
Broader Impact
We claim that this work does not present any foreseeable negative social impact.
References
[1] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 190.: 02893,, 2019.
[2] O Barndorff-Nielen. Identifiability of mixtures of exponential families. Journal of Mathematical Analysis and Applications, 12(1): 12(1):11--12,, 1965.
[3] Andrew Barron and Chyong-Hwa Sheu. Approximation of density functions by sequences of exponential families. The Annals of Statistics, pages 1347-1369, 1991.
[4] Alexis Bellot and Mihaela van der Schaar. Generalization and invariances in the presence of unobserved confounding, arXiv preprint mrXiv:2007.10533 2020.
[5] Shai Ben-David, John Blitzer. Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Advances in neural information processing systems, pages 137-144, 2007.
[6] Aharon Ben-Tal Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28, Princeton University Press, 2009.
[7] Yoshua Bengio. The consciousness prior. arXiv preprint 1701.:1709 2017.
[8] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1788)1898-11888, 2013.
Irving Biederman. accognition-by-componn review, 94(2):1 1987.
theory human image understanding. Psychological
[10] Peter Bühlmann. Invariance, causality and robustness. arXiv preprint arXiv: 2018.
[11] Krzysztof Chalupk,, Pietro Perona, and Frederick Eberhardt. Visual causal feature learning. arXiv preprint arXiv: 142.3394, 2014.
[12] Mike Davies. Identifiability issues in noisy ica. IEEE Signal processing letters, 1155:470-473, 2004.
[13] Christian Döbler et al. Stein's 's method of exchangeable pairs for the beta distribution and generalizations. Electronic Journal of Probahility, 20, 2015.
[14] Jan Eriksson and Visa Koivunen. Identifiability and separability of linear ica models revisited. In Proc. of ICA, volume 2003, pages 23-27, 2003.
[15] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette. Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17:1-35, 2016.
[16] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. A neural algorithm of artistic style. Journal of Vision, 2015.
10

---

[17] Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International Conference on Machine Learning, pages 2839-2848. 2016.
[18] Patrick J Grother. Nist special database 19 handprinted forms and characters database. National Institute of Standards and Technology. 1995.
[19] Rita Guerreiro and Jose Bras. The age factor in Alzheimer's disease. Genome medicine, 7(1):106, 2015.
[20] Yue He, Zheyan Shen, and Peng Cui. Towards non-iid image classification: A dataset and baselines. Pattern Recognition, 110:107383, 2020.
[21] Christina Heinze-Deml and Nicolai Meinshausen, Conditional variance penalties and domain shift robustness. Machine Leamiing pages 1-46, 2020.
[22] Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances neural information processing systems, 21:689-6964 (9696696, 2008.
[23] Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex J Smola. Correcting sample selection bias by unlabeled data. Advances Neural Information Processing Systems, pages 601-608, 2007.
[24] Christian Humpel and Tanja Hochstrasser. Cerebrospinal fluid and blood biomarkers in Alzheimer's disease. World journal of psychiatry, 1(1):8, 2011.
[25] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, pages 3765-3773, 2016.
[26] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 12(3)=429-439, 1999.
[27] Aapo Hyvärinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligncee and Stutistics, pages 859-868, 2019.
[28] Maximilian Tlse, Jakub M Tomezak, and Patrick Forré. Designing data augmentation for simulating interventions. arXiv preprint arXi::2005.0/855 2020.
[29] Maximilian llse. Jakub M Tomezak, Christos Louizos, and Max Welling. DIVA: Domain invariant variational autoencoders. arXiv preprint arXiv: 10427, 2019.
[30] D Janzing, Peters, JM Mooij, and B Schölkopf. Identifying confounders using additive noise models. In Proceedings of the 25th Conference Uncertainty in Artificial Intelligence (UAI 2009), pages 249-257 AUAI Press, 2009.
[31] Dominik Janzing, Jonas Peters, Joris Mooij, and Bernhard Schölkopf. Identifying confounders using additive noise models. arXiv preprint /205::2005.26400 2012.
[32] Dominik Janzing, Eleni Sgouritsa, Oliver Stegle, Jonas Peters, and Bernhard Schölkopf. Detecting low-complexity unobserved causes. arXiv preprint 120.:3700..3737, 2012.
[33] Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain-invariant reprosentations. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 527-536, 2019.
[34] Guoliang Kang, Xuanyi Dong, Liang Zheng, and Yang. Patchshuffle regularization. arXiv preprint arXiv:170.007003. 2017.
[35] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-220120 227 PMLR, 2020.
[36] Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based based deep models based on nonlincar ica. Advances in Neural Information Processing Systems, 33, 2020.
[37] Diederik P Kingma and Max Welling, Auto- eito-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR 2014), Banff, Canada, 2014. ICLR Committee.
11

---

[38] Mark A Kramer. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal, 37(2):233-243, 1991.
[39] David Krueger, Ethan Cahallero, Jbern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville. Out-of-distributie generalization via risk extrapolation (rex). arXiv preprint arXiv:2003. 006888 2020.
[40] Kun Kuang, Peng Cui, Susan Athey, Ruoxuan Xiong, and Bo Li. Stable prediction across unknown environments. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages (617-1626, 2018.
[41] Ciarán M Lee, Christopher Hart. Jonathan Richens, and Saurabh Johri. Leveraging directed causal discovery to detect latent common causes. arXiv preprint arXiv:1910.10174, 2019.
[42] Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[43] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018
[44] Chris J Maddison, Andriy Maih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint
2016.
[45] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 700.:00063,,,, 2017.
[46] Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In Advances in Neural Information Processing Systems, pages 10846-10856, 2018
[47] Diego Marcos, Michele Volpi, and Devis Tuia. Learning rotation invariant convolutional filters for texture classification. In 2016 23rd International Conference Patiern Recognition (ICPR), pages 2012-2017. IEEE, 2016.
[48] James A Mortimer. Brain reserve and the clinical expression of Alzheimer's disease. Geriatrics (Basel, Switzerland), 52:S50-3, 1997.
[49] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pages 10-18. 2013.
[50] Sinno Jialin Pan, Ivor W. Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEFE Transactions on Neural Networks, 22(2):199-21) 2010.
[51] Judea Pearl. Causality. Cambridge university press, 2009.
[52] Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 7845)1947-1012, 2016.
[53] Jonas Peters, Dominik Janzing, and Berhaard Schölkopf. Elements of causal inference: foundations and learning algorithms.. MIT press, 2017.
[54] Jonas Peters, Joris Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15(1):2009225) -0009, 2014.
[55] Mateo Rojas-Carss Carulla, Bernhard Schölkopf. Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309-1333 1342, 2018.
[56] Jan Jan-Willem Romeijn and Jon Williamson. Intervention and identifiability latent variable modelling. Minds and machines, 28(2):243- 224-244,, 2018.
[57] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias NieBner. Faceforensics*#: Learning to detect manipulated facial images. In Proceedings of the IEEE International Conference on Computer Vision, pages 1-11, 2019.
[58] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint 11Xiv.1911.08731. 2019.
12

---

[59] Bernhard Schölkopf. Causality for machine learning, arXiv preprint arXiv:1911. 2019.
[60] Bernhard Schölkopf. Dominik Janzing, Jonas Peters, and Kun Zhang. Robust learning via cause-effect models, arXiv preprint arXiv: 2011.
[61] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on mnist. arXiv preprint arXiv:1805.091900 2018.
[62] Eleni Sgouritsa, Dominik Janzing, Jonas Peters, and Bernhard Schölkopf. Identifying finite mixtures of nonparametric product distributions and causal inference of confounders. In Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence (UAI 20/3), pages 556-575. AUAI Press, 2013.
[63] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. In Proceedings of the International Conference on Learning Representations (ICLR 2018), 2018.
[64] Shohei Shimizu, Patrik O Hoyer, and Aapo Hyvärinen. Estimation of lincar non- ron--aussiin acyclic models for latent factors. Neurocomputing, 72(7-9):2024 2027, 2009.
[65] Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, 2019.
[66] Ricardo Silva, Richard Scheine, Clark Glymour, and Peter Spirtes. Learning the structure of linear latent variable models. Journal of Machine Learning Research, 7(Feb): 191-246, 2006.
[67] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint 1312v:1312.6034, 2013.
[68] Adarsh Subbaswamy and Suchi Saria. I-spec: An end-to-end framework for learning transportable, shift-stable models. arXiv preprint arXiv:2002. 08948. 2020.
[69] Adarsh Subbaswamy, Peter Schulam, and Suchi Saria. Preventing failures due to datasct shift: Learning predictive models that transport. In The 22nd International Conference on Artificial Intelligence and Stutistics, pages 31 18-3127. PMLR, 2019.
[70] Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60(4):699-746, 2008.
[71] Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, pages 6056-6065, 2019.
[72] M. Tan and Quoe V. L.e. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint, 1905. / 1946, 2019.
[73] Luke Taylor and Gcoff Nitschke. Improving deep learning using generic data augmentation. arXiv preprint 1oXiv:770,,0600 2017
[74] Damien Teney, Ehsan Abbasnejad, and Anton van den Hengel. Unshuffling data for improved generaliza- tion. arXiv preprint arXiv:2002. 11894, 2020.
[75] Takeshi Teshima, Issci Sato, and Masashi Sugiyama. Few-shot domain adaptation by causal mechanism transfer. arXiv preprint arXiv:2002.03499, 2020.
[76] Jose Vina and Ana Lloret. Why women have more Alzheimer's 's discase than men: gender and mitochondrial toxicity of amyloid-3 peptide. Journal of Alzheimer's 's disease, 20(s2):S527-S6) S533,, 2010.
[77] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhamhetov, and Gabriel Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition. pages 5028-5037 2017.
[78] Chuanlong Xie, Fei Chen, Yue Liu, and Zhenguo Li. Risk variance penalization: From distributional robustness to causality. arXiv preprint 20ti:2000.075444 2020.
[79] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 13..0016.1.03530 2016.
13

---

[80] Kun Zhang, Biwei Huang, Jiji Zhang, Clark Glymour, and Bernhard Schölkopf. Causal discovery from nonstaionary/heeeroggeeouu data: Skeleton estimation and orientation determination. In IJCAI: Proceedings of Conference, volume 2017. page 1347. NIH Public Access, 2017.
[81] Kun Zhang, Bernhard Schölkopf. Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. International Conference on Machine Learning, pages $19-827, 2013.
[82] Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant represen- tation for domain adaptation. arXiv preprint arXiv: 094.09953, 2019.
14


Reviews:
[{"review_id": "RuN_fDcmoiR", "reviewer": "Reviewer_1QgY", "summary": "This paper introduces the \"Latent Causal Invariance Model\" (LaCIM), a structural variational autoencoder-like approach for learning to isolate causal factors from \"spurious\" ones in prediction to improve generalization to new environments. The authors assume a latent generative structure for the data, with separate latent constructs for causal and contextual factors. Under strong parametric assumptions and requirements on the number of source environments the authors prove identifiability of the causal factors. Experiments on imaging datasets (2 benchmarks and a small medical dataset) show LaCIM outperforms some existing domain adaptation baselines as well as Invariant Risk Minimization (IRM). Further, qualitative assessment of images generated by ablating the causal and contextual factors show disentanglement of these factors.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To the best of my knowledge, the proposed LaCIM approach is a novel and interesting approach for avoiding \"spurious correlations\" to improve model generalizability to new contexts. The theoretical results are quite daunting to parse, in part because the methodological ideas build off of both literature on additive noise models (ANM) in causal discovery and independent component analysis (ICA). The authors define a two step process for learning: first learning the generative model (effectively a VAE), then computing the latent representation that maximizes the likelihood for each example and fitting a classifier on top of this. The approach is reasonable, and the experiments show its potential.\n\nI did have a number of minor concerns. The assumed causal structure makes some questionable decisions, there are untestable assumptions regarding the quality of the input data, and the experiments are designed to satisfy the assumptions of LaCIM (meaning comparisons to other methods are likely over optimistic). \n\nStrengths:\n- The approach is novel, synthesizing ideas from ANM and ICA to new effect\n- The authors prove identifiability of their model. These results are hidden behind rather dense theory, though, so I was unable to check these results.\n- Promising experimental evaluation: both quantitative comparisons to other approaches as well as qualitative analysis of disentanglement of factors.\n\nWeaknesses:\n- Under the causal model assumed by LaCIM, X and Y are generated concurrently from the contextual factors Z and causal factors S. The authors note this as a strength or unexplored direction in Remark 1 (Ln 148), but for many examples it doesn't seem to make as much sense as assuming X generates or is generated by Y. For example, in Remark 1 the authors talk about disease status (Y) and ultrasound tests (X), but surely whether or not a patient has the disease will effect the results observed in the ultrasound! In most classification tasks, even in imaging, the label is the concept of interest while the observed data (i.e., the image) is a realization of that label. In the qualitative analysis (Line 349), fixing S kept the class label fixed (which the authors state as evidence that LaCIM has learned the \"causal\" factors), but is it not inefficient to have this redundancy in the role of Y and S? Further, what value is there in having D and C be separate? Isn't it sufficient to just have D in Fig 1c? No additional conditional independence value is added.\n- The identifiablity results in Section 4.2 are very interesting, but are at times quite hard to follow. This is in part because of a general lack of clarity in this section (but in general the paper would benefit from one pass to improve writing). The authors justify many of the assumptions in Section 4.2 by pointing to other works that make these assumptions, but readers not familiar with these other works will not understand the significance or consequences of these assumptions. I think the authors assume readers will be familiar with, e.g., assumptions and techniques from ICA. \n- The authors discuss the diversity of environments. This is very important and I'm glad to see this! Condition i) (on Line 234) is untestable, right? It depends on R, the cardinality of the latent confounding construct... Does this mean, in general, a user will have no idea if they have enough data or high enough quality data to ensure the correctness of LaCIM?\n- In the experiments, the authors explicitly set up different environments to try to follow the generative structure of LaCIM. But this structure does not match up with, e.g., the way the original colored MNIST dataset was generated and thus favors LaCIM. Could LaCIM be applied to the original MNIST dataset? Does IRM (which was the paper which developed the original colored MNIST datasets) outperform LaCIM then? This would speak to sensitivity of results to the causal structural assumptions which would be important to know.\n- In the simulated experiment, I think it would be more relevant to examine how the number of domains and the diversity of the data affect the identifiability. Currently, the simulated experiment looks at 3 vs 5 environments, but according to diversity condition i), isn't the relevant factor how the number of environments relates to the cardinality of the latent factor C?\n- Minor question: Regarding the learning procedure, won't there be a posterior over S and Z? Couldn't predictions be made by marginalizing over this posterior, rather than optimizing to compute an estimate of s*,z* and then predict $P(Y | s^*)$?\n\n\n\n### UPDATES:\nThanks to the authors for their response. I think the greatest improvements to the paper can be made by increasing the clarity in Section 4. I think elements from the discussion of the identifiability and diversity conditions in the authors' response to my comments should be included in the main paper. I believe this will really help readers better understand the method. \n\nI would like to make one comment regrading the experimental methodology: In their response, the authors state that \"our method can outperform all compared methods on other benchmarks that satisfy our causal structure.\" It should be noted that this is expected in order to demonstrate the empirical soundness and validity of the method. For a comprehensive evaluation of the utility of the method, including its possible failure cases, it should also be applied in realistic settings which *may not satisfy the untestable assumptions of the method*. The authors assume a *latent* causal structure, and, in all experiments in the main paper, modify the datasets to satisfy this assumption. But in a real world practical application, a user will not know the latent causal structure. The authors pointed to experiments on the original colored MNIST dataset in the appendix. However, the authors modified the proposed LaCIM method (i.e., used LaCIM-REx) to fit the generative structure of the original colored MNIST dataset. In my view, this defeats the purpose of this experiment, because what we want to see is how the unmodified LaCIM method behaves on the original colored MNIST dataset. For example, from a user perspective, there is no way to differentiate the data generating process (DGP) of the original colored MNIST dataset from the DGP of the modified colored MNIST dataset used by the authors. Thus, in a real use case, the user would be unable to modify the LaCIM methodology as the authors did in the supplemental experiment. Put another way, users should be able to apply the method to problems as they appear (consider, e.g., the [WILDS](https://wilds.stanford.edu/) distribution shift benchmark tasks), or be able to determine that the method is not applicable to their particular problem/application.\n\nThus, while I remain positive about the work, I think the authors should more clearly investigate and discuss possible limitations of the method for this paper to be a strong and comprehensive . For this reason, I am maintaining my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors could put some more thought into possible impacts of applying LaCIM to more realistic imaging tasks. For example, in facial recognition tasks related to, e.g., law enforcement the distinction between \"causal\" vs \"contextual\" factors could have ethical implications or could imply limitations on how LaCIM models are interpreted.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "R02wDijPBxR", "reviewer": "Reviewer_wuYX", "summary": "In this paper, the authors propose a method to estimate the invariant causal mechanism. By modeling the process as the causal graph in Fig.1 (c), the authors first provide the condition when disentangling causal factors is feasible. Based on the theoretical result, they present the method to infer causal factor and non-causal factor with X based on VAE.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors address an important problem in this paper. The method is with solid theoretical guarantee. The experiments validate the effectiveness of the proposed method. I like this paper. Hence I give a positive score. I possibly change my score according to the comments of other reviewers.\n\nI have two suggestions:\n\n1. The writing could be improved. The theoretical part is quite hard to follow. There are many notations without clear illustration, e.g. k_t,q_t, \\theta. Does k_t imply the sample number and q_t imply the domain number? \n\n2. I suggest one additional citation, where the authors propose a method that also takes latent causal factors into account when there are distributional shifts. \n\nIJCAI 2017 Causal discovery from nonstationary/heterogeneous data: Skeleton estimation and orientation determination.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DeBiq63FAjU", "reviewer": "Reviewer_ss3N", "summary": "This work proposes an identifiable causal model with latent variables to model spurious correlations between labels and irrelevant context (such as background or light setting) in order to construct a classifier which is robust to certain distribution shifts. The authors show under which conditions the irrelevant context and important features can be disentangled from only input observations and labels. They propose a VAE-like approach to learn the model which they use to come up with an invariant classifier. The identifiability of the model is assessed on synthetic data and the robustness to dataset shift is evaluated on multiple real world data sets, drawing a favorable picture of their contribution. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nI have been following the literature on nonlinear ICA (from which the identifiability result is inspired) and it is the first time I see these ideas directly applied to out-of-distribution generalization. As far as I know, all important related works have been cited.\n\nQuality:\n\nI believe this paper presents interesting novel ideas with convincing experiments and I enjoyed reading it. However, some algorithmic decisions are not justified and the paper lacks clarity. The focus on identifiability and why it matters for dataset shift generalization is not well explained. I make some of these concerns more specific in what follows, with no specific order: \n\nSome assumptions appear to be implicit. For instance, the fact that q_s + q_z <= q_x. Otherwise, f_x cannot be bijective. Moreover, saying that f_x and f_y are bijective without specifying the domain and codomain of the functions is imprecise. For instance, if q_s + q_z < dim(x) and the codomain is R^dim(x), then f_x cannot be bijective. So what is the codomain?\n\nL172: The two points are said to “...ensure the robustness of such a two-step invariant prediction, ...”. However, the aforementioned two-step procedure is only very vaguely defined up to this point in the paper, thus it was hard for me to accept that we actually want these desiderata. Maybe presenting the procedure before would help? \n\nThat being said, even after understanding the procedure, I felt like I had to think a lot to understand why these two points (at line 172) are important to have (and I am still not sure). Could you provide a clear explanation of what would be the consequence of any of these desiderata not holding?\n\nDefinition 4.3:\nDesiderata 1 (line 172) does not require M_s and M_z to be permutation-scaling matrices, right? In principle, only having “block” identifiability would be enough, no? Do you believe one could come up with weaker assumptions? If not, please mention that having permutation matrices is “extra” and not really required. \n\nIs it possible that $[f\\_x^{-1}]\\_\\mathcal{S}$ maps some points of $\\mathcal{X}$ outside $\\mathcal{S}$? Similarly for $\\mathcal{Z}$? Since the definition supposes only $\\mathcal{X} \\supset f\\_x(\\mathcal{S}\\times\\mathcal{Z})$, it is possible.\n\nThis definition assumes implicitly that $f\\_x$ is invertible but this is not mentioned until the following theorem. This assumption should be mentionned earlier.\n\nTheorem 4.5 introduces a notion of “asymptotic identifiability”, however, I am not sure I understand its interpretation. It would be helpful to give some intuition for its meaning. That would help me understand its value. Also, I suspect the dimensionality of the sufficient statistic is allowed to grow, however, the definition of $\\mathcal{P}\\_\\text{exp}$ does not allow for this. Moreover, in order for the assumption of Theorem 4.4 to hold, the number of environments $m$ needs to grow as well, right? This should be clearer. I believe an interesting consequence of this Theorem is that even if the assumptions of Theorem 4.4 are not satisfied for some $p \\in \\mathcal{P}_\\text{exp}$, it can still be approximated by an identifiable model. Is that correct (assuming the exponential model is in the appropriate Sobolev space)? \n\nSection 4.3.2 presents the invariant prediction procedure which, as far as I know, is not standard (which is not necessarily a bad thing). I had a hard time interpreting the meaning of it and understanding why it makes sense. I believe the author should add a few lines motivating this approach. One possible interpretation for the max in (6) is that it corresponds to a maximum a posteriori (MAP) estimation of z and s in a counterfactual model in which p(s,z|c) is replaced by p(s)p(z) (thus without correlation). But even with this interpretation, I am not completely satisfied. Of course the procedure is invariant to the environment by design, but is it optimal in some sense?\n\nSection 5: I found the experiments sufficient and convincing. Identifiability is confirmed on synthetic experiments and out-of-distribution generalization is measured against multiple relevant baselines on three datasets. Since I do not follow closely the literature on OOD generalization, I cannot confirm that no important baseline is missing. \n\nI would like to see more than one latent interpolation for the dataset CMNIST. Do they all look as good as the one presented here?\n\nClarity:\n\nL96: The definition of [f]_A is imprecise, usually the word “restricted” refers to the domain being restricted. At that point I did not understand it and even after reading Definition 4.3, I had to infer it from context. Please clarify.\n\nL139: This sentence is contradictory: “In our scenario, we do not require D to be observed; rather, we only need the domain index d_e...”. This means D is observed, no?\n\nEquation (1): I believe some notation is unnecessarily hard to follow. For instance, the $|c$ symbol should be removed from the two left factors in the first line of (1). Also, having a superscript t on T and \\Gamma was a bit confusing. It led me to think that these things could depend on the actual *value* of z or s. Now I understand that this is just to allow different exponential families for Z and S. A solution would be to remove this superscript and replace it with a footnote mentioning that everything works even if we have different sufficient statistics for Z and S.\n \nSome notation is implicitly defined. For instance q_s and q_z being the dimensionality of S and Z, respectively. It would help the reader to mention it explicitly in the text. Similarly for k_s and k_q, which are the dimensionality of the sufficient statistic of each s_i and z_i, respectively. What is the dimensionality of x? Again, implicitly defined on L228.\n\nTheorem 4.4: The fifth assumption was hard to parse, even if I have some experience  with these kinds of assumptions.\n\nL237: “... can hold unless the space of \\Gamma belong to a zero-(Lebesgue) measure set.” This statement is not clear. It seems \\Gamma can always take only finitely many values and thus is contained in a measure zero set, even if the assumptions are satisfied. Am I misunderstanding something?\n\nThe writing of Section 4.3.1 should be completely revised. I had a very hard time actually understanding many steps of the derivations and even just what the actual inference model (q) versus the generative model (p) are. Some conditional density functions are introduced without definitions which might explain why I could not follow. Also, is there a superscript e missing on q_\\psi(y | s) at line 271. \n\nIn Section 5: Please mention explicitly which representations are used to compute the MCC. I suppose they are the ground-truth (z,s) and the one obtained by solving (6)?\n\nSignificance:\n\nI believe using ideas from nonlinear ICA to come up with identifiable latent models for out-of-distribution generalization is an interesting direction and could inspire more work. \n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "CQwwOM-fcmn", "reviewer": "Reviewer_nVGX", "summary": "To deal with the issue of distributional shifts between training and testing domains, the authors propose Latent Causal Invariance Models (LaCIM) consisting of both causal latent factors and non-causal latent factors and the extent of the correlations between them is governed by a domain variable. They theoretically show the identifiability of the causal latent factors and the ground-truth predicting mechanism in the proposed LaCIM. Based on the identifiability, they learn the model by reformulating VAE and then verify it on various real-world data. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I like the idea and technically the paper makes sense. Here are some of my concerns.\n\nMy main concern is that how practical the proposed LaCIM is. If I understand correctly, from Definition 4.1 we know that for any $e$, we have $S⫫Z | C$, that is, $S⫫Z | C, D$. Also, from the assumption over the prior $p_{\\boldsymbol{T}, \\boldsymbol{\\Gamma}}$, we see that $S_i ⫫ S_j | C, D$ for any $i \\neq j$ and $Z_i ⫫ Z_j | C, D$ for any $i \\neq j$. These conditional independence (given $C$ and $D$) assumptions play a key role in proving the identifiability. Conversely, if the latent variables $S$ and $Z$ do not satisfy these conditional independence assumptions, the identifiability would not hold true and the proposed method would fail to identify the causal factors. Am I right? Actually, in many real-world scenarios, these assumptions do not hold. For example, when some part of Z is affected by $Y$ [1-5], or $Z$ is directly affected by $S$ [6], etc. In fact, [3,7] provided some more general approaches to covering the dependent cases for identifiability, which might be helpful to relax the assumptions in this paper. \n\nAnother concern is that it seems in the paper that the authors did not explicitly provide the out-of-distribution generalization guarantees, i.e., some theoretical guarantees on that the learned invariant predictor can be generalized from the training environments in $\\mathcal{E}_{train}$ to all the environments in $\\mathcal{E}$. Proposition 4.2 is more of a definition than a proof. It might be better if the authors could provide more details about it.\n\nThe notations in Section 4.2 seem a bit messy. For example, what is $q_t$? what is $k_t$? etc. All these appear for the first time and should be explained better. In the last sentence of the caption of Figure 1, is it right that \"the read and blue respectively means the invariant and varied values/distributions\"? or the opposite?\n\n\n**References** \\\n[1] Invariant Risk Minimization. Arjovsky et al. 2019 \\\n[2] Invariant Risk Minimization Games. Ahuja et al. 2020 \\\n[3] Nonlinear Invariant Risk Minimization: A Causal Approach. Lu et al. 2021 \\\n[4] Elements of Causal Inference: Foundations and Learning Algorithms. Peters et al., 2017. \\\n[5] Domain Adaptation under Target and Conditional Shift. Zhang et al., 2013. \\\n[6] Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style. Kügelgen et al., 2021 \\\n[7] ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA. Khemakhem et al., 2020", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not include any societal impact in the paper. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "two hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the problem that distribution shift may degrade the prediction accuracy of the learned model. It proposes a way to distinguish between the underlying causal factors and other factors, together with a strategy to deal with distribution shift by using only causal factors for prediction. The studied problem is interesting, and reviewers agree that both theoretical studies and empirical results are convincing.
Decision: Accept (Poster)