Learning Causal Semantic Representation for Out-of-Distribution Prediction
Authors: Chang Liu
Xinwei Sun
Jindong Wang
Tao Li
Tao Qin
Wei Chen
Tie-Yan Liu
Venue:  (2021)
Keywords: out-of-distribution
causality
latent variable model
generative model
variational auto-encoder
domain adaptation

Abstract: Conventional supervised learning methods, especially deep ones, are found to be sensitive to out-of-distribution (OOD) examples, largely because the learned representation mixes the semantic factor with the variation factor due to their domain-specific correlation, while only the semantic factor causes the output. To address the problem, we propose a Causal Semantic Generative model (CSG) based on causality to model the two factors separately, and learn it on a single training domain for prediction without (OOD generalization) or with unsupervised data (domain adaptation) in a test domain. We prove that CSG identifies the semantic factor on the training domain, and the invariance principle of causality subsequently guarantees the boundedness of OOD generalization error and the success of adaptation. We also design novel and delicate learning methods for both effective learning and easy prediction, following the first principle of variational Bayes and the graphical structure of CSG. Empirical study demonstrates the effect of our methods to improve test accuracy for OOD generalization and domain adaptation.

Under review as a conference paper at ICLR 2021
LEARNING CAUSAL SEMANTIC REPRESENTATION FOR OUT-OF-DISTRIBUTION PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Conventional supervised learning methods, especially deep ones, are found to be sensitive to dfitr--diittibutioon (OOD) examples, largely because the learned rep- resentation mixes the semantic factor with the variation factor due to their domain- specific correlation, while only the semantic factor causes the output. To address the problem, we propose Causal Semantic Generative model (CSG) based on causality to model the two factors separately, and learn on a single training domain for prediction without (OOD generalization) or with unsupervised data (domain adaptation) in a test domain. We prove that CSG identifies the semantic factor the training domain, and the invariance principle of causality subse- quently guarantees the boundedness of OOD generalization error and the success of adaptation. We also design novel and delicate learning methods for both effec- tive learning and easy prediction. following the first principle of variational Bayes and the graphical structure of CSG. Empirical study demonstrates the effect of our methods to improve test accuracy for OOD generalization and domain adaptation.
INTRODUCTION
Deep learning has initiated a new era of artificial intelligence where the potential of machine learn- ing models greatly unleashed. Despite the great success, these methods heavily rely on the poppodently-nddiddeiiieee -istiiblly (IID) assumption. This does not always perfectly hold in practice, and the prediction of output (label, response, outcome) y may be saliently affected in out-of-distribution (OOD) cases, even from an essentially irrelevant change to the input (covariate) a, like a position shift or rotation of the object an image, or a change of background, illumination or style (Shen et al. 2018 He et al. 2019: Arjovsky et al.] (019).. These phenomena pose serious concerns on the robustness and trustworthiness of machine learning methods and severely impede them from risk-sensitive scenarios.
Looking into the problem, although deep learning models allow extracting abstract representation for prediction with their powerful approximation capacity, the representation may be overconfident in the correlation between semantic factors s (e.g., shape of an object) and variation factors v (e.g., background, illumination, object position). The correlation may be domain-specific and spurious, and may change drastically in a new environment. So it has become a desire to learn representation that separates semantics s from variations (Cai al. 2019 [sse et al. 2019). Formally, the impor- tance of this goal is that s represents the cause of Causal relations better reflect the fundamental mechanisms of nature, bringing the merit to machine learning that they tend to be universal and in- variant across domains (Schölkopf et al., the most transferable and confident information to unseen domains. Causality has also been shown to lead to proper domain adaptation (Schölkopf et al. 2012, Zhang et al., 2013), lower adaptation cost and lighter catastrophic forgetting (Peters et al. 2016 Bengio et al. 2019 et al. 2019).
et al. [ch 2019). thus providing
In this work, we propose a Causal Semantic Generative model (CSG) for proper and robust OOD prediction, including OOD generalization and domain adaptation. Both tasks have supervised data from single training domain, but domain adaptation has unsupervised test-domain data during learning, while OOD generalization has no test-domain data, including cases where queries come sequentially adaptation is unaffordable. (1) We build the model by cautiously following the prin- ciple of causality, where we explicitly separate the latent variables into a (group of) semantic factor and (group of) variation factor v. We prove that under appropriate conditions CSG identifies the

---

Under review as a conference paper at ICLR 2021
semantic factor by fitting training data, even in presence of an s-v correlation. (2) By leveraging the causal invariance, we prove that a well-learned CSG is guaranteed to have a bounded OOD general- ization error. The bound shows how causal mechanisms affect the error. (3) We develop domain adaptation method using CSG and causal invariance, which suggests to fix the causal generative mechanisms and adapt the prior to the new domain. We prove the identification of the new prior and the benefit of adaptation. (4) To learn and adapt the model from data, we design novel and delicate reformulations of the Evidence Lower BOund (ELBO) objective following the graphical structure of CSG, so that the inference models required therein can also serve for prediction, and model- ing and optimizing inference models in both domains can be avoided. To our best knowledge, our work is the first to identify semantic factor and leverage latent causal invariance for OOD prediction with guarantees. Empirical improvement in OOD performance and adaptation is demonstrated by experiments on multiple tasks including shifted MNIST and ImageCLEF-D4 task.
2 RELATED WORK
There have been works that aim leverage the merit of causality for OOD prediction. For OOD gen- eralization, some works ameliorate discriminative models towards a causal behavior. [Bahadori et al. (017)) introduce a regularizer that reweights input dimensions based on their approximated causal effects to the output, and [Shen et al. (2018) reweight training samples by amortizing causal effects among input dimensions. They are extended to nonlinear cases (Bahadori et al.] 2017 He et al.] 2019) via separable representations. [Heizze- Deml & Meinshausen (2019) enforce inference invariance by minimizing prediction variance within each label-identity group. These methods intro- duce no additional modeling effort, but may also be limited to capture invariant causal mechanisms (they are non- -enegeeerraiee) and may only behave quantitatively causal in the training domain.
For domain poggttation/gneralzzzztiii (Schölkopf et al. 2012; Zhang et al.] (203) or using learned causal relations (Rojas- (Rojas-Carulla, Carulla et al.) 1018|Magliacan et al. 2018). Zhang al. (013)); Gong et al. (016) 2018) also consider certain ways of mechanism shift. The considered causality is among directly observed variables, which may not be suitable for general data like image pixels where causality rather lies between data and conceptual latent factors (Lopez-Paz et al.) 2017 Besserve et al.] 2018 Kilbertus et al.] 2018). To consider latent factors, there are domain adaptation (Pan et 2010 Baktashmotlagh et al. 2013
methods are developed under various causal assumptions
Ganin et al.] 2016 Long et al. 2015: 2018) and generalization methods (Muandet et al. 2013 Shankar et al. (018) that learn representation with domain invariant marginal distribution, and domain-invariin
have achieved remarkable results. Nevertheless, Johansson et al. (2019); Zhao et al. (2019) point out that this invariance is neither sufficient nor necessary to identify the true semantics and lower the adaptation error (Supplement D). Moreover, these methods and invariance risk minimization (Ar-) jovsky et al. 2019) also assume the invariance in the inference direction (i.e., data - representation), which may not be general as causal invariance in the generative direction (Section 3.2)
There are also generative methods for domain adaptationn/enneralizattoon that model latent factors. Cai et al. (011)); Ilse et al.) (2019) introduce semantic factor and domain-feature factor. They assume the two factors are independent in both the generative and inference models, which may not meet reality closely. They also do not adapt the prior for domain shift thus resort to inference invariance. iZhang et af. (2020) consider a partially observed manipulation variable, while assume its independence from the output in both the joint and posterior, and the adaptation is inconsistent with causal invariance. [Atzmon et al. (2020) consider similar latent factors, but use the same (uniform) prior in all domains. These methods also do not show guarantees to identify their latent factors. Teshima et al. (2020) leverage causal invariance and adapt the prior, while also assume latent inde- pendence and do not separate the semantic factor. They require some supervised test-domain data, and their deterministic and invertible mechanism also indicates inference invariance. In addition, most domain generalization methods require multiple training domains, with exceptions (e.g., Qiia et al. 2020) that still seek to augment domains. In contrast, CSG leverages causal invariance, and has guarantee to identify the semantic factor from a single training domain, even with a correlation to the variation factor.
Generative supervised learning is not new (Mcauliffe & Blei, 2008 Kingma et al. 2014). but most works do not consider the encoded causality. Other works consider solving causality tasks, notably causal/treatment effect estimation (Louizos et al., 2017 Yao et al. 2018 Wang & Blei 2019). The task does not focus OOD prediction, and requires labels for both treated and controlled groups
2

---

Under review as a conference paper at ICLR 2021
Disentangling latent representations is also of interest in unsupervised learning. Despite some em- pirical success (Chen et al. 2016; Higgins et al.) 2017 Chen et al. 2018), Locatello et al. (2019) con- clude that it is impossible to guarantee the disentanglemenn in unsupervised settings. Khemakhem et al.] (2019) 2020) show an encouraging result that disentangled representation can be identified up to permutation with cause of the latent variable observed. But the methods cannot separate the semantic factor from variation for supervised learning, and require observing sufficiently many different values of the cause variable, making it hard to leverage labels.
Causality with latent variable has been considered in a rich literature (Verma & Pearl 1991 Spirtes al. 2000; Richardson et al. 2002, Hoyer et al., 2008; Shpitser et al.J 2014). while most works focus on the consequence on observation-leve causality. Others consider identifying the latent vari- able. Danzing et al. 22099; Lee et al. (2019) show the identifiability under additive noise or similar assumptions. For discrete data, "simple" latent variable can be identified under various specifica- tions (Janzing et al. 2011] Sgouritsa et al.) 2013] Kocaoglu et al., 2018). Romeijn & Williamson (2018) leverage interventional datasets. Over these works, we step further to separate and identify the latent variable as semantic and variation factors, and show the benefit for OOD prediction.
3. THE CAUSAL SEMANTIC GENERATIVE MODEL.
To develop the model seriously and soberly based on causality, we require the formal definition of causality: two variables have a causal relation, denoted as "cause-reffeet"" externally interven- ing the cause (by changing variables out of the considered system) may change the effect, but not vice versa (Pearl 2009; Peters al. 2017). We then follow the logic below to build our model.
(1) It may be general case that neither y + x (e.g. adding noise to the labels in a dataset does not change the images) nor + y holds (e.g.. intervening an image by e.g. break- ing camera sensor unit when taking the image, does not change how the photographer labels it), as also argued by Peters et al.] (2017, Section 1.4): Kilbertus et al.] (2018). So we employ a generative model (i.e., not only modeling p(ylz)), and introduce a latent variable z to capture factors with causal relations.
Figure 1: The graphical structure of the proposed Causal Semantic Gener- ative model (CSG) for the semantic s and variation v latent factors and su- pervised data (z. y)- Black solid ar- rows represent invariant causal gen- erating mechanisms p(rr|s, v) and p(y]s), ), the black undirected edge rep- resent domain-specift prior p(s, v), and blue dashed bended arrows repre- sent the inference model q(s, u/z) for learning and prediction.
(2) The latent variable x as underlying generating factors (e.g., object features like shape and texture, background and illumination in imaging) is plausible to cause both x (e.g., the change of object shape or background makes different image, but breaking camera sensor unit does not change the object shape or background) and y (e.g. the photogra- pher would give different label if the object shape, texture, etc. had been replaced by those of a different object, but noise-corupping the label does not change the object features). So we orient the edges in the gener-
ative direction = - (xx, y), as also adopted by Mcauliffe & Blei 2008); Peters et al. (2017); Teshima et al. (2020). This is in contrast to Cai et al. (2019): : Isse et al. (2019 2020): Castro et al. (2020)
who treat y as the cause of semantic factor, which, when y is also a noisy observation, makes unreasonable implications (e.g., adding noise to the labels in dataset automatically changes object features and consequently the images, and changing the object features does not change the label).
This difference is also discussed by Peters et al. (2017, Section 1.4); Kilbertus et al. (2018).
(3) We attribute all r-y relation to the existence of some latent factors ("purely common cause", Lee al. 2019; Janzing et al., 2009), and exclude z- edges. This can be achieved as long holds sufficient information of data (e.g., with shape, background etc. fixed, breaking a sensor unit does not change the label, and noise-corrupting the label does not change the image). Promoting this restriction reduces arbitrariness in explaining z-y relation and benefits the identification of 2. This in contrast to Kingma et al. (2014); Zhang et al. (2020); Castro al. (2000) who treat y as a cause of x since no latent variable is introduced between.
'Supplement provides more explanations on the model.
3

---

Under review as a conference paper at ICLR 2021
(4) Not all latent factors are the causes of y (e.g., changing the shape may alter the label, while changing the background does not). We thus split the latent variable as 2 = (s, v) and remove the edge y. where represents the semantic factor of that causes y, and describes the variation or diversity in generating æ. This formalizes the intuition on the concepts in Introduction.
(5) The variation often has relation to the semantics 8, which is often spurious correlation (e.g., desks prefer a workspace background, but they can also appear in bedrooms and beds can also appear in workspace). So we keep the undirected s-v edge. Although is not cause of y, modeling it explicitly is worth the effort since otherwise it would still be implicitly incorporated in 8 anyway through the s-v correlation. We summarize these conclusions in the following definition.
Definition 3.1 (CSG). A Causal Semantic Generative Model (CSG) p (p(s, v),, p(z p(y)()))) is a generative model on data variables x E X' C RRdx and y E y with semantic & E S C Rds and variation V C Rd latent variables, following the graphical structure shown in Fig. 1
3.1 THE CAUSAL INVARIANCE PRINCIPLE
The
of causal relations translates to the following principle for CSG:
Principle 3.2 (causal invariance). The causal generative mechanisms p(z|s, v) and p(yls) in CSG are invariant across domains, and the change of prior p(s, v) is the only source of domain shift.
It is supported by the invariance of basic laws of nature (Schölkopt al., 2012: Peters al. 2017
Besserve et al. 2018 Bühlmann 2018 Schölkopf 2019). Other works instead introduce domain index (Cai et al. 2019 llse et al. 2019; 2020 Castro al., 2020) or manipulation variables (Zhang et al. 2020; Khemakhem et al. 2019] 2020) to model distribution change explicitly. They require multiple training domains or additional observations, and such changes can also be explained under causal invariance as long as the latent variable includes all shifted factors (e.g., domain change of images can be attributed to different preference of shape, style, texture, background, etc. and their correlations, while the processes generating image and label from them remain the same).
3.2 COMPARISON WITH INFERENCE INVARIANCE
cetat--iivaaraaatteeeeeeettaaaie based adaptation and generalization methods, and invariant risk minimization (Arjovsky et al. 2019) for domain generalization, use shared feature extractor across domains. This effec- tively assumes the invariance of the process in the other direction, i.e., inferring the latent representation from data. We note that in its supportive examples (e.g., in- ferring the object position from an image, or extracting the fundamental frequency from a vocal audio), generat- ing mechanisms are nearly deterministic and invertible, so that the posterior is almost determined by the inverse function, and causal invariance implies inference invari- ance. For noisy degenerate mechanisms (Fig. 2 am- biguity occurs during inference since there may be multiple values of a latent feature that generate the same observation. The inferred feature would notably rely the prior through the Bayes rule. Since the prior changes across domains, the inference rule then changes by nature, which challenges the existence of domain- shared feature extractor. In this case, causal invariance is more reliable than inference invariance.
Figure 2: Examples of noisy (left) or de- generate (right) generating mechanisms that lead ambiguity in inference. Left: handwritten digit that may be generated either "3" or "5". Right: Schröder's stairs that may be generated with either A or B being the nearer surface. Inference results notably rely on the prior on the digits/surfaces, which is domain-specific.
To leverage causal invariance, adjust the prior conservatively for OOD generalization (CSG-ind) and data-driven for domain adaptation (CSG-DA), so together with the invariant generative mecha- nisms, gives difffrennt and more reliable inference rule than that following inference invariance.
4 METHOD
We develop learning, adaptation and prediction methods for OOD generalization and domain adap- tation using CSG following the causal invariance Principle and devise practical objectives using variational Bayes. Supplement(E.1 all the derivations.
4

---

Under review as a conference paper at ICLR 2021
4.1 METHOD FOR OOD GENERALIZATION
For OOD generalization, a CSG (p(s, v), p(a]s, v), p(yls)) needs to first learn from the super- vised data from an underlying data distribution p" p"(x, y) on the training domain. Maximizing likeli- hood (x..) [logg[( y)] is intractable since p(z, y) given by the CSG is hard to estimate effec- tively. We thus adopt the Evidence Lower BOund (ELBO) (a,r(1,)) == (gge.vlz..) Dog p(aa))))) g( (..))
(Jodaan et a 1999; Wainwright et al., 2008) as a tractable surrogate, which requires an auxiliary inference model q( 8, e 2| y) to estimate the expectation effectively. Maximizing L. w.r. q drives towards the posterior p(s, v/z, y) and meanwhile makes Lg a tighter lower bound of log p(æ, y). The expected ELBO (s.v)) (C..p 3)] then drives p(z, y) towards p" (xx, y).
However, the subtlety with supervised learning is that after fitting data, evaluating p(y|æ) for pre- diction is still hard. We thus propose to employ a model for g(s, v, y/r) instead. The required inference model can be then expressed as q(s, efar, y) ((s,0, U, y (/xl//()(xr) where g(vlx) [s,, v, y/2) dsdv. It reformulates the expected ELBO as:
log E (e y)1 = Epp()) Ep)()))) [log [b][=)] + B (y)x)) U,,, y) ((vlz)
(1)
((s,0
The first term is the common cross entropy loss (negative) driving (()()) towards p (y)xr). Once this is achieved, the second term becomes the expected ELBO Exy (o((s that drives q(s, v. y/ towards p(s, v, y/a:) (and p(z) towards p" (æ)). Since the target p(s, v. y/ admits the factorization p(s, u/z)p(yyl (y]s) (since (v, ()) IL y/s ) where p(yls) is already given by the CSG, we can further ease the modeling of g(s, v. y (r)) as q(s, vl plr)p(y)s). The ELBO is then reformulated as:
(yl2)) E ((s.waa plyls) log p(s, c)p(a)ss)) v)
1
(x,( v) log q(vl=))
(2)
q(s, v/2)
where <(y]=)) = (w) (w) (y)()))) The CSG p and g(s, o/a) are to be optimized. The expectations can be estimated by Monte Carlo, and their gradients can be estimated using the reparameterizztioo
trick (Kingma & Welling 2014). When well optimized, q(s, vfx) well approximates p(s, v/z)), so (y)22) then well approximates p(y|æ) (2)) [p(y]s) [s)] for prediction.
CSG-ind To actively mitigate the spurious s-v correlation from the training domain, we also con- sider CSG with an independent prior p ((sss) v) == p()) for prediction in the unknown test domain, where p(s) and p(v) are the marginals of p(s, v). The independent prior p= u) encour- ages the model to stay neutral on the s-u correlation. It has larger entropy than p(s, v) (Cover & Thomas, 2006 Theorem 2.6.6), so it reduces the information of the seeiiin-domain-speeiii prior. The model then relies more on the invariant generative mechanisms, thus better leverages causal invariance and gives more reliable prediction than that following inference invariance.
For the method, note that the prediction is given by p p'(ylz) Ep=(s.u(z)) [p(y)s)], we use an inference model for ( u|æ; that approximates ul(z). However, learning on the training domain still requires the original inference model q(s, v/z). To save the cost of building and learning two inference models, we propose to use q (ss, v/2r) represent q(s, u/z). Noting that their targets
related by p(s, u|æ² p ( p(z) accordingly, so that this q(s, v/2z) achieves its target once (s w/2x) does. The ELBO then becomes:
p(s.v) (w) '(t) u/ w/z), we formulate q(s, v/x) p(ar) p p'(x),
(a.o) p(z) q (s, u/a:))
1
p(s,u) E (c.l))) p p log (s. u))((sss)ss)
Cx v) log (y(z)) + (lylar
(3)
q (s v/a)
where (yylz) == (o Lp (e.,)) p(uls)]. The CSG p and g(s, u/z) be optimized (note that p= v) is determined by p(s, u) in the CSG p). Prediction is given by p (w(()))) ~ (n.
p(s.o)
4.2 METHOD FOR DOMAIN ADAPTATION
When unsupervised data is available from an underlying data distribution (x(()) (xr on the test domain, can leverage it for adaptation. According to the causal invariance Principle 3.2 we only need to adapt for the test-domain prior p(s, v) and the corresponding inference model g(s, u/z), while the causal mechanisms p(z|s, v) and p(yls) (s) are not optimized. Adaptation is done by fitting the test
5

---

Under review as a conference paper at ICLR 2021
data via maximizing Bir (x) where the ELBO is in the standard form:
(x) B((a)oo))) log (p(s, ()p(z)ss w)/((s v[]])]
(4)
Prediction is given by P(y]a) Fz eoje, ola) [p[y)s)]] Similar to the case of CSG-ind, we need 7(s, w/z)
for prediction, but q(s, w|z) is still required for learning on the training domain. When data from both domains are available during learning, we can save the effort of modeling and learning ((s, w|x)
using a similar technique. We formulate using ((s, v/x)) as q(s, v m/(r)
((z) p(x.n)
p(z) p(e,v) ple,vi((s, v/ar follow-
ing the same relation between their targets, and the ELBO on the training domain becomes:
p(s, v)
p(y]s) log p(s, v)p(als,
Lap( y) = log ((yar)
ग. (y(x)) ((..))
(5)
p(s, v)
((s, 0/a))
where (ylxr) B((..(())) (p(..)) p(e,v) P(yls)] The CSG p and q(s,, w//> are to be optimized (not for p(s, v)). The resulting method, termed CSG-DA, solves both optimizations
simultancously.
For implementing the three methods, note that only one inference model is required in each case. Supplement E2.] shows its implementation from a general discriminative model (e.g., how to select its hidden nodes as and r). In practice often has a much larger dimension than y. making the
supervised part of the training-domain ELBO (i.e., the first term in its formulation Eq. (T) scales smaller than the unsupervised part. So we include an additional cross entropy loss in the objectives.
5 THEORY
We now establish guarantee for the methods on identifying the semantic factor and the subsequent merits for OOD generalization and domain adaptation. We only consider the infinite-date regime to isolate another source of error from finite data. Supplement [A] shows all the proofs. Identifi- ability is hard to achieve for latent variable models (Koopmans & Reiersol 1950 Murphy 2012]
Yacoby et al. 20119 Locatello et al. 2019), since it a task beyond modeling observational relations (anzzin et al. 2009 Peters et al., 2017). Assumptions
required to draw definite conclusions.
Assumption 5.1 (additive noise). There exist nonlinear functions f and with bounded derivatives up to ohier-order, and independent random variables μ and v, such that p(z]s, v) = Pp f(s, v)), and p(y]s) (y g(s)) for continuous p(y(s) = Cat(yl (x)) for categorical y.
This structure disables describing bivariate joint distribution in both generating directions ZZang Hyvärinen (2009) Theorem 8). Peters et al. (2014) Proposition 23)), and widely adopted in directed causal discovery (Janzing et al.) 2009 Bühlmann al., 2014). CSG needs this since it should make the causal direction exclusive. is also easy to implement with deep models (Kingma)
& Welling 2014), so does not essentially restrict model capacity.
Assumption 5.2 (bijectivity). Function f is bijective and y injective.
It is a common assumption for identifiability (Janzing et al. 2009: Shalit et al. 2017 Khemakhem
al. 2019 Lee et al. 2 Under Assumption 5.1 it is sufficient condition (Peters al. 2014]
Proposition 17: Peters et 2017 Proposition 7.4) of causal minimality (Peters et al. 2014 2012.
Peters et 2017 Definition 6.33), a fundamental requirement for identifibilityy (Peters et al. 2014
Proposition Peters et al., 2017 p. 109). Particularly, and are otherwise allowed to have dummy dimensions that f and g simply ignore, raising another ambiguity against identifiability. On the other hand, according to the commonly acknowledged manifold hypothesis (Weinberger & Saul 2006 Fefferman et al., 2016) that data tends to lie on a nower-dimensional manifold embedded in the data space, we can take X as the manifold and such bijection exists as coordinate map. which is an injection to the original data space (thus allowing ds + dy < dx).
5.1 IDENTIFIABILTTY THEORY
We first formalize the goal of identifying the semantic factor.
Definition 5.3 (semantic-equivalence)) We say CSGs p and p' are semantic-equivalen,, if there exists homeomorphism 4 on S x V. such that (i) its output dimensions in S is constant of v:
A transformation is homeomorphism if is continuous bijection with continuous inverse.

---

Under review as a conference paper at ICLR 2021
Φ (s, (s) for any V. and (ii) it acts repanameteriization from p to p': # Pa.e. p(z)s, = p' (x (s,, v)) and p(y]s) p' (s))
It is an equivalent relation if V is connected and is either open or closed in de (Supplement A.1 Here, 2* [Ps.u] denotes the pushed-forward distribution formed random variable (ss 2) when (s, v) Ps..- As reparameterization, & allows the two models to have different latent- variable parameterizations while inducing the same distribution on the observed data variables (x. y) (Supplement Lemma A.2). At the heart of the definition, the e- constancy of & implies that & is gemantic-preservving one model does not mix the other's into its so that the variables of both models hold equivalent information.
f by D, i.e. the distribution of the trans-
We say that a learned CSG identifies the semantic factor it is semantic-equivalent to the ground- truth CSG p". This identification cannot be characterized by the statistical independence between and v (as in Cai et al. (211)); Isse et al. (201)) Zhang et al. (2020), which is not sufficient (Locatello al. 2019) nor necessary (due to the existence of spurious correlation). Another related concept disentanglement. It requires that a semantic transformation on r changes the learned only (Higgins al. 2018, Besserve et al. 2020), while the identification here does not require the learned to be constant of the ground-truth
To identify the semantic factor, the ground-truth model could at most provide its information via the data distribution p" (ar, y). Although semantic-equivalet CSGs induce the same distribution on (rr, y). the inverse is nontrivial. The following theorem shows that the semantic-identiffabiitty can be achieved under appropriate conditions.
Theorem 5.4 eeemanti-ideetifiabbln
well-learned CSG p with p(z, y) p (x, y) is remantic-equivalee to the ground-truth CSG fog p(s, v) and log p" (s, v) have bounded derivatives up to the second-order, and tha
With Assumptions 5.1 and 5.2
oo where of == Elu nu], or
oz (ii) Pa has a.e. non-zero characteristic function (c.g.. Gaussian distribution).
Remarks. (1) The requirement on p(s, v) and p" (s, v) excludes extreme training data that show a deterministic s-v relation, which makes the (s, v) density functions unbounded and discontinuous. In that case (e.g., all desks appear in workspace and all beds bedrooms), one cannot tell whether the label y is caused by s (e.g., the shape) or by u (e.g., the background).
(2) In condition (i), J measures the intensity of the causal mechanism p(z|s, v). A strong p(z|s, v)
helps disambiguating values of (s, in generating a given x. The condition makes p(æls, v) so strong that it is almost deterministic and invertible, so inference invariance also holds (Section 3.2). Supplement A.2 provides quantitative reference of large intensity for a practical consideration, and Supplement B gives a -symmttpt extension showing how the intensity trades-off the tolerance of equalities in Definition 5.3 Condition (fi) covers more than inference invariance. It roughly implies that different values of (s, v) a.s. produce different distributions p(z(s, v) on X' so their roles in generating x become clear which helps identification.
(3) The theorem does not contradict the impossibility result by Locatello et al. 2019), which con- siders disentangling each latent dimension with an unconstrained (s, v) + (z, y), while we identify as a whole with the edge v y removed which breaks the s-o symmetry.
5.2 OOD GENERALIZATION THEORY
The causal invariance Principle 3.2 forms the ground-truth CSG on the test domain as 3* (p" (s, v). p'p (I|s, v), (yls)) with the new ground-trutt prior p" (s, v). which gives the optimal predictor E.' [y]|:]] 5 on the test domain. The principle also leads the invariance of identified causal mechanisms, which shows that the OOD generalization error of a CSG is bounded:
Theorem 5.5 (OOD generalization error). With Assumptions
5.1 and 5.2 for a semantically- identified CSG p on the training domain with reparameterization 4 we have up to O that
'The definition of t [Ps..] requires & to be measurable. This is satisfied by the continuity of tb as a homeomorphism (as long as the considered o-field is the Borel fieldd) (Billingsey 2012 Theorem 13.2).
*To be precise, the semantic-equivalent conclusions are that the equalities Definition 5.3 hold asymptoti- cally in the limit
of - do for condition (i), and hold a.c. for condition (ii).
For categorical y, the expectation of y taken under the one-hot representation.

---

Under review as a conference paper at ICLR 2021
for any supp(p=) n supp(P?).
Bly|a]] - E' |ul=| & (
(=)) o)/p(s, 1//(ss,))lla U)ll (x))
(6)
where supp denotes the support of a distribution, Jn- is the Jacobian matrix of f 1-1 and Ps.v ## [a is the test-domain prior under the parameterization of the identified CSG p-
The result shows that when the causal mechanism p(z]s, v) is strong, especially in the extreme case Tn = 0 where inference invariance also holds, it dominates prediction over the prior and the generalization error diminishes. In more general cases where only causal invariance holds, the prior change deviates the prediction rule. The prior-change term || V (pg( 5, v)/p(s, e))l measures the hardness or severity of OOD. It diminishes in IID cases, and makes the bound lose its effect when the two priors do not share their support. Using a CSG to fit training data enforces causal invariance and other assumptions, so its B[y| behaves more faithfully in low p" (z) area and the boundedness becomes more plausible in practice. CSG-ind further actively uses an independent prior whose larger support covers more Pa.e candidates.
5.3 DOMAIN ADAPTATION THEORY
In cases of weak causal mechanism or violent prior change, the new ground-truth prior P?,, is im- portant for prediction. The domain adaptation method learns a new prior Ps.. by fitting unsupervised test-domain data, with causal mechanisms shared. Once the mechanisms are identified, p can also be identified under the learned parameterization, and prediction can be made precise.
Theorem 5.6 (domain udaptation error). Under the conditions of Theorem 5.4 for semantically- identified CSG p on the training domain with reparameterization 4 if its new prior p..o the test
domain is well-leurned with p(x) = p² (I), then Ps.0 = 4 Wo and E[yla] E [yla] for any x E suppl(²))
Different from existing domain adaptation bounds (Supplement D Theorems 5.5 and 5.6 allow different inference models in the two domains, thus go beyond inference invariance.
EXPERIMENTS
For baselines of OOD generalization, apart from the conventional supervised learning optimizing cross entropy (CE), we also consider causal discriminative method CNBB (He al. 2019), and a generative method supervised VAE (sVAE) which is a counterpart of CSG that does not separate its latent variable into and For domain adaptation, we consider well-acknowledged DANN
Ganin et al. 2016), DAN (Long et al. 2015) and CDAN (Long et al. 2018) methods implemented in the dalib package (Jiag al. 2020), and also sVAE using a similar method CSG-DA. All methods share the same optimization setup. We align the scale of the CE term in the objectives of all methods, and tune their hyperparameters lie on the margin that makes the final accuracy near 1 validation set from the training domain. See Supplement for details.
6.1 SHIFTED MNIST
We consider an OOD prediction task on MNIST to classify digits "0" and "1". In the training data, "0"s are horizontally shifted at random by d pixels with 8 ~ N((5,13) and "I"s by d ~ (5, 12) pixels. We consider two test domains where the digits are not moved, or are shifted 8 NO 2P) pixels. Both domains have balanced classes. We implement all methods using multilayer perceptron which not naturally shift invariant. We use a larger architecture for discriminative and domain adaptation methods compensate the additional generative components of generative methods.
The OOD performance is shown in Table I] For OOD generalization, CSG gives more genuine predictions in unseen domains, thanks to the identification of the semantic factor. CSG-ind performs even better, demonstrating the merit of approaching CSG with an independent prior. Other methods are more significantly misled by the position factor from the spurious correlation. CNBB ameliorates the position bias, but not as thoroughly without explicit structures for causal mechanisms. CSG
'The 2-norm ||- for matrices refers the induced operator norm (not the Frohenius norm).
8

---

Under review as a conference paper at ICLR 2021
Table I: Accuracy (%) of various methods (ours in bold) on OOD generalization (left) and domain adaptation (right) for shifted MNIST. Averaged over 10 runs.
shift
CE
CNBB
sVAE
CSG
CSG-ind DANN
DAN
CDAN
sVAE-DA CSG-DA
-0
53.3+ 74.4 62 60.5+13.9 90.9= 6.8 94.5+ 4.5 9916== 5.5 11.3+1.5 60.1
50.3
95.2+t119
.(0,23)
52.7+ 2.8 58.0# 58.6+ 64.8 64.8=23 66.5+ 4.7 47.0= 2.9 50.1 2.9 49.2+
68.0+ 7.5 76.0
Table 2: Results ImageCLEF-DA iim from
2014 Results of CE. DANN, DAN and CDAN are taken
Long et al. 2018).
task
CNBB
SVAR
CSG
CSG-ind
DANN
DAN
CDAN sVAE-DA CSG-DA
P 63 72.74 Ia 73.3+ 1o 73.4 73.7.
74.3: 69.2+
74.5 0.3 74.3+ 74.4 0.6
P-C 91.22 0. 91.7= 02 91.6# 09 92.3 0.5 92.5 0.3 91.5 0.6 89.8 0.4 93.5 0.4 92 92.9 0.3
also outperforms sVAE, showing the benefit of separating semantics and variation and modeling the variation explicitly, the model could consciously drive semantic representation into For domain adaptation, existing methods differ a lot, and are hard to perform well on both test domains. When fail to identify, adaptation sometimes on position gets strengthened on the unsupervised test data. CSG is benefited from adaptation by leveraging test data in proper way that identifies the semanties.
worsens the result, the misleading representation based
6.2 IMAGECLEF-DA
ImageCLBF-DA (ima 2014) is a standard benchmark dataset for the ImageCLEF 2014 domain adaptation challenge. We select a pair of adaptation tasks between two of its domains: Caltech-256 and Pascal VOC 2012. Each domain has 12 classes and 600 images following different distribution from each other. We adopt the same setup as in Long et al. (018), including the ResNet50 structure (He et al. 2016) pretrained on ImageNet as the backbone the diseriminnaiiee/ifeeenne model. For generative methods, we leverage the DCGAN generator (Radford et al. (015)) pretrained on Cifar 10.
Table 2 shows the results. We see that CSG(-ind) achieves the best OOD generalization result, and performs comparable with modern domain adaptation methods. On this task, the underlying causal mechanism may be very noisy (e.g., photos taken from inside and outside both count for the aircraft class), making identification hard. So CSG-DA does not make salient improvement.
7 CONCLUSION AND DISCUSSION
We tackle OOD generalization and domain adaptation tasks by proposing a Causal Semantic Gener- ative model (CSG), which builds upon a causal reasoning, and models semantic and variation factors separately while allowing their correlation. Using the invariance principle of causality, we develop effective and delicate methods for learning, adaptation and prediction, and prove the identification of the semantic factor, the boundedness of OOD generalization error, and the success of adaptation under appropriate conditions. Experiments show the improved performance in both tasks.
The consideration separating semantics from variation extends to broader examples regarding robustness. Convolutional neural networks are found to change its prediction under a different tex- ture but the same shape (Geirhos et al. 2019 Brendel & Bethge 2019). Adversarial vulnerability (Szegedy et al., 2014 Goodfellow et al. 2015; Kurakin et al. 2016 extends variation factors to human-imperceptible features, Lee. the adversarial noise, which shown to have a strong spuri- correlation with semantics (Ilyas et al.] 2019). The separation also matters for fairness when a sensitive variation factor may change prediction due to a spurious correlation. Our methods are potentially beneficial these examples.
REFERENCES
The imageclef-da challenge 2014. http://www.immgeelefoo/0000 2014.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Paze Invariant risk minimization. arXiv preprint arXiv: 1907.00907.000 00000000999,.. 2019.

---

Under review as a conference paper at ICLR 2021
Yuval Atzmon, Felix Kreuk, Uni Shalit, and Gal Chechik. A causal view of compositional zero-shot recognition. Advances in Neural Information Processing Systems, 33, 2020.
Mohammad Taha Bahadori, Krzysztof Chalupka, Edward Choi, Robert Chen, Walter F Stewart, and Jimeng Sun. Causal regularization. arXiv preprint arXiv: 2017.
Mahsa Bakrashmotlagh, Mehrtash T Harandi, Brian C Lovell, and Mathieu Salzmann. Unsupervised domain adaptation by domain invariant projection. Proceedings of the IEEE International Conference on Computer Vision, pp. 769-776, 2013.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1-))151-175, 2010a.
Shai Ben-David Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adap- tation. In Proceedings the Thirleenth International Conference on Artificial Intelligence and Statistics, pp. 129-136, 2010b.
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Rosemary Ke, Sébastien Lachapelle, Olexa Bila- niuk, Anirudh Goyal, and Christopher Pal. meta-transfer objective for learning to disentangle causal mechanisms. arXiv preprint arXiv:1901.10912 2019.
Michel Besserve, Naji Shajarisales, Bernhard Schölkopf, and Dominík Janzing. Group invariance principles for causal generative models. In International Conference on Artificial Intelligence and Statistics, pp. 557-565. PMLR, 2018.
Michel Besserve, Arash Mehrjou, Rémy Sun, and Bernhard Schölkopf. Counterfactuals uncover the modular structure of deep generative models. Proceedings of the International Conference on Learning Representations (ICLR 2020), 2020.
Irving Biederman. Recognition-by byw.ooonnnttsmssss: a theory of human image understanding. Psycho- logical review, 94(2):115, 1987.
Patrick Billingsley, Probability and Measure, John Wiley & Sons, New Jersey, 2012. ISBN 978-1- 118-12237-2.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Wieland Brendel and Matthias Bethge. Approximating CNNs with bag-of-locall-eattee models works surprisingly well on ImageNet. In Proceedings of the International Conference on Learn- ing Representations (ICLR 2019), 2019.
Peter Bühlmann. Invariance, causality and robustness. arXiv preprint arXiv: 111.11822..0222222
Peter Bühlmann, Jonas Peters, Jan Emnsst, al. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics, 42(())2556-255 2556, 2014.
Ruichu Cai, Zijian Li. Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning disentan- gled semantic representation for domain adaptation. In Proceedings of the Conference of IJCAI, volume 2019, pp. 2060. NIH Public Access, 2019.
Daniel Castro, Ian Walker, and Ben Glocker. Causality matters in medical imaging. Nature Communications. 11(1):1-10, 2020.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman. Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172-2180, 2016.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2006.
10

---

Under review as a conference paper at ICLR 2021
Dominik Maria Endres and Johannes E Schindelin. A new metric for probability distributions. IEEE Transactions on Information theory, 49(7):1888-188800 2003.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4):983-1049 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian approximation:
Representing model un- certainty in deep learning. Proceedings of the International Conference on Machine Learning, pp. 1050-1059, 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Romainn-averssan training of neural net- works. Journal of Machine Learning Research, 17:1-35, 2016.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. ImageNet-trained trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In Proceedings of the International Conference on Learning Representations (ICLR 2019), 2019.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International Con- ference on Machine Learning, pp. 2839-2848, 2016.
Mingming Gong, Kun Zhang, Biwei Huang, Clark Glymour, Dacheng Tao, and Kayhan Batmanghe- lich. Causal generative domain adaptation networks. arXiv preprint arXiv: 180400433333,,, 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR
2015), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Yue He, Zheyan Shen, and Peng Cui. Towards non-i.i.d. image classification: A dataset and base- lines. arXiv preprint arXiv: /906. 1906.029999 2019.
Christina Heinze-Deml and Nicolai Meinshausen. Conditional variance penalties and domain shift robustness. stat, 1050:13,2019
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAE: Learning basic visual concepts with constrained variational framework. Proceedings of the International Conference on Learning Representations (ICLR 2017), 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.022300 2018
Patrik O Hoyer, Shohei Shimizu, Antti J Kerminen, and Markus Palviainen. Estimation of causal effects using linear non-gause aon-gaussian causal models with hidden variables. International Journal of Approximate Reasoning, 49(2):362-378, 2008
Aapo Hyvärinen. Estimation of non-nommalized statistical models by score matching, Journal of Machine Learning Research, 6( 6(Apr):695-709, 2005.
Maximilian Ilse, Jakub M Tomezak, Christos Louizos, and Max Welling. DIVA: Domain invariant variational autoencoders. arXiv preprint 19Xiv:1905.1042)) 2019.
Maximilian llse, Jakub M Tomezak, and Patrick Forré. Designing data augmentation for simulating interventions. arXiv preprint arXiv:2005.01856 2020.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
11

---

Under review as a conference paper at ICLR 2021
Dominik Janzing, Jonas Peters, Joris M Mooij, and Bernhard Schölkopf. Identifying confounders using additive noise models. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009), pp.
AUAI Press, 2009.
Dominik Janzing, Eleni Sgouritsa, Oliver Stegle, Jonas Peters, and Bernhard Schölkopf. Detecting low-complexity unobserved causes. In 27th Conference Uncertainty in Artificial Intelligence (UA1 20/ 1), pp. 383-391. AUAI Press, 2011.
Junguang Jiang, Bo Fu, and Mingsheng Long. Transferlearming--bee https://github. haam:/htmm/rrafff--eeeaaee
2020.
Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain- invariant representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 527-536, 2019.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183225 1999.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv:1910.01075, 2019.
Ilyes Khemakhem, Diederik Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlin- ear ICA: A unifying framework. arXiv preprint
2019.
llyes Khemakhem, Ricardo Pio Monti, Diederik P Kingma, and Aapo Hyvärinen. ICE-BeeM: Iden- tifiable conditional energy-based -based deep models. arXiv preprint arXiv:2002. /1537, 2020.
Niki Kilbertus, Giambattista Parascandolo, and Bernhard Schölkopf. Generalization in anti-causal learning. arXiv preprint arXiv: 2018
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR 2014), Banff, Canada, 2014. ICLR Committee.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pp. 3581-3589, 2014.
Murat Kocaoglu, Sanjay Shakkottai, Alexandros Dimakis, Constantine Caramanis, and Sriram Vishwanath. Entropic latent variable discovery. arXiv preprint arXiv: 2018.
Tjalling Koopmans and Olav Reiersol. The identification of structural characteristics. The Annals of Mathematical Statistics, 21(2):165-181, 1950.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: arXiv: 1607.02333, 2016.
Ciarán M Lee, Christopher Hart, Jonathan G Richens, and Saurabh Johri. Leveraging directed causal discovery to detect latent common causes. arXiv preprint arXiv: 1910.. 10/74, 2019.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised leamn ing of disentangled representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, pp. 2114-4124 Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pp. 1640-1650, 2018.
12

---

Under review as a conference paper at ICLR 2021
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Schölkopf, and Léon Bottou, Dis- covering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1979-6987, 2017.
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent- variable models. In Advances in Neural Information Processing Systems, pp. 6446-6456, 2017.
Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In Advances in Neural Information Processing Systems, pp. 10846-10856, 2018.
Jon D Mcauliffe and David M Blei. Supervised topic models. In Advances in Neural Information Processing Systems, pp. 121-128, Vancouver, Canada, 2008. NIPS Foundation.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18, 2013.
Kevin P Murphy. Machine learning: probabillistc perspective. MIT press, 2012.
Radford M Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-210, 2010.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15(1):2009-2553, 2014.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series (Statistical Methodology), 78(5)9977-012, 2016.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556 12565, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1.Xiv:1511.004333 2015.
Thomas Richardson, Peter Spirtes, et al. Ancestral graph Markov models. The Annals of Statistics, 30(4):922-0030 2002.
Mateo Rajas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, (9(1):1309-1342 2018.
Jan Jan-Willem Romeijn and Jon Williamson. Intervention and identifiability latent variable mod- elling. Minds and machines, 28(2):233-264 2018
Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv:1911.10500 2019.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris M Mooij. On causal and anticausal learning. In International Conference on Machine Learning (ICML 2012). pp. 225511262. International Machine Learning Society, 2012.
Eleni Sgouritsa, Dominik Janzing, Jonas Peters, and Bernhard Schölkopf. Identifying finite mixtures of nonparametric product distributions and causal inference of confounders. In Proceedings of the 29th Conference on Uncertainty Artificial Intelligence (UAl 2013), pp. 556-575. AUAI Press, 2013.
13

---

Under review as a conference paper at ICLR 2021
Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: general- ization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning- Searning-Volume Volume 70, pp. 3076-3085. JMLR.org, org, 2017.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. In Proceedings of the Inter- national Conference on Learning Representations (ICLR 2018), 2018
Zheyan Shen, Peng Cui, Kun Kuang. Bo Li, and Peixuan Chen. Causally regularized learning with agnostic data selection bias. 2018 ACM Multimedia Conference on Multimedia Conference, pp. 411-419. ACM, 2018.
Iyy Shpitser. Robin J Evans, Thomas S Richardson, and James M Robins. Introduction to nested Markov models. Behaviormetrika, 41(1):3-39, 2014.
Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting The Journal of Machine Learning Research, (5()::192--1958 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR 2014), 2014.
Takeshi Teshima, Issei Sato, and Masashi Sugiyama. Few-shot domain adaptation by causal mech- anism transfer. arXiv preprint wxXiv:2002.03497, 2020.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-RMSprop:: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Thomas Verma and Judea Pearl. Equivalence and synthesis of causal models. UCLA, Computer Science Department, 1991.
Martin J Wainwright, Michael I Jordan, al. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008.
Yixin Wang and David Blei. The blessings of multiple causes. Journal of the American Statistical Association, 114(528):1574-1583 2019.
Kilian Weinberger and Lawrence K Saul. Unsupervised learning of image manifolds by semidef- inite programming, International Journal of Computer Vision, 70(1):77-90, 2006.
Yaniv Yacoby, Weiwei Pan, and Finale Doshi- Velez. Learning deep bayesian latent variable regression models that generalize: When non-identifiability is problem. arXiv preprint arXiv:1911.00569 2019.
Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Representation learn- ing for treatment effect estimation from observational data. In Advances in Neural Information Processing Systems, pp. 2633-2643, 2018
Kaichao You, Ximei Wang, Mingsheng Long, and Michael Jordan. Towards accurate model selec- tion in deep unsupervised domain adaptation. International Conference on Machine Learning, pp. 7124-7133, 2019.
Cheng Zhang, Kun Zhang, and Yingzhen Li. A causal view robustness of neural networks. In Advances in Neural Information Processing Systems, 2020.
Kun Zhang and Aapo Hyvärinen. On the identifiability of the post-nonlinear causal model. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAl 2009), pp- 647-655. AUAI Press, 2009.
14

---

Under review as a conference paper at ICLR 2021
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pp. 819-827, 2013.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pp- 7523-7532, 2019.
15

---

Under review as a conference paper at ICLR 2021
SUPPLEMENTARY MATERIALS
A PROOFS
We first introduce some handy concepts and results to make the proof succinct. We begin with extended discussions on CSG.
Definition A.1. A homeomorphism on S V is called reparameterization from CSG p to CSG p', if 2 [Pa.u] P's,.. and p[z|s, v) = p' p'(x)((s, v)) E S V. A reparameterization & is called to be semantic semantic-peeservngg if its output dimensions in S is constant of u: 4 (s, u) = $ (s) for any v E V.
and p(vls) = p' (s,v), u)) for any
Note that a reparameterization unnecessarily has its output dimensions in S, i.e. ( v). constant of v. The condition that p(yls) p' [y] (s. u)) for any E V does not indicate that o (s v) constant of v, since p' sy(ss'') may ignore the change of s' 4 (s. v) from the change of e. The following lemma shows the meaning of reparameterizatton: it allows CSG to vary while inducing the same distribution on the observed data variables (z., y) (i.e., holding the same effect on describing data).
Lemma A.2. If there exists reparameterization t from CSG p to CSG p', then p(æ, v) p/ (2r,
Proof. By the definition of reparamcterization, we have:
p(x, v) / p(s, u)(alss s)p(uss) dsdu = f + # ( os) (x) u))p (y) (s. u)) dsde
P (d', v')p (xls', o (yls") de'd'' = (x,
where we used variable substitution (s', =- (s,, v) in the second-last equality. Note that by the definition of porwad distribution and the bijectivity of 4 # p',. implies Ps," = = [V.,l. and f('', (365,"")) D''.. (s', , v') ds'de' = f f( v)) (ss, u) dsdv (can also be verified deductively using the rule of change of variables, i.e. Lemma A.4 in the following). O
The definition of semantic-equivalence (Definition 5.3 can be rephrased by the existence of semantic- semantic-preservingg reparameterization. With appropriate model assumptions, we can show that any reparameterization between two CSGs is semantic- preeeeriii....riingg. so that semantic-preservigg CSGs cannot be converted to each other by reparamcterization that mixes with v.
Lemma A.3. For two CSGs p and p', if (yl=) has statistics (s that is injective function of s, then any reparameterization & from to p', if exists, has its o constant v.
Proof. Let T ($5, 2D) be any reparameterization from p to p'. Then the condition that p(yls) p' (y $ (8, v)) for any E y indicates that M(s) = M' (1, v)) If there exist E S and (()) * (2)) E V such that Φ v(4)) * Φ (s. v(2)), then M' (s, (a)) * (s, since M' is injective. This violates (s) = M' (M($((s, which requires both (T and M M'
(5 (s, (())) to be equal to (s). So T (s. must be constant of e.
We then introduce two mathematical facts.
Lemma A.4 (rule of change of variables). Let z be random variable on Euclidean space Rdz with density function (2),), and let & be homeomorphism on RR.zz whose inverse 4- 1 is differen- tiable. Then the distribution of the transformed random variable x' =[)) has a density function 4 (p=[(2)) = (2)))) (2)), where (2') denotes the absolute value of the determi- nant of the Jacobian matrix (Ja-1 (2')) = 02. ( (2()) of 0- al
Proof. See e.g., Billingsey 2012, Theorem 17.2). Note that homeomorphism is (Borel) mea-
surable since it is continuous (Billingsley 2012 Theorem 13.2), so the definition of Φb # [p=] is valid.
16

---

Under review as a conference paper at ICLR 2021
Lemma A.5. Let be a random variable whose characteristic function is a.e. non-zero. For two functions f and f' on the same space, we have: f Рp = f * Pa - f = f a.e,, where = (
p1) du denotes convolution.
Proof. The function equality f Pr = = Pu leads to the equality under Fourier transformation
FI +py] F *Pu). which gives | [f] Pp
$ { " [y Since & [pu is the characteristic
function of Pu. the condition that it is a.c. non-zero indicates that [ [f a.c. thus f = f
a.c. See also Khemakhem et al. 22019 Theorem 1).
A.1 PROOF OF THE EQUIVALENCE RELATION
Proposition A.6. The ammantic-equivalenne defined Definition 5.3 is an equivalence relation if connected and is either open closed in Rdy
Proof. Let + be semantic-preserving reparameterization from one CSG p (p(s, v), p(x)s v), p(yls)) to another p' p''(s.)).p''z)se p'yyyys) It has its Ø stant of U, so we can write ((s, v) = (Ø (s), 4b (s. =: (o(s), (v)).
(1) We first show that o. and th for any E S, are homeomorphisms on S and V, respectively, and that + (s', v') = ( (s'), 1 (s)) (v')).
Since (S V)
V. so (88) = (S) =. surjective.
Suppose that there exists s' S such that o= (5') (o(0)) contains multiple distinct elements.
1. Since is surjective, for any o' V, there exist i I and
such that (s', (s((),, v) = (d(s()), ab (v)), which means that Ulez (l,oo (V)= =
Since & is injective, the sets (12,0 (V) must be mutually disjoint. Otherwise, there would exist o E I and o(2), (((2) V Vuuch that rhato) (x(2)) thaos (v((2) thus ((8(),, ((1)
(s', (o(1)) the injectivity of & since s(i) # s
(s', (acn
$(s6),, v((2), which violates
3. In the case where V is open, then is any 2a ((s(),, V) since & is contin- uous. But the union disjoint open sets Uler 2.0 (V) = V cannot be connected. This violates the condition that V is connected.
4. A similar argument holds in the case where V is closed.
So a (s')))) contains only unique element for any s' E S. So o is injective.
The above argument also shows that for any s' E S, we have Ulez 19000 (y00 = Ve (V) have (V) = V. So v. is surjective for any E S.
V. For any E S, there exists s' E S such that s o (s'), so we
Suppose that there exist (())) # v(2) E V such that (v())))) = th, (x))). Then (())- = (ő(s), tha (o())) = (ø(s), (v())) = i(s, o(2)), which contradicts the injectivity of + since (1)) # 2l2) So th is injective for any s S.S. That & is continuous and ((swv)
(sa)), indicates that and th S x V, we have ((0-1(5'), d - (v')
continuous. For any (5',0)
(#(0- () (0) (1) (v')) = (8',0). Applying Φ-1 to both sides gives Φ (s', e') = (o (s'), (0)) (60)(( (v')).
Since h- continuous, o and Φ- are also continuous.
(2) We now show that the relation is equivalence relation. It amounts to showing the following three properties.
Reflexivity. For two identical CSGs, we have p(s, v) p'(, v), p(rls,) p/ (z|s, v) and p(y|s) = p' (y|s). So the identity map as & obviously satisfies all the requirements. Symmetry. Let & be semantic-preserving
reparameterization from (p(s, v), p(z]s, v). p(y)s)) to = ='p''p'''s p') pyyss) From the above conclu- o- (s') is semantic preservvi.....ingg Also, क is
sion in (1). we know that (Ø ( a homeomorphism on S x since & is. So we only need to show that / is a reparame- terization from p' p for symmetry.
17

---

Under review as a conference paper at ICLR 2021
1. From the definition of mushed-forwand distribution, have $x [ps,el # [Ps.u] Pla..- It can also be verified through the rule of change of vari- ables (Lemma (A4)) when & and h- are differentiable. From P P+lps,ul P we have for any (s',""), Pa "o""""s''''))e =(''v')) = (',,('' Since for any (s, there exists (s',') such that (8,0) (-"(6, this implies (9, v)), or (s,
ps.e if
that for any (5,0). Ps.u(s, v) (0-),(( (*,,, o))/|Jo-1 (+() = (',s, ()). e)l (inverse function theorem), which means that Ps,v = o pl by the rule of change of variables.
2. For any (s',v",, there exists (s, v) such that (s',v") = (ss.)), so p'(xss', = p(a o " v'), and p'(uls') o' p((u)
p' ( v)) = p(zls,v)
p(y(s) = p(v ( (o-"))(s")).
So h- is reparameterization from p' to p. Transitivity. Given third CSG p"
(s, v), (zls, v). (y(s)) that is semantic- equivalent to p', there exists a semantic- preeerrrnggg reparameterization o' from p' to p". It is casy to see that (Ø's ()) v) o ( (s. v)) 4 (s)) is constant of thus semantic-preserving As the composition of two homeomorphisms & and D' on S x V. &' o & is also bomeomorphism. So we only need to show that 4' d is reparameteri- zation from to p" for transitivity.
1. From the definition of pushed-forward distribution,
have (Φ' o 1 D) [P D' # [+* P+.+]] = 4' [V',el = P",," if 4 [Po,ol = P'e. and &' [Pse = P",.- It can also be verified through the rule of change of variables (Lemma A.4 when Φ and d'1 are differentiable. For any (s", v²), we have
(4/ o +) #:#100 [ (8", v²" Ps.. oD) ()- "s",, """'') ((vo)))) (s",
-P,,o(D (Φ' (s", Ja- (0' x²")) J ,
#1Ip ( (5" ,²" [))]][ 1(8".
-D',000' ","""" J)))))= Jal-r-1(6 18", =
o").
2. For any (s, v). we have:
p(z|s, u)
-| p=' ()s,
p s)
~3))= ())))))))))))
So o' o is a reparameterization from p to p². This completes the proof for an equivalence relation.
A.2 PROOF OF THE SEMANTIC- DANNTICCIDENFFIABBLLTY 5.4
We present a more general and detailed version of Theorem 5.4 and prove it. The theorem in the main context corresponds to conclusions (ii) and (i) below by taking the two CSGs p' and p as the well-learned p and the goound-truth CSGs p", respectively.
Theorem 5.4 (semantic-identifaabilty... Consider CSGs p and p' that have Assumptions 5.1 and 5.2 hold, with the bounded derivative conditions specified to be thatfor both CSGs, f and g are twice and f thrice differentiable with mentioned derivatives bounded. Further assume that their priors have bounded densities and their log p(s, v) have bounded derivatives up to the second-order. If the two CSGs have p(x, y) = (z. y). then they are semantic- -etanniii--qvvalett. under the conditions that: (i) Pp has an a.e. non-zero characteristic function (e.g., Gaussian distribution):
(ii) + 00, where o2 == Elp pl:
- +3d Boggp* 3d B²_BB d B'-"),, where d == + + dy, and for both CSGs, the constant Bp bounds p(s, v). B By: Blog and By, By, B bound the 2-norm of the gradien//Jccoiun and the Hessians of the respective functions, and B bounds all the 3rd-order derivatives of f-
(iii) B7 max Bi p B3 By - B' B, By +
"To be precise, the conclusions are that the equalities Definition 5.3 hold a.e. for condition (i), hold asymptotically in the limit -+ oc for condition (ii), and hold up to negligible quantity for condition (iil).
z
As an induced operator for matrices (not the Frohenius norm).
18

---

Under review as a conference paper at ICLR 2021
Proof. Without loss of generality, we assume that μ and (for continuous y) have mean. If it is not, we can redefine f(s. := f(s. v) + Elp] and μ := l Elμ] (similarly for v for continuous y) which does not alter the joint distribution p(s, v, z, y) nor violates any assumptions. Also without loss of generality, we consider one scalar component (dimension) I of y, and abuse the use of symbols y and g for yi and Si to avoid unnecessary complication. Note that for continuous y, due to the additive noise structure as the same as the categorical y case (under the one-hot representation). We sometimes denote = (s, v) for convenience.
g(s) and that v has zero mean, we also have Blyls] g(s)
First note that for both CSGs and both continuous and categorical y. by construction g(s) is sufficient statistics of p(y]s) (not only the expectation E'yls),, and it is injective. So by Lemma A.3 we only need to show that there exists a reparameterizatioo from p to p/. We will show that o f is such reparameterization.
f
Since and f' are bijective and continuous, we have = f", so + is bijective and & and h- are continuous. So & is homeomorphism. Also, by construction, we have:
(7) So we only need to show that p(a, y) = (x, v) indicates #p = p's and p(uls) =
p(a|z) ( - f2)) Pa - (f())) = Pn ('((()))) = (xxx
p' (y) $ (s, u)), Ve E under the conditions.
Proof under condition (i). We begin with useful reformulation of the integral J [(<))((]]=) da for a general function & of z. We will encounter integrals in this form. By Assumption 5.1 we have p(z|z)= Pμ (x: - fz)). so we consider transformation (= == x - f() and let μ (z)..
is invertible, , (1) = (x 14), and J -- pt) By these definitions and the rule of change of variables, we have:
f (())p(=l=)
/ [(e))) / (w))(())) Jo.- (x) du
/ |15 (c (2 ())()))())) -a) w)
=Epn)((VV -x] -sm111 * Pp))());
(8)
where we have denoted functions 8= to f V := |ورر|| /. and abused the push-forward notation Iw [t] for general function to formally denote (l l رااررر IV.
According to the graphical structure of CSG, we have:
p(x)= = / p( p(=)
(10)
E[ylx] p(x) / up(a, v) dy = p(a) II mr(=)p(al=))((ll dady
1
p(x) / ((=))(al)) p(z) / ((s))())p(()))dd.
1 1
(11)
So from Eq. (9. we have:
Elyl = plaa (f +p))
p(z) = (f*[p=] * (x)),
(12)
p(x)
Matching the data distribution p(z, y) = p' (z. y) indicates both p(x) = p' (zr) and E[y|æ] = EE' []]
Using Lemma A.5 under condition (i), this further indicates f* [P.] = f [p's] a.e. and f [gp=
I'g [g' [p]] a.e. The former gives + ( = p'_- The latter can be reformed as gf = g [r
a.e., so g = T' a.e., where we have denoted ğ = go (f-1)S and g' := o (1-1) similarly. From g = g'. we have for any v V.
g(s) - s((S () - ((0)):)))=(((0)))((((((0000 -| g()(s.))
= =J(s = - (((0) v)).
(13)
For both continuous and categorical y, g(s) uniquely determines p(y]s). So the above equality means that p(yls) p' (y( $ v)) for any E V.
19

---

Under review as a conference paper at ICLR 2021
Proof under condition (ii). Applying Eq. (8) Eqs. 10 we have:
pa)) (Z p)],
E [|a]]
p(a) (0p (Ix - p)].
where we have similarly denoted P3 := P= f-1, Under condition (ii), BB(a T n1] is infinitesimal, so we can expand the expressions w.r.t μr. For p(z), we have:
p(z) = Bp(x) [D.V V) + 2" VV
= = + Eptuy Ik T VV T (D +O(o)))
2
where all functions are evaluated at x. For [[yla]]. we first expand 1/p(x) using x+s
+
O(e²) to get: p(z) P.V. 20g Bpi [n VV (p. + +O(2)) The second term is expanded
1
as: 0p2 V + 2 p [1= T VV (op=V)n) V)n + ((a)). Combining the two parts, we have:
Elyla | = 9 + Pln [n" (oVog, V\Vg
+ VV g) + +O6))) (14)
This equation holds for any supp(p*) since the expectation is taken w.r.t the distribution p(x, y); in other words, the considered æ here is any value generated by the model. So up to O(a)).
1 Bri 11 VV
p(x) (p.2 ()))
PP [N VV
<
(Ø. V)ll
2
Bol [1|x||2]
(Ø_ (6 ||21111111
Blu
1
اill
V ()
2
Biи 1Alp_
V. V) V) l2
<
Blu 14lp. V (V TogVTloogp.1 1 + |
(15)
E[y x] g(x))
Ep [H (VV (V VV + VV 3)
< Brl 1I
V) AA g
< 2
BP(n [Ile
+ Vg (V
VV [|اي|141|2
< Eiи A [[[
Vg + v
Elu" m) ([ V + 1 lll))
(16)
Given the bounding conditions in the theorem, the multiplicative factors to Elu ui] in the last pressions are bounded by a constant. So when /े - oo, i.e. Elu 14] + 0, we have p(x) and [[y]a]] converge uniformly to (p.V) = f* (p=((r) and g(æ), respectively. So p(a, y) = p((x, v) indicates SH [p=] = J |v and ğ = g'. which means + # [p=] = p's and plyls) = p' $ (s, for any U V. due to Eq. (33 and the explanation that follows.
Proof under condition (iii). We only need to show that when 1 is much larger than the given oz quantity, we still have y) = p(x, (z, y) = V.V = V g' up to a negligible effect. This task amounts to showing that the residuals (p(z) (z)) and [[[y|z] - g(x) controlled by Eqs. [3) 16 are negligible. To achieve this, we need to further expand the controlling functions using derivatives of f. 9 and P= explicitly, and bound them by the bounding constants. In the following, we use indices a, b, c for the components of x and i, j, k for those of 2. For functions of z appearing in the following (e.g.. J. g. Pa and their derivatives), they are evaluated since we are bounding functions of x.
--)((x)
(1) Bounding F[y| g(x)l Ely M[([VV log Vgl + 1
VV g|bb) from Eq. (16).
From the chain rule of differentiation, it is easy to show that:
V log P. Jy- V log
Vg Jo-1 Vg Jz V.g,
(17)
20

---

Under review as a conference paper at ICLR 2021
where Vag- (Vg] 07₂) (recall that g is function only of s). For the term V log V, we apply Jacobi's 's formula for the derivative of the log determinant:
Og log ((x) 0g Jog|Jy (()) tr (J (2) (+))) E
(z)mr)
b,
=
(z) = (J)(VVVT)'')
(18)
However, as bounding Eq. (17) already requires bounding J 1112- directly using this expression to bound V log Vil2 would require to also bound ||/ This requirement to bound the first-order derivatives of both f and f-1 is relatively restrictive one. To ease the requirement, we would like to express V log V in terms of J,- This can be achieved by expressing VV f: in terms of fe's s. To do this, first consider a general mwwertble-mmatrix-eeee = (0. 1
VV
function A(a) on a scalar a.
We have 0- = 0. ( (A(ox)
)) A + - A, so we have A A - -(0,
consequently A,A = -A(O_A-1)A4 Using this relation (in the fourth equality below), we have:
(VV) = = = 8, =
(0.J)-))
- (J)- = (J=r
= Σ (2a(0) = 1000000000000010000000000001
30
jok
=- -J [Un-)n((00000000))-.-)) = --U-))a,
or in matrix form,
VV Eiy
- 1)ad :VV f.)f- =: -Us Dakk
(19)
where we have defined the matrix K == J (VV which is symmetric. Substituting with this result, we can transform Eq. 188 into a desired form:
V log V(x) = E (J,(VVT
J Σ [U--bd)---V///1)...
= = Σ EJJ (VV 1.NT-.) -- (VVV f.NT--))
T.
= = - (V = = -- -(K)) = Σ K
(20)
so its norm can be bounded by:
Vog V(z)ll2 =
Kc = (J [
Σ )el
11
VV
fe 1ا < BBB (l(ll-))ll
< dBB-.Bj.
(21)
where we have used the following result in the last incquality:
ElI(,))alll < //22
Elo = = مdليم- -10->a==
& &dB
(22)
Integrating Eq.
and Eq. 21 we have:
(V log D.
(J, V logp, Vlog TT 1 V
(1]J || V log + | log /112) J|ر V
< (B/- + dBB- B3) B'
(Blog + d B3) B7- B'
(23)
21

---

Under review as a conference paper at ICLR 2021
For the Hessian of g. direct calculus gives:
V g Joy 11s (VV) 9) + > (VV (VV
= =Jy. Σ [V
To avoid the requirement of bounding both VV Se's and VV fs we substitute VV f using
Eq. 19
V J 1 (V. [V (v.g). .g): E(1 D)sk
=V - 3 (J) k.
So its norm can be bounded by:
VV
9 و|2
J 1 9|2 + [JJg- (V V29) K
< BZ-1B B" + Σ (Jy 1)c V V=9) 18
< B- BB" + B > (J 8-1 )x V=olla)
< B BB" + B"J B' E||(J)-))al1
< B--1 B" + dB ByJ
(24)
where we have used Eq. 22 in the last inequality. Assembling Eq. 233 and Eq. 24
into Eq.
(66)
we have:
1 D BB + B" +
3
Eylz] [(x) < Biu MBBBB (Biog
+ B
(25)
So given the condition (iii), this residual can be neglected.
(2) Bounding pp(x) - (z))
< 1 T Alr.
V + || | T pzl2
+
VV log V12 from Eq. 15
To begin with, for any x. (x) (x)) & Bp. and V(z) (x) is the product of absolute eigenvalues J (x). Since | l| the largest absolute eigenvalue of Jy-
so V(x) < (x)]1 < By
For the first norm in the bracket the r.h.s of Eq. 15 we have:
||V logg_,111 = V log
? ( p.) ) log V | log V 1
< Vogg 122 2 log V log Vl2 + ||V log Vll
BF. mg + 2dB B" + V log VI
(26)
where we have utilized Eq.
17 and Eq. 21 in the last inequality. We consider bounding
|| log VIl2 separately. Using Eq. 20p (in the second equality below), we have:
| log V12 = (V log V) (Vg = [((x)) Σ kd
Σ
KG KGK < >
KKKK
ad
cd
Σ
(J
VV Sc)J (VV (a(
ed
22

---

Under review as a conference paper at ICLR 2021
(J (Ur=)(())) folj ,J (VV fal 2
d
Σ (J )o (1 (
13 B7- By = BT By Σ -
)od
od
cd
< 1/ BB B3 J J ||
2 < / d B
(27)
where we have used the facts for general matrix A and (column) vectors a, that
1a A3 [1a)
112 - ||3 T A l12 < ||03 11 |121|1112 - |a A|1
(28)
in the fifth last inequality, and that
And Aodl < viz
Σ And 2
d|| < Al12
(29)
in the second last inequality. Substituting Eq. 27 into Eq. 26). we have:
||V log V12 Bf B 2d By B Blugr + 2/2 B3t By
(30)
For the second norm in the bracket of the r.h.s of Eq. (5) similar to Eq. (2). we have:
|| V log p2l12 < BB- (Bogg + dB B,, Blogp)
(31)
The third norm
VV log V11, in the bracket of the r.h.s of Eq. 115 needs some more effort. From
Eq. (20). we have O log V-
)ej. thus
0,0g log V Σ
0a(J (0,(0.fff)))))) 1)00 E(1, (00(000 1)0
cjj
aj
[0,0),,,,
cij
> (,,,,,1), )
cijk
(Jg (0.0, [U)))a(00, 14
cijd
ajd
(Jz-1)
cijk
Kd ac
Ki
+
Σ
K2 ab
K
-
(J)
od
ed
cijh
where we have used Eq. (19 in the third equality for the first two terms. In matrix form, we have:
VV log V =
kd K2
Σ
ed
ed
cijk
We now bound the norms of the three terms in turn. For the first term,
Σ k
Kg
Σ|Ka od
[K2K Kd
od
Σ (Jr 1a (VV faJT Jy (VV fa) D.o
1. (VV vVTTaal/
< BZ Byp
(J) & 18 By22 J J
cd
23

---

Under review as a conference paper at ICLR 2021
< < By
(32)
where we have used Eq. For the second term,
28) in the fourth last inequality and Eq.
29
in the second last inequality.
Σ K Kukk Kad
K
Kd 1|2 < B7
Σ
K
< d B7 B7 By
Σ Kidd 2 /2 BT 13 Σ
||
< Bd BiJ
Σ
VV f2) (J |111
d' B BT
(J
)al12
< 2 B
(33)
where we have used Eq. (22) in the last inequality. For the third term,
eigh
Σ1 [000) & ByT 122 By
(J )a
Σ (J
czih
d 1 11, Σ
(J (J (1 <
Σ jk
[J] ids-))
d BD By B/ J < d BT
(34)
where we have used Eq. (29) in the fourth last and second last inequalities. Finally, by assembling Eqs.
30
into Eq.
15 we have:
(p(x)) (Ø, [x)] <
2 Elu' B1 (B7 Bog + 2dB BB 8/2
+ BB7- (Blogg + ddB' By B2 p) + 3/2 227 B + d B(-i)
Elu T u B1 BygB4 Bd4+2 ( + Buerr +
= 2
3/2 BZ- B d B" B'-.))
So given the condition (iii), this residual can be neglected.
A.3 PROOF OF THE OOD GENERALIZATION ERROR BOUND THEOREM 5.5
We give the following more detailed version of Theorem 5.5 and prove it. The theorem in the main context corresponds to conclusion (ii) below (i.e., Eq. (37) below recovers Eq. (6) by taking the CSGs p', p and as the semantic-identified CSG p on the training domain and the ground- truth CSGs on the training p" and test 6* domains, respectively. Here, the semantic-identifccate requirement the learned CSG p is to guarantee that it is semantic-equivalent to the ground-truth CSG p" on the training domain, so that the condition in (ii) is satisfied.
Theorem 5.5' (OOD generalization error). Let Assumptions 5.1 and 5.2 hold. (i) Consider two CSGs p and p that share the same generative mechanisms p(a 18, v) and ply s) but have different priors Ps.v and Ps.." Then up to (67)) where o² == Elu p.]. we have for any x E supp(pæ) n supp(ā.).
Elylaj - -E(yxl < o -
log(p.. Pa. (0x0==)-)())
(35)
where J is the Jacobian of f-1 Further assume that the bounds B's defined in Theorem 5.4 iii)
hold. Then the error is negligible for any E ssup(pp)) O suupl(p.) if + > Blog B3 B and:
Ep(n) [[]]a]] - EEyya1] < B9 LEp,,, (24 log P... A P.. + V 1 logp,,e
(36)
if issup((>z) supp(ū.). where A denotes the Laplacian operator.
24

---

Under review as a conference paper at ICLR 2021
(ii) Let p' be CSG that is semantic-equivalens the CSG p introduced in (i. Then up ( we have for any supp(p() supp(pa),
E' y [r] Elul=] & o J 12 log(P,, (x,0)=/''(()
(37)
where V's == # We is the prior of CSG p under the parameterization of CSG pl, derived as the forward distribution by the reparameterization == f' from p to p'.
For conclusion (i), in the expected OOD generalization error Eq. (36), the term E 2 logPs. Ps.
A Alogps Ps.u + V Po,.s.. [l]]] is actually the score matching objective (Fisher divergence) Hyvärinen 2005) that measures the difference between Ps.v and Ps,v- For Gaussian priors p(s, v) NO Σ)
and f(8, u) = (0, E), the term reduces to the matrix trace, tr(-22- + Σ + E -- For Σ, the term vanishes.
For conclusion (ii), note that since and p' are semantic-equivalent, we have v/z Dr. and E [y[y[=]
[ly[ær (from Lemma A.2 So Eqs. 335 37 bound the same quantity. Equation (37 expresses
bound using the structures of the CSG
is considered since recovering the exact CSG p from
(x, y) data is impractical and we can only learn a CSG p' that semantic-equivalent to p.
we assume the additive noise variables μ and (for continuous y) have zero mean without loss of generality, and we denote 2 == (s, v).
Proof. Following the proof A.2 of Theorem 5.4
Proof under condition (i). Under the assumptions, we have Eq. (14) in the proof A.2 of Theo- rem 5.4 hold. Noting that the two CSGs share the same 9 and V (since they share the same v()(s, and p 8) thus f and g). we have for any E supp(p..) Suupp(p.).
Ely y [x] = g prin [M" ((V log V
Vg(V logg. V)T
g) +O(o)).
Blyla] g 2 Bplm [MT ((V log VV
g V)
VV [] + O(o)), (38)
where we have similarly defined P_ : p. o f- to Oaa)).
By subtracting the two equations, we have that up
1
Elyla] Blylæ]
2 Eplas
(Vlog(p,/ (
VVV loglog(p./b.)
1
Rew [n (Vlog(p.
log log(
2
<
2 Epin [[lm|| (I
1l + VgV log (p. /b.) T
Vg v log(p. (D. nl.
(39)
The multiplicative factor to Eiи²" μl on the right hand side can be further bounded by:
V Vlog(p./p.) =
(J log(p=/p.))
Vg JJ log(p=/0.)
< || Vlglp///Vooo///p
((V9) 00 )
log(p./p.)
log(p=//__)))
(40)
where Vg and V log(p=/// are evaluated at == f-) This gives:
[y|a Bly [r]
< 2 V | v log(p=/D.)ll.
i.e. Eq.
35 in conclusion (i). When the bounds B's in Theorem 5.4 (ii) hold, we further have:
E[yla]] [r]
[py|a]] < [ J, 1 V logp: V Voggzll2
V J |1 logg=ll2 + ||V |
25

---

Under review as a conference paper at ICLR 2021
< 20 Bo
So when o/2
1y Bo f-1. this difference is negligible for any E supp(p=) n supp(āz).
We now turn to the expected OOD generalization error Eq. (36 in conclusion (i). When supp(px) =
supp(ū,), Eq. 35 hold on P.. Together with the bounds in Theorem 5.4 iii), we have:
v Voog(p./D.)l. 1
Bp(a) [lyla] B
< 0 B BJJ px)()))
, B 9 By F. V loog(p=/p))ll
where the equality holds due to the generating process of the model. Note that the term Ep. V log(p=/p_)ll therein is the score matching objective (Fisher divergence). By Hyvärinen
2005 Theorem 1), we can reformulate it as Ep. [2 logp: A Alogg, V p=
l12] so we have:
Bp(a) B[y[a] - Blyla]] < of By
B By B p 224 logp: - A logē. + V 1og
Proof under condition (ii). From Eq. (1) in the proof A.2 of Theorem 5.4" we have for CSG p/ that for any x E supp(p')) or equivalently x E supp(pz),
E' [']y| - g' _ Pl ([ logg' Vg' (VVloo''' VV VV'g)mn + +O7))) (41)
where we have similarly defined p'g p/= o f and g' g'o (1'-1S. Since p and p' are
semantic-equivaleni with reparameterization & from p to p'. we have p(yls) = p' p'(y) (s,
thus g(s) g' g(("'(s. (s.)) for any V. So for any E supp(p=) equivalently supp(p'). we have ((-1))())) = ((())(())) (x). (F-))( = g' ((($((-1(x))) =
= ('((111)) Le.. = g. For another fact, since i/ p
(( [ by definition, we have [F fn (... V = V.V. Subtracting Eqs. (41 38
and applying these two facts, we have up to (0))), for any E supp(p!) nsupp(ā.),
['yy - Bly|a] = 2 Epcn i (V Log( /2 + (oV.l
log(d'/ Biu² T 14].
where the inequality follows Eq. 39
Using similar result of Eq. 40 we have:
E' E'[yla
[yyl
< xf V 1
(og(('//p)))
where Vig' and V Vlg((p'/p") are evaluated at z = (x)). This gives Eq. (37
A.4 PROOF OF THE DOMAIN ADAPTATION ERROR THEOREM 5.6
To be consistent with the notation in the proofs, prove the theorem by denoting the semantic- identified CSG p and the ground-truth CSG p on the test domain as p' and ÿ. respectively.
Proof. The new prior p'( (z) is learned by fitting unsupervised data from the test domain p(z). Ap- plying the deduction the proof A.. of Theorem [5.4" to the test domain, we have that under any
of the three conditions in Theorem 5.4" p(x) fx(x) indicates f# f= = f [P']. This gives
F' (f'-1 f = [-
From Eq. 12 in the same proof, we have that:
Mx = (f
* (x)
7 (<) (3's (1'4) pm)(c) = (V's 1i * (x))
From the proof[A.3 Theorem ii) (the paragraph under Eq. (4I ) the semantic- semantic-qquivalence be- tween CSGs p and p' indicates that g'. So from the above two equations, we have [[x))] = p' (a:) [0) (recall that p(x:) p' (a:) indicates [p.] 84 (T)) Since p(x) p' (x:) (that is how
26

---

Under review as a conference paper at ICLR 2021
p' is learned), we have for any supp((²) or equivalenty supp((?),
E [y|a] Ely|x]]
(42)
B ALTERNATIVE IDENTIFIABILITY THEORY FOR CSG
The presented identifiability theory, particularly Theorem 5.4, shows that the semantic-identifaabite can be achieved in the deterministic limit (/2 - oo). but does not quantitatively describe the extent
of violation of the identifiability for a finite variance oi Here we define a "soft" version of semantic- equivalence and show that it can be achieved with finite variance, with trade-off' between the "softesss" and the variance.
Definition B.1 (6- mennanti--dpendennge For 8 > 0 and two CSGs p and p', we say that they seenaantee-deendentt if there exists a homeomorphism & on S x V such that: (i) p(z]s, 8) = p' (s,) w)), (ii) (uppevlll(()) g ( 1)) 2 & where we have denoted g(s) == Efyys) and (iii) ( $ (s, (()) x (s, e(2)))l2 < E
In the definition, we have released the prior conversion requirement, and relaxed the exact likelihood conversion for p(y]s) in (ii) and the r-constancy of $ in (iii) to allow an error bounded by 5. When 5 0, the e-constancy of σ is exact, and under the additive noise Assumption 5.1 we also have the exact likelihood conversion p(y]s)= = p'(y |
(s, v)) for any E V. So O-semantic-dependenncy with the prior conversion requirement reduces to the semantic-equivalence.
Due to the quantitative nature, the binary relation cannot be made an equivalence relation but only a dependency. Here, dependency refers binary relation with reflexivity and symmetry, but no transitivity.
The d-semantic-dependenns is dependency relation if the function g E[yls] is bijective and its inverse g is 2 -Lipschhir.
Proposition B.2.
Proof. Showing a dependency relation amounts to showing the following two properties.
Reflexivity, For two identical CSGs p and p', we have p(x|s, v) = /' (arls, v) and p(yls)
s)) So the identity map as & obviously satisfies all the requirements in Definition B.
Symmetry. Let CSG be -ssmantic-dependem to CSG p' with homcomorphism & Ob-
viously Φ- is also a homeomorphism. For any (8', v') S x V, we have (z)ss' v')
p' ( (erl@ us),,"))
and [[g'(s")
(0((-1))(s+ (((s'sz))) r')]
(l'(( (s, -
8 where we have denoted (s, v) = x- (s', here. So Φ-
satisfies requirements (i) and (ii) in Definition B.1
For requirement (iii), we need the following fact: for any (1)) 5 S, - 212)||
[[ (g-(((((1))) g (g(s & +llg(s(") - g(s where the inequality holds
since g is 2 Lipschitz. Then for any s' E S, we have:
( 1)- (() (क 10-1))(6 (s/, /2)i
ton sup wo
(6), ((n) g(( -1:0011556
11s (s', (
Se sup
22
(oo (1)))
/ ((9)) (0(( (0((0-1)9()) ((11 ((1)) (9(9))-
a 1 dns sns
( ( |
+ (ost--a---
((aaa 1o0---/e - ((())
- (99))) /
dn sup
(laa
2
)(mn) - I s((3\\ + sus 1 1
11191 - 1 -
()) )
(s (laaa
sup
27

---

Under review as a conference paper at ICLR 2021
where in the last inequality we have used the fact that + satisfies requirement (ii). So p' is --semantic-dependentt to p via the homeomorphism d-1
The corresponding d-semantic-ideniiffabiity result follows.
Theorem B.3 (ô-semantic- iinatttiidddeeiffabiitttt)... Assume the same as Theorem 5.4 and Proposition B.2 and let the bounds B is defined in Theoremm5.4 iii hold. For two such CSGs p and p', if they have p(a, y) = (x. y), then they are d-semantie-dependent for any 8 2 of B (2B
log B + By +
3dB', 1.1 where d' := ds + dv.
Proof. Let : f-1 of where f and f' are given by the two CSGs p and p' via Assumption 5.1 We now show that and p' are e-semantic-dependee is a homeomorphism on S x V. and satisfies requirement (i) in Definition B.1 by construction due to Eq. 7 in the proof A.2 Theorem 5.4"
via this & for any 5 in the theorem. Obviously
Consider requirement (ii) in Definition B.1 have Eq. 25 hold for both CSGs:
Based on the same assumptions as Theorem 5.4 we
max RR [ x] - ((=) [F'[y]]] g' (2) & Bf- B' R" + 3 d By)
where we have denoted of == Ele" nd]. Since both CSGs induce the same p(ylx), so Elyla| E' E[y] This gives:
lØ(x) g (z) [E - - (E(yla) g(x))1l2
[ -
g(x)]12
B' (2 Bdd
So for any (s S V. by denoting == f(s, v). have:
[l9(s) ( (s, 2) 1))l| ||9((f)))(=z)) g (((("") =1g((>) - 12
< Bf- (2 p By Bn + By
So the requirement is satisfied.
For requirement (iii), note from the proof of Proposition B.2 that when is bijective and its inverse L-iiscchitz, requirement (ii) implies requirement (iii). So this 4 is homeomorphism that makes p 5-semantic-depeeddett to p' for any > o² B²_ (2 lugg By B" + 3 B.
Note that although the d-semantic-dependenng does not have transitivity, the above theo- rem still informative: for any two CSGs sharing the same data distribution, partieu- larly for a well-learned CSG p and the ground-truth CSG p", the likelihood conversion error suppsnnssxxlll(s)-- - (( by sup,en ov ( (s)) are bounded by o2 By + By +
and the degree of mixing v into 5, measured
3dB BdB
C MORE EXPLANATIONS ON THE MODEL
Explanations on our perspective. We see the data generation process as generating the concep- tual latent factors (s, v) first, and then generating both and y based on the factors. This follows Peters al.) (2017) Section 1.4) who promote the generation of an OCR dataset as the writer first comes up with intension to write character, and then writes down the character and gives its label based on the intension. It is also natural for medical image datasets, where the label may be diagnosed based more fundamental features (e.g., PCR test results showing the pathogen) that are not included in the dataset but actually cause the medical image. This generation process is also considered by Mcauliffe & Blei (2008): Kilbertus al. (2018) Teshima et al. (2020).
On the labeling process from images that one would commonly think of, we also view it as a s + process. Human directly knows the critical semantic feature (e.g., the shape and position of each
28

---

Under review as a conference paper at ICLR 2021
stroke) by seeing the image, through the nature gift of the vision system (Biederman 1987). The label is given by processing the feature (e.g.. the angle between two linear strokes, the position of a circular stroke relative to a linear stroke), which is a + y process.
The causal graph in Fig. [ implies that I 1 y/s. This does not indicate that the semantic factor generates image x regardless of the label y. Given the generated image is dictated to hold the given semantics regardless of randomness, so the statistical independence does not mean se- mantic irrelevance. If an image is given, the corresponding label is given by p(y(æ), which is p(s|a))p(() (y]s is) s)dss by the causal graph. So the semantic concept to cause the label through py]ss) is inferred from the image through p(s[z))
Comparison with the graph l/t: -> reviewers, under the perspective of a communication channel, where yox is a transmitted signal and Srx is the received.
Vrx- This graph is considered by one of our
If the observed label is treated as yex. the graph then implies y - This is argued at the end of item (2) in Section [ that may make unreasonable implications. Moreover, the graph also implies that y is a cause of z, as is challenged in item (1) in Section The unnatural implications arise since intervening y is different from intervening the truth"" label. We consider y as an observation that may be noisy, while the "ground-truth label" is never observed: one cannot tell if the labels at hand are noise-corrupted, based on the dataset alone. For example, the label of either image in Fig. 2 may be given by a labeler's 's random guess. Our adopted causal direction s y is consistent with these examples and is also argued and adopted by Mcauliffe & Blei (0008 : Peters et al. 20177 Section (il::ribbsrtus et al. (2018); Teshima et al. (2020).
If the observed label y is treated as Vix, the graph then implies x -> y, as is challenged in item (1) in Section 3 It is also argued by SSchllkoff et al. )0 |Peters et al. (2017) Section 1.4); Kilbertus label may be the motivation of this graph. But the graph implies that yox Il. Drx| that is, p(ytx (2r, Mrx) = p(yex (2r) and p(yrs Jr Utx) P(Uex (r). So modeling ytx (resp. Drx) does not benefit predicting Urx (resp. ytx) from T.
ct al. 2018). Treating the observed label y as 3ix and ytx as the "ground-truti
D RELATION TO EXISTING DOMAIN ADAPTATION THEORY
Existing DA theory In existing DA literature, the objective is find a labeling function h X
Y within hypothesis space H that minimizes the target-domain risk R(h) == E. (())((()))))) y)
defined with a loss function E: y x y R. Since p(z, v) is unavailable, it is of practical interest to consider the source-domain risk R(h) and investigate its relation to R(h). Ben-David -an-David al. (010a)
give a bound relating the two risks:
R(h) < R(h) + 2d (Po.Pa)
+ +min (Ep(,) (x) h (z)]]. Eppa)ll (x) h (+)I). sup |x[[X] P. [X]]
(43)
where: di (px,
XE
Here 2 denotes the o-algebra on X, di P.) is the total variation between the two distri- butions, and h' E argmingew R(h) and h² E argmingeиt R(h) are the oracle/ground-rruth labeling functions on the source and target domains, respectively (e.g. h²(x))
[[y]ar] and h <'xx) = E[ylæ] if supp(pz) supp(ūg)). Zhao et al. (2019) give a similar bound in the case of binary classification, terms of the H-divergence dæ in place of the total variance di, where H = {sign( h(x))= - h1(=)) - t): h, h² E H,t 1, E [],
Ben-David et al. (2) also argue that in this bound, the total variation di is overly strict and hard to estimate, so they develop another bound which is better known (asymptotically; omitting estimation error from finite samples):
R((h) < R(hh) (Pa.())))P) XH, I(M)))
(44)
where: dnAn(pP. P.) == sup
29

---

Under review as a conference paper at ICLR 2021
by == inf R(h) + R((h) MEH
Here dH P2) is the HAH H.HH-divergence measuring the difference between p(a) and j(x) under the discriminative efficacy of the labeling function family H. and Ky is the ideal joint risk achieved by H. Long et al. (2015) give a similar bound in terms of maximum mean discrepancy (MMD) dg in place of dHAR.
For successful adaptation, DA often makes the covariate shift assumption: h" ht (or p(y]>) p(y(z)) on supp(Pa, P²) := supp(p=) U supp(pz).
DA-DIR DA based on learning domain-invarian representations (DA-DIR) (Pan et al. 2010 Bak-
tashmotlagh et al. 2013 Long et al., 2015 Ganin et al. (016) aims to learn deterministic represen- to some representation space S, in order to achieve domain-
tation extractor r) X
representation (DIR): p(s) = p(s), where p(s) == 7g (s) and p(s) == 7# 1 [p)((s) are the rep- resentation distributions on the two domains. The motivation is that, if DIR is achieved, then the distribution difference term in bound Eq. 43) or Eq. (44) diminishes, thus the bound is hopefully tighter on the representation space than on the original data space X, so minimizing the source risk is more effective to minimizing the target risk. Let 9 S + y be labeling function the representation space. The end-to-end labeling function is effectively h = gon). The typical objective for DA-DIR thus combines the two desiderata:
min R(g o 17) + d(næ [p)) (p2).1=(7..11
nEEggE
where .(..-) is metric or discrepancy (d(g.p) = 0 - p) on distributions, and & and G are the hypothesis spaces for 7) and g, respectively.
For the existence of the solution of this problem, is often assumed stronger that there exist "y" E E and g" E G such that 7h [pm] = Vi [P,] and R(g rn" ) = R(h*). Assumption 3 of Johansson et al. 2019) further assumes covariate shift and that g* o 17" = h on supp(pe. Pg)) that is, there exist 7" E E and 9" E G such that 74 [Pa] = ni [P_.] and g o nn" h = h on supp(p=, P.). They also mention that this is not guaranteed to hold in practice.
Problems of DA-DIR Johansson al. (2019) Zhao et al. 2019) give examples where even under
as strong an assumption as Assumption 3 of Johansson et al. (2019) (i.e., covariate shift and a strong
existence assumption). the two desiderata of DI-DIR (i.e., minimal source risk R(g o 11) = R(h*) and DIR t [pz] = n ( still allow the bounds to be uselessly loose and the target risk R(g o 7) to take its worst value (particularly, the two desiderata cannot guarantee 1) 17" g g" or g o 1 = h² h supp(p.. P))). This is essentially an identifiability problem.
The examples do not contradict existing DA bounds. Consider a given representation extractor 1).
(1) Under Eq. (43. Applying the bound on the representation space S gives:
R(g n) < R(g o 17) + 2dh (1# [p=], 7# (b,,) + min [e 1o) (1s;, (s) - 37, (+)] E (107))) - (+)):
(45)
where S and 9ú are the optimal labeling functions on top of the representation extractor 7,. In the covariate shift case, DIR 7 [p=] " [P,.] and minimal source risk R(g o 1/) R(h²) are not sufficient guarantee 9;,- 9,
(Ben- et al. 20 Gong et al. 2016). Johansson et al. (019)) argue that they are still not sufficient even under their Assumption
In both examples by Johansson et al. 2019); Zhao et al. (2019), the considered 17, although achieving both desiderata, is not v/ and this 17 even renders different optimal g's: g o Fy. Johansson et al. (219) claim that it is necessary to require to be invertible to make g" = Jn and develop a bound that explicitly shows the effect of the invertibility of 17. The 1 in the examples is not invertible.
(2) Under Eq. 44). Applying the bound on the representation space S gives:
E BB( (((g(s). v)] & Ep(ss))((((ss) v)] (1 [pz), n [.))
+ inf [F ((((()) y)] + Eptan ((()))(((()))))))) gEg
30

---

Under review as a conference paper at ICLR 2021
where Pa, (n, idiyd)+ PPe.yl with idy (z, v) y and similarly for Ps.y Note that Bp(a,) [g)(s)) Ep(x, (((((()()(())))))))))))))))))))) v))] R(g 1)), the last term the r.h.s becomes:
infges Rlg (lgon)) RRg Agon where G on = {g o nj g E 9)- So the bound becomes: R(g & Rig + desg(n= [Pa). n P2)) + Xgon
(46)
This result is shown by Johansson et al. (2019). They argue that finding that achieves DIR and minimal training risk cannot guarantee tighter bound since the last term Agon may be very large.
In both examples by Johansson et al. (2019) Zhao et al. (2019). supp(p=) n supp(pE = 0. It may cause the problem that g o 77 can very different from h on supp(Ø>) even when R(g o n) R(h"). The developed bound by Johansson et al. (2019) also explicitly shows the role of support overlap. thus can be called a inpport-ivvettiilityy bound. They also give an example show that DIR (particularly implemented by minimizing MMD) is not necessary ("sometimes too strict") for learning the shared/invariant p(y(z).
(3) A third bound. Zhao et al.
(2119) develop another bound for binary classification, where )
{0, 1 and R(h) := Epiz) h" (x)
h(==)]]- Denote das (ps(p. q) := /JS(p. g) as the JS distance (Endres & Schindelin (2003). where (S q) is the JS divergence. It is bounded: 0 & das(p, g) & 1.
It is shown that (Zhao al.
2019 Lemma 4.8):
d3s(Pu. P.y) < das (1 TH (P.)) + VR(g o + R(g o n))
If dJs(Py, > (1 | n [x1) it is shown that (Zhao et al. 2019 Theorem 4.3): R(g - v) + ig o n)
(das(Pm. Pyy) - das(n (2) nM l..)))
(47)
or when the two domains are allowed to have their own repreeeennttioonllleelll labeling functions g and Corollary 4.1),
g (Zhao al. 2019
R(gon) + R(gon)
(ds((p.PPy) - das(n [Prl 7# (P.1))
So when p(y) # ply), we have dis (py. ( Pyy) O, DIR that minimizes dıs (1# [Pz], n# (.))) becomes harmful to minimizing the target risk R(g o 17).
Arjovsky al. 2019) point out that in the covariate shift case, achieving DIR p(s) p(s) implies ply) ply) (since p(s) = p(s) and p(y (s) = ))) This may not hold in practice. When it does hold, the bound above shows that DIR can limit prediction accuracy.
46 relate the source and target risks R(h), which is for bounding an objective; while our bound Eq. (36) relates the target risks of the optimal labeling functions on the source h² and target h domains, i.e., f((h*) - R((h") ) or (r(hh) - R((h") ) which measures the
Comparison with CSG Existing bounds Eqs. (43
44 45
of general and common labeling function h H. Le., R(h)
risk leap of the best source labeling function the target domain. After adaptation, the prediction
analysis (Eq. (42) shows that CSG-DA achieves the optimal labeling function on the target domain in the infinite data limit.
For Eq. (47). we are not minimizing (1 [p(x)], 7# [)) so our method is good under that view. In fact, in model the representation distributions on the two domains are p(s) ['((s, v) du and p(s) = p(s, v) du (replacing 7 [p(x)] and 7 [p())). We allow p(s, v) # p(s, v) of course and do not seek to match them. Essentially, we do not rely on the invariance of p(s(z) and p(y ()), or 1)" and h², but the invariance of p(r|s, v) in the other direction (generative direction). This thus allows p()]=) A p(s(ar) and p(y|z) * p(yl or 1)" + 7" and h / h"" so we rely on an assumption different from the idea of all the bounds above. Since the data at hand is produced following certain mechanism of nature anyway, the invariance in the generative direction p(xls, is thus
more plausible (see Section 3.2
Unfortunately, it seems that the opposite direction holds when there exist 1f and g" (unnecessarily the ones in the existence assumption or the Assumption 3 of Johansson et al. (2019) such that: Py (g" on7") "7n) [P=] and
p, (g" o 77" [i_] and that 7 is reparameterizztiio 17" due to the celebrated data processing inequality.
31

---

Under review as a conference paper at ICLR 2021
E METHODOLOGY DETAILS
E DERIVATION OF LEARNING OBJECTIVES
The Evidence Lower BOund (ELBO). A common and effective approach to matching the data distribution p (x, y) is to maximize likelihood, that is to maximize Ep- (x. logg (z, 4)] It is equiv- alent to minimizing KL(p" (æ, y) f)lpp( y)) (note that Ep- [log p" (x. y)] is a constant), so it drives p(x; y) towards y" (ar, y). But the likelihood function p(æ, y) = p(s, v, I, x, dsdv involves tractable integration, which is hard to estimate and optimize. To address this, the popular method of variational expectation expectation-maximizane maximization (variational EM) introduces tractable (has closed-form density function and easy to sample) distribution (s,, u/æ, y) of the latent variables given observed variables, and a lower bound of the likelihood function can derived:
plz,
in-
p(s. v, v. v.
pzz v) = log Ep(s.0) [p(s, v, v)] log (((,,ol..)))
g(s, u
log U8, g(8, ultr,y)
> (g(s.la.
La.p (),
where the inequality follows Jensen's inequality and the concavity of the log function. The function Cg,p(z; (z, y) is thus called Evidence Lower BOund (ELBO). The tractable distribution q(s, e|ar y) called variational distribution, and is commonly instantiated by standalone model (from the generative model) called an inference model. Moreover, we have:
Lgp v) + (L(g(s, v/2, v) [[p[s,s v/z, v))
log p(8, 22 3) ((8, 2|2, y) log p(s, U,,,0,
E Byl log ((8, u/ v)
p(s, u|,,
(o (ogg(sr
p(s, v/2,))
logp(æ, y),
maximizing (z, y) w.r.t q is equivalent to (note that the r.h.s log p(z, y) is constant of q) min- imizing KL(gllp(s, ofar, y)) which drives towards the true posterior (i.e., variational inference), and once this is (perfectly) done, L,, v) becomes lower bound of log p(a,, y) that is tight the current model so maximizing (I, 3) w.r.t effectively maximizes log p [., y), i.e. serves as maximizing likelihood. So the training objective becomes the expected ELBO, (x.))) [Cg,p(a, y)]- Optimizing it w.r.t 9 and p alternately drives p(x, y) towards p"(x, (z, y) and q(s, v|a,, y) towards p(s, u/z y) eventually. The derivations and conclusions above hold for general latent variable mod- els, with (s, v) representing the latent variables, and (xx, y) observed variables (data variables).
Variational EM for CSG. In the supervised case, the expected ELBO objective (ep (x. (x,,)) y) can also be understood the conventional supervised learning loss, i.e. the cross entropy, regularized by generative reconstruction term. As explained in the main text (Section 4). after training, we only have the model p(s, v, and an approximation q(s, v|r,,) v) to the posterior v|s, y), and prediction using p(y(ar) is still intractable. So we employ a tractable distribution q(s, v, v,yx) to model the required variational distribution as q(s, v/x,y) y) g(s, v. /)))))) where q(y)==)
(x.(())))) (
q(s, v/z² dsdv is the derived marginal distribution of y (we will show that it can be effectively estimated and sampled from). With this instantiation, the expected ELBO becomes:
Ep (v.w) (x,, ((s, v. v,ylz)
= =f 1/1( (x,
log p(s, u, u)(() dsdedady
((v(=))
g(s, v. v(z)
f (180)))3))
log (tyl)) dsdudady + / (xc, v) g(s u.la)
p(s: v, xr, v, dsdudady g(s, v(,)))
log
((vlx))
= (f
(((la)) g(s v, w/r) dadv
log (v)lz) dy) dr
+ /m(()) (1 ( ((,, (s, v/ log
p(s, v, x.
dodudy dx
((vla))
q(s, v. yzz)
32

---

Under review as a conference paper at ICLR 2021
p (2((z)
log p(s, v, u, 2.
= -Bp Bpp (ulz) (ul [log [(vl=)]] + (w) Byte..vi))))
g(8, v, vlar)
which is Eq. (T. The first term is the (negative) expected cross entropy loss, which drives the infer- ence model (predictor) g(ylæ) towards p p'(y) (y)zr) (z)) for p" (a.)-a.e. x. Once this is (perfectly) done, the second term becomes ".p((i) (g)... [log p(s, v, T, ///(/)//s/sss/)o/////))ss v, y/x]]]] which is the expected ELBO (x) ( y)] for q(,, v, y/z)). It thus drives g(s, U. y/ towards p(s, v, y/x)) and p(æ) towards p" (x). It accounts for a regularization by fitting the input distribution p (a) and align the inference model (predictor) with the generative model.
The target of q(s, v, y/æ), i.e. p(s, v, y yla), adopts a factorization p(s, v, y/2) = p(ss 2/z)p(y(s) due to the graphical structure (Fig. D of CSG (i.e. y 1 (zz, s)]s). The factor p(yls) ()) is known (the invariant causal mechanism to generate y in CSG), so we only need to employ an inference model q(s, e/(z) for the intractable factor p(s, v/ so g(s, v. y/2) q(s, ((x)p((s))) Using this relation, we can reformulate Eq. ①1 as:
Exp (2. (1,,
(x [og](()>))]]] Bp (x) f q(s, ((s) ('(v)x) ((yl)) (0,, u|x))
log p(s, v,)) dsdody
('(()
(ʃ (vlxz)
o(spvl log v,, v)
(o Bpr (x))
dsdv dy
[(yll)) Be(e,)))) p(yll)) log p(,, 0(8, v(x)
-Ep(as.)loog((/1))) (p
which is Eq. (2 With this form of q(s, v, y/a) = (ss w(r))( we have ((glr) Eg(e, [p $)] which can also be estimated and optimized using reparameterization. For pre- diction, we sample from the approximation ((ylr) instead of the intractable p(y]ær). This can be done by ancestral sampling: first sample (s, v) from q(s, u/ar), and then use the sampled to sample y from p(yls).
The conclusions and methods can also be applied to general latent generative models for supervised learning, with (s,v) representing the latent variables. When model does not distinguish the two (groups of) latent factors and v and treats them as one latent variable 2 = (s. v), following a similar deduction gives:
Bylll (((ll)) log p(z, 2))
Ep (x.)) ( Gg (2, v)1 = Rp (n.ww [log [(v|=r)] + Eep (u..v)
(48)
<(vl=))
<<>|<>)
where (112)) [a)) [p(l]=)] This is the conventional supervised variational auto-encoder (sVAE) baseline in our experiments.
Variational EM to learn CSG with independent prior (CSG-ind). See the main text in Sec- tion 4.1 for motivation and basic methods. Since the prior the only difference between p(s, u, y)
and p [s, v, x. y). we have p(s, v, x, v)/p v, v, x, y) = p(s, (///p(ss10 v) = p(s, v)/p(s)p(u) = p(uls) /p(e)) So p(s, v, u|æ² = p(vla) p p(())
p(v) p(a) p (s, v. vly)). As explained, inference models now only need approximate the posterior (s, v) Since p(s, u. ylx) p(s. w(z)p(yls) and p (ss, v. y/z) = p (ss, v p/yls( share the same p(y]s) factor, we have p(s, u|z) p(uis) (e()
p(v) p(z) p (s, u/z). The vari- and p (s, v|z) respectively, so we can ational distributions g(s, v/ær and (s, u/z) target p(s, v/r)
express the former with the latter:
q(s, e|.) p(u(s) (z)
p(u) p(x) q (s(s
Once (s, t/x) achieves its goal, such represented g(s, v/4) also does so. So we only need to construct an inference model for g+ v|z) and optimize it. With this representation, we have:
(p(ls)) pz(x)
p((x) E. (s,0o2))
p(u|s)
<()/xr= = E Egg. (p(yll)))) = -E,:=EE,,- (x(()))))
p(u) p(x) p(uls)
p(yl))
p(x)
p(r)
p ('()
p(x) (g)ar)
33

---

Under review as a conference paper at ICLR 2021
where x (y][z) E (s,o))) p(e) p(y]s)] as in the main text, which can be estimated and optimized using the reparameterization of q q v/r). From Eq. the expected ELBO training objective can be reformulated as:
p(els)
Ep (a.()))) L v)1
log (()l2) + <(vlæ:) Beca,ole) p(uls) log p(s, v,
(2.))
v(s,v(r))
log p (x)
(2.)))
+ (ulx)))
p(x)
p(x)
[p(v is) p (x))
p(s. v)p v²) p(uls) p (x)
+ p (x)
o ( (z) (e,(()))))
p(yls)
log
p(u) p(x)
p(o) p()) q (s, w/2)
log p (ar
(x.0)
+ +lgg
p(x)
p(uls))
p(v) p(yls) loog p(x)
p²(x) + log p(s))())(((sss0)
+ x(112)
Eg (1.0
q (5)
log p (x)
p(ols)
log p(z)
(p.(2.))
+ (ygg((//>) + (y]l)) E (o.))))
p(u) p(y)s)
p(x)
p (x)
p(u]s)
p(yls) log p u))(ps
+ x()(2)
E, (1
p(c)
q (s.
1 (x)
log p(x)
(p.(2.))
log p(x)
+ log
क (y)lx) T (y|z))
(x)
p(vls)
p (s, U,
Eg (x,0))))))
p(v) p(yls) log
x(y|z)
g (s, v|z)
p(((s) p(u)
(s v. x)
= -p,(())))
(ygz()(z))
T(y(>) E. (e
p(yls) log (= v/z)
where in the second-last equality we have used the definition of x (y)(z). This gives Eq. Note that (B)(2) is not used in prediction, so there is no need to sample from it. Prediction is done by ancestral sampling from q (y)lz) that is to first sample from q (s, v/ær and then from p(y]s)- Using this reformulation, we can train a CSG with independent prior even on data that manifests a correlated prior. The objective Eq. ( the training domain for domain adaptation can be derived similarly.
For numerical stability, we employ the log-sum-exp trick to estimate the expectations and compute the gradients.
E.2 INSTANTIATING THE INFERENCE MODEL
X
Although motivated from learning a generative model, the method can be implemented using a general discriminative model (with hidden nodes) with causal behavior. By parsing some of the hidden nodes as s and some others as u, discriminative model could for- malize distribution qs,, v. y/æ)), which implements the inference model and the generative mechanism p(y]s). The parsing mode is shown in Fig. which is based on the following consideration.
V
Figure 3: Parsing gen- eral discriminative model as an inference model for CSG. The black solid arrow spec- ifies p(yls) in the generative model, and the blue dashed arrows (representing computa- tional directions but not causal directions) specify q(s,, v/z) (or (s, r/zx) ğ(s, u|z)) as the inference model.
(1) The graphical structure of CSG in Fig. indicates that (v, _ y|s. so the hidden nodes for s should isolate y from v and The model then factorizes the distribution as q(s, y)z) q(s, vlz)q(y)s). and since the inference and generative models share the distribution on y/s (see the main text for explanation), we can thus use the component g given by the discriminative model implement the generative mechanism p(yss).
34

---

Under review as a conference paper at ICLR 2021
(2) The graphical structure in Fig. [ also indicates that L u/2: due to the v-structure (collider) x ("explain away"). The component g(s, u/z)) should embody this dependence, so the hidden nodes chosen should have effect those Note that the arrows in Fig. reppresent computation directions but not causal directions. We orient the computation direction v + 8 since all hidden nodes in discriminative model eventually contribute computing y-
After parsing, the discriminative model gives a mapping (s, v) = .)(2)) We implement the distribu- tion by _ (s, |/)r) = (s, u|n(1r), Egg). For all the three cases of CSG, CSG-ind and CSG-DA, only one inference model for (s, v) x is required. The component (s, v) j of the discriminative model thus parameterizes q (s. v) v|z) and q(s, u/z) for CSG-ind and CSG-DA. The expectations in all ob- jectives (except for expectations over p" which are estimated by averaging over data) are all under the respective (s. lal They can be estimated using 77(2) by the reparameterization trick (Kingma & Welling 2014). and the gradients can be pack-propagated.
We need two more components beyond the discriminative model to implement the method, i.e. the prior p(s, v) and the generative mechanism p(a[(,, v). The latter can be implemented using a gen- erator or decoder architecture comparable to the component g(s, o(z)). The prior can be commonly implemented using a multivariate Gaussian distribution, p(s, v) = (N((()) ). = E. Nov E., ma-
We parameterize E via its Cholesky decomposition, Σ = LL where L is a lower-triangul trix with positive diagonals, which is in turn parameterized as L = ( Leus o ) with smaller lower- M Luv))
triangular matrices Les and Lev and any matrix Mas. Matrices Les and Lov are parameterized by summation of positive diagonals (guaranteed via an exponential map) and a lower-triangular (excluding diagonals) matrix. The conditional distribution p(uls) required by training CSG-ind is given by p(vls) (ulMuoo Eujs). where Mujo M4 Mu Ma), Evl= = Liv (see e.g. Bishop (2006)). This prior does not imply a causal direction between and v (the linear Gaussian case of Zhang & Hyvärinen (009)) thus well serves as a prior for CSG.
F EXPERIMENT DETAILS
We use a validation set from the training domain for hyperparameter selection, to avoid overfitting to the finite training-domain data samples. The training and validation sets are constructed under 80%-20% random split in each task. We note that hyperparameter selection in OOD tasks is itself controversial and nontrivial, and it is still active research direction (You et al. 2019). It is argued that if a validation set from the test domain is available, a better choice would be to incorporate the semi-supervised adaptation task, instead of using it just for validation. As
it in learning our methods are designed to fit the training domain data and our theory shows guarantees under good fit of the naiming-domain data distribution, hyperparameter selection using a training-domain validation set is reasonable.
We align the scale of the CE term in the objectives of all methods, and tune the coefficients of the ELBOs to be their largest values that make the final accuracy near on the validation set, so that they wield the most power on the test domain while be faithful to explicit supervision. The coefficients are preferred to be large to well fit p" (a:) (and j (a:) for domain adaptation) to gain generalizability in the test domain, while they should not affect training accuracy, which is required for good fit of the training distribution.
The supervised variational auto-encoder (sVAE) baseline method a counterpart CSG that does not separate its latent variable = into and v. This means that all its latent variables in directly (i.e., not mediated by s) affect the output y. It is learned by optimizing Eq. (48) for OOD generalization, and adopts similar method as CSG-DA for domain adaptation. To align the model architecture for fair comparison, this means that the latent variable z of VVAE can only be taken as the latent variable in CSG.
All the experiments are implemented in Py Torch.
10Other approaches to introducing randomness are also possible, such employing stochasticity on the parameters/weights..c as in Bayesian neural networks (Neal) [Neal]1995) 1995). using dropout (Srivastava et al. 2014 Gal Ghahramani 2016). Here we adopt this simple treatment highlight the main contribution.
35

---

Under review as a conference paper at ICLR 2021
F.1 SHIFTED MNIST
We use a multilayer perceptron (MLP) with sigmoid activation with 784(for x)-400-200(first 100 for v)-50(for or 2)-1 y) nodes in each layer for the inference model of generative methods, and use an MLP with 50(for s)-(100(for v)+ 100))400-784(ffr 2)) nodes in each layer for their generative component p v). We use a larger architecture with 134-600--300-5-11 nodes in each layer for discriminative methods to compensate additional parameters of generative methods. For all the methods, we use a mini-batch of size 128 in each optimization step, and use the RMSprop optimizer Tieleman & Hinton 2012). with weight decay parameter 1 x 10-5, and learning rate x 10 for OOD generalization and 3 x 10 for domain adaptation. These hyperparameters are chosen and then fixed, by running and then validating using CE and DANN. For generative methods, we take the Gaussian variances of p(r|s, v) and q(s, v/zr) as 0.03². The scale of the standard derivations of these conditional Gaussian distributions are chosen small meet the intense causal mechanism assumption in our theory (e.g.. in Theorem 5.4p.
We train the models for 100 epochs when all the methods converge in terms of loss and training accuracy. We align the scale of the CE term in the objectives of all methods, and scale the ELBO terms with the largest weight that makes training accuracy near in OOD generalization. We then fix the tuned weight and scale the weight of adaptation terms in a similar way for domain adaptation. Other parameters are tuned similarly. For generative methods, the ELBO weight is 1 x 10 selected from {1,3) (1:33x10f0000 x0((-6,5)) x 106--2-1.0.1222 and the adaptation weights for SVAE-DA and CSG-DA and x 10-5 selected from 1 x
1 x 10 selected from x (,,0,-1,-,,,33
{1.3) x 10(-5,6,, 6,-)) For domain adaptation methods, the adaptation weight is x 10- 3, 3.-4) For CNBB, we
except for CDAN which adopts x 10-2, all selected from 1 x (0.0.-1 1,00-11-2,--00 2,
use regularization coefficients 1 10-1 and x 10-6 to regularize the sample weight and learned representation, and run 4 inner gradient descent iterations with learning rate x 10-3 to optimize the sample weight. These parameters are selected from a grid search where the range of the parameters are: (1,3) x 10(-2,-3,- {1,3} x 10(-4,-5.-0) {4, 1 x 10(-1,-2,-1
F.2 IMAGECLEF-DA
We adopt the same setup as in Long et al. (018 We use the ResNet50 structure pretrained on ImageNet as the backbone of the discriminative/nnerrenee model. Input images are cropped and resized to shape (3, 224, 224). For CSG, we select the first 128 dimensions of the bottleneck layer (the resized last fully connected layer of ResNet50, with output dimension 1024) as the variable v, and the output of subsequent fully connected layer with output dimension 1024 as the variable The output is produced by a linear layer built on s.
For generative methods (i.e.. our methods and sVAE(-DA)) we construct image de- coder/generator that uses the DCGAN model (Radford et al. 2013) pretrained on Cifarl0 as the backbone. The pretrained DCGAN is adapted from the PyTorch- Sachh--A---ore M The generator con- nects to the DCGAN backbone by an MLP layer to match DCGAN's input dimension 120, and generates images of desired size (3, 224, 224) by appending to DCGAN's output of size (3, 64, 64) with an transposed convolution layer with kernel size 4, stride size 4 and padding size 16.
Following Long al. (2018). we use mini-batch of size 32 in each optimization step, and adopt the SGD optimizer with Nesterov momentum parameter 0.9, weight decay parameter 5 x 10 and shrinking step size scheme with initial scale 1 x 1 shrinking exponent 0.75 and per-datum coefficient 6.25 x 10-6 For CSG methods, the Gaussian variances of p(z|s, v) and g(s, u/ær) are taken as 0.1 and 3.0, respectively. The ELBO weight is x 10- for CSG methods and 1 10-7 for s VAE, both selected from x -2-1112-1666
x -0,-7,-,,, The adaptation weights for s SVAE-DA and CSG-DA are 1 x 10 for task C-P and 1 x 10-7 for task P-C, selected from the same range. For CNBB, we use regularization coefficients 1 x 10 -6 and x 10-6
6 to regularize the sample weight and learned representation, and run 4 inner gradient descent iterations with learning rate x 10-4 to optimize the sample weight. These parameters are selected from grid search where the range of the parameters are: 1 10(-4,-5, U (3 10-6) {1,3} 10(-5,-6.777 {4).
1 x 10 - 2, -,,445
36


Reviews:
[{"review_id": "vkOkjAaNV_q", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThe paper focuses on the causal perspective of domain-generalization and domain adaptation setup for images. I.e. classifying an image under some distribution shift at test time. Similar to previous work [1-4], it assumes that some latent semantic-object representation (s) and semantic-domain representation (v) cause the image, and that these causal (generative) mechanisms (s,v ->x ) are stable, while their prior p(s,v) is prone to change at test time. It develops a new variational approach to estimate the generative distributions, and test the approach on two datasets for domain-generalization and domain-adaptation.\n\nOverall, the paper suggests a novel approach and theory to an important problem. Its major weaknesses are in its clarity and the experimental part. \n\n**Strong points**\nNovelty: The paper provides a novel approach for estimating the likelihood of p(class|image), by developing a new variational approach for modelling the causal direction (s,v->x).\nCorrectness: Although I didn’t verify the details of the proofs, the approach seems technically correct. Note that I was not convinced that s->y (see weakness)\n\n**Weak points**\nExperiments and Reproducibility:\nThe experiments show some signal, but are not through enough:\n• shifted-MNIST: it is not clear why shift=0 is much better than shift~$N(0,\\sigma^2)$, since both cases incorporate a domain shift\n• It would be useful to show the performance the model and baselines on test samples from the observational (in) distribution.\n• Missing details about evaluation split for shifted-MNIST: Did the experiments used a validation set for hyper-param search with shifted-MNIST and ImageCLEF? Was it based on in-distribution data or OOD data?\n• It would be useful to provide an ablation study, since the approach has a lot of \"moving parts\".\n• It would be useful to have an experiment on an additional dataset, maybe more controlled than ImageCLEF, but less artificial than shifted-MNIST.\n• What were the ranges used for hyper-param search? What was the search protocol?\n\nClarity:\n• The parts describing the method are hard to follow, it will be useful to improve their clarity.\n• It will be beneficial to explicitly state which are the learned parametrized distributions, and how inference is applied with them.\n• What makes the VAE inference mappings (x->s,v) stable to domain shift? E.g. [1] showed that correlated latent properties in VAEs are not robust to such domain shifts.\n• What makes v distinctive of s? Is it because y only depends on s?\n• Does the approach uses any information on the labels of the domain?\n\nCorrectness: I was not convinced about the causal relation s->y. I.e. that the semantic concept cause the label, independently of the image. I do agree that there is a semantic concept (e.g. s) that cause the image. But then, as explained by [Arjovsky 2019] the labelling process is caused by the image. I.e. s->image->y, and not as argued by the paper. The way I see it, is like a communication channel: y_tx -> s -> image -> y_rx. Could the authors elaborate how the model will change if replacing s->y by y_tx->s ?\n\n\n**Other comments:**\n• I suggest discussing [2,3,4], which learned similar stable mechanisms in images.\n• I am not sure about the statement that this work is the \"first to identify the semantic factor and leverage causal invariance for OOD prediction\" e.g. see [3,4]\n• The title may be confusing. OOD usually refers to anomaly-detection, while this paper relates to domain-generalization and domain-adaptation.\n• It will be useful to clarify that the approach doesn't use any external-semantic-knowledge.\n• Section 3.2 - I suggest to add a first sentence to introduce what this section is about.\n• About remark in page 6: (1) what is a deterministic s-v relation? (2) chairs can also appear in a workspace, and it may help to disentangle the desks from workspaces.\n\t\n[1] Suter et al. 2018, Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness\n[2] Besserve et al. 2020, Counterfactuals uncover the modular structure of deep generative models\n[3] Heinze-Deml et al. 2017, Conditional Variance Penalties and Domain Shift Robustness\n[4] Atzmon et al. 2020, A causal view of compositional zero-shot recognition\n\n\n\n\n\n**EDIT: Post rebuttal**\n\nI thank the authors for their reply. Although the authors answered most of my questions, I decided to keep the score as is, because I share similar concerns with R2 about the presentation, and because experiments are still lacking. \n\nAdditionally, I am concerned with one of the author's replies saying *All methods achieve accuracy 1 ... on the training distribution*, because usually there is a trade-off between accuracy on the observational distribution versus the shifted distribution (discussed by Rothenhäusler, 2018 [Anchor regression]): Achieving perfect accuracy on the observational distribution, usually means relying on the spurious correlations. And under domain-shift scenarios, this would hinder the performance on the shifted-distribution.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "n8An03Lahjr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The Causal Semantic Generative Model (CSG) presents an approach for learning both semantic and diverse latent causal variables in supervised settings using variational Bayes.  Contrary to many similar approaches which assume the label as the causal variable, this assumes a hidden causal variable which produces both the labels and observed features. Furthermore, this approach separates this latent causal variable into two components one for semantics, which impacts label generation and one for the diversity/variation which in combination with the semantic variable generates the observed features.\n\nThe authors present variations for learning this model which account for correlation/independence between semantic and diversity latent variables (CSG and CSG-ind) and also extend to settings where some data does contain labels (CSG-DA). The authors also show under which data generating assumptions the modeling holds and show how changes to the prior distribution of semantic and diversity variable distributions impact Out-of-Distribution Generalization error and Domain Adaptation Error.\n\nThe work presents empirical results demonstrating the effectiveness of this approach in both OOD settings (no test adaptation) and domain adaptation settings (test adaptation). For the presented experiments the authors show superior results in OOD generalization and competitive results in the domain adaptation setting. While the experiments present a compelling proof of concept, the tasks Shifted MNIST and ImageCLEF-DA are not the most representative challenges in their respective domains.  Would be interested to see performance in ColoredMNIST task for causal identification and OOD generalization as the generative structure is well understood as well as performance capabilities. The same could be said for the domain adaptation task, with Office-Home, VisDA-17, DomainNet or a variety of more challenging and representative tasks giving more empirical credibly to the experiments performed.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DOZBr3QfkR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Causal Semantic Generative model (CSG) to model semantic and variation factors separately and also provides a variational approach to learn the model.  It is proved that under some (perhaps strong) assumptions, CSG identifies the semantic factor on the training domain. Authors further show the boundedness of OOD generalization error based on the above result. Two  experiments are presented to validate the proposed method.\n\nWhile the paper seems to have many interesting ideas and theoretic results, it is poorly presented and not well prepared, making it hard to evaluate the technical correctness. I also notice that the paper changes the original latex formatting by reducing much vertical space for almost all section titles, theorems, and equations. Also, the top margin of page 8 is heavily reduced. There is approximately 2/3 - 1 page more content than that of the original format with required page limit. Given this consideration, I also lower my score. \n\nIn summary, I recommend a rejection for the present version. I highly suggest the authors revise the paper to make it more readable, self-contained, and concise, and resubmit the paper to another top conference/journal.\n\nPlease find my questions/suggestions below.\n\n1. The variables $s$, $v$, $x$, $y$ are introduced in the introduction part but then not mentioned when they are actually used in Section 3 to describe CSG. I suggest introducing the variables in Section 3.\n\n2. Authors use many italic words, and more crucially, lots of mixed uses of italic and normal fonts, particularly in Section 3. It really affects the reading flow. For introducing the CSG part, I recommend firstly introduce the model and then use another few paragraphs to present the examples for illustration.\n\n3. For the invariance principle 3.2, $p(s,v)$ is the only source of domain shift. Does it allow $p(v)$ and $p(s|v)$ (or $p(s)$ and $p(v|s)$) change while keeping $p(s,v)$ unchanged?\n\n3. (a) Sometimes you use 'Theorem' and also 'Thm.' in the main text, please be consistent. (b) The places for citation also affects the reading flow. For example, right after Assumption 5.2,  'It is a common (Janzing et al., 2009; Shalit et al., 2017; Khemakhem et al., 2019; Lee et al., 2019) sufficient condition for the fundamental (Peters et al., 2014, Prop. 7) requirement of causal minimality\n(Peters et al., 2014; 2017) for identifiability.' (c)  Some notations are not introduced, e.g.,  $q^{\\indep}(x)$ in CSG-ind. \n\n4. Why is it necessary to introduce CSG-ind? And in what case shall we consider CSG-ind? I did not find it well explained.\n\n5.  Also for Assumption 5.2, 'It is a common sufficient condition for the fundamental requirement of causal minimality for identifiability.' However, causal minimality is not equivalent to that $f$ being bijective. \n\n6. The paper should provide more details in comparing with IRM, and in fact I didn't get much information about the difference with IRM from Section 3.2.  Can the authors give more details about 'For noisy or degenerate mechanisms, ambiguity\noccurs during inference (Fig. 2), and the inferred result notably relies on the prior.'? Also, IRM is not compared in the experiments.\n\n7. In my opinion, the biggest contribution of this paper is to propose CSG, within which several theoretic results (identifiability, OOD error bound, etc.) are established under some necessary but maybe strong conditions. However, there is much less content on verifying that CSG does bring many benefits. One way is to conduct extensive experiments to verify so, but I find the experiments are not sufficient and also several methods (like IRM) are not included for comparison.\n\n** after reading rebuttal**\n\nThanks for clarifications and an improved version. I decide to increase my evaluation to 5.\n\nHowever, I think that the paper needs to take more content to illustrate the practical benefits of the proposed CSG frameworks, for the following reasons: \n- the framework is proposed based on empirical observations like 'intervening an image by e.g. breaking a camera sensor unit when taking the image, does not change how the photographer labels it', which is not mathematically rigorous; \n- the principles and assumptions are rather strong (though I understand one generally has to make assumptions in causality), and in practice it is not clear when such assumptions hold and how many applications satisfy these assumptions; \n- the interesting derivations and theorems are also based on the CSG framework, which means, if the framework is incorrect, then these results may fail; \n- the experiment settings are rather limited in the current version. I hope the authors to add further content in their next version, regardless of whether the paper gets accepted or rejected.\n\nLastly, I still feel it a bit tricky to change the original formatting in the previous submission.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper formalizes domain adaptation by taking the causal (generative) direction of dependencies p(image | class, domain).  They evaluate an ELBO surrogate loss by fitting a reverse q function that is new for this setup, and add a term to the loss that induces independence between class and domain. The paper also  proves identifiability conditions. The approach is then evaluated on two semi-synthetic and small datasets, showing some improvement. 

Reviewers were concerned about presentation and the experimental validation. The authors addresses some of the concerns in their rebuttal, but several reviewers found that the experimental evidence was still lacking, and that the authors should evaluate their approach in more standard and realistic benchmark datasets.  As a result, the paper cannot be accepted in its current form



 


Decision: Reject