Learning Causal Semantic Representation for Out-of-Distribution Prediction
Authors: Chang Liu
Xinwei Sun
Jindong Wang
Haoyue Tang
Tao Li
Tao Qin
Wei Chen
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: causal representation learning
out-of-distribution generalization
generative model
variational auto-encoder
variational inference
domain adaptation

Abstract: Conventional supervised learning methods, especially deep ones, are found to be sensitive to out-of-distribution (OOD) examples, largely because the learned representation mixes the semantic factor with the variation factor due to their domain-specific correlation, while only the semantic factor causes the output. To address the problem, we propose a Causal Semantic Generative model (CSG) based on a causal reasoning so that the two factors are modeled separately, and develop methods for OOD prediction from a single training domain, which is common and challenging. The methods are based on the causal invariance principle, with a novel design in variational Bayes for both efficient learning and easy prediction. Theoretically, we prove that under certain conditions, CSG can identify the semantic factor by fitting training data, and this semantic-identification guarantees the boundedness of OOD generalization error and the success of adaptation. Empirical study shows improved OOD performance over prevailing baselines.

Learning Causal Semantic Representation for Out-of-Distribution Prediction
Chang Liu' Xinwei Sun', Jindong Wang', Haoyue Tang"; i Tao Li
Tao Qin', Wei Chen', Tie-Yan Liu'
Microsoft Research Asia, Beijing, 100080.
2 Tsinghua University, Beijing, 100084. 3 Peking University, Beijing, 100871.
Abstract
Conventional supervised learning methods, especially deep ones, found be sensitive to cuttof-distribution (OOD) examples, largely because the learned representation mixes the semantic factor with the variation factor due their domain-specific correlation, while only the semantic factor causes the output. To address the problem, propose Causal Semantic Generative model (CSG) based on a causal reasoning so that the two factors are modeled separately, and develop methods for OOD prediction from single training domain, which is common and challenging. The methods are based on the causal invariance principle, with a novel design in variational Bayes for both efficient learning and easy prediction. Theoretically, we prove that under certain conditions, CSG can identify the seman- tic factor by fitting training data, and this -iaantiiiidetiiiictiinn guarantees the boundedness of OOD generalization error and the success of adaptation. Empirical study shows improved OOD performance over prevailing baselines.
Introduction
Deep learning has initiated new era of artificial intelligence where the potential machine learning models is greatly unleashed. Despite the great success, these methods heavily rely on the assumption that data from training and test domains follow the same distribution (i.e., the IID assumption), while in practice the test domain is often out-of-distribution (OOD), meaning that the test data distribute differently from the training data. Popular models for predicting the output (or label, response. outcome) from the input (or covariate) z have been found erroneous when confronted with a distribution change, even from an essentially irrelevant perturbation like a position shift or background change for images [91 6 102 41 2 27 These phenomena pose serious concerns on the robustness and trustworthiness of machine learning methods and severely impede them from risk-sensitive scenarios.
Looking into the problem, although deep learning models allow extracting abstract representation for prediction with their powerful approximation capacity, the representation may unconsciously mix up semantic factors s (e.g., shape of an object) and variation factors v (e.g., background, object position) due to a correlation between them (e.g., desks often appear in workspace background and beds in bedrooms), so the model also relies on the variation factors for prediction via this corrllation. However, this correlation tends to be superficial and spurious (e.g., a desk can also appear bedroom, but this does not make bed), and may change drastically a new domain, making the effect from u misleading. So it is desired to learn representation that identifies against v.
Formally, the essence of this goal distinction between s and is that only a is the cause of Causal relations better reflect basic
leverage causal relations for prediction, since the fundamental
"Correspondence Chang Liu xchangliu@microsseee Work done during an internship at Microsoft Research Asia.
com-.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

mechanisms of nature. They bring the merit to machine learning that they tend to be universal and
invariant across domains [97 87 93
96 98,, thus provide the most transferable and reliable information to unseen domains. This causal invariance has been shown to lead to proper domain
adaptation 997 123 lower adaptation cost and lighter catastrophic forgetting 87 56].
In this work, we propose a Causal Semantic Generative model (CSG) following causal consideration to separately model the semantic (cause of prediction) and variation latent factors, and develop OOD prediction methods with theoretical guarantees on identifiability and the boundedness of OOD prediction error. Addressing the complaint that OOD prediction and causality methods often require multi-domain or intervention data, we focus on the most common and also challenging tasks where only one single training domain is available, including OOD generalization and domain adaptation. where in the latter, unsupervised test-domain data are additionally available for training. The methods and theory are based on the causal invariance principle, which suggests to share generative mechanisms across domains, while the latent factor distribution (i.e., the prior p(s, vi) changes. We argue that this causal invariance is more reliable than inference invariance in the other direction adopted by many existing methods [33 J01 2 66 79). For our method, we design novel and delicate reformulations of the ELBO objective so that we avoid the cost to build and learn two inference models. Theoretically, we prove that under certain conditions, CSG can identify the semantic factor on the single training domain, even in presence of an S-U correlation. We further prove the merits from this identification: prediction error is bounded for OOD generalization, and for domain adaptation, the test dosmin prior is identifiable which leads to an accurate prediction. To sum up our contributions,
Up to our knowledge, we are the first to show theoretical guarantee (under appropriate conditions) to identify the latent cause of prediction (i.e., the semantic factor) on a single training domain, and also the first show the theoretical benefits of this identification for OOD prediction.
The results also contribute to generative representation learning for revealing what is learned. We develop effective methods for OOD generalization and domain adaptation, and achieve mostly better performance than prevailing methods on real-world image classification tasks.
2 Related Work
OOD generalization with causality. There are trials that ameliorate discriminative models towards a causal behavior. Bahadori al. 44) introduce a regularizer that reweights input dimensions based on their approximated causal effects to the output, and Shen et al. [102] reweight training samples by amortizing causal effects among input samples. Their linear input-output assumption then extended [4, 41 by learning a representation. Some recent works require identity data (finer than label) and enforce inference invariance via variance minimization 42). or leverage strong domain knowledge to augment images as an independent intervention on variation factors (79]. These methods introduce no additional generative modeling efforts, at the cost of limited capacity for invariant causal mechanisms.
with causality. There are methods developed under various causal assumptions 197, 123) or using learned causal relations (7). Zhang et al. [123], Gong al. B5 36) also consider certain ways of mechanism change. The considered causality is among directly observed variables, which may not well suit general data like image pixels where causality
Domain adaptatiin//eeneralizzii
rather lies in the conceptual latent level 75 59
To consider latent factors, there are domain adaptation 83 5 33 73
and generalization meth- ods [80 101 [T]] that learn a representation with domain-invariant marginal distribution. Remark- able results have been achieved. Nevertheless, it is found that this invariance is neither sufficient nor necessary to identify the true semanties or lower the adaptation error (54 125]; see also Appx. EE Moreover, these methods are based on inference invariance, which may not be as reliable as causal invariance (see Sec. 3.2).
There are also generative methods for domain adapttation/generaaiiaaioo that model latent factors. Cai et al. [8] and Ilse et al. [49] introduce semantic factor and a feetuue factor. They assume the two latent factors are independent in both generative and inference models, which is unrealistic. Correlated factors are then considered B.. But all these works do not adapt the prior for domain change thus resort to inference invariance. Zhang et al. [12 consider a partially observed manipulation variable, while still assuming its independence from the output in both the joint and posterior, and the adaptation is inconsistent with causal invariance. The above methods also do not show guarantees to identify their latent factors. Teshima al. (008) leverage causal invariance and

---

(s,, v)
training
test
== ==s)))=pps)p(00)
B(s. v)
(s, p/x)
domain domain
q (s,))x)
as, 1/2)
v
plys)
pOyls)
p(1ls)
px]ss v)
(x)ss
p(x|s, v)
(a) CSG
(b) CSG-ind
(c) CSG-DA
Figure 1: (a) Graphical structure of the proposed CSG. Solid arrows represent causal mechanisms p(z]s, v) and p(y(s) the undirected s-u clique represents a domain-specific prior p(s, v), and the dashed bended arrows represent the inference model q(s, v|æ) for learning. (b, e) Graphical struc- the test domain. An independent prior p (s, v)
tures of CSG-ind and CSG-DA for prediction (constructed from p(8, v)) and a new prior p(s,, v)) (the dotted 8-u clique) are introduced reflecting the intervention on the test domain. Respective inference models q (s, w/z) and 0(s, u/z) are also shown. All three models share the same causal mechanisms p (zr|s, v) and p(y]s).
adapt the prior, yet also assume latent independence and do not separate the semantic factor. They require some supervised test-domain data, and their deterministic and invertible mechanism also indicates inference invariance. In addition, most domain generalization methods require multiple training domains, with exceptions [9] that still seek to augment domains. In contrast, CSG leverages causal invariance, and has guarantee to identify the semantic factor from a single training domain, even with correlation to the variation factor.
Disentangled latent representations is also of interest in unsupervised learning. Despite em- Locatello al. [70] conclude that it is impossible to guarantee the disentanglement in unsupervised settings. Subsequent works then introduce ways of supervision like few latent variable observations [7] or sample similarity [20] 72 [04]. Identifiable VAE (57) and extensions [58] [17] leverage the data of cause variable of the latent variables and have established theoretical guarantees under diversity condition. But the works do not depict domain change thus not suitable for OOD prediction. Instead of disentangling latent factors, we focus on identifying the semantic factor s (Sec. 5.1 and its benefit for OOD prediction. Appx. Dishows more related work.
pirical success 22
43 21.
The Causal Semantic Generative Model
To develop the model soberly based on causality, we require its formal definition: two variables have a causal relation, denoted as "cause effect", if intervening the cause (by changing external variables out of the considered system) may change the effect, but not vice versa 85
88
We follow
this definition to build our model (Fig. Tab by analyzing the example that photographer takes photo in a scene as x and labels it as y. Appx. prooides more explanations under other perspectives.
(1) It is likely that neither y - (e.g., intervening the label with noise by distracting the photographer does not change the image) nor x: -> y holds (e.g., intervening an image by breaking a camera sensor unit does not change how the photographer labels it), also argued in [88] Sec. 1.4; [59]. So we introduce a latent variable to capture factors with causal relations. Also for this reason, we need a generative model (vs. discriminative model that only learns + y).
(2) The latent variable as underlying generating factors (e.g., object shape and texture, background and illumination during imaging) is plausible to cause both x (e.g., changing object shape or back- ground makes a different image, but breaking the camera does not change the shape or background) and y (e.g., the photographer would give different label if the object shape had been different, but noise-corrupting the label does not change the shape). So we orient the edges in the generative direc- tion (x., y), as also adopted in [78 88 [08]. This is in contrast to prior works [8] 44) 48 19] that treat y as the cause of a semantic factor, which, when y is also a noisy observation, makes unreasonable implications (e.g., adding noise to the labels in a dataset automatically changes object features and consequently the images, and changing the object features does not change the label).
This difference is also discussed in [88] Sec. 1.4: 59
(3) We attribute all z-y relation to the existence of some latent factor 68] "purely common cause"; 51 and exclude r-y edges. This can be achieved as long as holds sufficient information of data (e.g., with shape, background etc. fixed, breaking the camera does not change the label, and noise- corrupting the label does not change the image). Promoting this structure reduces arbitrariness in explaining z-y relation thus helps identify (part of) z. This is in contrast to prior works that treat y cause of z as no latent variable is introduced between.
[ 121 19
3

---

(4) Not all latent factors are the causes of y (e.g., changing the shape may alter the label, while changing the background does not). We thus split the latent variable as z = (s, v) and remove the + y edge, where represents the semantic factor that causes y, and describes the variation or diversity generating z. This formalizes the intuition on the concepts in Introduction (See. I.
(5) The two factors s and often have a relation (e.g., desk/bed shape tends to appear with a workspace/bedroom background), but it is usually a spurious correlation (e.g., putting a desk in bedroom does not automatically change the room as a workspace, nor does it turn the desk into a 108 79]
bed). So we keep the undirected s-v edge. This in contrast to prior works [18 49 which assume independent latent variables. Although v is not cause of y, modeling it explicitly worth the effort since otherwise would still be implicitly mixed into s anyway through the s-v correlation. We summarize these conclusions in the following definition.
Definition (CSG). A Causal Semantic Generative Model (CSG), p: is generative model on data variables z E X C R and y E Y with semantic E S C Rds and variation e E V C Rdy latent variables, following the graphical structure shown in Fig. la
:- (p(s, p(rss p(y)(,)))
3.1 The Causal Invariance Principle
Through the above process, we see that the s-v correlation embodied in the prior p(s, v) tends to change across domains. Under a causal view, this means that the domain change comes from (soft) intervention on 8 or e or both, leading to a different prior. On the other hand, the generative processes are likely causal mechanisms, so they enjoy the celebrated Independent Causal Mechanisms principle [8 98] indicating that they are unaffected under the intervention on prior. This leads to the following causal invariance principle for CSG.
Principle 2 (causal invariance). The causal generative mechanisms p(ær|s, v) and plyls) in CSG are invariant across domains, and the change of prior p(s, v) is the only source of domain change.
This invariance reflects the universality of basic laws of nature and is considered in some prior works 197, 88 10
16. Other works instead introduce domain index [18] 49 48 19] or manipulation
variables 121 57 58 to model distribution change explicitly. They then require multiple training
domains or additional observations, while such changes can also be explained under causal invariance as long as the latent variables include all changing factors.
3.2 Comparison with Inference Invariance
Most domain adaptation and generalization methods (inel. invariant-represemee
based [33] [O1], invariant- Lutent-predice based [2) 666 (79) use shared representation extractor across domains. This effectively assumes the in- variance in the other direction, i.e. inferring latent factors from observed data x. We note in its supportive examples (e.g., inferring object position from image, extracting the fundamental frequency from audio), the causal mechanism p(z|z) is nearly deterministic and invertible such that it pre- serves the information of z. Formally, for given only one single z value achieves a positive p(x|2) while all other values lead to zero. The inferred representation given by the posterior via the Bayes rule p(2|z)) x p(=)p(z) 22) then concentrates on this value, which is determined by the causal mechanism p( 2) alone, regardless of the domain -speaffe prior p(=). Causal invariance then implies inference invariance.
Figure 2: Examples of noisy (left) or degenerate (right) generating mech- anisms that lead to ambiguity in in- ference. Left: handwritten digit that may be generated as either "3" or "5". Right: Schröder's stairs that may be generated with either A or B being the nearer surface. Inference results notably rely on the prior on the dig- its/surfaces, which is domain-specific.
In more general cases, the causal mechanism may be noisy or degenerate (Fig. [ such that there are multiple z values that give positive p( 22), i.e. they all could generate the same x. Inference is then ambiguous, and the posterior relies on the prior to choose from these z values. Since the prior changes across domains (e.g., different labelers have different mindset), the inference rule then changes by nature and is not invariant reliable. To leverage causal invariance, we use different prior for the test domain (CSG-ind and CSG-DA), which gives a different and more reliable prediction than following inference invariance.
5) while the causal invariance is rather more fundamental and
'Particularly, although Mitrovic et al. 79 consider a similar causal structure and promote the invariance of p(yls), actually depends on for a given even when they are independent in the prior. So p(s[z) must depend on the nomain-speciff p(v). and a domain-invarian representation extractor does not exist.

---

4
Method
We now develop methods based on variational Bayes [55 62 for OOD generalization and domain adaptation using CSG. Appx. F.I shows all details.
4.1 Method for OOD Generalization
For OOD generalization, one only has supervised data from the underlying data distribution p= (xx y) the training domain. Fitting a CSG p == (p(s, v), ( v), p(yss)) to data by max- imizing likelihood (z..)) [logp(r; y)] is intractable, since p(z, y) == p(s, u, æ; y) dsdu where p(8 U, k) == p(8, ( v)p(y)s), is hard to estimate. The Evidence Lower BOund (ELBO) pp. v. (I, == Bu(s.ola.w)) [log ((s)))p:)))
is tractable surrogate with the help of an inference model g(s, u/æ, y) that enjoys easy sampling and density evaluation. It is known that (I. y) drives g(s, u/2, y) towards the posterior p(s, v u/2, y) == p(s,v,a.v)) mean- p(a.))
55 112
vl. ois.v fp.
while makes Cp. ge,to, (z. v) tighter lower bound of log p(r, 3) for optimizing CSG p.
However, the subtlety with supervised learning that prediction still hard, as the introduced model q(s, e/zr y) does not help estimate p(y|z)) To address this, we propose to employ an auxiliary model q(s, v, y/ targeting p(s, v, y/ y/z).. It allows easy sampling of y given x for prediction, and can also serve as the required inference model: g(s, w/z, y) = -(., where ((yl2)) == f g(s, y/r) dsde
is also determined by q(8, v, y/r)). The ELBO objective (x.() (Cp, ae, (1r, y)] then becomes:
1 ["(((( log p(s, v,,,
Ber(() Pa Dog [()l(( + Exp(() (e(e.)))))))
((y(2))
q(s, v, v,)lz)
As a functional of q(s, u. y/r) (instead of q(s, o|z, u)) and the CSG p. this objective also drives them towards their targets: the first term is the negative of the standard cross entropy (CE) loss which drives (()(z)) towards p" (vllz), and once this is achieved, the second term becomes the expected ELBO (s)pp_ s....i (z) that drives q(s, u. y|z) towards p(s, v, y/z) and p(z) towards p" (xx). Furthermore, as the target of g(s, v, y/(r factorizes as p(s, v, y/z)) = p(s, e|z)p(yls)) (due to Fig. Lab where p(y|s) is already known (part of the CSG), we can instead employ a lighter inference model (s, v/ for the minimally intractable component p(s, w/z) therein, and use q(s, e p(y))s as q(8, v, v/a.). This turns the objective Eq. ( to:
logg(v/x) + ((vlo)) Pe(o,el)) p(yls) log p(s, p(s,))p(a)ss00)
max (x Pr P
(2)
g(s, 0(x)
where ((yla) = Eg (2() [p(]]s) The expectations can be estimated by Monte Carlo after applying the reparameterization trick 622 This is the basic CSG method.
CSG-ind To actively improve OOD generalization performance, we consider using an independent prior v)) := (()) for prediction in the test domain (Fig. Ib), where p(s) and p(v) are the marginals of the training-domaii prior p(s, v). Intuitively, (s v) discards the spurious correlation between 8 and v on the training domain (e.g., the workspser"s "bed-bedroom" association), and promotes a cautious neutral belief on the unknown test-domain correlation in defence against Sped-workspace" association). Formally, p=(s v) has information
all possibilitics (e.g., a larger entropy than p(s, v) [24] Thm. 2.6.6] it reduces beaiing-domminn-pee and encourages reliance on the causal mechanisms for better generalization. It also amounts to applying the do-operator [85 to Fig. representing randomized experiment by independently soft-intervening s or U. In this way, causal invariance is properly leveraged, making a different and more reliable prediction than following inference invariance, Our theory below also shows that p "ss, v) leads to a smaller generalization error bound (Thm. Remark).
Methodologically, we need the test-domain inference model q (/,z)z) for prediction p² (yllx se [P(r/s))) (e,(e [[]]ss)], but also need q(s, v/z) for learning on the training domain. To save the cost of building and learning two inference models, propose to
(s, v|x) represent q(s, u(z).
Noting that their targets are related by p(s, v|x) p(a.v) x+(x), p(s.a) p-(a)
r(a) p p w/x), we formulate g(s, u|z) (=(o)) p(z) q (s, u/z) accordingly, so that this g(s, 2/22) achieves its target if and only if q (s, f/z)
5

---

does. The objective Eq. ① then becomes:
p(s, v) (* p(y) log (s, (s, (l),),,
max Bp((a.)) (yga())z) <(yl>)
Bg v())))
(3)
(8,
P: Pool
where π (v]x) Byy (ss,)))r) p(a.)) [=,00 (Note (s, v) is determined by p(s, v) in the CSG p.)
4.2 Method for Domain Adaptation
In domain adaptation, also has unsupervised data from the underlying data distribution ji (xx) on the test domain. We can leverage them for better prediction. According to the causal invariance prin- ciple 2 we only need new prior p(s, v) for the test-domain CSG == (f(s, p(a|s, v), p(yss)
(Fig. Ic Fitting test-domain data can be done through the standard ELBO objective with the lest-domain inference model ū(s, u/zr):
max (x) (x)], where Cp. (x) Eg(s,ola) log p(s. p(s, e)p(((sss)) 9.
(4)
ū(s,
Prediction is given by P(y|z) = [((,)/= [p(y|s)]] Similar to the CSG-ind case, we still need ((s, v/z) for fitting training-domain data, and we can also avoid a separate q(s, v/o) model by representing it using ğ(s, ufæ:). Following the same relation between their targets, we let g(s, u|z)
p(a) pla.)) p(m) P(a,o) 7(s, u|z), which reformulates the same training-domant objective Eq. (I as:
u(s.0) p(s, p(y (s) log
((s. (1s.))((rsss 7(x, u/x))
max Epp( log (v(2)) + <(llx) (l(s.
(5)
where =(ylxr = B ((s..) p(uls)]- The resulting method, termed CSG-DA, solves both optimization problems Eqs. simultaneously.
4.3 Implementation and Model Selection
To implement the three CSG methods, we only need one inference model in each. Appx. H2 shows its construction from general discriminative model (e.g., how to select its hidden nodes as s and v). In practice often has much larger dimension than y. making the first supervision term overwhelmed So we downscale the second term.
by the second unsupervised term in Eqs. 2 3 5
As recently emphasized (39,, OOD method should include model selection method, since it nontrivial and significantly affects performance [95] 120]. For our methods, we use a validation set from the training domain for model selection. This complies with the OOD setup, and is also suggested by our theory below which gives guarantees based on a good fit to the training-done domain data distribution. For CSG-ind/DA, the learned predictor targets the test domain, so we do not use it directly for evaluating validation accuracy, but by normalizing xy)lz). Appx. F3 shows details.
Theory
We now establish theory for the identification of the semantic factor (cause of prediction) and subsequent merits for OOD generalization and domain adaptation. We focus on the distribution-level generalization instead of from finite samples to unseen samples under the same distribution, so we only consider the infinite-data regime. Appx. shows all the proofs and auxiliary theory.
Latent variable identification is hard 65 Assumptions are thus required draw definite conclusions.
116 70) as it is beyond observational relations 51 81
Assumption 3. (Additive noise) There exist nonlinear functions f and with bounded derivatives up to the third-order, and independent random variables μ and v, such that p(x]s, v) P (x f(s v)). and p(y|s) = (y g(s)) for continuous or p(y(s) = Cat (s))) for categorical y. (Bijectivity) Assume is bijective and g is injective.
The additive noise assumption is widely adopted in causal discovery 51 17 It disables expressing
the same joint in the other direction 122. Thm. 8: 86) Prop. 23] so that CSG unnecessarily indicates
inference invariance. For this reason, we exclude GAN [37] and Now-based [61] implementations.
Bijectivity is a common assumption for identifiability
[86] Prop. 17: 1 109] requirement of causal minimality p. 20121 88) Def. 6.33). Particularly, and may otherwise have dummy dimensions that / and
51 100 68, It sufficient
88
Prop. 7.4] for the more fundamental [86] Prop. 88
86
g simply ignore, raising another ambiguity against identifiability. On the other hand, according the

---

commonly acknowledged manifold hypothesis [15 311, we can take X' as the lower diver-dimensional data manifold and such bijection exists as a coordinate map, which is an injection the original data space and also allows ds + dy < dx.
5.1 Identifiability Theory
We first formalize the goal of identifying the semantic factor.
Definition 4 ceemannic-identffcation a homeomorphismrf & on S x V. such that (i) its output dimensions in S is constant of v: 4 (s, u) = g. v'), Vu, e' E V (hence denote o (s. v) 4 (s)), and (ii) it is a reparameterization of the ground truth CSG p": 2 [22,ll = Ps... p" (a]s, v) = p(z] (s,, v)) and p" (y]s) ) = p(y) (s)).
We say learned CSG p is semantic- if there exists
Here, 2g [P,,e] denotes the pushed- forward distributior of p by D, i.e. the distribution of o v) when (s) ~ Ps.u- As the ground-truth CSG could at most provide its information via the data distribution p" (x, y), well-learned CSG that achieves p(z, y) (xx v) still has the degree of freedom in parameterizing (s, v). This is described by this reparameterization & (Appx. Lemma 9 At the heart of the definition, the e-constancy of & implies that & is semantic-presee preserving: the learned model does not mix the ground-truth to the ground- truth s. The definition can thus be seen as the semantic equivalence (Appx. Def. IO Prop. 14p to the ground -ruth CSGp".
into its s, so that the learned holds equivalent information
For related concepts, this identification cannot be characterized by the statistical independence between and a (vs. [8 49 [21]), which is not sufficient 770 nor necessary (due to the existence of spurious correlation). It is also weaker than disentanglement 44 11, which additionally requires the learned e to be constant of the ground-truth s. The following theorem shows that semantic- identification can be achieved on single domain under certain conditions.
Theorem (semantic-identiffallitt)) With Assumption 3 a CSG p is semantic-identifed, if it is well-learned such that p(z, v) = (z(x,,) u). under the conditions that p(s, v) and log p" (8, v) are bounded up to the second-order and that (i) 1/0 + oe where i == Elu" pl) or (ii) Pu (e.g. a Gaussian) has a.e. non-zero characteristic function.
Remarks. (1) (Condition and Intuition) Compared with the multi-domain case [8 93 2] identifi- ability on a single training domain comes at a cost and requires certain conditions. One may imagine that in some extreme cases e.g., all desks appear in workspace and all beds in bedrooms, it is impossi- ble to distinguish whether labels the object or the background (unlearnable OOD problem (119). The theorem finds an appropriate condition that excludes such cases: when log p" (s, v) is bounded, deterministic 8-0 relations are not allowed as they concentrate p"(s, v) on a lower-dimensional subspace in S V thus make it unbounded.
also leads to the intuition of identifiability: a bounded log p" (s, v) indicates a stochastic s-v relation, so mixing the ground-trume diversity/unceerainng of this v. As prediction is made via the inferred 8, this worsens prediction accuracy thus violates the "well-learned" requirement. Compared with discriminative models, CSG makes more faithful inference, and its causal structure leads to proper description of domain change.
into the learned s makes the inference of more noisy due to the intrinsic
(2) In condition (i), 1/02 measures the intensity of the causal mechanism p(x|s,, v)). When it is large, the "strong" p(ar|s, v)) helps disambiguating values of (s, v) in generating given The formal version in Appx. Thm. 5" shows a quantitative reference for large enough intensity, and Appx. B gives non-asymptotic extension showing how the intensity trades-off the tolerance of equalities in Def. Condition (ii) goes beyond inference invariance. It roughly implies that different (s, v) values a.s. produce different p(r)s, (r]ss,0 v). so their roles in generating become clear which helps identification.
(3) The theorem does not contradict the impossibility result by Locatello et al. [70], which considers disentangling each latent dimension with an unconstrained (s, v) - (a, y), while we only identify as a whole, with the e + y edge removed which breaks the s-v symmetry.
A transformation is homeomorphism if it is continuous bijection with continuous inverse. 'The definition of & [p,.e] requires & be measurable. This is satisfied by the continuity of as homeomorphism (as long as the Borel o-field is considered) [13] Thm. 13.2].
'To be precise, the conclusions are that the equalities in Def. 4 hold asymptotically in the limit 1/0 + 00
for condition (i), and hold a.e. for condition (ii).
7

---

5.2 OOD Generalization Theory
Now we show the benefit of is bounded. Note the optimal predictor E [y] on the test domain is defined by the corresponding ground-truth CSG p" which differs from p only in the test-domain prior p (s, u) (Principle 2
for OOD generalization that the prediction error
Theorem 6 (OOD generalization error).
g With Assumption 3 for a semantic-identifed CSG p on the
training domain with semontic-preserving reparameterization b we have up to O(a)).
(x) | EE [ & o,, Es,,. 1. Loo((,,,
(6)
where B', and B'g bound the 2-norms of the Jacobians of f-1 and g, respectively, and Ps.o # [e is the test-domain prior under the parameterization of the CSG p.
In the bound, the term F | log.P...//P...).l. is the Fisher divergence measuring the difference between the two priors. As the prior change is the only source of domain change, this term also measures the "OODness" in terms of the effect on prediction. The bound also shows that when the causal mechanism p(ar|ss, v) is strong (small o₂),, it dominates prediction over the prior change, as the generalization error becomes small. Compared with other methods, using a CSG enforces causal invariance, the boundedness of OOD generalization error becomes more plausible in practice.
The bound also shows the advantage of CSG-ind (Sec. 4.1) The Fisher divergence is revealed (28 to have similar behavior as the forward KL divergence Ps." H KL IlP...) that it is very sensitive to the insufficient coverage of Ps," on the support of P..v 44 109], since PP...) is infinitely large on the uncovered region. As the independent prior p3,u has a larger support than Pa... it is less likely to miss the support of Pa... so it induces a generally smaller Fisher divergence. CSG-ind thus generally has smaller OOD generalization error bound than CSG.
Remark.
5.3 Domain Adaptation Theory
CSG-DA (Sec. 4.2 learns a new prior Ps.. by fitting unsupervised test-domain data, with causal mechanisms shared. If the mechanisms are semantic-identifed, the ground- -tuth test-domain prior Ps.u can also be identified under the learned parameterization, and prediction is made precise.
Theorem 7 (domain adaptation error). With conditions Tf 5 for semantic-identifer CSG p on the training domain with semantic-preservinn reparameterization ④;, if its new prior Ps.y is well- learned such that p(z) = p" (z), then Ps,v = 4 [.. and E[yyl]] = E [ylæ] for any x E supp(ēt).
Different from existing domain adaptation bounds (Appx. Ep. Theorems 7 allow different inference models in the two domains, thus go beyond inference invariance.
6 Experiments
For OOD generalization baselines, there is not much choice beyond the standard CE loss optimization, as domain adaptation methods require test-domain data and most domain generalization methods degenerate to CE with training domain. The exception within our scope is causal discriminative method CNBB [1]. For domain adaptation, we consider well well-acknowledged methods DANN [3),
DAN 73 CDAN 74 and recent compelling methods MDD [14 and BNM 25
(shown in Appx.
Tables
2 Appx.
G
shows more details, results, and discussions. 10
Shifted-MNIST. We first consider an OOD prediction task on MNIST to classify digits "O"s and "I"s. To make a spurious correlation, in the training data, we horizontally shift each "0" at random by ón ~ (-5,, 12) pixels, while each "1" by 5, ~ A5 1²²) pixels. We consider two test domains with different digit-position distributions: each digit is not moved 6 = O₁ = the first, and is shifted at random by do, 81 NN 22) pixels in the second. We implement all methods using multilayer perceptron which is not naturally shift invariant. We use a larger architecture for non-generative methods to compensate the additional generative component of generative methods.
The performance is shown in Table toop 2 rows). For OOD generalization, CE is misled by the more noticeable position factor due to the spurious correlation to digits, and resorts to random guess (even
Forr categorical y, the expectation of y is taken under the one-hot representation.
$See Appx. Thm. 66' for the formal version.
9 the induced operator norm for matrices (not the Frohenius norm).
10 Codes are available at ht/ptss: //ggi wig.thu..co... http/changglln 1000//uusaaaa wem.ntic-geneaatiieeeeeeeeeeeeee
8

---

Table I: Test accuracy (%) by various methods (ours in bold) for OOD generalization (left 4 cols) and domain adaptation (right 5 cols) on Shifted-MNIST (top 2 rows), ImageCLEF-DA (middle 4 rows) and PACS (bottom 4 rows) datasets. Averaged over 10 runs. Appx. Tables show more results.
task
CE
CNBB
CSG
CSG-ind
DANN DAN
CDAN
MDD
CSG-DA
60 = 61 = 60,61 (1002~
42.9-3.1 54.7:33
81.4
82.614.0
40.943.0
40.4+2.0
41.040.5
41,9+08
97.6-140
47.8=15 59.2:24
61.7:3.6
62.3+2.2
46.2:077
45.640.7
46.340.6
45.8+033
72.0=9.2
C-P
63.5=03 72.7=11
73.6=06
14.0=1.3
74.3+0.5
69.2+004
74.5+003
74.1=0.7
7,.1
P-C
91.2=03 91.7=02 92.3=0.4
92.7=0.2
91.5==0.6
89.8+0.4
93.5=004
92.
0334
I-pP
74.810.3 75.4.006 76.910.
77.2.10.2
75.0.in.6
74.5.004
76.710.3
76.810.
77.4
P-I
83.910.1 88.710.5 90.410.3
90.9
86.0.n.3
82.2.102 90.6..0.3
90.211.1
91.1 1.5
others- P
97.8=000 96.9=02
97.7=002
97.8+002
97.6=002
97.6+0.4
97.0=00.
9..6=0.3
9,2
others- A
88.110.1
73.1
88.5.006
88.61006
85.9
84.5
84.0
88.1
0.7
others-XC
1..9=1. 1..2+112 84.4 84.611..
19.9=114 81.9
1.
1.
84.7
others S
79.1 43.3+12 80.7 81.1
75.2+ 77.4+3.1 71.8 80.2+ 0.8
worse) when position is not informative for prediction. CNBB ameliorates the position confusion, but not as thoroughly without modeling causal mechanisms. In contrast, our CSG gives more genuine predictions unseen domains, thanks to the identification of the semantic factor. CSG-ind performs even better, justifying the merit of using an independent prior for prediction. For domain adaptation, CSG-DA achieves the best results. Existing adaptation methods even worsen the result (negative transfer), as the misleading position representation gets strengthened on the unsupervised test data. CSG is benefited from adaptation in a proper way that identifies the semantic factor.
ImageCLEF-DA is a standard benchmark for domain adaptation [.. It has 12 classes and three domains of real-world images: Caltech-256, ImageNet, Pascal VOC 2012. We select four OOD prediction tasks CeP, I+PP that have not seen good enough results. We adopt the same setup as [74). As shown in Table I mmiddle+ 4 rows), CSG-ind again achieves the best OOD generalization results, and even outperforms some domain adaptation methods. Our CSG also outperforms the baselines mostly. For domain adaptation, CSG-DA is the best in most cases and on par with the best in others.
PACS is a more recent benchmark dataset [69] It has 7 classes and is named after its four domains: Photo, Art, Cartoon, Sketch; each contains images of certain style. We follow the same setup as (39; particularly, we pool together all domains but the test one as the single training domain. Results in Table I [bottomm 4 rows) show the same trend. CSG-DA even outperforms most domain generalization methods reported in (39], which are fed with more information. Appx. Tables 23 also show the results on an even larger dataset VLCS 30) which present similar observation.
Visualization. Appx. Fig. 5 visualizes the learned models using LIME 91. The results show our methods focus more the semantic regions and shapes, indicating a causal representation is learned.
Dataset analysis. The results indicate our methods are more powerful on Chifted-MNSS" and PACS (and VLCS) than ImageCIEF--AA This meets the intuition of identifiability (Thm. 55 Remark (I)): the random position or pooled training domain shows a diverse v for each (while with a misleading spurious correlation), so identification is better guaranteed to overcome the spurious correlation.
Ablation study. To show the benefit of modeling and u separately, we compare with counterpart of CSG that treats and v as whole (equivalently, v + y is kept; see Appx. F.1.4 for method details). Appx. Tables 23 show that our methods outperform this baseline in all cases. This shows the separate modeling makes CSG consciously drive semantic representation into the dedicated variable s.
7 Conclusion and Discussion
We propose Causal Semantic Generative model for single-domain OOD prediction tasks, which builds upon a causal reasoning, and models the semantic (cause of prediction) and variation factors separately. By the causal invariance principle, we develop novel and efficient learning and prediction methods, and prove the semantic-identifaabee the success of adaptation. Experiments show the improved performance over prevailing baselines.
and the subsequent bounded generalization error and
Notably, we answered the questions in the recent farseeing paper 998 on causal representation learning: we found an appropriate condition under which "causal variables can be recovered", and provided "compelling evidence on the advantages (of causal modeling) in terms of generalization" Also, separating semantics from variation extends to broader examples. Neural nets are found to

---

change their prediction under a different texture [3 151.
Adversarial vulnerability [07 38 67
extends variation factors to human -ippeneeptttibllee features, i.e. adversarial noise, which found to have strong correlation to the semantics [SO]. The separation also matters for fairesss when sensitive variation factor may affect prediction. This work also inspires the dual connection between causal representation learning ("fill in the blanks" given graph) and causal discovery ("link the nodes" given observed variables). Our theory shows the identifiability condition for causal discovery (the additive noise assumption) also makes causal representation identifiable. Studying the general connection between the two tasks an interesting future work.
References
[1] The imageclef-de challenge 2014. http::/www.immagclle.oo//20444 2014.
[2] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 02893, 2019.
[3] Y. Atzmon, F. Kreuk, U. Shalit, and G. Chechik. A causal view of compositional zero-shot recognition. Advances in Neural Information Processing Systems, 33, 2020.
[4] M. T. Bahadori, K. Chalupka, E. Choi, R. Chen, W. F. Stewart, and J. Sun. Causal regularization. arXiv preprint arXiv: 02604, 2017.
[5] M. Baktaasmottlagh, M. T. Harandi, B. C. Lovell, and M. Salzmann. Unsupervised domain adaptation by domain invariant projection. In Proceedings of the IEEE International Confer- ence on Computer Vision, pages 769-776, 2013.
[6] S. Beery, G. Van Horn, and Perona. Recognition terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pages 456 456-473, 2018.
[7] S. Ben-David, J. Blitzer, Crammer, A. Kulesza, F. Pereira, and W. Vaughan, A theory of learning from different domains. Machine learning. 79(1-2):151-11) 2010.
[8] S. Ben-David, T. Lu, Luu, and D. Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 129-136, 2010.
[9] Y. Bengio, T. Deleu, N. Rahaman, N. R. Ke, S. Lachapelle. O. Bilaniuk, A. Goyal, and C. Pal. A meta-transfer objective for learning to disentangle causal mechanisms. 8th International Conference Learning Representations, ICIR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020.
[10] M. Besserve, N. Shajarisales, B. Schölkopf, and D. Janzing. Group invariance principles for causal generative models. In International Conference on Artificial Intelligence and Statistics, pages 557-565. PMLR, 2018.
[11] M. Besserve, A. Mehrjou, R. Sun, and B. Schölkopf. Counterfactuals uncover the modular structure of deep generative models. In Proceedings of the International Conference Learning Representations (ICLR 2020), 2020.
[12] I. Biederman. Recognition--y-componee logical review, 94(2):115, 1987.
theory of human image understanding, Psycho-
[13] P. Billingsley. Probability and Measure. John Wiley & Sons, New Jersey, 2012. ISBN 978-1-118-12237-2.
[14] C. M. Bishop. Pattern recognition and machine learning. springer, 2006.
[15] W. Brendel and M. Bethge. Approximating CNNs with bag-of- models works surprisingly well on ImageNet. In Proceedings of the International Conference on Learning Representations (ICLR 2019), 2019.
[16] P. Bühlmann. Invariance, causality and robustness. arXiv preprint arXiv: 11X.:v88122.82333 2018
10

---

[17] P. Bühhmann. J. Peters, J. Emesst et al. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics, 42(6)) 2014.
[18] R. Cai, Z. Li, P. Wei, J. Qiao, K. Zhang, and Z. Hao. Learning disentangled semantic representation for domain adaptation. In Proceedings of the Conference of IJCAI, volume 2019, page 2060. NIH Public Access, 2019.
[19] D. C. Castro, I. Walker, and B. Glocker. Causality matters medical imaging. Nature Communications, 11(1):1-10, 2020.
[20] J. Chen and K. Batmanghelich. Weakly supervised disentanglement by pairwise similarities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3495-3502, 2020.
[21] R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pages 2610-2620, 2018.
[22] X. Chen, Y. Duan, R. Houthooft, J. Schulman, Sutskever, and P. Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2172-2180, 2016.
[23] C.-Y. Chuang, A. Torralba, and S. Jegelka. Estimating generalization under distribution shifts via domain-iivanttei invariant representations. In International Conference on Machine Learning, pages 1984-1994. PMLR, 2020.
[24] T. M. Cover and J. A. Thomas. Elements of information theory. John Wiley & Sons, 2006.
[25] S. Cui, S. Wang, J. Zhuo, L. Li, Q. Huang. and Q. Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3941-39500 2020.
[26] B. Dai and D. Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
[27] A. D' D'Amorr, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Fisenstein, M. D. Hoffman, al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.00: 2020.
[28] C. Durkan and Y. Song. On maximum likelihood training of score-based generative models. arXiv preprint arXiv:2101.09258, 2021.
[29] D. M. Endres and J. E. Schindelin. A new metric for probability distributions. IEEE Transac- tions on Information theory, 49(7): 1987))885-1186) 2003.
[30] C. Fang, Y. Xu, and D. N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pages 1657-1664, 2013.
[31] Fefferman, S. Mitter, and H. Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 29(4))983-1049 2016.
[32] Y. Gal and Z. Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the International Conference on Machine Learning, pages 1050-1059, 2016.
[33] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-ade adversarial training of neural networks. Journal of Machine Learning Research, 17:1-35, 2016.
[34] Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. ImageNet- trained CNNs are biased towards texture; increasing shape bias improves accuracy and ro- bustness. In Proceedings of the International Conference on Learning Representations (ICLR 2019), 2019.

---

[35] M. Gong, K. Zhang. T. Liu. D. Tao, C. Glymour, and B. Schölkopf. Domain adaptation with conditional transferable components. In International Conference on Machine Learning, pages 2839-2848 2016.
[36] M. Gong, K. Zhang, B. Huang, C. Glymour, D. Tao, and K. Batmanghelich. Causal generative domain adaptation networks. arXiv preprint arXiv: 100..80444444 000400 2018.
[37] I. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 1672-2680, Montréal, Canada, 2014. NIPS Foundation.
[38] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations (ICLR 2015), 2015.
[39] I. Gulrajani and D. Lopez-Pazz search of lost domain generalization. arXiv preprint 2XXi:2007.014433 2020.
[40] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[41] Y. He, Z. Shen, and P. Cui. Towards non-i.i.d. image classification: A dataset and baselines. arXiv preprint arXiv: /906. 02899, 2019.
[42] Heinze-Deml and N. Meinshausen. Conditional variance penalties and domain shift robustness. stat, 1050: 13, 2019.
[43] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. Beta-VAE, Learning basic visual concepts with a constrained variational framework. In Proceedings of the International Conference on Leurning Representations (ICLR 2017), 2017.
[44] I. Higgins, D. Amos, D. Pfau, S. Racaniere, L. Matthey, D. Rezende, and A. Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 1812.0023, 2018.
[45] P. Hoyer, S. Shimizu, A. J. Kerminen, and M. Palviainen. Estimation of causal effects using linear non-gaussian causal models with hidden variables. International Journal of Approximate Reasoning, 49(2):362-378 2008.
[46] F. Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adver- sary? arXiv preprini arXiv: /5/ 1,05/01, 2015.
[47] A. Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Muchine Learning Reseurch, 6(Apr):695-709, 2005.
[48] M. Ilse, M. Tomezak, and Forré. Designing data augmentation for simulating interventions. arXiv preprint arXiv:2005.01887 2020.
[49] M. Ilse, J. M. Tomezak, C. Louizos, and M. Welling. DIVA: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322-348. PMLR, 2020.
[50] Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pages 125-136, 2019.
[51] D. Janzing, Peters, J. M. Mooij, and B. Schölkopf. Identifying confounders using additive noise models. In Proceedings of the 25th Conference Uncertainty in Artificial Intelligence (UAI 2009), pages 249-257, AUAI Press, 2009.
[52] D. Janzing, E. Sgouritsa, O. Stegle, Peters, and B. Schölkopf. Detecting low-complexity unobserved causes. In 27th Conference Uncertainty in Artificial Intelligence (UAI 2011), pages 383-391. AUAI Press, 2011.
[53] J. Jiang, B. Fu, and M. Long. Transfer-eamminn-lliaay https://github..com Transfer- Learning Bratiinng-bbbbrry Library 2020.
12

---

[54] F. D. Johansson, D. Sontag, and R. Ranganath. Support and invertibility in representations. The 22nd International Conference on Artificial Intelligence and Statistics, pages 527-536, 2019.
[55] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-22 233, 1999.
[56] N. R. Ke, O. Bilaniuk, A. Goyal, S. Bauer, H. Larochelle, C. Pal, and Y. Bengio. Learning neural causal models from unknown interventions. arXiv preprint arXiv:1910.01075. 2019.
[57] I. Khemakhem, D. P. Kingma, P. Monti, and A. Hyvärinen. Variational autoencoders and nonlinear ICA: A unifying framework. In S. Chiappa and R. Calandra, editors. The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo. Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pages 2207-2217, 2020.
[58] I. Khemakhem, R. P. Monti, D. P. Kingma, and A. Hyvärinen. ICE-BeeM: Identifiable conditional energy-based deep models. arXiv preprint arXiv: 15337, 2020.
[59] N. Kilbertus, G. Parascandolo, and B. Schölkopf. Generalization in anti-causal learning. arXiv preprint arXiv: 1812000524 2018.
[60] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998) 2014.
[61] P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, 2018.
[62] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR 2014), Banff, Canada, 2014. ICLR Committee.
[63] D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pages 3581-3589, 2014.
[64] M. Kocaoglu, S. Shakkottai, A. G. Dimakis, C. Caramanis, and S. Vishwanath. Entropic latent variable discovery. arXiv preprint arXiv:1807. 10399, 2018.
[65] T. C. Koopmans and O. Reiersol. The identification of structural characteristics. The Annals of Mathematical Statistics, 2 11 1950.
[66] D. Krueger, E. Caballero, J. Jacobsen, A. Zhang, Binas, R. L. Priol, and A. Courville. Out- of-distribution generalization via risk extrapolation (REx). arXiv preprint vrXiv:2003.00683 2020.
[67] A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv 2016.
preprint
[68] C. M. Lee, C. Hart, J. G. Richens, and Johri. Leveraging directed causal discovery to detect latent common causes. arXiv preprint 11X11:1910.10174, 2019
[69] D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
[70] F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem. Chal- lenging common assumptions in the unsupervised learning of disentangled representations, In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Confer- ence on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 4114-4224 Long Beach, California, USA, 09-15 Jun 2019. PMLR.
[71] F. Locatello, M. Tschannen, S. Bauer, G. Rätsch, B. Schölkopf, and O. Bachem. Disentangling factors of variation using few labels. arXiv preprint 4rXiv:1905.012 2019.
13

---

[72] F. LLocatello, B. Poole, G. Rätsch, B. Schölkopf. O. Bachem, and M. Tschannen. Weakly- supervised disentanglement without compromises. In International Conference on Machine Learning, pages (348-6359) PMLR, 2020.
[73] M. Long, Y. Cao, J. Wang. and M. Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105, 2015.
[74] M. Long, Z. Cao, J. Wang, and M. Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, pages 1640-1650, 2018.
[75] D. Lopez-Paz, R. Nishihara, S. Chintala, B. Schölkopf, and L. Bottou. Discovering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
2017.
[76] C. Louizos, U. Shalit, M. Mooij, D. Sontag, R. Zemel, and M. Welling. Causal effect inference with deep latent-variable models. In Advances in Neural Information Processing Systems, pages 1446-6456 2017.
[77] S. Magliacane, T. van Ommen, T. Claassen, S. Bongers, P. Versteeg, and J. M. Mooij, Domain adaptation by using causal inference to predict invariant conditional distributions. In Advances in Neural Information Processing Systems, pages 10846-10856, 2018.
[78] J. D. Mcauliffe and D. M. Blei. Supervised topic models. In Advances in Neural Information Processing Systems, pages 12 121-128, Vancouver, Canada, 2008. NIPS Foundation.
[79] J. Mitrovic, B. McWilliams, J. C. Walker, L. H. Buesing, and C. Blundell. Representation learn- ing via invariant causal mechanisms. In International Conference on Learning Representations, 2021. URL htR.itttpp::/ppeeevieee.. http://foorrm11159222200000
[80] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In International Conference Machine Learning, pages 10-18, 2013.
[81] K. P. Murphy, Machine learning: a probabilistic perspective. MIT press, 2012.
[82] R. M. Neal. Bayesian learning for neural networks. PhD thesis, University of Toronto, 1995.
[83] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 2292):199-210, 2010.
[84] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killcen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, segh-performance deep learning library. Advances in Neural Information Processing Systems, 32:8026-800 2019.
[85] J. Pearl. Causality. Cambridge university press, 2009.
[86] J. Peters, J. M. Mooij, D. Janzing, and B. Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15(1):2009-2055 2014.
[87] J. Peters, P. Bühlmann, and N. Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 7875):477-1022 2016.
[88] J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. MIT press, 2017. 2017.
[89] F. Qiao, L. Zhao, and X. Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12556- 12565, 2020.
[90] A. Radford, L. Metz, and Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Y. Bengio and Y. LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
14

---

[91] M. T. Ribeiro. S. Singh, and C. Guestrin. "Why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144, 2016.
[92] T. Richardson, P. Spirtes, et al. Ancestral graph Markov models. The Annals of Statistics, 30 (4):962-1030, 2002.
[93] M. Rojas-Carulla, B. Schölkopf, R. Turner, and Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):109-1142, 2018.
[94] J.-W. Romeijn and J. Williamson. Intervention and identifiability in latent variable modelling. Minds and machines, 28(2):243-265 264, 2018.
[95] Rothenhäusler, Meinshausen, Bühlmann, and Peters. Anchor regression: heteroge- neous data meets causality. arXiv preprint arXiv:1801.06229, 2018.
[96] B. Schölkopf. Causality for machine learning, arXiv preprint arXiv: 191199111..00... 2019.
[97] Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. M. Mooij. On causal and anticausal learning. In International Conference on Machine Learning (ICML 2012), pages 1255-1262. International Machine Learning Society, 2012.
[98] Schölkopf. F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-66) 2021.
[99] E. Sgouritsa, D. Janzing, J. Peters, and B. Schölkopf. Identifying finite mixtures of nonpara- metric product distributions and causal inference of confounders. In Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence (UAI 2013), pages 556-575, AUAI Press, 2013.
[100] U. Shalit, F. D. Johansson, and D. Sontag. Estimating individual treatment effect: generaliza- tion bounds and algorithms. In Proceedings of the 34th International Conference on Machine Coarning-Volume 70, pages 3076-3085. JMLR.org, 2017.
[101] S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi. Generalizing across domains via cross-gradient training. In Proceedings of the International Conference on Learning Representations (ICLR 2018), 2018.
[102] Z. Shen, P. Cui, K. Kuang, B. Li, and P. Chen. Causally regularized learning with agnostic data selection bias. In 2018 ACM Multimedia Conference on Multimedia Conference, pages 411-419. ACM, 2018.
[103] I. Shpitser, R. J. Evans, T. S. Richardson, and M. Robins. Introduction to nested Markov models. Behaviormetrika, 41(1):3-39, 2014.
[104] R. Shu, Y. Chen, A. Kumar, S. Ermon, and B. Poole. Weakly supervised disentanglement with guarantees. In International Conference on Learning Representations, 2020.
[105] P. Spirtes, C. N. Glymour, R. Scheines, and D. Heckerman. Causation, prediction, and search. MIT press, 2000.
[106] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-199) 2014.
[107] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. In Proceedings of the International Conference on Learning Representations (ICLR 2014), 2014.
[108] T. Teshima, I. Sato, and M. Sugiyama. Few-shot domain adaptation by causal mechanism transfer. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages
2020.
15

---

[109] L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. In International Conference on Learning Representations (ICLR 2016), pages 1-10, 2016.
[110] T. Tieleman and G. Hinton. Lecture 6.5-RMSprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
[111] T. Verma and J. Pearl. Equivalence and synthesis of causal models. UCLA, Computer Science Department, 1991.
[112] M. Wainwright, M. I. Jordan, al. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008.
[113] J. Wang. C. Lan, C. Liu, Y. Ouyang, and T. Qin. Generalizing to unseen domains: A survey on domain generalization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4627-4635 International Joint Conferences on Artificial Intelligence Organization, 2021. Survey Track.
[114] Y. Wang and D. M. Blei. The blessings of multiple causes. Journal of the American Statistical Association, 114(528) 2019.
[115] K. Weinberger and L. K. Saul. Unsupervised learning of image manifolds by semidefinite programming. International Journal of Computer Vision, 70(1):77-90, 2006.
[116] Y. Yacoby, W. Pan, and F. Doshi-Velez. Learning deep bayesian latent variable regression mod- els that generalize: When non-identifiability is problem. arXiv preprint 1r11v:1911.005: 2019.
[117] M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. CausalVAE, Structured causal disentanglement in variational autoencoder. arXiv preprint arXiv:2004. 006997, 2020.
[118] L. Yao, S. Li, Y. Li, M. Huai, J. Gao, and A. Zhang. Representation learning for treatment effect estimation from observational data. In Advances in Neural Information Processing Systems, pages 2633-2643, 2018.
[119] H. Ye, C. Xie, T. Cai, R. Li, Z. Li, and L. Wang. Towards a theoretical framework of out-of-distribution generalization. arXiv preprint arXiv:2 106::2006.04496, 2021.
[120] You, X. Wang, M. Long, and M. Jordan. Towards accurate model selection in deep unsupervised domain adaptation. In International Conference on Machine Learning, pages 7124-7133, 2019.
[121] C. Zhang, K. Zhang, and Y. Li. A causal view on robustness of neural networks. In Advances in Neural Information Processing Systems, 2020.
[122] K. Zhang and A. Hyvärinen. On the identifiability of the post-nonlinear causal model. In Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009), pages 647-655. AUAI Press, 2009.
[123] Zhang, B. Schölkopf, K. Muandet, and Z. Wang. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pages 819-827, 2013.
[124] Y. Zhang, T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pages 7404-7413, 2019.
[125] Zhao, R. T. Des Combes, K. Zhang, and G. Gordon. On learning invariant representations for domain adaptation. In International Conference Machine Learning, pages 7523-7532, 2019.
16


Reviews:
[{"review_id": "xyGSc6OzQwp", "reviewer": "Reviewer_iwHL", "summary": "The paper addresses the problem of out-of-distribution prediction by proposing a causal model which captures the underlying data generation process. The causal model consists of a latent variable which generates both the input images and the associated output labels. The novelty of the paper stems from the fact that the authors split the latent variable into two parts. The first part (s) captures the semantics of the generation process and the second part (v) captures the spurious domain specific variations. Since maximizing the likelihood of the data under this model is intractable, the authors propose an approximate and efficient training and inference mechanism. The authors show theoretically that their model will learn semantic features which are responsible for the final prediction. The claims are backed by a collection of experiments on simple datasets.  \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper seems to be technically sound and the claims are backed up by the experiments. The generative model proposed in the paper is fairly intuitive and well motivated. The ideas are reasonably novel, though incremental. I do have a couple of objections with the way the paper currently stands: \n\n1. The paper is very poorly written and it is extremely difficult to follow the arguments and the motivations behind the arguments. The authors really need to revise the presentation of the paper to make it easier to understand before it is accepted. \n\n2. It is not clear what is the full benefit of splitting the latent variable z, which is generating the x and y. While the authors claim to show some results of the ablation study in the appendix by combining the two variables, the error bars of the two methods have a significant overlap (especially on the non MNIST datasets). \n\n3. There are no insights provided around what semantic features is the model learning on the ImageCLEF-DA and PACS datasets. \n\n4. The experimental section backing up the claims of out-of-domain generalization is somewhat weak. The images in the ImageCLEF-DA datasets are not very different visually. In the PACS datasets the images are fairly different however pooling across three domains to create the training set is bound to make the task easier. It would be interesting to see the performance of the model on one domain when trained on images only from a single domain. \n\nOriginality\nThe ideas in the paper, although somewhat incremental, are fairly original. \n\nQuality\nThe research presented in this paper is of good quality however the presentation of the research is extremely poor. \n\nClarity\nThe paper is rather poorly written and extremely hard to follow. Furthermore it is certainly not a self contained paper with a lot of important details left to the appendix.  \n\nSignificance\nI believe more thorough empirical evidence is required to infer the true significance of this approach. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not have any concerns around societal impact and they seem to have called out most of the limitations of their model.\n", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns pertaining to the methodology proposed in the paper or the data sets used in the paper. \n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "rdjPgSMLWfE", "reviewer": "Reviewer_9SYR", "summary": "The authors propose a specific graphical model, Causal Semantic Generative model, which is based on the principles of casual reasoning. The graphical model attempts to explicitly distinguish between meaningful semantic factors of variation and spurious correlations. The design of the model is targeted at out-of-domain generalization and domain adaptation performance. The authors describe an approach for training such a model using variational inference, and demonstrate theoretically and experimentally that their approach gives superior OOD generalization and domain adaptation performance compared to relevant baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: While latent variable models aiming to separate out some form of content and style have been proposed before, the specific model proposed by the authors along with in-depth intuitive and formal justification represents a novel contribution.\n\nQuality: The authors make theoretical claims about the properties of their model, and experimental claims about the value of their formulation. As far as I can tell their theoretical claims are well supported.\n\nHowever, I feel their specific experiment claims become lost in jargon. Specifically, the authors make a number of strong claims based on intuition (e.g. lines 68, 72-73, 79-80, 83, 100-101, all of section 3.2). These are central claims to the justification of the authors’ model, and are made repeatedly and strongly, but I don’t feel that the experimental evidence is commensurate with the strength of the authors’ claims. For example, there are a number of works the authors do no cite that outperform their method on ImageCLEF-DA or PACS (e.g. Wang, Qian, and Toby Breckon. \"Unsupervised domain adaptation via structured prediction based selective pseudo-labeling.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020, Huang, Zeyi, et al. \"Self-challenging improves cross-domain generalization.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.). Similarly, despite making claims about the shortcomings of many other works (lines 66-67, 74-75, 81-85), the authors do not compare to these models. \n\nIf the authors' claims are solely supported by empirical evaluation, then they should be evaluated against actual state of the art methods. Moreover, if the authors are going to make such strong claims about the shortcomings of many other models they need to demonstrate that their approach is superior. Instead of pure accuracy results, these claims could alternatively be supported by analysis showing that the representations learned by their method do indeed separate out true signal from spurious signal better than competing methods. But they do not provide any such evidence as it stands.\n\nClarity: The organization of the paper is strong, although as a reader I felt there was a large amount of unnecessary language and appeal to intuition which hides what the authors actually did and how they evaluated their hypotheses. Section 3.1 has a lot of this in my opinion, with the e.g. very formal Principle 2 and section 3.2.\n\nSignificance: Learning models that are robust to domain shift is one of the major open challenges in machine learning right now in my opinion, and methods that attempt to improve this robustness are very much worthwhile.\n\n\nOverall this paper presents an interesting model along with theoretical analysis and a small amount of experimental validation. The model itself and theoretical analysis may merit publication on its own, but I am concerned with the amount and strength of claims made by the authors in proportion to the evidence for their claims. As a reader I felt like formality and jargon were used as a tool to convince me of the correctness of claims, instead of evidence. It would not take a huge amount of experimental evidence or experimentation at a large scale to push me to accept, and in fact it may be as simple as toning down the unfounded rhetoric, but as written I do not feel the authors have supported their claims with good-faith experimentation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss the limitations of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "0LiyxT7j5zF", "reviewer": "Reviewer_bSHd", "summary": "This paper focuses on addressing the domain generalization problem through the lens of causality. Instead of learning environment-invariant representation/inference p(z|x) like [31,97], the paper proposes to learn the other direction p(x|z). The paper proposes a novel graph, where it assumes the latent semantic-object representation (s) and domain-specific representation (v) cause the image. The generative mechanism works (s,v) -> x, while the prior p(s, v) can be prone to environment change at test time.   Therefore, the model proposes a CSG-ELBO to maximize the likelihood of E_{p(x, y)}[log p(x,y)], and further derive a CSG-ind to disentangle v from s. The paper conducts three experiments on different datasets for both OOD generalization and domain adaptation to show the advantage of their proposed model.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. I think the causal graph proposed to estimate (x -> y) is novel. Instead of directly maximizing p(y|x), the paper proposes to model the causal direction (s,v->x) and uses the variational method to approximate the original objective function. \n2. The paper establishes a theory for the identification of the semantic factor (cause of prediction) and subsequent merits for OOD generalization and domain adaptation.\n\nCons:\n1. The paper is somewhat elusive. Maybe because I'm not an expert in Causality/Domain-Adaptation area, it took me very long to understand section 3. It would be good to make this paragraph more understandable. Due to the clarity issue, I'm not convinced of the design of the proposed CSG. \n2. The used datasets are somewhat toy and not quite representative in the field. Is there any chance to apply your method on ColoredMNIST or Office to see its performance?\n\nQuestions:\n1. In line 166, why do you suggest p(x|z) is nearly deterministic and invertible? Are you suggesting that once we know the object position z, we can deterministically generate an image x? While in line 173, you suggest that multiple z can lead to the same x, therefore, identifying z from a single x is difficult, which seems to contradict the original assumption. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "-IdX84HbccO", "reviewer": "Reviewer_SkAw", "summary": "This paper proposes a Causal Semantic Generative model (CSG) for out-of-distribution (OOD) generalization. It assumes a semantic latent variable $s$ and a variation latent variable $v$, following the causal graph that $s,v$ are the causes of the input $x$ and $s$ is the only cause of the label $y$. The prior $p(s,v)$ is domain-specific, but the causal mechanisms $p(x|s,v)$ and $p(y|s)$ are invariant across domains. To learn the model, the authors adopt variational inference by introducing $q(s,v|x)$ and training with ELBO. The authors consider single-domain OOD prediction and domain adaptation. For single-domain, they advocate to use an independent prior p(s)p(v) to eliminate spurious correlation. For domain adaptation, a new prior is learned on the target domain.\n\nThe authors provide theoretical analysis, showing that under additive noise assumption, the ground-truth CSG is identifiable, and subsequently the OOD generalization error and domain adaptation error are bounded. Experiments are conducted on Shifted MNIST, ImageCLEF-DA and PACS datasets. The proposed model achieves superior performance compared to CNBB in single domain and DANN, DAN, CDAN in domain adaptation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Causal learning and OOD generalization are very important problems. This paper proposes a novel model that's well motivated, theoretically justified, and empirically achieves good results. I'd like to recommend acceptance.\n\nThe paper is well-written and I enjoyed reading it. I just found two typos. Line 203 \"use $q(s, v|x)p(y|x)$ as $q(s, v, y|x)$\", should be $p(y|s)$ instead of $p(y|x)$. Line 321 \"it is scales larger\", remove \"is\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential negative societal impact are not discussed in the paper.\n\nAs the proposed CSG is a generative model, if you can visualize the effects of $s$ and $v$ on $x$ and $y$, it will help understand what the model has learned.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attempts to robustify models learned on a particular dataset to OOD examples which could plausibly be addressed if the model learns based on the semantics of training observations, rather than spurious factors. The proposed solution is—inspired by recent work in causality—to introduce a pair of latent variables modelling the semantic relationship between inputs and labels, and the natural spurious variations/noise in the input, respectively. Neural variational inference is used to jointly learn this generative model. Experiments show it works on some relatively toy data, which is arguably complex enough to prove the concept.

I personally like the idea, and would like to have seen a richer discussion here. I have read all the reviews and rebuttals carefully, it is seems like the main issues had to do with clarity, the strength of some of the claims in the paper, and the appropriateness of comparisons and datasets used. I feel in the rebuttals the authors have responded to the latter two points to my satisfaction, and if the concerned reviewers are not satisfied with those rebuttals, they have not made this known to me. The authors have committed to reviewing the strength of some the assertions about the approach in their rebuttal, and I count on them to make good on this in the preparation of their camera ready. Regarding clarity, I did find section 4 a bit hard to read due to the density of mathematical expressions. My suggestion would be to summarise the findings of section 5 in the main paper, move the rest of section 5 to the appendix, and expand section 4 to give a little bit of visual breathing room and more detail where needed. I don't think this sort of change warrants asking for further reviews, and believe this paper will be of interest to NeurIPS attendees. I recommend acceptance.
Decision: Accept (Poster)