Invariant and Transportable Representations for Anti-Causal Domain Shifts
Authors: Yibo Jiang
Victor Veitch
Venue: NeurIPS 2022 Accept (2022)
Keywords: causality
spurious correlation
invariant prediction
domain shift
image classification
domain adaptation

Abstract: Real-world classification problems must contend with domain shift, the (potential) mismatch between the domain where a model is deployed and the domain(s) where the training data was gathered. Methods to handle such problems must specify what structure is held in common between the domains and what is allowed to vary. A natural assumption is that causal (structural) relationships are invariant in all domains. Then, it is tempting to learn a predictor for label $Y$ that depends only on its causal parents. However, many real-world problems are ``anti-causal'' in the sense that $Y$ is a cause of the covariates $X$---in this case, $Y$ has no causal parents and the naive causal invariance is useless. In this paper, we study representation learning under a particular notion of domain shift that both respects causal invariance and that naturally handles the ``anti-causal'' structure. We show how to leverage the shared causal structure of the domains to learn a representation that both admits an invariant predictor and that also allows fast adaptation in new domains. The key is to translate causal assumptions into learning principles that disentangle ``invariant'' and ``non-stable'' features. Experiments on both synthetic and real-world data demonstrate the effectiveness of the proposed learning algorithm. 

Invariant and Transportable Representations for Anti-Causal Domain Shifts
Yibo Jiang' and Victor Veitch
Department of Computer Science, University of Chicago Department of Statisties, University of Chicago Google Research
Abstract
Real-world classification problems must contend with domain shift, the (potential) mismatch between the domain where model is deployed and the domain(s) where the training data was gathered. Methods to handle such problems must specify what structure is common between the domains and what varies. A natural assumption is that causal (structural) relationships are invariant in all domains. Then, it tempting to learn a predictor for label that depends only on its causal parents. However, many real-world problems are "anti-causal" in the sense that Y cause of the covariates X-in this case, Y has no causal parents and the naive causal invariance is useless. In this paper, we study representation learning under a particular notion of domain shift that both respects causal invariance and that naturally handles the 'anti-causal" structure. We show how to leverage the shared causal structure of the domains learn representation that both admits an invariant predictor and that also allows fast adaptation in new domains. The key is to translate causal assumptions into learning principles that disentangle "invariant" and "non-stable" features. Experiments on both synthetic and real-world data demonstrate the effectiveness of the proposed learning algorithm. Code is available at
cow wommybjiaang///// (bbiiaang/ACTIRR ACTIR
Introduction
This paper concerns the problem of domain shift supervised learning, the phenomenon where predictor with good performance in some (training) domains may have poor performance when deployed novel (test) domain. There are two goals when faced with domain shifts. First, we would like to learn fixed predictor that is domain-invarian in the sense that it has good performance domains. Note, however, that even a good domain-invariant predictor may still be far from optimal in any given target domain. In such cases, we't d like to learn an optimal domain-specinfe predictor as quickly as possible. Then, the second goal is to learn a representation for our data that transportable in the sense that, when given data from to learn a domain-specific predictor using only a small number of examples.
domain, we
the representation
Domain shifts plague real-world applications of machine learning and there is a large and active literature aimed at mitigating the problem [e.g., Arj+119 Vei+21 [BM16; Rot+2TT Wan+21 /ho+21
Koh+2I Zhu+200 Cai+21 Shi+21 Sag+ 19; Bai+20 SSSI9: Zhe+21l Liu+21] Lu+21]. Empirically, when domain shift methods are applied to wide-ranging benchmarks, there is no single dominant
-inded,, it's common for methods that work well in one context to do worse than naive
empirical risk minimization (i.e., ignore the shift problem) in another context [Koh+21] C2 This problem is fundamental: is impossible build predictors that are robust to all possible kinds of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure I: Causal model for the data generating process. We decompose the ohserved covariate X into latent parts defined by their causal relationships with Solid circles denote observed variables, while shaded circles denote hidden variables.
shifts Accordingly, it necessary to specify the manner in which the training and test domains related to each other: that is, what structure is common to all domains, and what structure can vary across them. Then, make progress on the domain shift problem, the task is to identify structural assumptions that are well matched to real-world problems and then find methods that can achieve and transportability under this structure.
In this paper, we rely on a particular variant of the assumption that structural causal relationships invariant across domains, but certain "non-causal" relationships may vary. The motivation is that relationships fixed by the underlying dynamics of a system are the same regardless of the domain [PIS77. A similar causal domain-structure assumption is already well-studied in the domain-shift literature [e.g.. PBM16; 19) Rot+21] Roj+ 18] MBST3]. There, the goal is to predict the target label Y using only causal parents (reconstructed from observed features X). In particular, the aim is to build predictors that do not rely on any part of the features that is causally affected by Y. However, many problems, can happen that the observed covariates X are all caused by Y. that the causal parents of Y are the empty set. In this case, the naive causally invariant predictor is vacuous.
The purpose of this paper is to study alternative causal notion of domain shift that handles such "anti-causal" (Y causes X) problems, and that maintains the interpretation that structural causal relationships are held fixed across all domains. Specifically,
1. We formalize the anti-causal domain shift assumption.
2. We show how the causal domain shift assumption can be leveraged to find an invariant predictor and transportable representation.
3. We use this as the basis a concrete learning procedure for domain-invariant and domain- adaptive representations.
2
Causal Setup
The first step is to make precise what structure is preserved across domains, and what structure varies. Once we have this, we'll make the notions of invariant and fast-adapting predictor precise.
iid In each domain we have observed data (X;, Yi) ~ pe, where pe is domain-specific data- generating distribution. We will mainly consider problems where we have (finite) datasets sampled from multiple distinct domains at training time, and wish to make predictions on data sampled from some additional domains not observed during training. This paper discusses classification problems where Y; is discrete.
To formalize the causal structure assumption, we'll introduce additional latent (unobserved) variables. First, Z, (subset of) the causes of X. Second, U, a confounding variable that affects both Z and Y; see Figure [ Conceptually, Z are the factors of variation where the association with Y can vary across domains. The confounder U is the reason the association can vary. The relationship between Y and Z induced by U needs not be stable across domains. Slightly abusing notation, we have that Z,, U,, X consider.
lid pe in each domain e. We can now describe the set of domain shifts
For any given predictor, it's possible adversarially construct domain where that predictor does poorly.

---

Definition 1. (Compatible Cati-Causal Shift Domains) Distributions {P*) (over X, Y) are compati- ble anti-causal shift domains if the following conditions hold. First, there are unobserved variables 7, U such that causal graph in Figure 1 holds in all domains. Second, there is fixed distribution P and for each e there some distribution Q (U) such that P ( J P(X, Z UU)Q (U).
Informally: The causal structure is fixed all domains (implying the conditional distribution over X, Z given U is the same). We allow only the distribution of the unobserved common cause U to vary.
This notion of domain shift respects the preserved- seerveddccusa---eee causal-structure desiderata. However, it is not obvious that it suggests any useful algorithms for learning robust predictors. This is the subject of the remainder of the paper.
2.1 Invariant Prediction
Intuitively, a predictor will be robust against domain shifts it depends only on causes of X that have stable relationship with Y in all domains. In our setup, these the factors of variation that are not included in Z. Accordingly, we want a predictor that depends only on the parts of X that are not causally influenced by Z.
To formalize this notion, we'll use the concept of counterfactual invariance to Z [ei+21]. A function f is counterfactually invariant to Z if (()()) = f(X(=') for all z, z, where (=) denotes the counterfactual X we would see had Z been 2. Learning a predictor that does not depend on the factors of variation Z that induce unstable relationships means learning a predictor that is counterfactually invariant to Z.
Part of our goal in the following will be to learn counterfactually invariant predictor. This causal notion of invariance closely related to the notion of invariance that requires a predictor to be the risk minimizer in all domains [PBM16 1rj+19]. Specifically, under the anti-causal structure, if prior distributions (Y) are the same in all domains, then Veitch et al. [Vei+21 Thm. 4.2] shows that if f is the counterfactually invariant predictor with the lowest risk in any training domain, it is also the counterfactually invariant predictor with the lowest risk in all domains. Even when P< is not constant across domains memains-there's 's prior shift- imposing counterfactual invariance should still improve out-of-domain performance since removes the domain domain-varying part of the features X. This is supported by experiments in Appendix B.2 Throughout the paper, we use the term "invariant" in the sense of counterfactual invariance.
2.2 Causal Decomposition of X
To go further in our formalization, we'll need another idea from Veitch al. [Vei+2]] : the decompo-
sition of X into (latent) parts defined by their causal relationship with Z. We define X 1 to be the part of that not causally affected by Z. More precisely, - is the part of such that any function of is counterfactually invariant if and only if it a function of x alone (that is, f(X) is X measurable). Under weak conditions on Z, X is well defined (e.g., discrete Z suffices) Vei++21
We also introduce X₂ to denote the part of X that not invariant to Z. We make the extra assumption that X does not have causal effect on X (the other direction is ruled out by the definition of X The meaning of this assumption that the Z-specific parts of X can be disentangled it's possible to vary the other parts of X without affecting X., For example, we can change the object of the image without changing the background. This is a non-trivial assumption about the structure of the anti-causal shift domains, baked into the causal compatibility assumption by the absence of an arrow between X₂ and X...
the sense that
2.3 Rapid Adaptation
Even if P(Y) is held fixed, the optimal counterfactually invariant predictor g(X) is unlikely to be the best predictor in any given domain. The reason is that it excludes Z-dependent information that may in fact be highly predictive in a given domain. Given new domain e, we would like to be able to quickly learn a new predictor f"(X that updates the invariant predictor with domain-specific associations. This update should only depend on X; because the relation between X and Y is stable. Accordingly, we want to learn representation h(X) that encapsulates the information in
3

---

X,. Moreover, this should be done in a manner such that, given g(X) and h(X), we can learn good predictor for p with only a small number samples.
To formalize this, we'll introduce the following domain-specifie predictors:
(X) = ( + (X)
(2.1)
Here (xX is logits. In words: f adds correction to the invariant predictor that is logit-linean linear in the learned representation ((X). We take the correction to be linear map because, once is known, linear maps very sample efficient to learn. Accordingly, we can formalize "learn h such that we can rapidly adapt in new domains" as "learn such that the domain-specific predictor with optimal M has low risk under pe". Then, our second goal to learn such representation h.
2.4 Learning Goals
We have now given a causal formalization the domain transfer scenario we consider, and formaliza- tions of the problems of learning invariant and rapidly adapting predictors. With the causal notation in hand, our goal can plainly stated. We want to learn an invariant g( () and a domain-varying h(X) with the following properties.
1. g(X) depends only on X
2. g(X) has low risk in each training domain.
3. ((X) depends only on X
4. (x) = g(X) + MXX) should have low risk in each training domain, where M the linear map that minimizes the domain-specific risk.
The challenge is that we do not observe Z (or U) for any data point and do not know the decomposition of into X and X,, As we will see in the next section, we can find a relaxation that is enforced with observed data which relies on the particular anti-causal structure.
3 Observable Signature
The first problem we must confront is how to learn a function g(X) that depends on X alone, and (xX) that depends on X2 alone. Strictly speaking, learning such functions precisely would be impossible, even we observed Z [Vei+2]]. The reason is that we have access to only observational data, but the two parts of X are defined in terms of the underlying causal structure. Instead, the best we can hope for is to require that g(X) and h(X) satisfy the observable implications of the causal structure. That is, the properties of the causal assumption that actually be measured using the observed data.
When Z is observed, signature for g(X) is that g(X) is conditionally independent of Z given Y challenging to learn g(X) and h (X), as these representa-
[ei+21 But when 7 is unobserved, tions of X are intimately tied to Z.
The key observation is that there are two relations that connect (XX) and h(X). The first relation comes from the causal graph. In particular, we want to impose the requirement that g(X) X) and ((X) satisfy the observable implications of the causal structure. The next theorem gives such an observable implication, which can serve an observable signature of the causal decomposition.
Theorem 2. If (()) depends only on x and h(X) depends only X then, under the causal graph in Figure ! g(X) 1 h(X) Y.
The usefulness of this theorem is that the conditional independence statement can be measured from data, and enforced in the model training.
The second relation is subtler and it comes from our formulation of Nomain-specifii predictors f" Specifically, f", linear combination of g and h. should minimize the risk in every domain. And by only allowing coefficients and h would learn "unstable" information (X2).
to change, we hope would capture "invariant" information (X!)
Therefore, try to enforce conditions and in Section 2.4 we can learn g and jointly to minimize domain-specific risk, while enforcing that g(X) and h(X) satisfy the conditional independence

---

implied by the causal structure. Since the observable signature is only necessary (not sufficient) for the causal decomposition and there could be multiple candidates of g and h pairs that can parameterize Ie in the aforementioned way, it's not guaranteed to recover and h that only rely on X and X; respectively. However, does strongly constrain the functions we can learn. And, as we will see in Section 6 enforcing the signature does lead to predictors with good robustness and fast adaptation properties.
3.1 Causal Regularization
We enforce g(X) and h(X) to satisfy the conditional independence condition via regularization. Specifically, want define regularizer such that its value goes to zero whenever the conditional independence requirement is met. In general, measuring conditional independence is hard [Zha+ 12 Fuk+07 TSS16). Instead, enforce weaker condition that uses the following fact.
Lemma 3. If A B D. then, E[A (B E[B|D)]
This is necessary but sufficient condition for conditional independence. However, easy to compute and leads to good results in practice (as shown in Section 6). With this identity in hand, we define Ceond (A, B, D), the (infinite data) conditional independent regularization term between random variables A, B given random variable D, and its empirical estimate Gcond as follows:
Ceond (A, D) = B[A (B - D[)
Grond (a; by d,) =
1 Σ
ai 6
Σ
b;
#j d
j:dy=d:
where {a:}i=1- (b, 12-1 {d are samples of A, B and D. Here, the conditional random variable D is assumed to be discrete which is true for the case in this paper.
4 Learning Algorithm
We have reduced our goal to learning g and h such that
1. For training domains, ( has low risk.
2. For given domain e, there exists M such that f*(X) g(X) + M*h(X) is the risk minimizer that domain.
3. 9(X), ((X) constrained by the conditional independence regularization.
We now design specific algorithm that accomplishes the learning task. First, parameterize the learning problem in a form that's convenient to use with neural networks. Then, we translate our learning objectives into bi-level optimization problem. Finally, we introduce practical algorithm to solve the bi-level optimization problem. We name our method ACTIR for Anti- Causal Tranportable and Invariant Representation.
4.1 Reparameterizztioon
In principle, we could learn two completely separate functions g(X) and h(X). However, this can be wasteful. For instance, in vision, many low-level features can be reused by different predictors. To address this, we first notice that we can always rewrite (2.I) as follows
(4.1) That is, shared representation & followed by fixed linear map W defining y and domain- specific linear map W defining M*{(X) P The task is then learning the representation (which we'll parameterize by neural network), and the invariant and domain-spectie linear maps.
(*(X) (W W=)O(X).
In fact, a further simplification is possible: we can fix W to
The reason is that, because
0
unconstrained, learning 4 doesn't actually add expressive
power any non-zero map suffices.
I 0 and W
0
*Cossider ((X) = [o(X)"",( W
0
0
M
5

---

4.2 Bi-Level Optimization
We have now reduced our task bi-level optimization problem
min Σ
(R ( +)) + (1 (w
cEET.
(4.2)
st W E argmin ((W + w) + ACcond (W WD, Ve E Etr
W
where 7 E [0, 1I, > O. The set E consists of all training domains, and ft (f) is the domain-specific population risk defined (J) (xx (x), with the cross-entropy loss function l.
In words: we try to learn a representation & such that the invariant predictor has low risk (second term), the domain-specifie predictor has low risk in each domain (first term), and the domain-specific perturbation V is optimal given W and while satisfying the observable signature the causal condition (constraint, with the Ceond regularization).
4.3 Practical Algorithm
is a challenging optimization problem. In general, each constraint calls for an inner optimization routine. So instead of solving 4.22 directly, we use gradient penalty to make the problem more tractable. Specifically, we translate the condition that the domain-specifie risk is optimal (the inner loop) into the condition that the gradient of the domain-specific risk with respect to W is O. Then, we regularize the ---norm of this gradient. This is inspired by a similar trick used in Invariant Risk Minimization 19]. The finite sample objective function can be expressed
L(W", W +))
Σ Σ 7(( W=)0(z)))) + + (1 + Xg
Σ
V
(W")
BEEt (R.N)DD=
eEEer
where A > is regularization coefficient for the gradient penalty, a labeled dataset collected from training domain pe and Tineer is given by
Linner (W) E (IN*+W) + ACcond
(((r")0(s) (x.g)ED=
4.4 Invariant and Adaptive Prediction
After training, suppose the returned representation is ②. Then the invariant predictor is
g(X) (x((X)
Moreover, given few labeled examples from a new domain, we can find domain speciffc predictor by fine-tuning the linear layer Whb
Related Work
Causal Prediction Several papers connect causality and robustness to domain shifts. PBM16
HDPMI8 Arj+19 Lu+2I]. These papers usually assume that all domains share common causal structure, and consider the set of domains induced by arbitrary intervention any node other than the label Y. In this case, the predictor that has invariant risk across domains is the one that depends only on the causal parents of Y. By contrast, in this paper, only allow changes of unobserved confounders resulting in a much smaller set of possible shifts. Restricting the possible shifts enlarged the set of possible invariant predictors, allowing for invariant predictors that depend on the descendants of Y.
A closely related work is Invariant Risk Minimization 19j+19], that also seeks to learn representa- tion of X such that fixed linear map on top of the representation yields an invariant predictor. The major distinction with the approach here is that have a different notion of invariance (sec

---

Section 2.I), and we rely on simultaneously learning the non-stable factors of variation in order to Identity the representation.
Other papers also consider settings where the covariates X are not direct causes of Y Liu+22 M [s+ 'hey assume that both X and Y are caused by latent variables that can be divided into stable and non-stable parts. Then, they use generative models reflecting this assumption. By contrast, the approach in this paper fully nonparametric- there is no explicit modeling of the generative process. Prediction in the anti-causal direction has also been studied in other contexts
[Sch+112 Li+18 Wal+21 KPST8 Gon+16 HM2T]. In particular, Schölkopf al. [Sch+12 study the role of anti-causal learning in semi-supervised learning and transfer learning.
This work fits into the emerging literature causal representation learning [e.g.. Bes+ Loc+20
Sch+21 WJ21]. This literature seeks to find representations that disentangle causally meaningful components of the data-here,, we disentangle the factors of variation that have domain-stable or domain-varying relationships with the target Y.
Veitch et al. [vi++2]] introduce the notion of counterfactual invariance to a spurious factor and make some connections with domain shifts. However, they assume is known in advance and observed, and rely on this to learn the
predictor. In contrast, in this paper we merely assume the existence of some we don't 't need to know it in advance, and we don't need to measure it directly. And, they use data from only single domain, whereas require observations from several distinct domains. We also treat the problem of learning transportable representations, but they only handle invariant learning.
Domain Adaptation and Meta Learning There have been numerous fruitful developments in the fields of domain generalization and adaptation [e.g.. Zho+21| Wan+2T], including ones under various causal assumptions (Zha+ 13 Mag+18; CB21 SSS19 Sch++ 12] Lv+22]. J. A distinctive aspect of the work in this paper is that we consider the interplay between both the problem of invariant/robust
learning and adaptation.
The adaptive part of the learning model in this paper is also related to meta learning, where the goal learn predictors that can quickly adapt to new tasks. Meta learning has been used for supervised learning [6,n+16), reinforcement learning [Wan+16] and even unsupervised learning [JV19). Tra- ditional approaches to meta learning include defining distribution over the structure input data to perform inference [T or to use a memory model such as LSTM Wan++18). But the dominant models for meta learning are generic gradient-basee learning methods such as MAML [FAL17] and Reptile (NAS18). Theoretically, Tripuraneni et al. [JJ2T and Du et al. [Du+20] also examine the rep- resentation power of meta-learning. Although not motivated by causality, they show that if there is a shared common structure, meta learning can be used to reduce sample complexity in unseen domains.
6 Experiments
The main claims of the paper are:
1. The invariant predictor ((X) will have good performance in new domains, so long as the shifts obey the anti-causal structure.
2. The learned representation & enables rapid adaptation to new domains by learning only linear adjustment term on top of क.
3. The learned & disentangles the parts of X that are not affected by Z from the parts that are.
To evaluate the above claims, we conduct experiments on synthetic and real-world data. While causal structures of real-world problems like image classification are usually unknown, we find that the anti-causal based method works well on many such problems- sugggesinn the anti-causal structure is appropriate. For space reasons, in addition to the experiments reported in this section, also report experiments on PACS [Li+17] and VLCS [XXR33) in Appendix B These experiments also strongly support the effectiveness of the causal adaptive model. We also provide a counterexample showing that ACTIR can fail when causal assumptions fail to hold in, A.pppddidxx B.3
For each experiment, all methods share common architecture; they differ only in objective functions or optimization procedures. For invariant learning, we compare with empirical risk minimization (ERM), IRM 1rj+19 and the MAML [FAL17] base learner. To test how well learned
Baselines
7

---

Z -1
Z =
Density
Density
Density
Densin
-45 Activation Value
Activation Value
Synthetic Dataset
(b) Color MNIST
Figure 2: The learned representation disentangles stable and unstable factors of variation. Plots show activation levels of different representation units. Units and are trained as the invariant part of the representation. The invariant units have no dependency on the unstable factor Z, but the other unit has a strong dependency. For synthetic dataset, Z is the random variable defined in the structural equations. For Color MNIST, Z is the color.
representations & can enable fast adaptation, we fine-tune linear models on top of the representation. For comparison, we also fine-tune linear layers on top of the representations (penultimate layers) from ERM, IRM, and MAML. has been shown recently that fine-tuning the last layer of models trained by ERM has surprisingly good performance on many real-world datasets [RRR22] For MAML, the last layer is trained using the MAML update rule.
6.1 Synthetic Dataset
We generate synthetic data according to the following structural equations (which obey the anti-causal structure):
Y (0.5)
X y (0.55) Z Rad(3e) X= t 7
where input X is (X,, x+) and (ad(3) means that a random variable is -1 with probability - and +1 with probability B. We create two training domains with Bg E {0.95, 0.7). one validation domain with Re 0.6 and one test domain with B. = 0.1. Prediction with X is stable but has lower accuracy compared to prediction with X, during training. If a learning model only chooses the classifier with the best prediction accuracy in training domains and ignores its instability, it will choose X, as its predictor and end up with only 10% accuracy on the test set. The robust predictor would be X with 75% accuracy. On the other hand, in the test domain, -x, predicts Y with 90% accuracy-so an adaptive predictor better than the invariant one.
We use three-layer neural network with hidden size 8 and ReLU activation for d and train the neural network with Adam optimizer. The hyperparameters are chosen based on performance on the validation set. For the fine-tuning test, we run 20 steps with a learning rate 10-2 The result is shown in Table Both IRM and ACTIR learn good invariant predictors. But ACTIR is also equipped with the ability to adapt given very small amount of data points while the performance of IRM stays the same after fine-tuning. Perhaps unsurprisingly, ERM has a test accuracy of 10%, suggesting that uses only spurious features Xz.
6.2 Color MNIST
Color MNIST modifies the original MNIST dataset Arj+19). First, we assign label 0 to digits 0-4 and label to digits 5-9, We then flip the label with probability 25% and assign colors to the original images based on the label but with flip rate 1 1 fr. That is, we assign color 0 to images with label with probability 3 Here, color is naturally the factor of variation Z. We create two training domains with 5 E (0.95, 0.7), a validation domain with 8r 0.2 and a test domain with 8 0.1. Ideally, we want to learn invariant predictors based on the shape of the digit-this will achieve 75% accuracy. But the problem is significantly more challenging than the synthetic example because the color is much easier feature to learn than the shape of the digit, making models more susceptible spurious correlations. We use three-layer convolutional neural network for & and train the neural network with Adam optimizer. The hyperparameters are chosen based performance on the validation set.

---

Fine-tuned Accuary
FRM
RM
MAML
Fine tuning Points
Figure 3: ACTIR has good accuracy after fine-tuning with small datasets on Camelyon 17. Graph shows accuracy (%) Camelyon17 7 for different numbers of fine-tuning examples. Standard errors over 5 runs, For each run, the accuracy averaged for 100 fine-tuning tests.
Table ACTIR has good invariant adaptive performance on synthetic datasets and Color MNIST, and it outperforms baseline methods on Camelyonl7 for invariant prediction. The table shows accuracy (%). Note that adaptation (n) means that the predictor is tuned on number of labeled examples. For synthetic datasets, standard errors are 100 runs. For Color MNIST, it is over 50 runs. For Camelyon17 17, it over 5 runs. In each run, the adaptive accuracy is determined by the average of 100 fine-tuning tests for both synthetic datascts and Color MNIST.
SYNTHETIC DATASET
COLOR MNIST
CAMBLYON17 TEST ACc.
METHOD
TEST ACC.
ADAPTATION (10)
TEST Acc.
ADAPTATION (10)
ERM
9.95+0.10
11.57:10.71
28. 28.2440.51
27.26:10.48
70.77
IRM
74.9110.13
74.2710.47
59.9710.91
60.16:10.90
71.5912.76
MAML
17.1412.22
44.0113.48
22.1811.001
75.0313.30
70.22:12.40
ACTIR
74.77+00.44
0..28
70.30++0.71
85.25+1.11
77.73+1.74
For the fine-tuning test, we run 20 steps with a learning rate 20 The result is shown in Table ACTIR learns both the invariant and adaptive structure significantly better than reference baselines.
Causal Disentanglement To understand why ACTIR can adapt to test domains in both synthetic and Color MNIST datasets, we plot distributions of activation values of Φ. See Figure We see that the first two used as the invariant part of the representation in training (see Seection 4.3) have distributions that do not depend on the value of Z. On the other hand, some other inonrinvariant) representation coordinates have activations that change dramatically depending on the value of Z. Thus, the representation effectively disentangles the X and X, features. Importantly, this is achieved with no priori knowledge of what Z might be, and no observations of it.
6.3 Camelyon17
The goal of the Camelyon17 dataset Ban+18 is to predict the existence of a tumor given region of tissue. This is a binary classification problem. Data are collected from a small number of hospitals. But there are variations in data collection and processing that could negatively impact model performance on data from a new hospital. We take the individual hospitals to be separate domains. The objective is to generalize to new hospitals not seen in training. The dataset consists of input images with size 96 x 96 and binary labels that indicate the central 32 x 32 regions contain any tumor tissues. The dataset can be divided into 5 subsets, each from a different hospital. Following the WILDS benchmark Koh+21
we use 3 for training, for validation, and the last one for test.
We use pre-trained ResNet- 18 model for our and train the whole model using Adam optimizer with learning rate 10-4, For the fine-tuning test, we run 20 iterations with a learning rate 10-2, As shown in Table ACTIR has the best invariant accuracy. For adaptive performance, Figure Tigure]shows shows that ACTIR has large performance improvement when given small fine-tuning dataset, while other models require more fine-tuning examples to see significant increase in accuracy.

---

7 Discussion
This paper studies learning invariant and transportable representations for specific class of anti- causal shift domains, We assume that all domains have common anti-causal structure and are differentiated only by the distribution of certain unobserved confounders. This setup reasonable match for many practical problems.
This work serves as proof of concept for this anti-causal domain shift notion, showing that it can be translated into useful learning principles for domain adaptation. This opens the door for substantial future work. In particular, the practical training procedure be nice to find formal guarantees for robust models trained under this setup-e.g relying on some notion of diversity of training domains.
likely be refined. It would also
8 Acknowledgments
Thanks to Jacob Eisenstein for feedback on an earlier version. We also acknowledge the University of Chicago's Research Computing Center for providing computing resources. This work partially supported by Open Philanthropy.
References
[Arj+19]
M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. "Invariant risk minimization". In: arXiv preprint 1907:v:907.02200 (2019).
[Bai+20]
H. Bai, R. Sun, L. Hong, F. Zhou, N. Ye, H.-J. Ye, S.-H. G. Chan, and Z. Li. "Decaug: duttr---isttibuuiion generalization via decomposed feature representation and semantic augmentation". In: arXiv preprint arXiv:2012.0 2.02222 (2020).
[Ban+18]
P. Bandi, O. Geessink, Q. Manson, M. Van Dijk, M. Balkenhol, M. Hermsen, B. E. Bejnordi, B. Lee, K. Paeng, A. Zhong, et al. "From detection of individual metastases classification of lymph node status the patient level: the camelyon17 challenge". In: IEEE Transactions on Medical Imaging (2018).
[Bes+18]
M. Besserve, A. Mehrjou, R. Sun, and B. Schölkopf. "Counterfactuals uncover the modular structure of deep generative models". In: arXiv preprint arXiv: 1812.03253 (2018).
[Cai+21]
T. Cai, R. Gao, D. Lee, and Q. Lei. "A theory of label propagation for subpopulation shift". In: arXiv preprint arXiv:2102.1
(2021).
[CB21]
Y. Chen and Bühlmann. "Domain adaptation under structural causal models". In: Journal of Machine Learning Research 261 (2021).
[Du+20]
S. S. Du, W. Hu, S. M. Kakade, J. D. Lee, and Q. Lei. "Few-shot learning via learning the representation, provably". In: arXiv preprint arXiv:2002.088 09434 (2020). C. Fang, Y. Xu, and D. N. Rockmore. "Unbiased metric learning: on the utilization of multiple datasets and web images for softening bias". In: IEEE International Con- ference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013. 2013.
[FXR13]
[FAL17]
C. Finn, P. Abbeel, and Levine. "Model-agnostic meta-learning for fast adaptation of deep networks". In: International Conference on Machine Learning. PMLR. 2017.
[Fuk+07]
K. Fukumizu, A. Gretton, X. Sun, and B. Schölkopf. "Kernel measures of conditional dependence." " In: NIPS. 2007.
[Gon+16]
M. Gong, K. Zhang, Liu, D. Tao, C. Glymour, and B. Schölkopf. "Domain adaptation with conditional transferable components", In: International conference machine learning. PMLR. 2016.
[GL21]
I. Gulrajani and D. Lopez-Paz. "In search of lost domain generalization". In: 9th International Conference
Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. 2021.
[HM21]
C. Seinze-Demll and N. Meinshausen. "Conditional variance penalties and domain shift robustness". In: Mach. Learn. 2 (2021).
[HDPM18]
C. Heinze-Deml, J. Peters, and Meinshausen. "Invariant causal prediction for non- linear models". " In: Journal of Causal Inference 2 (2018).
10

---

[1ls+20]
M. Ilse, J. M. Tomczak, C. Louizos, and M. Welling. "Diva: domain invariant variational autoencoders". In: Medical Imaging with Deep Learning. PMLR. 2020. Y. Jiang and N. Verma. "Meta-learning to cluster". In: arXiv preprint WrXi:1910.14134 (2019).
[JV19]
[KPS18]
N. Kilbertus, G. Parascandolo, and B. Schölkopf. "Generalization in anti-causal learn- ing". In: arXiv preprint arXiv:1812.00522 (2018).
[Koh+21]
P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M.
Yasunaga, R. L. Phillips, Gao, et al. "Wilds: a benchmark wn-the-wild distribution shifts". In: International Conference on Machine Learning. PMLR. 2021.
[Lak+11]
B. Lake, R. Salakhutdinov, J. Gross, and Tenenbaum. "One shot learning of simple visual concepts". In: Proceedings of the annual meeting of the cognitive science society.
33. 2011.
[Li+17]
D. Li, Y. Yang, Y.-Z. Song, and T. M. Hospedales. "Deeper, broader and artier domain generalization". In: Proceedings of the IEEE international conference on computer vision. 2017.
[Li+18]
Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao. "Deep domain general- ization via conditional invariant adversarial networks". In: Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[Liu+21]
C. Liu, X. Sun, J. Wang, H. Tang, T. Li, T. Qin, W. Chen, and T.-Y. Liu. "Learning causal semantic representation for -fitt--distriuttionn prediction". In: Advances in Neural Information Processing Systems (2021).
[Loc+20]
F. Locatello, B. Poole, G. Rätsch, B. Schölkopf. O. Bachem, and M. Tschannen. "Weakly- supervisde disentanglement without compromises". In: International Confer- ence Machine Learning. PMLR. 2020.
[Lu+21]
C. Lu, Y. Wu, J. M. Hernández-L Lobato, and B. Schölkopf. "Nonlinear invariant risk minimization: a causal approach". In: arXiv preprint
(2021). F. Lv, J. Liang, S. Li, B. Zang, C. H. Liu, Z. Wang, and D. Liu. "Causality inspired representation learning for domain generalization". In: Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition. 2022.
[Lv+22]
[Mag+18]
S. Magliacane, T. Van Ommen, T. Claassen, S. Bongers, P. Versteeg, and J. M. Mooij. "Domain adaptation by using causal inference to predict invariant conditional distribu- tions". In: Advances in neural information processing systems (2018).
[Mit+21]
J. Mitrovic, B. McWilliams, C. Walker, L. H. Buesing, and C. Blundell. "Representa- tion learning via invariant causal mechanisms". In: 9th International Conference on Learning Representations, ICIR 2021, Virtual Event, Austria, May 3-7, 2021. 2021.
[MBS13]
K. Muandet, D. Balduzzi, and B. Schölkopf. "Domain generalization via invariant fea- ture representation" In: Proceedings of the 30th International Conference on Machine Learning. ICML 2013, Atlanta, GA, USA, 16-21 June 2013. 2013..
[NAS18]
A. Nichol, J. Achiam, and J. Schulman. "On first-order meta- meta-learnin algorithms". In: arXiv preprint arXiv::8033,0 02999 (2018).
[PBM16]
Peters, Bühlmann, and N. Meinshausen. "Causal inference by using invariant
prediction: identification and confidence intervals". In: Journal of the Royal Statistical Society, Series B (Statistical Methodology) (2016).
[PJS17]
J. Peters, D. Janzing, and B. Schölkopf. Elements of causal fnferencee,oundatioon foundations and learning algorithms. 2017.
[Roj+18]
M. Rojas-Carulla, B. Schölkopf, R. E. Turner, and J. Peters. "Invariant models for causal transfer learning". In: J. Mach. Learn. Res. (2018).
[RRR22]
E. Rosenfeld, P. Ravikumar, and A. Risteski. Domain-Adjuset Adjusted Regression ERM May Already Learn Features Sufficient for Dutt-of-Dsttribue
Generalization. 2022.
[Rot+21]
D. Rothenhääsler, N. Meinshausen, P. Bühhmann, and J. Peters. "Anchor regression: heterogeneous data meet causality". In: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 2 (2021).
[Sag+19]
S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. "Distributionally robust neural networks for group shifts: on the importance of regularization for worst-case general- ization". In: arXiv preprint arXiv:1911.0873) (2019).

---

[San+16]
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. "Meta-learning with nemory-augmenter neural networks", In: International conference on machine learning. PMLR. 2016.
[Sch+12]
B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. M. Mooij. "On causal and anticausal learning". In: ICML. 2012.
[Sch+21]
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio. "Towards causal representation learning". In: CoRR (2021).
[Shi+21]
Y. Shi, J. Seely, H. Torr, N Siddharth, A. Hannun, N. Usunier, and G. Synnaeve. "Gradient matching for domain generalization", In: arXiv preprint arXiv:2104.099) (2021).
[SSS19]
A. Subbaswamy, P. Schulam, and S. Saria. "Preventing failures due to dataset shift: learning predictive models that transport". In: The 22nd International Conference Artificial Intelligence and Statistics. PMLR. 2019.
[TSS16]
I. O. Tolstikhin, B. K. Sriperumbudur, and B. Schölkopf. "Minimax estimation of maximum mean discrepancy with radial kernels". In: Advances in Neural Information Processing Systems (2016).
[TJJ21]
N. Tripuraneni, C. Jin, and M. Jordan. "Provable meta-learning of linear representa- tions". In: International Conference on Machine Leurning. PMLR. 2021.
[Vei+21]
V. Veitch, A. D' Amour, S. Yadlowsky, and J. Eisenstein. "Counterfactual invari- ance spurious correlations: why and how pass stress tests". In: arXiv preprint OrXiv:2106.. 00545 (2021).
[Wal+21]
Y. Wald, A. Feder, D. Greenfeld, and U. Shalit. "On calibration and out-of-domain generalization" In: Advances in neural information processing systems (2021). J. X. Wang, Z. Kurth-Nelson, D. Kumaran, D. Tirumala, H. Soyer, Z. Leibo, D. Has- sabis, and M. Botvinick. "Prefrontal cortex as meta-reinforceement learning system". In: Nature neuroscience 6 (2018).
[Wan+18]
[Wa++16]
J.X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell,
D. Kumaran, and M. Botvinick. "Learning to reinforcement learn". In: arXiv preprint 161iv:661.00566 (2016).
[Wan+21]
J. Wang, C. Lan, C. Liu, Y. Ouyang, and T. Qin. "Generalizing to unseen domains: A survey on domain generalization". In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021. 2021.
[WJ21]
Y. Wang and M. I. Jordan. "Desiderata for representation learning: a causal perspective".
In: arXiv preprint arXiv:2109.0377 (2021).
[2ha+12]
K. Zhang. J. Peters, D. Janzing, and B. Schölkopf. "Kerne--base conditional indepen- dence test and application in causal discovery". In: arXiv preprint arXiv:1202.377 (2012).
[7ha+13]
K. Zhang, B. Schölkopf, Muandet, and Z. Wang. "Domain adaptation under target and conditional shift". In: International Conference on Machine Learning. PMLR. 2013.
[Zhe+21]
X. Zheng, X. Sun, W. Chen, and T.-Y. Liu. "Causally invariant predictor with shift- robustness", In: arXiv preprint 2xXiv:2107.01 (2021).
[Zho+21]
K. Zhou, Z. Liu, Y. Qiao, T. Xiang, and C. C. Loy. "Domain generalization: survey". In: CoRR (2021).
[7hu+20]
F. Zhuang, 7. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. "A compre- hensive survey on transfer learning". In: Proceedings of the IEEE (2020).
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
12

---

Did you include the license to the code and datasets?
[No]
The code and the data are
proprictary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(C) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ns3pwSUar0", "reviewer": "Reviewer_wAxr", "summary": "In this paper, a setting is studied where data come from different domains, the features X are causal descendants of Y, and for some part of X, the relation with Y will change between domains. Based on a notion of counterfactual invariance, an optimization objective and an algorithm are then proposed to disentangle the different parts of X. This allows prediction using just the invariant part of X, but also fine-tuning the predictor on a few samples of a new domain to leverage the information in the non-invariant part of X.", "questions": "1. **Definition 1**: I think the crucial aspect of this definition is not that the causal graph is shared, but that $P(X,Y,Z|U)$ is shared between domains, which is now only mentioned implicitly.\n\n2. **counterfactual invariance**: A \"counterfactual\" notion of invariance is both a stronger condition than you actually need, and also stronger than can be verified from your data. It is in principle possible to detect if the *distribution* of f(X) (jointly with Y?) given z remains invariant when z changes, but not whether the specific value of f(X) for a given individual is invariant in such a way. Since the term \"counterfactual\" (as the third level of Pearl's hierarchy) is specifically about this latter case, I feel this term is not the best choice in this context. And more specifically, I would disagree that (line 103) learning a stable predictor \"means\" learning a counterfactually invariant predictor.\n\n3. **latent parts of X**: Could you clarify in the paper what you mean by \"part\": a subset of the coordinates of a long vector X, or any function of X (that preserves some of the data in X but discards other parts)?\n\n4. **Causal decomposition of X**: In the second paragraph of section 2.2, I think what you want to work towards is that the two parts of X are independent given Y. If you want to phrase this in terms of the causal diagram, I think you also need to assume here that the two parts have no latent confounders between them, and I suppose selection bias could be a problem as well. I'm not sure what is meant with this extra assumption being \"baked into the causal compatibility assumption\" (line 123), because the two parts of X were not defined yet when Definition 1 was posed.\n\n5. **$W^b$ and $W^e$**: In line 215, the possibility of making $W^b$ a fixed matrix is mentioned. I assume this is also being done for the experiments? Would it also be a good simplification to restrict $W^e$ to be of the form suggested in footnote 2, or would that reduce the expressive power in an undesirable way? Relatedly, I think the $W^b$ in line 237 (about fine-tuning) should read $W^e$.\n\nMinor points:\n\n* it looks like the display above line 196 doesn't use \\lVert and \\rVert\n\n* In three displays on page 6, summation signs are ambiguous as to which terms are part of the expression being summed.", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The anti-causal setting studied in this paper has not been studied before to my knowledge (except under an additive noise assumption). I think its exploration of what is possible for domain adaptation with such anti-causal variables, is a significant contribution (mainly but not exclusively for enabling future work). The derivation and methods include some elements that are derived from earlier work, but for the main part, the work is novel.\n\nThe paper is very clearly written, though I do have some questions below where e.g. a definition doesn't seem to express exactly what is intended. I hope that addressing these points will further improve this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mdrjoFgVAr", "reviewer": "Reviewer_UXLv", "summary": "The present work address generalization and transfer of learnable representations in the setting of anti-causal domain shifts. To this end, the work formalize the anti-causal domain shift assumption by adopting the notion of counterfactual invariance (Veitch et. al., 21). Based on this formulation, the work proposes a bi-level optimization based objective to learn domain-invariant and domain-adaptive predictors in a multi-distribution training setup.  Eventually, the proposed algorithm is evaluated on a synthetic task, colored-MNIST and some real-world datasets against ERM, IRM, MAML and show promising performance.", "questions": "- During fine-tuning, are both the invariant features and the non-stable features adapted or only the non-stable features? Did the authors also consider to finetune with different learning rates, in order to allow an even more rapid adaptation?\n- Is any regularization technique employed in ERM? ", "limitations": "Fine", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n- Anti-Causal problem setting is well motivated and the work is well written in general\n- I appreciate the formulation based on counterfactual invariance to separate stable from unstable knowledge\n- Supportive visual analysis in figure 2\n- Valid sets of experimental baselines where proposed approach shows promising performance\n\n**Weaknesses / Suggestions for improvements**\n- **Evaluation**: I have some concerns about the real-world applicability of the proposed algorithm and would appreciate some discussion on this matter  \n  - There is a significant gap between performances on manually crafted tasks (synthetic setting, color MNIST) which match the underlying assumptions of the framework and on the real-world datasets (Camelyon17, PACs, VLCS). I would like to encourage the authors to address this issue and it's implications for applications on real-world datasets in more detail in the main part (especially including the important findings on real-world datasets in the main part of the paper).\n   - What conditions need the training environments to fullfill and what dataset characteristics can break the algorithm? From my point of view, it looks like diversity of the training environment is a key property?\n   - Would it be possible to show a similar analysis as figure 2 for one of the real-world datasets?\n- **Assumptions**\n   - The work assumes that $X_{z}^{\\perp}$ does not have a causal effect on $X_z$. What are the implications on potential application(especially on structured datasets)? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-roHZ163Z-M", "reviewer": "Reviewer_ABF4", "summary": "This paper proposes an invariant representation learning method across multi-domain from the perspective of the anti-causal learning process. They learn invariant and variant representation simultaneously. It is an interesting idea that considers the anti-causal direction in invariant representation learning. Experimental results show the effectiveness of the proposed method on both synthetic data and real-world datasets.", "questions": "1. Consider the case (Example 1) in Invariant Risk Minimization, the difficulty of anti-causal learning is that one can not have a stable prediction function if they use the effect variable to predict the cause variable (use X2 to predict Y in IRM is regarded as anti-causal learning). Therefore if the authors claim that they can achieve invariance across multi-domain, they should better give a simple example to support their motivation. \n2. This algorithm can also be considered a causal learning method. if we change the direction of the edges between (y,z) and (x), the method will also hold. What makes the proposed method better for anti-causal learning should have more discussion. Causal learning and anti-causal learning should be considered separately since the independent causal mechanism(ICM) will not hold in the anti-causal direction. Note that ICM is not conditional independent, that is 'if X is cause, X is independent of the causal mechanism p(Y|X)'. If this issue can not be solved, why this method is better than IRM or other invariant representation learning where variant and invariant representation are learned simultaneously.\n3. Authors should add some ablation study about the performance of the proposed method in both causal and anti-causal datasets. \n4. Authors should better provide some theoretical analysis about whether they can actually learn the invariant representation, which can be regarded as the identifiability property.\n5. The linear setting is limited, can the method be extended to non-linear settings?\n6. Should add more state-of-art causal invariant learning methods as baselines.\n\n\n", "limitations": "Some parts of the method should have more discussion. 1. What makes this method hold in anti-causal learning 2. Lack of theoretical analysis of identifiability 3. The linear setting is simple. More details are shown in the question part.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper seeks an invariant learning method considering the anti-causal prediction task. They formed a causal graph to describe anti-causal prediction and learn adaption and invariant representation together. Experimental results show the effectiveness of proposed method. However, some parts of the method should have more discussion. 1. What makes this method hold in anti-causal learning 2. Lack of theoretical analysis of identifiability 3. The linear setting is simple.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept