Identifying Representations for Intervention Extrapolation
Authors: Sorawit Saengkyongam
Elan Rosenfeld
Pradeep Kumar Ravikumar
Niklas Pfister
Jonas Peters
Venue: ICLR 2024 poster (2024)
Keywords: causality
extrapolation
exogenous variables
causal representation learning
identifiable representation learning
control functions
instrumental variables
invariance
Primary area: causal reasoning

Abstract: The premise of identifiable and causal representation learning is to improve the current representation learning paradigm in terms of generalizability or robustness. Despite recent progress in questions of identifiability, more theoretical results demonstrating concrete advantages of these methods for downstream tasks are needed. In this paper, we consider the task of intervention extrapolation: predicting how interventions affect an outcome, even when those interventions are not observed at training time, and show that identifiable representations can provide an effective solution to this task even if the interventions affect the outcome non-linearly. Our setup includes an outcome variable $Y$, observed features $X$, which are generated as a non-linear transformation of latent features $Z$, and exogenous action variables $A$, which influence $Z$. The objective of intervention extrapolation is then to predict how interventions on $A$ that lie outside the training support of $A$ affect $Y$. Here, extrapolation becomes possible if the effect of $A$ on $Z$ is linear and the residual when regressing Z on A has full support. As $Z$ is latent, we combine the task of intervention extrapolation with identifiable representation learning, which we call $\texttt{Rep4Ex}$: we aim to map the observed features $X$ into a subspace that allows for non-linear extrapolation in $A$. We show that the hidden representation is identifiable up to an affine transformation in $Z$-space, which, we prove, is sufficient for intervention extrapolation. The identifiability is characterized by a novel constraint describing the linearity assumption of $A$ on $Z$. Based on this insight, we propose a flexible method that enforces the linear invariance constraint and can be combined with any type of autoencoder. We validate our theoretical findings through a series of synthetic experiments and show that our approach can indeed succeed in predicting the effects of unseen interventions.

Published as a conference paper at ICLR 2024
IDENTIFYING REPRESENTATIONS FOR INTERVENTION EXTRAPOLATION
Sorawit Saengkyongam'
Elan Rosenfeld
Pradeep Ravikumar
Niklas Plister Jonas Peters'
ETTH Zürich
Carnegie Mellon University University of Copenhagen
ABSTRACT
The premise of identifiable and causal representation learning is to improve the current representation learning paradigm in terms of generalizability robust- ness. Despite recent progress in questions of identifiability, more theoretical re- sults demonstrating concrete advantages of these methods for downstream tasks are needed. In this paper, we consider the task of intervention extrapolation: pre- dicting how interventions affect outcome, even when those interventions are not observed at training time, and show that identifiable representations can provide an effective solution this task even if the interventions affect the outcome non- linearly. Our setup includes an outcome variable Y, observed features X, which are generated as a nonlinear transformation of latent features Z, and exogenous action variables A, which influence Z. The objective of intervention extrapola- tion is then to predict how interventions on that lie outside the training support of A affect Y. Here, extrapolation becomes possible if the effect of A Z is linear and the residual when regressing Z on A has full support. As is latent, we combine the task of intervention extrapolation with identifiable representation learning, which we call Rep4Ex: we aim to map the observed features X into a subspace that allows for nonlinear extrapolation in A. We show that the hidden representation is identifiable up to an affine transformation in Z-space, which, we prove, is sufficient for intervention extrapolation. The identifiability is character- ized by a novel constraint describing the linearity assumption of A on Z. Based on this insight, we propose flexible method that enforces the linear invariance constraint and can be combined with any type of autoencoder. We validate our theoretical findings through a series synthetic experiments and show that our approach can indeed succeed in predicting the effects of unseen interventions.
INTRODUCTION
Representation learning (see, c.g., Bengio et al., 2013, for an overview) underpins the success of modern machine learning methods evident, for example, in their application to natural language processing and computer vision. Despite the tremendous success of such machine learning methods, is still an open question when and to which extent they generalize to unseen data distributions. is further unclear, which precise role representation learning can play in tackling this task.
To us, the main motivation for identifiable and causal representation learning (e.g., Schölkopf al., 2021) is to overcome this shortcoming. The core component of this approach involves learning a representation of the data that reflects some causal aspects of the underlying model. Identifying this from the observational distribution is referred to as the identifiability problem. Without any assump- tions on the data generating process, learning identifiable representations is not possible (Hyvärinen & Pajunen, 1999). To show identifiability, previous works have explored various assumptions, in- cluding the use of auxiliary information (Hyvarinen et al., 2019; Khemakhem et al., 2020), sparsity (Moran et al., 2021; Lachapelle et al., 2022), interventional data (Brehmer et al., 2022: Seigal et al., 2022: Ahuja et al., 2022a; 2023; Buchholz et al., 2023) and structural assumptions (Hälvä et al., 2021: Kivva et al., 2022). However, this body of work has focused solely the problem of identifi- ability. Despite its potential, however, convincing theoretical results illustrating the benefits of such identification in solving tangible downstream tasks arguably scarce; few recent works have

---

Published as a conference paper at ICLR 2024
Bep4
Tue condeamal meen
A
Rep4Ex
Learning Identifablle roprosontation
X
-15
Intarvention
Predicting the effect onseen interventions
ETVid//Auo"))
(b) Example illustrating intervention extrapola- tion; during training, X, and Y are observed
(a) Graphical model of the problem setup
Figure I: In this paper, we consider the goal of intervention extrapolation, see (b). We given training data (yellow) that cover only a limited range of possible values of A. During test time (grey), we would like to predict E[[| do(AA = a for previously unseen values of a". The function a' + B[{| do(A- = a²)] (red) can be nonlinear in a". We argue in Section 2 how this can be achieved using control functions if the data follow a structure like in (a) and Z is observed. We show in Section 3 that, under suitable assumptions, the problem is still solvable if we first have to reconstruct the hidden representation Z (up to a transformation) from X. The representation used to predict B[Y do(A (A- = a") )], so we learn a representation for intervention extrapolation (Rep4Ex).
provided theoretical evidence for the advantages of identifiable representations in tasks such es- timating treatment effects (Wu & Fukumizu, 2022), improving generalization in multi-task learning (Lachapelle et al., 2023a) and generating novel object compositions (Lachapelle al., 2023b).
In this work, we consider the task of intervention extrapolation, that is, predicting how interventions that were not present the training data will affect an outcome. We study a setup with an outcome Y; observed features X which are generated via nonlinear transformation of latent predictors Z: and exogenous action variables A which influence Z. We assume the underlying data generating process depicted in Figure la. The dimension of X can be larger than the dimension of 2 and we allow for potentially unobserved confounders between Y and Z (as depicted by the two-headed dotted arrow between Z and Y). Adapting notation from the independent component analysis (ICA) literature (Hyvärinen & Oja, 2000), we refer to 9o as a mixing (and 90 as an unmixing) function.
In this setup, the task of intervention extrapolation is to predict the effect of a previously unseen intervention on the action variables A (with respect to the outcome Y). Using do-notation (Pearl, 2009), thus aim estimate E| do a²) a²)], where a* lies outside the training support of A. Due to this extrapolation, E[Y] do(A = a²)], which may be nonlinear in a", cannot be consistently estimated by only considering the conditional expectation of Y given A (even though A is exogenous and E|| do(AA A = a)] A = a] for all a in the support of A), see Figure 1b. We formally prove this in Proposition 1. In this paper, the central assumption that permits learning identifiable representation and subsequently solving the downstream task is that the effect of A on Z is linear, that is, Z A] A for unknown matrix My.
The approach we propose in this paper, Rep4Ex- CF, successfully extrapolates the effects outside the training support by performing two steps (see Figure la): the first stage, we use (A,X) to learn an encoder Φ X + Z that identifies, from the observed distribution of (A, the unmixing function 90 up to an affine transformation and thereby obtains a feature representation d(X). To do that, we propose to make use of a novel constraint based on the assumption of the linear effect of A on Z, which, as we are going to see, enables identification. Since this constraint has simple analytical form, it can be added as regularization term to an auto-encoder loss. In the second stage, use (A, @(X), Y) to estimate the interventional expression effect EE[ do(AA a²)]. The model in the second stage is adapted from the method of control functions in the econometrics literature (Telser, 1964; Heckman, 1977; Newey et al., 1999), where views A as instrumental variables. Figure lb shows results of our proposed method (Rep4Ex-CF) on a simulated data set, together with the outputs of and standard neural-network- -aase regression (MI.P).
2

---

Published as a conference paper at ICLR 2024
We believe that our framework provides complementary perspective causal representation learn- ing. Similar to most works in that area, we also view as the variables that we ultimately aim to control. However, in view, direct (or hard) interventions on Z are inherently ill-defined due to its latent nature. We, therefore, consider the action variables A as a means to modify the latent variables Z. As example, in the context of reinforcement learning, one may view X as observable state, Z as a latent state, AA as an action, and Y as a reward. Our aim is then to identify the actions that guide us toward the desired latent state which subsequently leads to the optimal expected reward. The ability to extrapolate to unseen values of A comes (partially) from the linearity of A on Z; such extrapolation therefore becomes possible if we recover the true latent variables Z up to an affine transformation. The problem of learning identifiable representations can then be understood as the process of mapping the observed features X to a subspace that permits extrapolation in A. We refer to this task of learning representation for intervention extrapolation as Rep4Ex.
1.1 RELATION TO EXISTING WORK
Some of the recent work on representation learning for latent causal discovery also relies on (un- observed) interventions to show identifiability, sometimes with auxiliary information. These works often assume that the interventions occur one or fixed group of nodes in the latent DAG (Ahuja et al., 2022a; Buchholz et al., 2023; Zhang et al., 2023) or that they are exactly paired (Brehmer et al., 2022; von Kügelgen et al., 2023). Other common conditions include parametric assumptions on the mixing function (Rosenfeld et al., 2021; Seigal et al., 2022; Ahuja et al., 2023; Varici et al., 2023) or precise structural conditions the generative model (Cai et al., 2019; Kivva et al., 2021; Xie et al., 2022; Jiang & Aragam, 2023; Kong et al., 2023). Unlike these works, we study interven- tions on exogenous (or "anchor") variables, akin to simultaneous soft interventions Identifiability is also studied in nonlinear ICA (e.g., Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Khemakhem et al., 2020; Schell Oberhauser, 2023), we discuss the relation in Appendix A.
the latents.
The task of predicting the effects of new interventions has been explored in several prior works. Nandy et al. (2017); Saengkyongam & Silva (2020); Zhang al. (2023) consider learning the ef- fects of new joint interventions based on observational distribution and single interventions. Bravo- Hermsdorff al. (2023) combine data from various regimes to predict intervention effects in previ- ously unobserved regimes. Closely related to our work, Gultchin et al. (2021) focus on predicting causal responses for new interventions the presence of high-dimensional mediators X. Unlike our work, they assume that the latent features are known and do not allow for unobserved confounders.
Our work is related to research that utilizes exogenous variables for causal effect estimation and distribution generalization. Instrumental variable (IV) approaches (Wright, 1928; Angrist et al., 1996) exploit the existence of the exogenous variables to estimate causal effects the presence of unobserved confounders. Our work draws inspiration from the control funetion approach in the IV literature (Telser, 1964; Heckman, 1977; Newey et al., 1999). Several works (e.g., al., 2018; Arjovsky et al., 2019; Rothenhäusler et al., 2021; Christiansen et al., 2021; Rosenfeld et al., 2022; Saengkyongam et al., 2022) have used exogenous variables to increase robustness and perform distribution generalization. While the use of exogenous variables enters similarly in our approach, these existing works focus on a different task and do not allow for nonlinear extrapolation.
Carulla
2 INTERVENTION EXTRAPOLATION WITH OBSERVED Z
To provide better intuition and insight into our approach, we start by considering a setup in which Z is observed, which is equivalent to assuming that we are given the true underlying representation. We now focus on the intervention extrapolation part, see Figure la (red box) with Z observed. Consider an outcome variable Y E y C R, predictors Z E Z C Rd. and exogenous action variables A E A C RR We assume the following structural causal model (Pearl, 2009)
S: {A=
= MoA+ + V
Y ((Z) + U,
(1)
where e,, V, U are noise variables and we assume that ε JL (V,U), EUU] 0, and Mo has full row rank. Here, V and U may be dependent.
Notation. For a structural causal model (SCM) S, denote by pS the observational distribution entailed by S and the corresponding expectation by ES, When there is ambiguity, we may omit
3

---

Published as a conference paper at ICLR 2024
the superscript S. Further, we employ the do-notation to denote the distribution and the expectation under intervention. In particular, we write pS:do(A=a) and E [-| do(A a)] to denote the distribution and the expectation under an intervention setting A == a, respectively, and pdo((A=)) and [E|| do(A = a)] if there is no ambiguity. Lastly, for any random vector B, we denote by supp"(B) (B) the support' of B in the observational distribution IP. Again, when the SCM is clear from the context, we may omit and write supp(B) B) as the support in the observational distribution.
Our goal is to compute the effect of an unseen intervention on the action variables A (with respect to the outcome Y), that is, BB| do (AA = a²)], where a* & supp(A). A naive approach to tackle this problem is to estimate the conditional expectation E[YJA a] by regressing Y on A using sample from the observational distribution of (Y, A). Despite being exogenous, from (1) we only have that [E[ (A = a)] = E a] for all supp(AA). As a" lies outside the support of A. we face the non-trivial challenge of extrapolation. The proposition below shows that in our model class dA (A- a²)] is indeed not identifiable from the conditional expectation [[ alone. Consequently, E[ do(A A = a²)] cannot be consistently estimated by simply regressing Y on A. (The result is independent of the fact whether Z is observed or not and applies to the setting of unobserved Z in the same way, see Section 3. Furthermore, the result still holds even when V and U are independent.) All proof' can be found in Appendix D.
Proposition 1 (Regressing Y on A does not suffice). There exist SCMs S, and S₂ of the form (1) (with the same set A) that satisfy all of the following conditions: (i) supp
supp>> = R.
(ūi) supps:
S: (A) supp (A) A; (iii) Va E suppSi
(A) ES [Y|A- al E A = a): (iv)
BB C A with positive Lebesgue measure s.t. Va E B: EY do = a)] + E do a)]
(the latter implies B n supp (A)= Ø).
Proposition 1 affirms that relying solely the knowledge of the conditional expectation E|Y| is not sufficient to identify the effect of an intervention outside the support of A. It is, however, possible to incorporate additional information beyond the conditional expectation help us identify E| do = a²) ]]. In particular, inspired by the method of control functions in econometrics, we propose identify do = a²)] from the observational distribution of (A, X, 2) based on the following identities,
B|y do a²)] = BB((Z)) do = a²) E(U - aa]]
= Ele(MMa" + do = a")] + BiU do(A= = a*)l
E[ + V)),
(2)
where the last equality follows from IE[U]- 0 and the fact that, for all a" E A, = A=o) U.V
Now, since A IL V. we have E[Z A] Mo and Mo can be identified by regressing Z on A. V is then identified with V = 2 Mo A. V is called a control variable and, as argued by Newey et al. (1999), for example, it can be used to identify t: defining À : + BUUVV = 2, E supp Z, V)
v). we have for all
B|y Z = z, 2,V = v] = B(((Z) + UIZ 2, V = o) x)= [(== EIUIZ = V u
= =(=)) + BLUIV u] = ((2) + A(v),
(3)
where in the second last equality, we have used that U 1L Z V (see Lemma in Appendix C). In general, (3) does not suffice to identify É (c.g.. V and Z are not necessarily independent of each other). Only under additional assumptions, such as parametric assumptions on the function classes, E and X are identifiable up to additive constants In our work, we utilize an assumption by Newey et al. (1999) that puts restrictions on the joint support of A and V and identifies l on the set Mo supp (A) + supp(V). Since Mo and V are identifiable, too, this then allows us to compute, by (2), | do = a²)] for all a* s.t. Moa* + sVpp(V) C Mo supp( A) + supp(V); thus, sV) Rd is a sufficient condition to identify assumption, together with the additivity of V in (1), is key to ensure that the nonlinear function ě can be inferred on all of Rdd allowing for nonlinear extrapolation. Similar ideas have been used for extrapolation in a different setting and under different assumptions by Shen & Meinshausen (2023). In some applications, we may want to compute the effect of intervention on A conditioned on Z, that is, | = 2, do A = at) "). We show in Appendix C. that this expression is identifiable, too.
do(A = a")] for all a* A. This support
'The support of random vector B E C RD (for some g E N+) is defined as the set of all b E Q for which every open neighborhood of b (in ?) has positive probability.
"The constant can he identified by using the assumption E.U] = 0.
4

---

Published as a conference paper at ICLR 2024
3 INTERVENTION EXTRAPOLATION VIA IDENTIFIABLE REPRESENTATIONS
Section 2 illustrates the problem of intervention extrapolation in the setting where the latent pre- dictors Z are fully observed. We now consider the setup where we do not directly observe Z but instead we observe X which are generated by applying a nonlinear mixing function to Z. Formally, consider an outcome variable Y E y S R. observable features X X C RTT latent predictors Z E 2 = IR and action variables A E A IR*. We model the underlying data generating process by the following SCM.
Setting (Rep4Ex). We assume the SCM
A cA x= x= 90(
Z= MA ==
S:
(4)
where C,. V.U U are noise variables and we assume that the covariance matrix of C.A is full-rank, CA IL (V,U), B[U] 0, supp(V)= = TR and Mo has full row rank (thus k 2 d). Further, go and l are measurable functions and go is injective. In this work, we only consider interventions Z correctly. Possible
on A. For example, we do not require that the SCM models interventions relaxations of the linearity assumption between and Z and the absence of noise in X are discussed in Remark 6 in Appendix B.
Our goal is to compute E|y do = a²)] for some a" & supp(A). As in the case observed Z, the naive method of regressing Y on A using non non-parametric regression fails to handle the extrapolation a" (see Proposition 1). We, however, can incorporate additional information beyond a²) through the method of control functions.
the conditional expectation to identify E[Y]
From (2), we have for all a* E A that
(5) Unlike the case where we observe Z, the task of identifying the unknown components on the right- hand side of (5) becomes more intricate. what follows, we show that if we can learn an encoder X + Z that identifies 9. up to affine transformation (see Definition 2 below), we can construct a procedure that identifies the right-hand side of (5) and can thus be used to predict the effect of unseen interventions on A.
E[y| do(A = a²)] = Ele(Maa" + V))-
Definition 2 (Affine identifiability). Assume Setting 1. An encoder & X + Z is said to identify Io up to an affine transformation (aff afff-identify for short) if there exists an invertible matrix H gaxd and a vector Co Ra such that
Yz E Z : go)(2) = Haz + co.
(6)
We denote by Ks : + H&z + co the corresponding affine map.
Under Setting 1, we show equivalent formulation of affine identifiability in Proposition 7 stressing that Z can be reconstructed from ((X).
Next, let X + be an encoder that aff- aff-identifies In and Kig 2 H HØ2 + co be the corresponding affine map. From (5), we have for all a* that
E|y do(A a²)) E[e(Moa" + V)] B[(e Kg (Moaa + V])]
E[[( [[[((ox]]]] (HH Moe' + cæ - Ho B[V] + Ho - E[V)
=
E[(lo K5 (Maa + 20 Vo)l.
where we define
Mo HoMo, %s cs + Ho EV], and V = - E(V).
(8)
We now outline how identify the right-hand side of (7) by using the encoder é and formalize the result in Theorem 3.
Identifying Má. 9s and Vo Using that ó aff-identifies Do we have (almost surely) that ((X) = (o o go) = H3 + Go = HaMoA + H&V + Cs M3A + Go + Vg- Now, since Vg IL A (following from JL A), we can identify the pair (M.. (s) by regressing ((X) on A. The control variable Vo can therefore be obtained as V = ((X) (M, A go).
5

---

Published as a conference paper at ICLR 2024
Identifying& lo Kig Defining Ag : 2 V 2/, we have, for all w. v supp((6(X), Vo)).
E (XX) = w, V = w] [[) = w, V. = = = = = (w), vo= =o]
=E]((Z) + U| = x (w), = r|
(lo (w)
na (w), V8 = uj
( (eo (w)) = o] = (eo (oogg ~ (w) + (
(10)
where the equality (*) holds since aff-identifiess 9T and (*+) holds by Lemma 9, see Appendix C. Similarly the case in Section 2, the functions e' o x and A are identifiable (up additive con- stants) under some regularity conditions on the joint support of A and Vo (Newey et al., 1999). We make this precise in the following theorem, which summarizes the deliberations from this section.
Theorem 3. Assume Setting and let Φ X + Z be an encoder that aff- idd-tifiiiis gu Further, define the optimal linear function from A ((X) as*
(W,
argmin
(X) - (WA+ a))1
(11)
WER .aERd
and the control variable V o(X) (W&A + as). Lastly, let v Z + y and sb V - y be additive regression functions such that
Yu,e v E supp((((X), Va)) BYY (X)) w, Vo = u| v(w) + (v))
(12)
If f, Xa are differentiable and the interior of (A) is convex, then the following statements hold
(i) Va" AA do = a² Elv(W 30 + + Voo1 (E[v((())) -E((1)
(13)
(ii) Ya Im(go), at E A: E| = x, do(A a²)] +((l)) (Waa* * + az)) (14)
4 IDENTIFICATION OF THE UNMIXING FUNCTION 90
Theorem illustrates that intervention extrapolation can be achieved if one can identify the unmixing function 90 up to an affine transformation. In this section, we focus on the representation part (see Figure la, blue box) and prove that such identification is possible. The identification relies on two key assumptions outlined in Setting 1: (i) the exogeneity of and (ii) the linearity of the effect of A on Z. These two assumptions give rise to conditional moment restriction
the residuals
obtained from the linear regression of 96 (X) on A. Recall that for all encoders 6 : X + Z we defined (W p. ap) == argmitll cRaxi .RE [B[ [ (X) (W A + ax) ) Under Setting 1, we have
A x1))) :) A a] = Va supp(A) 1 (W 9o
(15)
The conditional moment restriction (15) motivates us to introduce the notion of linear invariance of an encoder & (with respect to A). Definition 4 (Linear invariance). Assume Setting 1. An encoder & X + Z is said to be linearly invariant (with respect to A) if the following holds
Ya E supp(A) BB((X) (WoA+ag) A = a] = 0.
(16)
To establish identifiability, we consider an encoder Φ : X + Z satisfying the following constraints.
(i) Φ is linearly invariant and
(ii) (m bijective,
(17)
where @lim(g)) denotes the restriction of & to the image of the mixing function 90- The second constraint (invertibility) rules out trivial solutions of the first constraint (linear invariance). For
(for some Idd) satisfies the linear invariance constraint instance, constant encoder ò -1 Theorem 5 shows that, under the assumptions listed below, but it clearly does not aff-identify 9o the constraints (17) are necessary and sufficient conditions for an encoder & to aff-identify gu
Here, Wa 0g and V, are equal to Ma. 9e and Va as shown in the proof. We introduce the new notation (c.g., (11) instead of (8) to emphasize that the expressions are functions of the observational distribution.

---

Published as a conference paper at ICLR 2024
Assumption (Regularity conditions gu). Assume Setting 1. The mixing function go is differen- riable and Lipschitz continuous.
Assumption (Regularity conditions on V). Assume Setting First, the characteristic function of the noise variable V has no zeros. Second, the distribution Pv admits a density fv w.r.t. Lebesgue measure such that fv is analytic on IR...
Assumption 3 (Regularity condition on A). Assume Setting 1. The support of A, supp(A), contains a non-emply open subset of IR*
In addition to the injectivity assumed in Setting Assumption 1 imposss further regularity condi- tions on the mixing function 90- As for Assumption 2, the first condition is satisfied, for example, when the distribution of V is infinitely divisible. The second condition requires that the density function of V can be locally expressed are the exponential functions, trigonometric functions, and any linear combinations, compositions, and products of those. Hence, Gaussians and mixture of Gaussians are examples of distributions that satisfy Assumption 2. Lastly, Assumption 3 imposes a condition on the support of Mc that is, the support of A has non-zero Lebesgue measure. These assumptions assumptions for bounded completeness in instrumental variable problems (D' Haultfoeuille, 201 1).
convergent power series. Examples of such functions
closely related to the
Theorem 5. Assume Setting I and Assumptions 1, 2, and 3. Let be a class functions from X to Z that are differentiable and Lipschitz continuous. It holds for all & E & that
& satisfies (17) & aff-identifies 9o
(18)
5 A METHOD FOR TACKLING REP4EX
5.1 FIRST-STAGE: AUTO-ENCODER WITH MMR REGULARIZATION
This section illustrates how to turn the identifiability result outlined in Section 4 into practical method that implements the linear invariance and invertibility constraints in (17). The method is based auto-encoder (Kramer, 1991; Goodfellow et al., 2016) with regularization term that enforces the linear invariance constraint (16). In particular, we adopt the the framework of maxi- mum moment restrictions (MMRs) introduced in Muandet al. (2020) a representation of the constraint (16). MMRs can be seen as the reproducing kernel Hilbert space (RKHS) representations of conditional moment restrictions. Formally, let H. be the RKHS of vector-valued functions (Al- varez et al., 2012) from A to Z with a reproducing kernel k and define e == vPx. (xz, a, Φ) + (xx) (W 3a + aia) (recall that Wa and as depend on the observational distribution 1. We can turn the conditional moment restriction in (16) into the MMR as follows. Define the function
Q(0)= = sup
(((A)))
(19)
A
If the reproducing kernel & is integrally strictly positive definite (see Muandet et al. (2020, Defini- tion 2.1)). then Q()) if and only if the conditional moment restriction in (16) is satisfied.
One of the main advantages of using the MMR representation is that it can be written as closed- form expression. We have by Muandet et al. (2020, Theorem 3.3) that
(())= = Bi A, 6) K(A, A' (')''''''' A))
(20)
where (X', A') is an independent copy of (X, A).
We now introduce our auto-encoder objective function' with the MMR regularization. Let Φ X - Z be an encoder and 1) : 2 + X' be a decoder. Our (population) loss function is defined as
Cle, m) = E|||X- ((((X))) + XQ(o),
(1))
where À is a regularization parameter. In practice, we parameterize & and by neural networks, use plug-in estimator" for (21) to obtain empirical loss function, and minimize that loss with
We consider basic auto-encoder, but one can add MMR regularization other variants too, e.g., (Kingma Welling, 2014), adversarial- (Makhzani et al., 2015), or diffusion- (Preechakul et al., 2022). 'More precisely, we replace the expectations in (21) and (11) by empirical means (the latter expression enters through vi and (())..
7

---

Published as a conference paper at ICLR 2024
The dimension of =
The dimension of
1.0
Xome
0.9
R-quuared
0.8
0.8
AE-MMR-Oracle
0.6
AE-Vanilia
VAE
0.6
0.4
0.5
1.0
2.0
2.5
3.0 3.5
4.0
0.5
1.0
2.0
2.5
3.0
3.5
Intervention Strength (a)
Intervention Strength (a)
Figure 2: R-squared values for different methods as the intervention strength (a) increases. Each point represents an average over 20 repetitions, and the error bar indicates its 95% confidence inter- val. AE-MMR yields an R-squared close to as a increases, indicating its ability to aff-identify 9o while the two baseline methods yield significantly lower R-squared values.
standard (stochastic) gradient descent optimizer. Here, the role of the reconstruction loss part in (21) is to enforce the bijectivity constraint of Φ [m(go)) in (17). The regularization parameter À controls the trade-off between minimizing the mean squared error (MSE) and satisfying the MMR. We discuss procedures to choose À in Appendix E.2.
5.2 SECOND-STAR :TAGE: CONTROL FUNCTION APPROACH
Given a learned encoder o, we can now implement the control function approach for estimating E|Y] do a" a²)] per Theorem 3. We call the procedure Rep4Ex-CF, Algorithm 1 in Ap- pendix E outlines the details. In summary, we first perform the linear regression of d(X) on A to obtain (s)) allowing us compute the control variables V <(X) (Wa. âœ). Subse- quently, we employ an additive regression model on (ø(X), V) to predict Y and obtain the additive regression functions & and vÙ. Finally, using the function iz we compute an empirical average of the expectation the right-han hand side of (13).
EXPERIMENTS
We now conduct simulation experiments empirically validate our theoretical findings. First, we apply the MMR based auto-encoder introduced in Section 5.1 and show in Section 6.1 that it can suc- cessfully recover the unmixing function 9n up to an affine transformation. Second, in Section 6.2, apply the full Rep4Ex-C2 procedure (see Section 5.2) to demonstrate that one can indeed pre- dict previously unseen interventions as suggested by Theorem 3. The code for all experiments is included in the supplementary material.
6.1 IDENTIFYING THE UNMIXING FUNCTION 96
This section validates the result of affine identifiability. see Theorem 5. We consider the SCMs
S(a) (A= EA z: aMMA+V
X = ==))
(22)
where the complete specification of this SCM is given in Appendix G.1. The parameter a controls the strength of the effect of A on Z. We set the dimension of X to 10 and consider two choices E (2, 4} for the dimension of Z. Additionally, we set the dimension of the dimension of Z.
We sample 1 1'000 observations from the SCM (22) and learn encoder á using the regularized auto-encoder
as outlined in Section 5. As our baselines, we include a vanilla auto- encoder (AE-Vanil) la)) and variational auto-encoder (VAE) for comparison. We also consider where we train the encoder and decoder using the true latent
an oracle model (AE-MMR-OzacC predictors Z and then these trained models to initialize the regularized auto-encoder. We refer to Appendix G.2 for the details on the network and parameter choices. Lastly, we consider iden- -1 only up affine transformation, Definition 2. To measure the quality of tifiability of go
8

---

Published as a conference paper at ICLR 2024
Rep4Ex:CF
Rep4Ex-CF-On
conditional mean
Training data
lest data
A- 2--URR--22.000 0.2)
A- -.7.f.0070...
A- Unift-12, 1.2)
10
0-
-10
0
0
0
Figure 3: Different estimations the target of inference C[Y]do(A = )] as the training support increases. The error bars represent the 95% confidence intervals over 10 repetitions. The training points displayed are subsampled for the purpose visualization. Rep4Ex-CF demonstrates the ability to extrapolate beyond the training support, achieving nearly perfect extrapolation when y 1.2. In contrast, the baseline MLP shows clear limitations in its ability to extrapolate.
an estimate we therefore linearly regress the true Z on the representation @(X) and report the R-squared for each candidate method. This metric is justified by Proposition in Appendix C.2.
Figure illustrates the results with varying intervention strength (a)). As a increases, our method, AE-MMR, achieves higher R-squared values that appear to approach 1. This indicates that AE-MMR can indeed recover the unmixing function 9n up an affine transformation. baseline methods, AE-Vanil 1 la and VAE, achieve significantly lower R R-squared values. indicating non-identifiablity without enforcing the linear invariance constraint, see also the scatter plots in Figures 5 (AE-MMR) and 6 (AE-Vanille la)) in Appendix H.
contrast, the two
6.2 PREDICTING PREVIOUSLY UNSEEN INTERVENTIONS
In this section, we focus on the task predicting previously unseen interventions as detailed in Section 3. We the following SCM data generating process.
8(7): {A==E
== MoA+V
X 90
(() +U))+U,
(23)
where c2 ~ Unif([->,,>'] Hence, the parameter ? determines the support of A in the observa- tional distribution. The complete specification of this SCM is provided in Appendix G.I.
Our approach, denoted by Rep4Ex-CF. follows the procedure outlined in Algorithm 1. In the first stage, employ AE-MMR the regularized auto-encoder. In the second stage, neural network that enforces additivity in the output layer for the additive regression model. For comparison, we include neural-network-bassee regression model (MLP) Y on A as baseline. We also include an oracle method, Rep4Ex-CF 1144EX-FF-Oracee where we use the true latent Z instead of learning a representation in the first stage. In all experiments, we use a sample size of 10'000.
Figure presents the results obtained with three y values (0.2, 0.7, 1.2), one- me-dimensional A and two- dimensional X. As anticipated, the neural-network-ber -et--eet-oobbeeedddd regression model (MLP) fails extrapo- late beyond the training support. Conversely, our approach, Rxp4Ex-CF, demonstrates successful extrapolation, with increased performance multi-dimensional A and present the results in Appendix H.I. Solving the optimization problem becomes more difficult but the outcomes echo the results observed with one-dimensional A.
higher y. Furthermore, we conduct experiments with
DISCUSSION
Our work highlights concrete benefits of identifiable representation learning. We introduce Rep4Ex, the task of learning a representation that enables nonlinear intervention extrapolation and propose corresponding theory and methodology. We regard this work only as a first step toward solving this task. Developing alternative methods and relaxing some of the assumptions (e.g., al- lowing for noise in the mixing function 9o and more flexible dependencies between A and Z) may yield more powerful methods for achieving Rep4Ex.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Nicola Gnecco and Felix Schur for helpful discussions. NP was supported by a research grant (0069071) from Novo Nordisk Fonden. ER and PR acknowledge the support of DARPA via FA8750--3-2-1015, ONR via N00014-23-1-2366 and NSF via IIS- IIS-1955532
REFERENCES
Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 15:15516-15528, 2022a.
Kartik Ahuja, Divyat Mahajan, Vasilis Syrgkanis, and Ioannis Mitliagkas. Towards efficient repre- sentation identification in supervised learning. In Conference on Causal Learning and Reasoning, pp. 19-43. PMLR, 2022b.
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representa- tion learning. In International Conference on Machine Learning, pp. 372-407. PMLR, 2023.
Mauricio A Alvarez, Lorenzo Rosasco, Neil D Lawrence, et al. Kernels for vector-valued functions: A review. Foundations and Trends@ in Machine Learning, 4(3):155-26,, 2012.
Joshua Angrist, Guido W Imbens, and Donald Rubin. Identification of causal effects using instrumental variables. Journal of the American statistical Association, 91(434):444 455, 1996.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. ArXiv e-prints (1907.02893), 2019.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 3588),1798-1888, 2013.
Gecia Bravo-Hermsdorf David S Watson, Jialin Yu, Jakob Zeitler, and Ricardo Silva. Intervention generalization: A view from factor graph models. arXiv preprint arXiv:2306.04027, 2023.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal repre- sentation learning. Advances in Neural Information Processing Systems, 35:38319-38333 2022.
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general non- linear mixing. arXiv preprint arXiv:2306.02335, 2023.
Theo Bühler and Dietmar A Salamon. Functional analysis, volume 191. American Mathematical Society, 2018
Ruichu Cai, Feng Xie, Clark Glymour. Zhifeng Hao, and Kun Zhang. Triad constraints for learn- ing causal structure of latent variables. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Rune Christiansen, Niklas Pfister, Martin Emil Jakobsen, Nicola Gnecco, and Jonas Peters. A causal framework for distribution generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6614-6633 2021.
Panayiota Constantinou and Philip Dawid. Extended conditional independence and applications in causal inference. The Annals of Statistics, pp. 2618-2653, 2017.
Xavier D' D'Haultfoeuille. On the completeness condition in nonparametric instrumental problems. Econometric Theory. 27(3):40077 2011.
Kenji Fukumizu, Arthur Gretton, Gert Lanckriet, Bernhard Schölkopf, and Bharath K Sriperum- budur. Kernel choice and classifiability for rkhs embeddings of probability distributions. Ad- vances in Neural Information P Processing Systems 22 (NeurIPS). Curran Associates, Inc., 2009.
10

---

Published as a conference paper at ICLR 2024
Nicola Gnecco, Jonas Peters, Sebastian Engelke, and Niklas Pfister. Boosted control functions. arXiv preprint arXiv:2310.05805, 2023.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Limor Gultchin, David Watson, Matt Kusner, and Ricardo Silva. Operationalizing complex causes: A pragmatic view of mediation. In International Conference Machine Learning, pp. 3875- 3885. PMLR, 2021.
Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling identifiable features from noisy data with structured nonlinear ica. Advances in Neural Information Processing Systems, 34:1624-6633, 2021.
James J Heckman. Dummy endogenous variables in a simultaneous equation system. Technical report, National Bureau of Economic Research, 1977.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications.
Neural networks, 134-5))441-4300 2000.
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: existence and uniqueness results. Neural Networks, 12(3):429-43, 1999.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868. PMLR, 2019.
Martin Emil Jakobsen and Jonas Peters. Distributional robustness of K-class estimators and the PULSE. The Econometrics Journal, 25(2):404-432 2022.
Yibo Jiang and Bryon Aragam. Learning nonparametric latent causal graphs with unknown inter- ventions. arXiv preprint 23v:2336600299, 2023.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference Artificial Intel- ligence and Statistics, pp. 2207-2217 PMLR, 2020.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference Learning Representations, 2014.
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal graphs via mixture oracles. Advances Neural Information Processing Systems, 34:18087- 18101, 2021.
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687-15701, 2022.
Lingjing Kong, Biwei Huang, Feng Xie, Eric Xing, Yuejie Chi, and Kun Zhang. Identification of nonlinear latent hierarchical models. arXiv preprint (rXiv:2306.07916 2023.
Mark A Kramer. Nonlincar principal component analysis using autoassociative neural networks.
AIChE journal, 37(2):233-244 1991.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference Causal Learning and Reasoning, pp. 428-484. PMLR. 2022.
Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Simon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: Gen- cralization and identifiability in multi-task learning. In International Conference on Machine Learning, 2023a.
11

---

Published as a conference paper at ICLR 2024
Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian crressaan-prouuct extrapolation. arXiv preprint arXiv:2307.02258, 2023b.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey, Adversarial autoencoders, arXiv preprint arXiv:1511.055 2015.
Gemma E. Moran, Dhanya Sridhar, Yixin Wang, and David M Blei. Identifiable deep generative models via sparse decoding. arXiv preprint arXiv:21 10804, 2021.
Krikamol Muandet, Wittawat Jitkrittum, and Jonas Kübler. Kernel conditional moment test via maximum moment restriction. In Conference on Uncertainty in Artificial Intelligence, pp. 41-50.
PMLR, 2020.
Preetam Nandy, Marloes H. Maathuis, and Thomas S. Richardson. Estimating the effect of joint in- terventions from observational data in sparse high-dimensional settings. The Annals of Statistics, 45(2):647 674, 2017.
Whitney K Newey, James L Powell, and Francis Vella. Nonparametric estimation of triangular simultaneous equations models. Econometrica, 67(3):565-603, 1999.
Judea Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, USA, 2nd edition, 2009.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10619-10629, 2022.
Thomas Richardson. Markov properties for acyclic directed mixed graphs. Scandinavian Journal of Statistics, 30(1):145-155)) 2003.
Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In International Conference on Machine Learning, pp. 9030-9039, PMLR, 2021.
Mateo Rojas-Carulla Carulla, Bernhard Schölkopf. Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research, 19(1):1309-1342 2018.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In International Conference on Learning Representations, volume 9, 2021.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint arXiv:2202. 068566 2022.
Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor regres- sion: Heterogeneous data meet causality. Journal of the Royal Statistical Society: Series B (Sta- fistical Methodology), 83(2):215-246, 2021.
Walter Rudin. Real and complex analysis, 3rd Edition. McGraw-Hill, 1987.
Sorawit Saengkyongam and Ricardo Silva. Learning joint nonlinear effects from single-variable interventions the presence of hidden confounders. In Conference on Uncertainty in Artificial Intelligence, pp. 300-309. PMLR, 2020.
Sorawit Saengkyongam, Leonard Henckel, Niklas Pfister, and Jonas Peters. Exploiting indepen- dent instruments: Identification and distribution generalization. In International Conference on Machine Learning, pp. 18935-18958. PMLR, 2022.
Alexander Schell and Harald Oberhauser. Nonlinear independent component analysis for discrete- time and continuous-time signals. The Annals of Statistics, 51(2):887-518, 2023.
Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke. Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEF, 10915)6612-633 2021.
12

---

Published as a conference paper at ICLR 2024
Anna Seigal, Chandler Squires, and Caroline Uhler. Linear causal disentanglement via interventions. arXiv preprint arXiv:2211.1646,, 2022.
Xinwei Shen and Nicolai Meinshausen. Engression: Extrapolation for nonlinear regression? arXiv preprint arXiv:2307. 00033, 2023.
Lester Telser. Iterative estimation of set of linear regression equations. Journal of the American Statistical Association, 59(307):845-862 862, 1964.
Burak Variei, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning with interventions. arXiv preprint arXiv:2301.0822 2023.
Julius von Kügelgen, Michel Besserve, Wendong Liang, Luigi Gresele, Armin Kekić, Elias Barein- boim, David M Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal represen- tations from unknown interventions. arXiv preprint 230,::::300000542,, 2023.
Norbert Wiener. Tauberian theorems. Annals of Mathematics, pp. 1-100, 1932.
Philip G Wright. The Tariff Animal and Vegetable Oils. Investigations in International Commer- cial Policies. Macmillan, New York, NY, 1928.
Pengzhou Abel Wu and Kenji Fukumizu. $Abetaš- - Identifying and estimating causal effects under limited overlap. In International Conference Learning Representations, 2022.
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identification of linear non-Gaussian latent hierarchical structure. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 24370-24387. PMLR, 2022.
Jiaqi Zhang, Chandler Squires, Kristjan Greenewald, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. arXiv preprint arXiv:2307.062500 2023.
13

---

Published as a conference paper at ICLR 2024
Appendices
A RELATED WORK: NONLINEAR ICA
Identifiable representation learning has been studied within the framework of nonlinear ICA (e.g.. Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Khemakhem et al., 2020; Schell & Oberhauser, 2023). Khemakhem al. (2020) provide unifying framework that leverages the independence structure of latent variables Z conditioned on auxiliary variables. Although our actions A could be considered auxiliary variables, the identifiability results and assumptions in Khemakhem et al. (2020) do not fit our setup and task. Concretely, key assumption in their framework is that the com- ponents Z are independent when conditioned on A. In contrast, our approach permits dependence among the components of Z even when conditioned on A the components of V in our setting can have arbitrary dependencies. More importantly, Khemakhem et al. (2020) provide identifiability up to point- wise nonlinearities which is not sufficient for intervention extrapolation. The main fo- cus of our work is to provide an identification that facilitates a solution to the task of intervention extrapolation. Some other studies in nonlinear ICA have shown identifiability beyond point-wise nonlincarities (e.g., Roeder et al., 2021; Ahuja et al., 2022b). However, the models considered in these studies are not compatible with our data generation process either.
B REMARK ON THE KEY ASSUMPTIONS SETTING 1
Remark (i) The assumption of linearity from Z on A can relaxed: if there is known nonlinear function h such that Z == Mgh((A) + V. we can define A = h(A) and obtain an instance of Setting Similarly, if there is an injective such that 7 h(M,A + VV) and X = go ()) we can define 7 == MoA + V and X = (go ()). (ū) The assumptions of full support of V and full rank of Mo can be relaxed by considering Z C Rd to be a linear subspace, with supp(V) and MoA both being equal Z. (iii) Our experimental results in Appendix H.3 suggest that it may be possible to relax the assumption of the absence of noise in X.
C FURTHER THEORETICAL RESULTS
C.1 IDENTIFYING E 2 = 2, do = a²)] WITH OBSERVED
For all z E supp(Z) and a" E A. we have
E|y Z z, do a') {(2) + BLUIZ 2, do(A= a')l
= E((2) + | Maa" +V = z, do = a")]
<((=) + ELUIV = Moa" do(A a")]
= =((x) + BLUIV = Moa"]
since Puvv = 1
(ddo(A=a))
= -((x) A(= Moa*).
where, f and X are identifiable by (3) if the interior of sApp(A) is convex (we still assume that supp(V)= = Rdd,, and the functions e and À are differentiable (Newey et al., 1999).
C.2 EQUIVALENT FORMULATION OF AFFINE IDENTIFIABILITY
Under Setting 1, we show an equivalent formulation of affine identifiability in Proposition stressing that Z can be reconstructed from o(X). In empirical evaluation (see Section 6), we adopt this formulation to define metric for measuring how well encoder o aff-identifies 9n _
Proposition 7 (Equivalent definition of affine identifiability). Assume Setting 1. An encoder Φ : X + Z idf-tieetsss 96 if and only if there exists matrix Jo E dRd and a vector dg E R2 s.t.
Vz E 2 Joo(x) + døg where == g0())-
(24)
14

---

Published as a conference paper at ICLR 2024
C.3 SOME LEMMATA
Lemma 8. Assume the underlying SCM (1). We have that U 11 Z V under ip..
Proof. In the SCM (1) it holds that (U,V) which by the weak union property of conditional independence (e.g., Constantinou & Dawid, 2017, Theorem 2.4) implies that A -l U V. This
in turn implies (A, V) U V (e.g., Constantinou & Dawid, 2017, Example 2.1). Now, by
Proposition 2.3 (ii) in Constantinou & Dawid (2017) this is equivalent to the condition that for all measurable and bounded functions g A x R : R it almost surely holds that
(lg(A, U, V1 Eig(A, V) V].
(25)
Hence, for all f 2 IR measurable and bounded it almost surely holds that
[Z) V = [[/(MM+VV) V
E[f(M V = EZ( Vj.
by (25) with g (a, + f( + v)
(26)
Again by Proposition 2.3 (ii) in Constantinou & Dawid (2017), this is equivalent to U 1L Z as desired.
As an alternative to our proof, one can also argue using SCMs and Markov properties in ADMGs (Richardson, 2003).
Lemma 9. Assume Setting 1. We have that U 1L Z Ve-
Proof.
Since the function u + Hạ(u -F[V] is bijective, the proof follows from the same arguments given in the proof of Lemma 8.
Lemma 10. Assume Setting I. Let Φ X - Z be encoder. We have that
@ o go is bijective
= Im is bijective.
(27)
Proof. Let & be an encoder such that á o 9o is bijective. We first show that (m is injective by contradiction. Assume that (mm(( is not injective. Then, there exist $1, T2 E Im(go) such that ((x1) = (x22) and Iı f x2. Now consider 21, 22 E with x = (2)) and æ2 = go g0(22); ); clearly, 21 22. Using that 6o go injective, we have (gogu) d(x1) & Φ(x2) (ø gu) (22) which leads to the contradiction. We can thus conclude that @ (mm))) is injective.
Next, we show that (m(()) is surjective. Let æ1, 22 E 2. Since Φ go surjective, there exist 21 22 Z such that 21 (Ø (z,) and 22 (o (ā2). Let #1 _)( E Im(g)) and I2 = (x2) Im(go). We then have that 21 = )(x1) and 22 = Φ(x2) which shows that (m surjective and concludes the proof.
D PROOFS
D.1 PROOF OF PROPOSITION 1
Proof. We consider k = d = 1, that is, A E R 2 E RR E R. We define the function pt R + IR.
for all u E R by
if (-4,2) 2) if E (2, x)) if (-00, -4)
(v)
1 expl - (v (--2)) 2))
7 eepp( 4)
and the function p R IR for v by
if E (-2,1)
12
1
if E -5,-2)
(2) exp(-(v- (c 1))
24
if E (1,00)
T6
exp(u + 5)
if E (-00,-1 -5).
I6
15

---

Published as a conference paper at ICLR 2024
These two functions valid densities have for all R that (b) > 0, Ve E R. pf- O, and S (v) de 1. f_ p (v) de 1. Furthermore, these two densities p1/(v) and py (v) satisfy the following conditions,
(1) for all a E (0, 1), it holds that
a+1
(b( du - a-
(v)
(28)
(2) for all (-3, 22) the following holds
a++
(u)
exp(v + 4) de
4
a-1
exp((a 1) +4)
1
2
12
(v) du.
(29)
a-
Next, let S be the following SCM
A = EA
SI:
= A+V
(30)
Y = 1(121 s UU
where ε Uniform(O.1), ~ IPt. ~ Pt independent such that e JL (V,U), and E(U] 0.
Further, we assume that V admits density Pv as defined above.
Next, we define the second SCM S2 as follows
A c cA Z -A
S2: :
(31)
y = 1(2
where 4 ~ Uniform(O,1), ~ Pe. U Pt independent such that eΛ 11 (V,U), E[U] = 0 and V has the density given by pt. By construction we have that supp (V) (upps>(()) Supps (A) Su (A). Now, we show that the two SCMs and Sn satisfy the third statement of Proposition I. Define 0 and = 1. For i E {1, 2}, we have for all E R that
and
ES. Y do(A a)) ( (1 al <
do a)] + S [U] do A a)] 1 + 1) do(A = a)] + E [U] do = a)]
B..[1((V a c, 1) do = a))
E [ a c| [1]
1(10 1 a c/ < (v) de,
(32)
where (*) holds because Va E A Py = pdo(A=a) and S [U] = =0 and (**) holds because Va E pdo(A=a) Since A is exogenous, have for all i {1,2) and suppSi (A) (0.
A Py y
16

---

Published as a conference paper at ICLR 2024
that E 1 { do(A a)] E Y A a]. From (32), therefore have for all (0.1)
S Y A = d) =
1( al s (u) do
(v)
p de
by (28)
a-2
1(1: - a + 11 1/ (v) de
E a
We have shown that the two SCMs S1 and Sn satisfy the first statement of Proposition 1. Lastly, we show below that they also satisfy the fourth statement of Proposition 1. Define B = (-3, 22- S R which has positive measure. From (32), we then have for all E (-3, -2)
E do( = a)]
1(v al S 1)py (v) de
a++
(v()) de
pV (v) du
by (29)
o-
1(1 + 1 S (v) de
E do a))
which shows that S and 82 satisfy the forth condition of Proposition 1 and concludes the proof.
D.2 PROOF OF PROPOSITION 7
Proof. We begin by showing the 'only if" direction. Let á X - Z be an encoder that aff-identifies 9o Then, by definition, there exists an invertible matrix Ho E ddddd and vector Go Rdd such that
Vz E z (Ø (z) = H8z + co.
(33)
We then have that
Ho. xzz)
Ho' cs. where r == =0]))),
(34)
which shows the required statement.
Next, we show the 'if' direction. Let ø X - be an encoder for which there exists a matrix
Jg E Rdxd and a vector dg E Rd such that
V2 E Z : Joo(z) + do. where I = go)).
(35)
Since Z = IRd, this implies that Jo is surjective and thus has full rank. We therefore have that
V= E 2 ( (=) Jo J 1
(36)
which shows the required statement and concludes the proof.
D.3 PROOF OF THEOREM 3
Proof. Let Ks + Ho2 + cø be the corresponding affine map of d. From (7), we have for all a" E A, that
E|Y| a²)] E((e Nz (Maa* + go + Vo)],
(37)
17

---

Published as a conference paper at ICLR 2024
where M = Ho. 90 co + H E and Va H&(V [V] as defined in (8). To prove the first statement, we thus aim to show that, for all a" E A,
Ely(Woa" + as + Vo]] - (B - B[y]) El( o Ki (Maa* + qs + Vo)].
(38)
To begin with, we show that We Ma and as qs- We have for a E R. W Rdxd
B[ll((X) (W A + a)
= [E[| M + ds + V - a W All
from (9)
[ - W)A + (9p a) + Voll
[BIl(Ms W)A+ + (gs a)l²
[Vo] + + - A + (gs - a)) Vol + B|I
[M W)A+ + (go a)]l" =1IIV=
since A IL Vo and E[V;]
Since the covariance matrix of A has full rank, therefore have that
(cx0, W&) argmin GER",WER*X
B[ll((X) a- WAll (gp, Ma),
(39)
and that Vo x(X) (M&A + go) Va. where the last equality holds by (9).
Next, we show that (fox,') ). Since differentiable, the function Loxz is also differentiable. We have supp(A, (A, V/)= = supp(A, V) supp(A) x Rd. Thus, the interior of supp(A, Vo) is convex (as the interior (Ap is convex) and its boundary has measure zero. Also, the matrix My has full row rank. Moreover, using aff- iffniiitifiiiiiitty and (4) we can write
<(X) = MM. + V = K (x()) U.
where JL (V&, U). This is a simultaneous equation model (over the observed variables ((X), A, and Y) for which the structural function is 15g" and the control function is Ag. We can therefore apply Theorem 2.3 in Newey et al. (1999) (see Gneeco al. (2023, Proposition 3) for a complete proof, including usage of convexity, which believe is missing in the argument of Newey et al. (1999)) to conclude that E o and As are identifiable from (10) up to a constant. That is,
&
v == (L ) and
+ Ao -
(40)
for some constant d E IR. Combining with the fact that We MM and as = 9á. we then have, for all a' A.
Elp(Waa* + as Vo]] E[(e k. (Maa' + qs + Vo)] + 5.
(41)
Now, we use the assumption that REUU- = 0 to deal with the constant term 5.
B|y] BiClyo
((X() (X))]
since B(U] = 0
(42)
E[((E o xz 2 (Nig 90 (x))
(43)
= E[(E x²6 (b(x))
since & aff-identifies 96
(44)
Thus, we have
B[r(((X)) E|y = E[(e =(((ox=))(0(X)) (o(x)) - E|y
by (40)
E((A o o ())))) + 5] E[(e o K5 ( (o(x))
by (44)
(45)
Combining (45) and (41), we have for all at A that
x B[p(Waa" + as + Vo)] (E(v(((X)) B[y)) = E[(e K3 (Maa' + + + Vo)).
which yields (38) and concludes the proof of the first statement.
18

---

Published as a conference paper at ICLR 2024
Next, we prove the second statement. We have for all Im(g)) and a' A, that
E| = 2, do(A = a²)] = El((() X = x, do( - + E = 2, do(A -- = a²)]
(6 90 (xx E|u - x. do a')]
(60 B(U E(V (Maa" + +V) - x, do A a')]
do a")]
(cog5 (x)) E/U
(6 (x)
90 (x) Moa' do(A a²)
( (eog0 (x)) = 90 (x) - Muaa
(Clong (KS 96' ( ()())))))))))))))))))))) BUUVV 90 (x) - Moa"l
KLo (Ø((z) + ElUi = 90 (x) - Moa"l
(46)
where the equality (+) hold because Va E AA Pv,v Ido(A=a") and (**) follows from the fact that Φ aff-identifies 9o Next, define = + Holv - B[V]). We have for all x Im(gg) and
a' A that
Mgr (x)) Maa*) = Halso (x) - Mha - E[V])
Hagn (x) HaMaa* Ho E[V]
Hogo (x) cœ (Msa' + 9o)
(Ø 0 go o 9o (x)) (Maa" + (0)
=(x) - (Maa + 96)
((x)) (Waa" + a.))-
from (39)
(47)
Since the function h is bijective, combining (47) and (46) yields
X| = x, do = a a²)] = (lo x (d(x)) + (EU[)(() h(9o ()) - Moa")
(f (d(x)) = 6(x) (Wau* a+)l
= (e (d(=)) + {s(0(x) - (Woa" + az)).
Lastly, as argued in the first part of the proof, holds from Theorem 2.3 in Newey et al. (1999) that (E n. -1 8 and = A 5, for some constant 8 E R. We thus have that
Vz Im(go), a" A: | x, do(A a a")] =(((=)) + (((()) (Woa" + ta)),
which concludes the proof of the second statement.
D.4 PROOF OF THEOREM 5
We begin the proof by showing the forward direction (ø satisfies (17) Let E & be an encoder that satisfies (17). We then have for all E supp(A)
Proof.
& satisfies (6)).
Waa + as = [[o(X) A = a]
[ go) (MoA + V) AA a]
El( o 9o) g)) + V)]
since A JL V.
Define h := 6 Tu- Taking derivative with respect to both sides yields
i EEh + V)]
Wa
da Next, we interchange the expectation and derivative using the assumptions that & and go have bounded derivative and the dominated convergence theorem. We have for all supp(A)
Maa + V)
Wa E
Oh(u))
a(Moa + V)
E
by the chain rule
Ou
aa
Maa++
ah(uv)
Mo].
(48)
Ou
M=M
19

---

Published as a conference paper at ICLR 2024
Defining N' @h(u) du
and g g:2t. 22
1x221(12)) Mo Wa, we have for a supp(A)
0 = (Maa + V)Mo Wol
Elg(M + v)]
/ g(Moa + ()fv
Define t: MMa E Fd and 7 + v, then have for all t (A that
/ o(r)fv(T - ()
0
I (r - I ((t)sv((-T]dd
(49)
Recall that is function from IRd to Rdxk Now, for an arbitrary (i,j) E IRd x IR* define the function Psy Rd + IR g(-) We then have for each element (i,5) and all supp(M,A) that
f (u(()f-v((t T)dT.
(50)
Next, let us define Cig : LE E Tdd + ( 11-vv(l--00 - E IR. We now show that Gj = 0 where adapt the proof of D' H'Hallffouillle (2011, Proposition 2.3). By Assumption 2, f-v analytic Rdd we thus have for all r E Rd that the function 1 ~ 9b5 ( is analytic on TRd. Moreover, (
since 9ij is bounded the function
7) bounded, too. Thus, by (Rudin, 1987, page 229), the function Cjj is then also analytic on Rd Using that Mo is surjective, we have by the open mapping theorem (see e.g., Bühler & Salamon (2018), page 54) that My is an open map. Now, since supp(AA) contains non-empty open subset of R and My is an open map, we thus have from (50) that Ciy (t) = 0 on non-empty open subset of Rd. Then, by the identity theorem, the function Gjj is identically zero, that is,
Gj = 0.
(51)
Next, we show that Pi = O. L' denote the space of equivalence classes of integrable functions from Rd to R. For all Rd let define f((-) f-v (t- and Q {ffe t E R By Assumption 2, the characteristic function of does not vanish. This implics that the characteris- tic function of -V does not vanish either (since the characteristic function of -V is the complex conjugate of the characteristic function of V). We therefore have that the Fourier transform of f-v has real zeros. Then, we apply Wiener's Tauberian theorem (Wiener, 1932) and have that Q is dense in L Using that Q is dense in L combining with (51) and the continuity of the linear form E L + (i5(( (15(7)((T)d (continuity follows from boundedness of gij and Cauchy-Schwand holds that
f (g(())(()dd
Vo E L'
(52)
From (52), we can then conclude that
-v,(-)
(53)
Next, from (53) and the definition of g. we thus have for all supp(A) and v Rd
h' HMMaa + uo Wa.
(54)
As Mo has full row rank, it thus holds that
h hM + +e) - W.
(55)
We therefore have that the function h = و go is an affine transformation. Furthermore, using that Do is injective and (m)())) is bijective, the composition h pog go is also injective. Therefore, there exists an invertible matrix H Rdxd and vector E Rd such that
V2 I do (=) H +
(56)
20

---

Published as a conference paper at ICLR 2024
which concludes the proof of the forward direction.
Next, we show the backward direction of the statement (Ø satisfies (6) = satisfies (17)). Let á & satisfy (17). Then, there exists invertible matrix H Raxd and vector E R' such that Vz E R² (o (z) = Hz + c. We first show the second condition (17). By the invertibility of H, the composition Do is bijective. By Lemma 10, we thus have that @lim(an) is bijective. Next,
we show the first condition (17). Let jV E.V. We have for all Rd W Rdxd
(X) WAll
o - a WAl
+c a- WAI
+ [I]IH(MAA++]+
WAI
[E] W) - a) +
[E] W)A (c+ Huy - a) H(V -vv)
MM W)A - (c+ HNY -a]ll²
=
2 M W)A + (c+ Huy a)) H(V - pv)] + (V uv)l since 1
= (M W) (c + Hpv - at)l + BBI| - -v
Since the covariance matrix of A is full rank, we therefore have that
(0'., w
argmin a WeRe
B[(((())
W A[ = (c+ Huv, HMo).
(57)
Then, we have for all E supp(A) A) that
B[o(X) - as - W&A A = a]
- - Hpv) M. A = a)
-EHHZ + (c+ - HMAA| A = a)
EE[H(MA V) +c (c+ Hpv) - MMM A=
-[HV -Hpv AA
(.)) Hμy Hμy 0,
where the equality (*) follows from (57) and (**) holds by IL V. This concludes the proof.
E DETAILS ON THE ALGORITHM
E. THE ALGORITHM FOR REP4EX
We here present a pseudo algorithm for Rep4Ex- CF, see Algorithm 1.
E.2 HEURISTIC FOR CHOOSING REGULARIZATION PARAMETER À
To select the regularization parameter À in the regularized auto-encoder objective function (21), we employ the following heuristic. Let A = {Ai..... Am be our candidate regularization parameters, ordered such that λ > A2 > > Am- For each A,, we estimate the minimizer of (21) and calculate the reconstruction loss. Additionally, we compute the reconstruction loss when setting as the baseline loss. We denote the resulting reconstruction losses for different X as Rx, (and Rg for the baseline loss). Algorithm 2 illustrates how À is chosen.
In our experiments, we set a cutoff parameter at 0.2 and for each setting execute the heuristic al- gorithm only during the first repetition effectiveness our heuristic. Here, our algorithm would suggest choosing = 10?, which also corresponds to the highest R-squared value.
computation time. Figure 4 demonstrates the
Another approach to choose À is to apply the conditional moment test in Muandet et al. (2020) to test whether the linear invariance constraint (16) is satisfied. Specifically, in similar vein Jakobsen
21

---

Published as a conference paper at ICLR 2024
Algorithm 1: An algorithm for Rep4Ex
Input:
observations (Iz, 1-) target interventions (az) auto-encoder AE, additive
regression AR
// Train the auto-encoder
AE((,;; i;);;
// Regress ((X) on A
(W p: âه) argmin E [(((;;) (Wa, + a)ll:
// Obtain the control variables
for i to do
8 ((x)) (Wa++a)
end
// Train additive regression
8, b = AR(ы ~ v(o(z;)) + i=...))
// Est imate B[Y do(A=
for j = to do
9j i-1 D(W 2a + as u.) En (p(0(z))) vi)
end
Output: (ū,)
Algorithm 2: Choosing À parameter
Input: cut off parameter a + Am
for m
1 do
6, = Rx. 1;
Ro
if 6, < then
< A;;
break
return A
& Peters (2022); Saengkyongam al. (2022), we may select the smallest possible value of À for which the conditional moment test is not rejected.
F POSSIBLE WAYS OF CHECKING APPLICABILITY OF THE PROPOSED METHOD
Due to the nature of extrapolation problems, is not feasible definitively verify the method's underlying assumptions from the training data. However, we may still be able to check and poten- tially falsify the applicability of our approach in practice. To this end, we propose comparing its performance under two different cross-validation schemes:
(i) Standard cross cross- validation, where the data is randomly divided into training and test sets.
(ii) Extrapolation-awee aware cross- cross-validation, in which the data is split such that the support of A in the test set does not overlap with that in the training set.
By comparing method's performance across these two schemes, of our overall method. A significant performance gap may suggest that some key assumptions are not valid and one could consider adapting the setting, e.g., by transforming A (see Remark 6).
assess the applicability
further option of checking for potential model violations is to test for linear invariance of the fitted encoder, using for example the conditional moment test by Muandet al. (2020). If the null hy- pothesis of linear invariance is rejected, this indicates that either the optimization was unsuccessful or the model is incorrectly specified.
22

---

Published as a conference paper at ICLR 2024
MSE AE-Vanilla
R Squaree
0.925
0.025
0.900
0.020
R-Squared
0.015 MW
0.850
0.825
0.010
0.800
0.005
0.775
10
109
10
10
10a
10a
10a
Regularization parameter (1)
Figure 4
G DETAILS ON THE EXPERIMENTS
G.1 DATA GENERATING PROCESSES (DGPs) SECTION 6
DGP for Section 6.1 We consider the following underlying SCM
S(a): {A= Z = aMM +V
X x
(58)
where сд Unif(-1,,1) and V (N., Σ) are independent noise variables. Here, we consider four-layer neural networks with Leaky ReLU activation functions the mixing function go. The parameters of the neural networks and parameters the SCM (22) including E and Mo are randomly chosen, see below for more details. The parameter controls the strength of the effect of A on Z. In this experiment, we set the dimension X to 10 and consider two choices E {2,4) for the dimension of Z. Additionally, set the dimension of A to the dimension of Z.
DGP for Section 6.2
We consider the following underlying SCM
S(7):
{A (A=6%
= MoA+1
X 9o) Y ((2) U.
(59)
where ei ~~~ ~ Unif([-n: 71a) and ~ (0, Ev) are independent noise variables. is then gener- ated 0 = h(V) EU. where eg N 1). The parameter 7 determines the support of A in the observational distribution. Similar to Section 6.1, we consider a four-layer neural networks with Leaky ReI.U activation functions the mixing function To and the parameters of go- Ey, and Mo are randomly chosen as detailed below.
Details on other parameters In all experiments, we employ a neural network with the following details as the mixing function 90:
Activation functions: Leaky ReLU
Architecture: three hidden layers with the hidden size of 16
Initialization: weights are independently drawn from Unif( 11,
As for the matrix MM. each element is indepedently drawn from Unif(- 2). The covariance Σv is generated by Σy : AAT + diag(V) V), where A and V are indepedently drawn from Unif([0, 1
In Section 6.2, the case of one-dimensional A, we specify the functions h and & as o + Lu f: + 22 + si)) In the case of multi-dimensione A (sce Appendix H.1), employ the following neural network for both h and E:
Activation functions: Tanh
23

---

Published as a conference paper at ICLR 2024
Architecture: one hidden layer with the hidden size of 64
Initialization: weights are independently drawn from Unif(-1,,)
Lastly, in all experiments, we use the Gaussian kernel for the MMR term in the objective function (21). The bandwidth of the Gaussian kernel is chosen by the median heuristic (e.g., Fukumizu al., 2009).
G.2 AUTO-ENCODER DETAILS
We employ the following hyperparameters for all autoencoders in our experiments. The same archi- tecture is utilized for both the encoder and decoder:
Activation functions: Leaky ReLU
Architecture: three hidden layers with the hidden size of 32
Learning rate: 0.005
Batch size: 256
Optimizer: Adam optimizer with 31 = =0.9, 33 = 0.999
Number of epochs: 1000.
For the variational auto-encoder, we employ standard Gaussian prior with the same network archi- tecture and hyperparameterss as defined above.
H FURTHER DETAILS ON EXPERIMENTAL RESULTS
Figures 5 and show reconstruction performance of the hidden variables for the experiment de- scribed in Section 6.1.
Method AE-MMR
3
-2
True
-1
Taae
22
-4
0.4 o
Estimated 2
Estimated 22
-02 0.0
0.1
0.2 0.3
-0.4
-0.3
0.2 0.1 0.0
Estimated Zz
Estimated Z₁
Figure 5
H.1 SECTION 6.2 CONTINUED: MULTI-DIMENSIONAL.
Here, we consider multi-dimensional variables Z and A. Similar to Section 6.1, we set the dimen- sion of X to 10, vary the dimension d of Z, and keep the dimension of A equal to that of Z. We specify the functions h and using two-layer neural networks with the hyperbolic tangent activation functions. For the training distribution, we generate A from uniform distribution over |-1, 1]d. To assess extrapolation performance, we generate 100 test points of A from a uniform distribution over (-3,- -1"" and calculate the mean squared error with respect to the true conditional mean. In addition to the baseline MLP, we also include an oracle method, denoted as Pep4ExxCF-Oree where directly the true latent predictors Z instead of learning representation in the first
24

---

Published as a conference paper at ICLR 2024
Method AE-Vanilla
3
-6
2
1
-2
True N
0
0
1
-2
10 Sarn SatimatedZZ
-3
0 Estimated 2
-5 -10 15-20
1510.5 0 0-5 Estimated 2;
10 Estimated Z2
5
Figure 6
The dimension of
The dimension of
The dimension of A 10
102
10a
10
Meen squared
101
101
10"
10
10a
10a
Rep4Ex-CF
Rep4Ex-CF-Oracle
Figure MSEs of different methods for three distinct dimensionalities of The box plots illustrate the distribution of MSEs based on 10 repetitions. Rep4Ex-C7 yields substantially lower MSEs in comparison to the baseline MLP. Furthermore, the MSEs achieved by Rxp4Ex--0 are comparable to those of Eep44xxCF-Oracle underscoring the effectiveness of the representation learning stage.
stage. The outcomes for d E 42,4, 2,,4,00 are depicted in Figure Across all settings, our pro- posed method, Rep4Ex-CF, CF, consistently achieves markedly lower mean squared compared to the baseline MLP. Furthermore, the performance of Rep4Ex-CF CF is on par with that of the ora- indicating that the learned representations are close the true
cle method Rep4Ex-CF-Oral latent predictors (up to an affine transformation).
H.2 SECTION 6.2 CONTINUED: IMPACT OF UNOBSERVED CONFOUNDERS
Our approach allows for unobserved confounders between Z and Y. This section explores the impact of such confounders on extrapolation performance empirically. We consider the SCM as in (59) from Section 6.2, where set 7 1.2 and generate the noise variables U and V from joint Gaussian distribution with the covariance matrix Σu,v (2 Here, the parameter controls the dependency between U and V, representing the strength of unobserved confounders. Figure presents the results for four different confounding levels p (0, 0.1, 0.5, 0.9). Our method, Rep4Ex-CE demonstrates robust extrapolation capabilities across all confounding levels.
H.3 SECTION 6.2 CONTINUED: ROBUSTNESS AGAINST VIOLATING THE MODEL ASSUMPTION OF NOISELESS
In Setting I, we assume that the observed features are deterministically generated from Z via the mixing function 9o- However, this assumption may not hold in practice. In this section, investigate the robustness of our method against the violation of this assumption. We conduct an experiment with the setting similar to that with -iedimensioonll A in Section 6.2 but here in- troduce independent additive random standard Gaussian noise X, i.c., X == go (Z) + éx, where ex NO0 o² J.). The parameter controls the noise level. Figure illustrates the results for
25

---

Published as a conference paper at ICLR 2024
RepAEx-CF-Oracle
conditional
Training data
P-00
10
-5
-10
Figure 8: Different estimations of the target of inference E|y do(A = -) as the strength of unob- served confounders (p) increases. Notably, the extrapolation performance of Rep4Ex-CE remains consistent across all confounding levels.
Rep4Ex-CF
Rep4Ex.CF -F-rralee
MLP
True conditional
Training data
0=
0-
-10
Figure 9: Different estimations of the target of inference E|Y do(A )] in the presence noise in X. Our method, Rep4Ex-CE demonstrates the ability to extrapolate beyond the training support when the noise is not too large, suggesting the potential relax the assumption of the absence of noise in X.
different noise levels = (1, 4). The results indicate that our method maintains successful ex- trapolation capabilities under moderate noise conditions. Therefore, believe it may be possible to relax the assumption the absence of noise in X.
26


Reviews:
[{"review_id": "B9KZfISwox", "reviewer": "Reviewer_iVVC", "summary": "The authors propose a method to learn an effect through learning a representation of latents (Z) from observed data (X) where X and Z are related through some injective function (in practice an encoder model). It is assumed that interventions in A act on Z through which they act on some required outcome variable Y through linear functions. In such a setting the authors propose a method for learning the effects of ood interventions a^* in A. They show through experiments the efficacy of their method.", "questions": "- Proposition 1: If for all a in the support of A, E^S1[Y|A=a]=E^S2[Y|A=a], then how is there a set with positive lebegue measure over the support s.t the do distributions are not equal? The issue is one of measure of sets outside support being 0. Some clarification remarks would be helpful as to what positive measure outside supp(A) means.\n- The linear assumption is reasonably strong. Future work may be required to extend it to GLM's or non-linear representation learning.\n- Is the functional form of the SCM necessary for extrapolation? Can there be such analyses on CBNs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Important problem. Can be thought of as OOD estimation of intervention effects through learning latent representations.\n- Extremely well written paper!\n- Crucially E[Y | do(A=a')] \\neq E[Y | A=a'] for a' not in support of A.\n- The propositions are exactly at the places that the reader thinks about the question, and are easily understandeable.\n- The proofs of extrapolation are not straightforward. There have been several causal tools brought together to show the validity of extrapolation (invariance principle, mixing-unmixing, instrumental variable approaches. I quite liked the work.", "weaknesses": "- I did not see any major weakness. One model assumption that could be weakened in future work is the linearity assumption.\n- The role of the Wiener’s Tauberian theorem in the proof of hidden representation being identifiable, upto affine transformation, is not clear to me. Since this has been claimed in the abstract it would be helpful to delineate where it has been used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSbH3o3hzP", "reviewer": "Reviewer_k2zA", "summary": "**Post rebuttal update**: I'd really like to see a couple of real-world examples that might satisfy the assumptions, but my other concerns are largely addressed. I raise my score to 8 accordingly.\n\nThe paper considers the causal effect of an intervention $A=a^*$, where value $a^*$ is not observed in the data; A affects the outcome Y through unobserved variable Z, and covariates X relates to Z through an injective function g. The paper uses a conditional moment restriction (CMR) implemented by a kernel method called maximum moment restriction (MMR), and it then uses the control function (CF) approach to identify the outcome function between Z and Y. Both the CMR and CF depend on the assumption that A affects Z linearly. The approach is theoretically guaranteed, and experiments on synthetic datasets support the theoretical analysis.", "questions": "I cannot understand the importance of Proposition 3, and it seems just a trivial restatement of Def 2 and adds confusion to me.\n\nI think the title would be better to stress the CF approach because this arguably contributes more to intervention extrapolation than “identifying representation”.\n\nIt is weird that Wiener’s Tauberian theorem is mentioned in the Abstract but not the main text.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Using CF to achieve intervention extrapolation is a nice idea.\n\nThe theoretical analysis is serious and detailed (but I did not check the proofs in Appendix).\n\nThe paper is quite well written.", "weaknesses": "*Technical novelties seem to be weak*. Theorem 4 seems to be an adaptation of the CF approach in (Newey et al., 1999), and Theorem 6 seems to be an adaptation of the IV approach in (D’Haultfoeuille, 2011). If there are some technical novelties, they should be discussed and compared to the original works; otherwise, I suggest being more explicit about this weakness.\n\n*Some assumptions are strong*; particularly, the linear model between Z and A, and the injective model and noiseless model between Z and X. Intuitively, X is an observable proxy of the hidden Z, and the assumption means there is no information loss in this proxy, which is strong. Moreover,  both assumptions involve hidden variable Z and add difficulty to practical judgments. Since both assumptions are inherent in the current approach, I do not expect the author(s) to address this weakness in the rebuttal, but the following could certainly be done.\n\n*Discussion of the setting and comparison to related work*. The discussion of the relationship to reinforcement learning is interesting but does not touch on when can we possibly expect linearity and noiseless injectivity. It would be more interesting to draw and discuss a couple of real-world problems that might satisfy the assumptions. \n\nOn the other hand, (Khemakhem et al., 2020) and [1, 2], which are based on the former, are important related work that needs more discussions, and this would clarify the current approach.  For example, (Khemakhem et al., 2020) recover Z based on exactly the same graph as A→Z→X, and also assume g is injective but *allows an additive noise on X*; the identification also relies on assumptions on the A→Z part, where p(Z|A) is assumed to be an exponential family distribution but *allows nonlinearity*. Overall, I do not think the assumptions in (Khemakhem et al., 2020) are clearly stronger than those in the current work. Further, [1, 2] uses (Khemakhem et al., 2020) to estimate treatment effects, though not considering intervention extrapolation, [2] mentioned the ideas of CMR and CF in Sec 4.4.\n\n[1] Wu, Pengzhou Abel, and Kenji Fukumizu. \"$\\beta $-Intact-VAE: Identifying and Estimating Causal Effects under Limited Overlap.\" International Conference on Learning Representations. 2022.\n[2] Wu, Pengzhou and Kenji Fukumizu. Towards Principled Causal Effect Estimation by Deep Identifiable Models”. In: arXiv preprint arXiv:2109.15062 2021\n\n*Additional experiments could be added*.\n\nI think the ability to deal with unobserved confounders is a strength of the work. So, why not add experiments on this? I know eq24 contains hidden confounding, but this direction is not examined, e.g., by adjusting the strength of confounding and comparing to other methods.\n\nAs indicated above, adding real-world problems in experiments can greatly strengthen the work. Also, it would be interesting to replace the CMR part with iVAE (Khemakhem et al., 2020) and see how the results would change.\n\nI will read the rebuttal and revised paper and raise my score to 8 if the issues/questions above are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UwIWJzEqfK", "reviewer": "Reviewer_dGv9", "summary": "The paper discusses identification strategies for effect extrapolation when the treatment of interest is unobserved during the experiment. The key assumptions are that \n\n- (exogenous) treatment A affects outcome Y through and only through an unobserved mediator Z\n- can observe a rich feature $X=g_0(Z)$ that is and only is a (injective) function of Z\n- the relationship between A and Z is linear\n\nTogether with some other regularity assumptions, the author shows that, given an encoder that aff-identifies $g_0^{-1}$, it is possible to identify $E[Y|do(A=a^*)]$ and $E[Y|X=x,do(A=a^*)]$ for a treatment $a^*$ that has never been observed. They also propose a method for locating such an encoder.", "questions": "- It seems that this approach relies heavily on the linear structure between A and Z. Can this structural model be extended to $M_0 t(A) + V$ for some transformation $t(A)$ (e.g., $A^2$)? What if the relationship between A and Z is not linear, but rather through a known class of models $m_{\\theta}(A)+V$ with parameters $\\theta$? How would this affect the result?\n\n- Is it possible to view the linear assumption as an approximation through a Taylor expansion around supp(A)? How would this compare with an extrapolation that is solely based on Lipschitz assumptions (see, e.g., Ben-Michael et al. 2021)?\n\n- Although this paper is only about identification, I am curious about how the errors would accumulate.\n\n- I find the sudden change of notation in Theorem 4 confusing.\n\n- Can there be randomness in X, i.e., can X be a noisy observation of $g_0(Z)$?\n\n- On page 2, the authors wrote \"we allow for potentially unobserved confounders between Y and Z\". How could there be confounding when the action is exogenous?\n\n- Although the goal is very different, the approach reminds me of the negative control literature. The authors may want to discuss the connection.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper studies intervention extrapolation under a well-chosen set of assumptions, which I find more appealing than the previously studied scenarios in the literature.\n\n- To the best of my knowledge, the proposed identification strategy is novel.\n\n- The manuscript is clear and well-written. \n\n- The proposed algorithm is straightforward and practical.", "weaknesses": "- The assumptions are clear mathematically but might seem opaque to readers unfamiliar with the literature. The authors may want to give an example of what some of the key assumptions would imply in a simple setup.\n\n- Many of the structural assumptions are not testable, and it is unclear to me when one shall be comfortable using the proposed method.\n\nAlso see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kiTFzv2pN7", "reviewer": "Reviewer_wfVE", "summary": "This work introduces a type of interventional extrapolation which consists in predicting the effect of an unseen intervention. The causal model considered is given by A -> Z -> X and Z->Y where Z is latent, A -> Z is linear, Z->X is invertible and Z -> Y is an additive noise model. The goal is then to estimate E[Y | do(A = a*)] where a* is outside its training domain. The authors separate this problem in three subproblems: 1) estimation given that Z is observed, 2) estimation this given that Z is known only up to an affine transformation, and 3) how to identify the invertible map Z->X up to an affine transformation. These identifiability results are then transferred into a practical algorithm: a) The mixing function is recovered by performing SGD on a loss consisting of a reconstruction term and a regularizer based on maximum moment restriction (MMR), and b) the learned encoder is then used to estimate the desired “do” expectation following the procedure of step 2) above.  The proposed estimation procedure is then validated on a synthetic dataset.\n\nReview Summary: I gave the main text a thorough reading and checked the large majority of the math it presents. Despite the few problems I raised in my review, I believe this work is sufficiently novel, interesting, rigorous and well written to warrant acceptance to ICLR. I was planning to give a score of 7, since this score is not available, I'm rounding up to 8. I'm giving this score assuming that a discussion of the limitations will be added, as I suggest in my review.", "questions": "- Lemma 7 in Appendix A: Could you explain the argument a bit more? I’m unaware of this proof technique, so citing the result used here would be useful. Or is it an alternative definition of conditional independence that I’m not aware of?\n- Paragraph after (3):\n    - Could you give some insight as to why $\\ell$ and $\\lambda$ are not identifiable up to additive constant without further assumptions in (3)? I’m not sure I see what could go wrong since we observe Y, Z and V here (V is identifiable). Also, can you state the assumption from Newey et al. (1999) that allows you to identify $\\ell$? I suspect it has something to do with the differentiability + supp(A) convex assumption from Theorem 4?\n    - It is written “for all $a^* \\in M_0 supp(A) + supp(V)$” somewhere, but it seems wrong since $M_0 supp(A) + supp(V)$ is an event for Z, no? I’m not sure I’m following this part of the argument and onward.\n    - Same paragraph: I think we need to compute $\\ell(z)$ only for $z \\in M_0\\mathcal{A} + supp(V)$, no? \n- Eq. (16): I’m not sure why this holds. Is it because the inside of the conditional expectation is equal to V? But we don’t assume E[V] = 0 right? What am I missing? Since this is so crucial to the algorithm, it might be worth expliciting further.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- I agree that we need more theory to demonstrate concrete benefits of identifiable representation learning. I enjoyed this perspective.\n- The paper is very clearly written and structured and relatively easy to follow despite its technicality. The level of mathematical rigor in this work is very high in my opinion and the notation is always very crisp and transparent. \n- I thought the theoretical contributions were novel and interesting. \n- I appreciated that the main ideas about the proofs were provided in the main text. Most works in this literature only give very short proof sketches if any. \n- Theoretical results are well modularized to facilitate reuse.\n- I thought the method to learn the encoder up to affine transformation was novel and interesting.\n- I’m left with a good feeling of having learned something new.", "weaknesses": "- The limitations of the proposed approach could be discussed further. For example, how does the method perform under various assumption violations? A few experiments could provide some insight into this. \n- The paper ends a bit abruptly. There’s no conclusion nor discussion of limitations. I guess this is the cost of having so much technical details in the main text (which, as I said, I appreciated, but I’m unsure whether this is a good balance.)\n- I think the paper could do a better job of contrasting its results with what already appears in the literature, especially regarding the affine identifiability results. How does this compare to iVAE for example, which has a very similar data generating process with an auxiliary variable (which would be A here)? A few papers have similar results: [1,2,3,4].\n- I couldn’t grasp a few reasoning steps in the main text (see Questions below).\n\nMinor:\n- $M_0$ has full row rank implies that dim(A) >=  dim(Z). Might be worth making explicit.\n- I always forget which noise variable U or V is associated with which variable. How about replacing V by V_z and U by V_y?  \n- Footnote one: what is $\\mathcal{B}$?\n- The introduction rightfully points out that the literature has almost no works giving theoretical arguments for why the identifiable representation learning is important. You might want to consider citing [1, 5] as examples of work pushing that direction.\n\n[1] S. Lachapelle, T. Deleu, D. Mahajan, I. Mitliagkas, Y. Bengio, S. Lacoste-Julien, and Q. Bertrand. Synergies between disentanglement and sparsity: a multi-task learning perspective, 2022. \n\n[2] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ¨ ica: A unifying framework. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, 2020.\n\n[3] Ahuja, K., Mahajan, D., Syrgkanis, V., and Mitliagkas, I. Towards efficient representation identification in supervised learning. In First Conference on Causal Learning and Reasoning, 2022\n\n[4] Roeder, G., Metz, L., and Kingma, D. P. On linear identifiability of learned representations. In Proceedings of the 38th International Conference on Machine Learning, 2021.\n\n[5] S. Lachapelle, D. Mahajan, I. Mitliagkas, and S. Lacoste-Julien. Additive decoders for latent variables identification and cartesian-product extrapolation. In Advances in Neural Information Processing Systems, 2023b.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)