Towards Characterizing Domain Counterfactuals for Invertible Latent Causal Models
Authors: Zeyu Zhou
Ruqi Bai
Sean Kulinski
Murat Kocaoglu
David I. Inouye
Venue: ICLR 2024 poster (2024)
Keywords: counterfactual
domain
causal representation learning
Primary area: causal reasoning
TL;DR: We build generative models by learning latent causal models from data observed from different domains for the purpose of generating domain counterfactuals.

Abstract: Answering counterfactual queries has important applications such as explainability, robustness, and fairness but is challenging when the causal variables are unobserved and the observations are non-linear mixtures of these latent variables, such as pixels in images.
One approach is to recover the latent Structural Causal Model (SCM), which may be infeasible in practice due to requiring strong assumptions, e.g., linearity of the causal mechanisms or perfect atomic interventions.
Meanwhile, more practical ML-based approaches using naive domain translation models to generate counterfactual samples lack theoretical grounding and may construct invalid counterfactuals.
In this work, we strive to strike a balance between practicality and theoretical guarantees by analyzing a specific type of causal query called *domain counterfactuals*, which hypothesizes what a sample would have looked like if it had been generated in a different domain (or environment).
We show that recovering the latent SCM is unnecessary for estimating domain counterfactuals, thereby sidestepping some of the theoretic challenges.
By assuming invertibility and sparsity of intervention, we prove domain counterfactual estimation error can be bounded by a data fit term and intervention sparsity term.
Building upon our theoretical results, we develop a theoretically grounded practical algorithm that simplifies the modeling process to generative model estimation under autoregressive and shared parameter constraints that enforce intervention sparsity.
Finally, we show an improvement in counterfactual estimation over baseline methods through extensive simulated and image-based experiments.

Published as a conference paper at ICLR 2024
TOWARDS CHARACTERIZING DOMAIN COUNTERFAC- TUALS FOR INVERTIBLE LATENT CAUSAL MODELS
Zeyu Zhou', Ruqi , Sean Kulinski', Murat Kocaoglu, David I. Inouye Elmore Family School of Electrical and Computer Engineering Purdue University (zhou1059, bail16, skulinsk, mkocaoglu, dinouye (anouye]@purddee @pudduee
ABSTRACT
Answering counterfactual queries has important applications such as explainabil- ity, robustness, and fairness but is challenging when the causal variables are un- observed and the observations are non-linear mixtures of these latent variables, such as pixels in images. One approach is to recover the latent Structural Causal Model (SCM), which may be infeasible in practice due to requiring strong as- sumptions, e.g., linearity of the causal mechanisms or perfect atomic interven- tions. Meanwhile, more practical ML-based approaches using naïve domain trans- lation models to generate counterfactual samples lack theoretical grounding and may construct invalid counterfactuals. In this work, strive strike balance between practicality and theoretical guarantees by analyzing a specific type of causal query called domain counterfactuals, which hypothesizes what a sample would have looked like if it had been generated in a different domain (or envi- ronment). We show that recovering the latent SCM is unnecessary for estimating domain counterfactuals, thereby sidestepping some of the theoretic challenges. By assuming invertibility and sparsity of intervention, we prove domain counterfac- tual estimation error can be bounded by a data fit term and intervention sparsity term. Building upon our theoretical results, we develop a theoretically grounded practical algorithm that simplifies the modeling process to generative model esti- mation under autoregressive and shared parameter constraints that enforce inter- vention sparsity. Finally, show improvement in counterfactual estimation over baseline methods through extensive simulated and image-based experiments.
INTRODUCTION
Causal reasoning and machine learning, two fields which historically evolved disconnected from each other, have recently started to merge with several recent results leveraging the available causal knowledge develop better ML solutions (Kusner et al., 2017; Moraffah et al., 2020; Nemirovsky et al., 2022; Calderon et al., 2022). One such setting is causal representation learning (Schölkopf et al., 2021; Brehmer et al., 2022), which aims to take data from a complex observed space (e.g., images) and learn the latent causal factors that generate the data. A common scenario is when we have access to diverse datasets from different domains, where from a causal perspective, each do- main is generated via unknown intervention some domain-specifff latent causal mechanisms. With this in mind, we focus on a specific causal query called a domain counterfactual (DCF), which hypothesizes: "What would this sample look like if it had been generated in different domain (or environment)?" For example, given a patient's 's medical imaging from Hospital A, what would it look like if it had been taken at Hospital B? Answering this DCF query could have applications in fairness, explainability, and model robustness.
naive ML approach to answering this query is to simply train generative models to map between the two distributions without any causal assumptions or causal constraints (c.g., Kulinski and Inouye (2023)); however, this lacks theoretic grounding and may produce invalid counterfactuals. One common causal approach for answering such a counterfactual query would be two-stage method of first recovering the causal structure and then estimating the counterfactual examples (Kocaoglu
"Equal contribution. Listing order random.

---

Published as a conference paper at ICLR 2024
Table 1: This table of related causal representation learning works, focuses mostly on works that study learning a latent SCM, shows that most prior works in this area aim for identifiability of the (latent) SCM, and thus require strong technical assumptions which may not hold in real-world
scenarios ge.g., perfect single-node interventions for each variable).
Observ,
Observ. Function
Characterization of Counterfactual Equiv. Single mechamism
SCM type
Function
Other Assumptions
Edentifiability
Nate-Essahang Esfahany et al. (2023)
Invertible observed
N/A
1) Access to ground- truth DAG
N/A
counterfactuals under specific contexts
1) Atomic stochastic hard interv
Brehmer al. (2022)
Invertible lntent
Invertible
Mixing elementwise
N/A- Counterfactuals as input
2) Training set is
counterfactuals pairs
Squires et al. (2023)
Lincar latcat
Lincar
1) Atomic bard interv.
Sealing
No
Lio ct (2022a)
Lincar latent
Non-lincar
Signilicant causal
Mixing and scaling
No
weights variation
Varici al. (2023)
Latent non min-uineur
Lineur
Atomic stochastic'
Mixing scaling
No
hard interv,
Khemakhem ct al. (2021)
Invertible observed
(implicit)
Afficc
1) Bivariate requirement for identifibilitty
Full (bivaristc only)
No
Ours
Inverttble Jatent
Invertible
1) Access domain labels
No
Domain counterfactual
et al., 2018; Sauer and Geiger, 2021; Nemirovsky et al., 2022). However, most of the existing
methods for causal structure learning either assume the causal variable to be observed (as opposed to our setting where the causal variables are latent) or require restrictive assumptions for recovering the latent causal structure, such as atomic interventions (Brehmer et al., 2022; Squires et al., 2023; Varici et al., 2023), or access to counterfactual pairs (Brehmer et al., 2022), or assume model structures like linearity or polynomial (Khemakhem et al., 2021; Squires et al., 2023), which often do not hold in practice. A summary of existing works can be found in Table 1. In this paper, strive to balance practicality and theoretical guarantees by answering the question: "Can we theoretically and practically estimate domain counterfactuals without the need to recover the ground-truth causal structure?"
With weak assumptions about the true causal model and available data, we analyze invertible latent causal models and show that it is possible to estimate domain counterfactuals both theoretically and practically, where the estimation error depends on the intervention sparsity. We summarize our contributions as follows:
C1 For a class of invertible latent domain causal models (ILD), we show that recovering the true ILD model is unnecessary for estimating domain counterfactuals by proving necessary and sufficient characterization of domain counterfactual equivalence.
C2 We prove bound on the domain counterfactual estimation error which decomposes into a data fit term and intervention sparsity term. If the true intervention sparsity is small, this bound suggests adding a sparsity constraint for DCF estimation.
C3 Towards practical implementation, we prove that any ILD model with intervention sparsity k can be written in canonical form where only the last k variables are intervened. This significantly reduces the modeling search space from (T) causal structures to only one.
C4 In light of these theoretic results, we propose an algorithm for estimating domain counterfac- tuals by searching over canonical ILD models while restricting intervention sparsity (inspired by C2 and C3). We validate our algorithm on both simulated and image-based experiments
Notation We denote function equality between two functions X + ) and I' X + y as simply ʃ f', which more formally can be stated as v E (xx) fax)). Similarly, f + f' means that there exists x E fxx) * f((x). We use o to denote function composition, e.g., g(f(x)) = go (x) or simply h f.
f. We use subscripts to denote particular indices (e.g., xy E R
the j-th value of the vector æ and æej E Ri is the subvector corresponding to the indices I to j - 1). For function outputs, we use bracket notation to select a single item (e.g.. [f(=]] E IR refers to the j-th output of f(x)) or subvector (e.g.. [f(x)] E Rd refers to the subvector for indices to j inclusive). Similarly, for (unbound) functions, let [f]; Rm - IR refer to the scalar function corresponding to the j-th output or [fls] Im + R refer to the vector function corresponding to first j outputs. For any positive integer m, we define [m] e (l...., m). We denote Na as number of domains in the ILD model.
'Code he found in http://ttiiiccoiinnyyilliiiiiiit
2

---

Published as a conference paper at ICLR 2024
2
DOMAIN COUNTERFACTUALS WITH INVERTIBLE LATENT DOMAIN CAUSAL MODELS
Given a set of domains (or environments), a domain counterfactual (DCF) asks the question: "What would sample from domain look like if it had (counterfactually) been generated from a dif- ferent domain?" Each domain represents different causal model on the same set of causal vari- ables, i.e., they can be viewed as interventions of baseline causal model. If we let D be an aux- iliary indicator variable denoting the domain, a DCF can be formalized as the counterfactual query p(XD-e æ, D d), where z is the observed evidence, d is the original domain, and XD=d' is the counterfactual random variable when forcing the domain to be d'. In this work, we aim to find DCF for a class of invertible models (which we define in Section 2.1) and we will assume that the causal variables are unobserved (i.e., latent). To compare. Causal Representation Learning (CRL) has a similar latent causal model setup (Schölkopf et al., 2021). However, most CRL methods aim for identifiability of the latent representations, which is unsurprisingly very challenging. In contrast, we show that estimating DCFss is easier than estimating the latent causal representations and may require fewer assumptions in Section 2.2.
2.1 ILD MODEL
We now define the causal model based primarily on the assumption of invertibility, First, we assume that the observation function (or mixing function) shared between all domains is invertible (as in Liu et al. (2022a): Zhang et al. (2023); von Kügelgen et (2023)). This means that the latent causal variables are invertible functions of the observed variables. Second, we assume that the latent SCMs for each domain are also invertible with univariate exogenous noise terms per causal variable. We assume the standard Directed Acyclic Graph (DAG) constraint on the SCMs. For notational simplicity, we will assume w.l.o.g. that the DAG is complete graph (i.e., it includes all possible edges), but some edges could represent zero dependency which is functionally equivalent to the edge being missing. Given the topological ordering respecting the complete DAG, we prove that an invertible SCM can be written as unique autoregressive invertible function that maps from all the exogenous noises to the latent endogenous causal variables (See Appendix B.1). Note that the SCM invertibility assumption excludes causal models where causal variables have multivariate exogenous noise. Given all this, we now define our ILD model class that joins together the shared mixing function and the latent SCMs for each domain.
Definition 1 (Invertible Latent Domain Causal Model). An invertible latent domain causal model (ILD), denoted by F), combines shared invertible mixing function g Z - X with a set of Na domain-specife latent SCMs F & {Ja: : Rn + +Z) 2 where Ia are invertible and autoregressive. The exogenous noise is assumed to have a standard normal distribution, i.e., 6 ~ (00 Im).
While we discuss the model in depth in Appendix A, we first briefly discuss why the autoregressive and standard normal exogenous noise assumptions are not restrictive. For any model that violates the topological ordering, an equivalent ILD model can be constructed by merging the original mixing function with a variable permutation. Similarly, for any continuous exogenous distribution, we can construct equivalent Gaussian noise-based ILD model via merging the original SCM with the Rosenblatt transform (Rosenblatt, 1952) and inverse element-wise normal CDF transformation. Moreover, prove in the appendix that for any observed domain distributions, there exists ILD model that could match these domain distributions. Therefore, these two assumptions are not critical but will simplify theoretical analysis.
Given our definition, we note that interventions between two ILDs are implicitly defined by the difference between two domain-specific causal models and the intervention is denoted by I(fa, fa) S. [m], which is the set of the intervened causal variables' indices. In Appendix B.3 in the appendix, we prove that the standard notion of causal intervention is equivalent checking if the inverse subfunctions are equal, i.e., j E I(Ja,aa) e [Ja # [Jx'] 1 We further define
the ILD intervention set as the union over all pairs of domains, i.e., I(F) A Ure.ferr I(Ja, sa UReN, I(fi, fa). These implicit ILD interventions could be a hard intervention (i.e., remove depen- dence parents) soft intervention (i.e., merely change the dependence structure with parents). Because any intervened causal mechanism is invertible by our definition, ILD interventions must be stochastic rather than do-style interventions, which would break the invertibility of the latent SCM.
3

---

Published as a conference paper at ICLR 2024
Finally, we define notion of two ILD models being equivalent with respect to their observed distri- butions based on the change of variables formula. This notion, which is a true equivalence relation because the equation in (2) has the properties of reflexivity, symmetry. and transitivity by the proper- ties of the equality of measures, will be important for defining an upper bound on DCF estimation in Section 3.1 and for developing practical algorithms that minimize the divergence between the ILII) observed distribution and the training data in Section 3.3.
Definition 2 (Distribution Equivalence),
Two ILDs F) and FF'F')) are distributionally equiv- alent, denoted by Fg. DD (g',F"), the induced domain distributions are equal, i.e. dd, PN (Jz (x)) (x)) = PN (f"z) o 11
2.2 ILD DOMAIN COUNTERFACTUAN
With our ILD model defined, we now formalize DCF query for our ILD model. For that, we remember the three steps for computing (domain) counterfactuals (Pearl, 2009, Chapter 1.4.4): abduction, action, and prediction. The first step is to infer the exogenous noise from the evi- dence. For ILD models, this simplifies tion and latent SCM, i.c.. e fa g (x). The second step and third steps are to perform the target intervention and run the exogenous noise through the intervened mechanisms. For ILD, this is simply applying the other domain's causal model and the shared mixing function, i.e., ad-ad' = go (e). Combining these steps yields the simple form of DCF for ILD models: Ip-d' goofo fa o fa og og-gggg-oo (z), where fa. fa F. DCF for ILD models are deterministic coun- terfactuals (de Lara et al., 2023) since they have unique mapping, i.c., given the evidence from d, the counterfactual da--d'' is deterministic. We now provide a notion that will define which ILDs have the same DCFs (see Appendix B.4 for the equivalence relation proof).
deterministic function that inverts the mixing func-
Definition 3 (Domain Counterfactual Equivalence). Two ILDs (g. and (g',F)) are domain coun- terfactually equivalent, denoted by (g. F) = (g', F'). if all domain counterfactuals are equal, i.e., for all d, d' La J -
= o 8''. o S' og" og'
While Definition succinctly defines the equivalence classes of ILDs, does not give much in- sight into the structure of the equivalence classes. To fill this gap, now present one of our main theoretic results which characterizes a necessary and sufficient condition for being domain counter- factually equivalent and relates proves that their intervention set size must be equal.
Theorem (Characterization of Counterfactual Equivalence). Two ILDS are domain counterfactu- ally equivalent, i.e., (g, F) =c F") if and only if:
3h,, h2 E Fr s.l. g' g hT E F, and S'a = hn o la o h2 E FFA,Vd,
(1)
and moreover, counterfactually equivalent models share the same intervention set size, iee. if F) =c (F)F'), then [IFF] [(F').
See Appendix B.5 for proofs. Importantly, Theorem can be used to construct domain counter- factually equivalent models and verify if two models are domain counterfactually equivalent (or determine they are not equivalent). In fact, for any two invertible functions hı and he that satisfy the implicit autoregressive constraint, i.e., for all d, hı o fa h2 E FA. we can construct coun- terfactually equivalent model
can have arbitrarily different latent representations defined by g' = g o h since hı can be an arbitrary invertible function. Ultimately, this result implies that to estimate domain counterfactuals, we indeed do not require the recovery of the latent representations or the full causal model.
ESTIMATING ILD DOMAIN COUNTERFACTUALS IN PRACTICE
While the previous section proved that recovering the latent causal representations not neces- sary for DCFs, this section seeks to design practical method for estimating DCFss Since we only assume access to i.i.d. data from each domain, one natural idea is to fit an ILD model that is dis- tributionally equivalent to the observed domain distributions. Yet. distribution equivalence is only distribution-level property while counterfactual equivalence is a point-wise property, i..., the domain distributions match while the counterfactuals could be different. Indeed, we show in Theorem
4

---

Published as a conference paper at ICLR 2024
that even under the constraint of distribution equivalence, the counterfactual error can be very large. To mitigate this issue, we choose a relatively weak assumption called the Sparse Mechanism Shift (SMS) hypothesis (Schölkopf et al., 2021), which states that the differences between domain distri- butions are caused by a small number of intervened variables. Given this assumption about the true IL.D model, it is natural impose this intervention sparsity on the estimated ILD model. Therefore, we now have two components to ILD estimation: a distribution equivalence term and sparsity con- straint which are based on the dataset and our assumption respectively. We first prove that both of these components are important for DCF estimation by providing bound on the counterfactual er- ror (defined below). Then, we prove that the sparsity constraint can be enforced by only optimizing over canonical version of ILD models, which have all intervened variables last in a topological or- dering. This greatly simplifies the practical optimization algorithm since only one sparsity structure is needed than the potentially ((t) different sparsity structures, where & is the sparsity level. Finally, we bring all of this together to form a practical optimization objective with sparsity constraints.
3.1 DOMAIN COUNTERFACTUAL ERROR BOUND
In this section, will prove bound counterfactual error that depends both distribution equivalence and intervention sparsity. Towards this end, let us first define a counterfactual pseudo- metric between ILD models via RMSE (proof of pseudo-metric in Lemma 6 in the appendix).
Definition 4 (Counterfactual Pseudo-Metric for ILD Models), Given joint distribution p(æ, d). the counterfactual pseudo metric between two HLDs (g. and (g',F') is defined as the RMSE over all counterfactuals, i..,
(g.
Eppeadd fa fa fa oa-')g-(()))
f'd fd g' (x)
where p(d') = p(d) is the marginal distribution of the domain labels.
Given this aseudo-metric, we can now derive a bound on the counterfactual error between an esti- mated ILD (ő. F) and the true ILD FF" defined as E(. & dc((}, (o"
Theorem (Counterfactual Error Bound Decomposition), Given a max intervention sparsity 2 and letting M(k) - ((g, F) F) =D (g", "a",F"","""""""""""""")" (F*,,II(F)) S max(k; [IFF) the counterfac- tual error can be upper bounded follows:
F) S
min W (P)MM) F),
(F.R)
(A) Error lock of distribution equivalence
(R) Worst-case evror given distribution equivalence
Furthermore, if we assume that the HLD mixing functions are Lipschitz continuous, we can bound the worst-case error (B) as follows:
(B) <
max To LG max B[[Fa(c) - Fa (c))) - L3 max f2 iE(m]
(EM((
sE|m]
Error depends on since S max RT}
only depends on ground model
where To = (IF) and k = [I(F'),, Ly is the Lipchitz constant of g, and the expectation is over p(d, d', c) p(d))(d))p(e).
Please check proof in Appendix B.6. The first term (A) corresponds to data fit term and could be reduced by minimizing the divergence between the ILD model and the observed distributions. If the estimated ILD already matches the ground truth distribution, then this term would be zero. The second term (B), however, does not involve the data distribution and cannot be explicitly reduced. Yet, the bound on this second error term shows that it can be implicitly controlled by constraining the target intervention sparsity h of the estimated model. Informally, the (B) term depends on the intervention sparsity, Lipschitz constant, and a term that corresponds to the largest feature difference between domain SCMs. This last term can be interpreted as the worst case single-feature difference between latent counterfactuals, We do not claim this bound is tight, but rather simply aim to show that the domain counterfactual error depends the target intervention sparsity i such that reducing (as long as 2 k*) can improve DCF estimation. Therefore, our error bound elucidates that both data fit and intervention sparsity needed for DCF estimation.
5

---

Published as a conference paper at ICLR 2024
3.2 CANONICAL ILD MODEL
While the last section showed that imposing intervention sparsity helps control the counterfactual error, imposing this sparsity constraint can be challenging. In particular, the ground truth sparsity pattern, i.e., which of k causal mechanisms are intervened, is unknown. A naïve solution would be to optimize an ILD model for all possible (*) sparsity patterns. In this section, we prove that we only need to optimize one sparsity pattern without loss of generality. In particular, we can assume that all intervened mechanisms are on the last variables. We refer to such a model as canonical ILD model which we formalize next.
F) is a canonical domain coun- terfactual model (canonical ILD), denoted by (g. F) E C. if and only the last variables are inter- vened, i... (g. F) E C + I(F) = {m - j : < [IF)
Definition (Canonical Domain Counterfactual Model), An ILD (g.
While this definition may seem quite restrictive, we prove that (perhaps surprisingly) any ILD can be transformed to an equivalent canonical ILD.
Theorem (Existence of Equivalent Canonical ILD), Given an ILD [g, F), there exists a canonical ILD that is both counterfactually and distributionally equivalent to Fg while maintaining the
size of the intervention set, i.e., F), FF) C s.t. (g,F') CCD Fg,F) and [I(F) [I(F')
See Appendix B.7 for full proof and Example in the appendix for a toy example. This result is helpful for theoretic analysis and, more importantly, it has great practical significance as now we can merely search over canonical ILD models.
3.3 PROPOSED ILD ESTIMATION ALGORITHM
Given the error bound in Theorem 2, the natural approach is to minimize the divergence between the observed domain distributions (represented by the training data) and the model's induced dis- tributions while constraining h interventions. From Theorem 3, can simply optimize canonical ILD models without loss of generality. Therefore, we optimize the following constrained objective given target intervention size k:
min Ep(a,a)) - log (2r, d)) s.t. [Ja]sm= [arlsm Ad d' 9.F
Concretely, the practical algorithm means training a normalizing flow for each domain while sharing most (but not all) parameters and enforcing autoregressiveeesss for part of the model. The non-shared domain-specific parameters correspond to the intervened variable(s). For higher dimensional data, we also relax the strict invertibility constraint and implement this design using VAEs.
RELATED WORK
Causal Representation Learning Causal representation learning a rapidly developing field that aims to discover the underlying causal mechanisms that drive observed patterns in data and learn representations of data that are causally informative (Schölkopf et al., 2021). 1). This is in contrast to traditional representation learning, which does not consider the causal relationships between vari- ables. As this is a highly difficult task, most works make assumptions on the problem structure, such as access to atomic interventions, the graph structure (e.g., pure children assumptions), or model structure (e.g., linearity) (Yang et al., 2022; Huang et al., 2022; Xie et al., 2022; Squires al., 2023; Zhang et al., 2023; Sturma et al., 2023; Jiang and Aragam, 2023; Liu al., 2022a). Other works such as (Brehmer et al., 2022; Ahuja et al., 2022; Von Kügelgen et al., 2021) assume ceakly--upprrisse
setting where one can train on counterfactual pairs (zx .)) during training. In our work, we aim to maximize the practicality of our assumptions while still maintaining our theoretical goal of equivalent domain counterfactuals (as seen Table 1).
Counterfactual Generation A line of works focus on the identifiability of counterfactual queries (Shpitser and Pearl, 2008; Shah et al., 2022). For example, given knowledge of the ground-truth able to recover the structural causal models up to
causal structure, Nast-Esfahany et al. (2023) equivalence. However, they do not consider the latent causal setting and assume some prior knowl- edge of underlying causal structures such the backdoor criterion. There is weaker form of

---

Published as a conference paper at ICLR 2024
counterfactual generation without explicit causal reasoning but instead using generative models Zhu et al. (2017); Nemirovsky et al. (2022). These typically involve training a generative model with a meaningful latent representation that can be intervened to guide counterfactual generation (Ilse et al., 2020). As these works do not directly incorporate causal learning in their frameworks, we con- sider them out of scope for this paper. Another branch of works estimate causal effect without trying to learn the underlying causal structure, which typically assume all variables are observable(Louizos al., 2017). An expanded related work section in Appendix F.
EXPERIMENTS
We have shown theoretically the benefit of our canonical ILD characterization and restriction of intervention sparsity. In this section, we empirically test whether our theory could guide us to design better models for producing domain counterfactuals while only having access to observational data a and the corresponding domain label d. In our simulated experiment, under the scenario where all of our modeling assumptions hold, we try to answer the following questions: (1) When we know the ground truth sparsity, does sparse canonical ILD lead to better domain counterfactual generation over naïve MI. approaches (dense models)? (2) What would happen if there is mismatch of sparsity between the dataset and modeling and what is good model design strategy in practice? After this simulated experiment, we perform experiments on image datasets to determine if sparse canonical models are still advantageous in this more realistic setting. In this case, we assume the latent causal model lies in a lower dimensional space than the observed space and thus we use autoencoders to approximate an observation function that is invertible on lower-dimensional manifold.
5.1 SIMULATED DATASET
Experiment Setup To extensively address our questions against diverse causal mechanism settings, for each experiment, we generate 10 distinct ground truth ILDs. The ground truth latent SCM fā FIA takes the form fe(e) F3 e + 6;17 where F -1- L,)= LA E mmxm is domain-specific lower triangular matrix that satisfies the sparsity constraint, by E R is a domain- specific bias, 1 is indicator vector where entries corresponding
the intervention set are 1, and L3 and by are randomly generated for each experiment. The observation function takes the form g²(x) G LeakyReLU (ar) where G Rixx and the slope of LeakyReLU is 0.5. We use maximum likelihood estimation to train two ILDs (like training of a normalizing flow): ILDDCan as introduced in Section 3.2 and a baseline model, ILD-Dense, which has no sparsity restrictions on its latent SCM. To evaluate the models, we compute the mean square error between the estimated counterfactual and ground truth counterfactual. More details on datasets and models, and illustrating figures of the models can be found in Appendix C.I.
Result To answer whether sparse canonical ILD provides any benefit in domain counterfactual gen- eration, we first look at the simplest case where the latent causal structure of the dataset and our model exactly match. In Figure 1a, we notice that when the grounth truth intervention set I" is 6) (i.e. the last two nodes), Can significantly outperforms few harder and more practical tasks where the intervention set size is still 2 but not constrained to the last few nodes. Again, in Figure la, we observe that no matter which two nodes inter- vened on, ILD-Can performs much better than the naïve ML approach IL.D-Desse. This first checks that restricting model structure to the specific canoncial form does not harm the optimization even though the ground truth structure is different. Furthermore, it validates the benefit of our model design for domain counterfactual generation. More results with different number of domains and latent dimensions can be found in Appendix C.2, which all show that IL.D-Can consistently perform better than -LD-De We also include an illustrating figure visualizing how ILD-Can achieves lower counterfactual error. We then transition to the more practical scenario where the true sparsity [T'] is unknown. In Figure 1b, at first glance, we observe a trend of the decrease in counterfactaul eтror as we decrease [Z]. For the case where [Z] 2 |T"| (i.e. when [Z] = 2, 3, 4), this aligns with our intuition that the smaller search space of ILD-Can leads to a higher chance of finding model with low counterfactual error. For the case where [Z] 1, we notice that it performs better than the canonical model that matches the true sparsity. Though it cannot reach distribution equivalence, the reduction in worst-case error (see Theorem 2) seems to be enough to enable comparable or better counterfactuals on average. We further check the performance of the data fitting and see a signif- icant decrease in the fit of II.D-Can once [Z| I'', which supports that the performance in data
Then we create
7

---

Published as a conference paper at ICLR 2024
12.5
ID Denge
15.0
LD--erse
10.0
IDD-an
7.5
am
10.0
7.5
5.0
2,5
0.0
0.0
2' (5.0)(4.5)
(3.4)
(2.3)
0.2)
[4]-4
(a) With knowledge of [I"| and [T'
(b) Without knowledge of [T' and I"
[ 2.
(5, 6)
Figure 1: Simulated experiment results (N₂ = 3) averaged over 10 runs with different ground truth SCMs and the enror bar represents the standard error. (a) This shows IIDD-Can is consistently better than H.D-DDense regardless of intervened nodes the dataset. (b) Here we test varying [Z] while holding I' fixed. The performance of ILD-Can approaches to that of ILD-Dense as we increase [I]. An unexpected result is that ILD-Can performs best when [Z] = 1 and that results from a worse data fitting which is more carefully investigated in Appendix C.2.
fitting can be used as an indicator for whether we found the appropriate [Z]. Additional results on data fitting performance and experiments with different setups, including more complex g based on normalizing flows and VAEs, can be found in Appendix C, and they all lead to the conclusion that ILD-Can produces better counterfactuals than ILD-Dense even though we do not know I
5.2
IMAGE-BASED COUNTERFACTUAL EXPERIMENTS
Here we seek to learn domain counterfactuals in the more realistic image regime. Following the manifold hypothesis (Gorban and Tyukin, 2018; Schölkopf et al., 2021), we assume that the causal interactions in this regime happen through eever-dimensional semantic latent factors as opposed to high-dimensional pixel-level interactions. To allow for learning of the lower dimensional latent space, we relax the invertibility constraint of our image-based II.D) to only require pseudoinvertibility and test our models in this practical setting.
High-dim ILD Modeling We modify the ILD models from Section 5.1 to fit a VAE (Kingma and Welling, 2013) structure where the variational encoder, via g+ to produce the latent encoding 2, which is then passed to two domain-specific latent causal models fāz fā. which produce the parameters of posterior noise distribution. The decoder, (g.F), follows the typical ILD structure: g fa, where, and fa can be viewed as pseudoinverse of ft, gndg* A detailed description and diagram of the models can be found in Figure 19, but informally, these modified ILD models can be seen as training VAE per domain with the restriction that each VAE shares parameters for its initial encoder and final decoder layers (i.e. g is shared). As an additional baseline, we compare against the naive setup, which we call ILD-Independent, where each VAE has shared parameters (i.e. a separate g is learned for each domain). These models were trained using the 3- VAE framework (Higgins et al., 2017). Further details can be found in the Appendix D.4. After training, can perform domain counterfactuals as described in Section 2.2.
first projects to the latent space
We apply our methods to five amage-based datasets: Rotated MNIST (RMNIST), Rotated NNSHionMNIST (RFMNIST)(Xian) et al., 2017), Colored Rotated MNIST (CRMNIST). 31) Shapes (Burgess and Kim, 2018) and CausaI3DIdent (Von Kügelgen et al., 2021), which all have both domain information (c.g... the rotation of the MNIST digit) and class information (e.g... the digit number). For each dataset, we split the data into disjoint domains (e.g., each rotation in CRM- NIST constitutes a different domain) and define class variables which are generated independently of domains (e.g., digit class in CRMNIST), to evaluate our model's 's capability of generating domain counterfactuals. Specifically, for RMNIST, REMNIST and 3D Shapes, all latent variables are in- dependently generated, and for CRMNIST and Cadsal there is more complicated causal graph containing the domain, class and other latent variables. Further details on each dataset and (assumed) ground-truth latent causal graphs could be found in Appendix D. and Appendix D.3.
Dataset
Metries Inspired by the work in Monteiro et al. (2023), we evaluate the image-based counterfac- tuals with latent SCMs via the following metrics, where hdomain and htlnss represents pretrained domain classifier and class classifier respectively: (1) Effectiveness whether the counterfactual truly changes the domain defined as PP(hoomiin (fa-dr) = d'); ); (2) Preservation whether the domain counterfactual only changes domain specific information defined as P (hettsss (# = y); (3) Com- position whether the counterfactual model is invertible defined as P( (Fd-rd) y): and (4)
8

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative result for Composition (Comp.), Reversibility (Rev.). Preservation (Pre.), and Effectiveness (Eff.), where higher is better. CRMNIST, 3D Shapes, Causal3DIdent are averaged 20, 5, 10 runs respectively. Best models are bold (within standard deviation) and due to space constraints, expanded tables with additional datasets and standard deviation are in Appendix D.5.
CRMNIST
3D Shapes
CausaBBDleen
Comp.
Rev.
Eft.
Prc.
Comp.
Rev.
Eff.
Pre.
Comp.
Rev.
Eft.
Pre.
H.D-bneepoden
87.24
59.88
94.65
60.39
99.79
32.56
94.97
32.49
88. 15
51.43
91.05
51.94
ILD-Deuse
88.18
62.29
92.72
59.60
99.76
32.60
80.92
32.64
83.59
49.17
92.17
48.83
ILD-Can
92.10
85.74
91.18
72.95
99.85
79.84
96.72
61.99
86.00
79.73
84.15
79.73
d
d2 d3
d1
d2 dg
dx
d
d2
d
d2
Wrrry
Wyn
d2
42
de - de
62-3
d=
42 d2
ILü-Denie
ILD-Con
ILD-Dense
ILD-Con
(a) 3D Shapes
(b) Causalldent
Figure 2: Domain counterfactuals with 3D Shapes and Causalident. Expanded figures can be found in Appendix D.5 (a) For 3D Shapes, only the object shape should change with domain counterfac- tuals the other latent factors such as the hue of object, floor, background, should not change. (b) For Causalldent, as the domain changes, the color of the background should change while holding all else unchanged. ILD Can clearly performs better than the baseline ILDDDense in terms of pre- serving non-domain features while changing domains for all datasets.
Reversibility- whether the counterfactual model is cycle- consiseen defined as (hetass (fd-dd' y). For example, in the case of CRMNIST, a model might be able to rotate the image but cannot preserve the digit class during rotation, which would be high in effectiveness but low in preserva- tion score. Details on the computation of these metrics and causal interpretations can be found in Appendix D.2 and Appendix D.3 respectively.
Result Due to space constraints, we put all results with RMNIST and REMNIST in Appendix D.5. In Figure 2 we can see examples of domain counterfactuals for both ILD-Dense and ILD-Can. We note that no latent information other than the domain label was seen during training, thus suggesting the intervention sparsity is what allowed the canonical models to preserve important non non-domain-- specific information such as class information when generating domain counterfactuals, In Table 2, we include quantitative results using our metrics, which shows IIDD-Can having significantly better reversibility and preservation while maintaining similar levels of counterfactual effectiveness and composition than the non-sparse counterparts. In Appendix D.5, we further investigate our model's sensitivity to the choice of sparsity by tracking how each metric change w.r.t. [Z]. We observe that reversibility and preservation tends to decrease while effectiveness tends to increase as we increase [Z]. which aligns with our findings here nodes. In summary, our results here indicate our theory-inspired model design leads to better domain counterfactual generation in the practical pseudo-invertibl setting.
is equivalent to making I contain all latent
CONCLUSION
In this paper, we show that estimating domain counterfactuals given only i..d. data from each domain feasible without recovering the latent causal structure. We theoretically analyzed the DCF problem for particular invertible causal model class and proved bound on estimation error that depends on both a data fit term and an intervention sparsity term. Inspired by these results, implemented practical likelibood-based algorithm under intervention sparsity constraints that demonstrated better DCF estimation than baselines across experimental conditions. We discuss the limitations of our methods in Appendix E. We hope findings can inspire simpler causal queries that are useful yet practically feasible to estimate and begin bridging the gap between causality and machine learning.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Z.Z., R.B., S.K., and D.I. acknowledge support from NSF (IS--2112097) ARL (W91 INF-2020- 221), and ONR (N00014-23-C-1016), M.K. acknowledges support from NSF CAREER 2239375.
REFERENCES
Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 15:1561-155528 2022.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco S Cohen. Weakly supervised causal repre- sentation learning. Advances in Neural Information Processing Systems, 35:38319-38331, 2022.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https//gghhbbboommmmmpmmm/////ss 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv: 03599,. 2018.
Nitay Calderon, Eyal Ben-David, Amir Feder, and Roi Reichart. Docogen: Domain counterfactual generation for low resource domain adaptation. arXiv preprint arXiv:2202.123550 2022.
Tianfeng Chai and Roland R Draxler. Root mean square error (rmse) or mean absolute error (mae)?- arguments against avoiding rmse in the literature. Geoscientific model development, 7(3):1247- 1250, 2014.
Zhengming Chen, Feng Xie, Jie Qiao, Zhifeng Hao, Kun Zhang, and Ruichu Cai. Identification of linear latent variable model with arbitrary distribution. Thirty-Sixit Sixth AAAI Conference on Arti- ficial Intelligence, AAAI 2022, Thirty-Fouthh Conference Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelli- gence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 6350-6357 AAAI Press, 2022. URL https::////m /ojs....a/////o aaai.osao....o index...x.. php/ AAAI
Innovative Applications of Artificial
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8789- 8797, 2018
Lucas de Lara, Alberto González-Sanz Nicholas Asher, Laurent Risser, and Jean-Michel Loubes. Transport-baad based counterfactual models, 2023.
Laurent Dinh, Jascha Sohk-Dickst,,n, and Samy Bengio. Density estimation using real nvp. arXiv 2016.
preprint I6X5v:1605.00
Alexander N Gorban and Ivan Yu Tyukin. Blessing of dimensionality: mathematical foundations of the statistical physics of data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences,
2018.
Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf. and Michel Besserve. Independent mechanism analysis, new concept? Advances neural information processing systems, 34:28233-28248, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Christina Geinze-Deml. Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 602):20170016, 2018.
10

---

Published as a conference paper at ICLR 2024
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations. 2017.
Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchical causal structure discovery with rank constraints. CoRR, abs/2210.01198,, 2022. doi: 10.48550/ arXiv.2210.01798. URL ht / /tt//do../ www.oo..r.......eeeoeooooeeeee.e wrgg/orgg//.o.o 48550/ 2210 01798.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 459-868, PMLR, 2019.
Maximilian llse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322-348. PMLR, 2020.
Yibo Jiang and Bryon Aragam. Learning nonparametric latent causal graphs with unknown inter- ventions. In Shirty-seventh Conference on Neural Information Processing Systems, 2023. URL
ftctps://openrettt .nt forum? d= S000mmmgg................
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International Conference on Artificial Intel- ligence and Staristics, pages 2207 2217, PMLR, 2020.
Ilyes Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. In International conference on artificial intelligence and statistics, pages 3520-3528. PMLR, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 1rXiv:1412.6980, 2014.
arXiv preprint
Diederík P Kingma and Max Welling. 1rXiv:1312.6114,222 2013.
Auto-encoding variational bayes.
arXiv preprint
Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, and Sriram Vishwanath. Causalgan: Learning causal implicit generative models with adversarial training. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Truck Proceedings. OpenReview.net, 2018. URL ptatt::://pppreeeeee Forum?id=BJFF 4xWOW.
net/
Sean Kulinski and David I Inouye. Towards explaining distribution shifts. In International Confer- ence on Machine Learning, pages 17931-17952. PMLR, 2023.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
Sebastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Si- mon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and sparsity: Generalization and identifiability in Multi-Tass learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Ma- chine Learning Research, pages 18171-18206. PMLR, 2023.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Identifying weight- variint latent causal models. arXiv preprint 220i /4153, 2022a.
Yuhang Liu, Zhen Zhang, Dong Gong, Mingming Gong, Biwei Huang, Kun Zhang, and Javen Qin- feng Shi. Identifying latent causal content for multi-source domain adaptation. CoRR, abs/2208.144 14161, 2022b. doi: 10.48550/axxiv22208.. 14161. URL https:/ /oii. wtt.do..o.....oooooo org/10. 48550/am a wt..v.220................ 14161.

---

Published as a conference paper at ICLR 2024
Christos Louizos, Uni Shalit, Joris M. Mooij, David A. Sontag, Richard S. Zemel, and Max Welling. Causal effect inference with deep latent-variable models. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6446-6456, 2017. URL https: http//pooceeenngss/e/ee /ttps//pcccs neurips http:/seeeerrppseppapa/p/////11 9 4b55bde6de888 8ddf 9cde 67 48d222332233 Abstract html.
ccc/paper/201/hhaa//
Robert E Melchers and André T Beck. Structural reliability analysis and prediction. John wiley & sons, 2018.
Miguel Aires Barros Monteiro, Fabio De Sousa Ribeiro, Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Measuring axiomatic soundness of counterfactual image mod- els. ArXiv, 0///2303,01174, 2023. URL https: /api. CorpusID:25728040::
org./
Raha Moraffah, Mansooreh Karami, Ruocheng Guo, Adrienne Raglin, and Huan Liu. Causal inter- learning-problems,
pretability for machine learning- problems, methods and evaluation. ACM SIGKDD Explorations Newsletter, 22(1):18-33 33, 2020.
Gemma E Moran, Dhanya Sridhar, Yixin Wang, and David M Blei. Identifiable deep generative models via sparse decoding. arXiv preprint arXiv:21 10804, 2021.
Arash Nasr-Esfahany, Mohammad Alizadeh, and Devavrat Shah. Counterfactual identifiability of bijective causal models. arXiv preprint arXiv:2302. 0222220 2023.
Daniel Nemirovsky, Nicolas Thiebaut, Ye Xu, and Abhishek Gupta. Countergan: Generating coun-
terfactuals for real-time recourse and interpretability using residual gans. In Uncertainty in Arti- ficial Intelligence, pages 1488-1497. PMLR, 2022.
Judea Pearl. Causality. Cambridge University Press, Cambridge, UK, 2 edition, 2009. ISBN 978- 8-521-895604 doi: 10.1017/CB0978051 1803161.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant pre- diction: identification and confidence intervals. Journal of the Royal Statistical Society: Series (Statistical Methodology), 78(5):947-1012, 2016.
Murray Rosenblatt. Remarks on multivariate transformation. The annals of mathematical statis- fies, 23(3):470-722, 1952.
Axel Sauer and Andreas Geiger. Counterfactual generative networks. In International Confer- ence on Learning Representations, 2021. URL httts:://ppenrvieew net fiet/formm? did= BXew fA FMMJJw.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965)612-664, 2021.
Abhin Shah, Raaz Dwivedi, Devavrat Shah, and Gregory W Wornell. On counterfactual inference with unobserved confounding. arXiv preprint arXiv:2211.08209, 2022.
lya Shpitser and Judea Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9:1941-1979, 2008
Chandler Squires, Anna Seigal, Salil S. Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learming Research, pages 32540-32560 PMLR, 2023. URL /ttpp//eeeeens...//////// mmlr. press/ /teas/v2022 stt/squirrreeeee re32aa html.
Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler. Unpaired multi-domain causal representation learning. In seventh Conference on Neural Information Processing Systems, 2023. URL https: peatpss//opperev//e net. / fo forum? d=zW1 UVN
12

---

Published as a conference paper at ICLR 2024
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning with interventions. arXiv preprint arXiv:2301.88230 2023.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supprvised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 14:16651--6467 2021.
Julius von Kügelgen, Michel Besserve, Wendong Liang, Luigi Gresele, Armin Kekić, Elias Barein- boim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representa- tions from unknown interventions. In Thirty-seventh Conference on Neural Information Process- ing Systems, 2023. URL http::t/////mm httttpps//openreviee....o wwwp:/ntttooummmmmemme v8 7 gZeSO44.
Pengzhou Wu and Kenji Fukumizu. Causal mosaic: Cause-Effect inference via nonlinear ICA and ensemble method. January 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint vrXXvv1708.001 2017.
Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Identification of linear non-gaussian latent hierarchical structure. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 24370-24387. PMLR, 2022. URL https: // //proceedings mlr www.tngsssm.....s.........e..e.e / /1///v112////1 2eeeaxieeaa.... html.
Feng Xie, Yan Zeng. Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang. Causal discov- cry of I-factor measurement models lincar latent variable models with arbitrary noise dis- tributions. Neurocomputing, 526:48-61, 2023. ISSN 0925-2312. doi: https://doi.orr/11.. neucom.2023.01 .044, URL https: www. wec.enneeireeeem com/ htmm//ccence/aaee article/ pl1/50925231220000
Yuqin Yang, AmirEmad Ghassami, Mohamed S. Nafca, Negar Kiyavash, Kun Zhang, and Ilya Sh- pitser. Causal discovery in linear latent variable models subject to measurement error. CoRR, abs/2211.03984, 2022. doi: 10.485500aaxx.:2211.33388. URL attps://doi.crr//1 48550/arX0t wtt...2....119. 03984.
Jiqqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https: / /twp///eeee..eeeeeeeee wtw.::/wwwnnnoootneooeaa d d=o oalfoo6ssee YKHK333..
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. pages 164]1-16422, June 2022.
Zeyu Zhou, Ziyu Gong, Pradeep Ravikumar, and David Inouye. Iterative alignment flows. In International Conference on Artificial Intelligence and Statistics, pages 6409-6444, PMLR, 2022.
Zeyu Zhou, Sheikh Shams Azam, Christopher Brinton, and David I Inouye. Efficient federated do- main translation. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
13

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Discussion of Invertible Latent Domain Causal Model
14
A.1 Invertible SCM as Global Invertible Autoregressive Function
14
A.2 Interventions for Invertible SCMs
15
A.3 Invertible Latent Domain Causal Model (ILD)
16
B Proofs and Auxiliary Results
16
B.1 Proof of Proposition
17
B.2 Proof of Proposition 2
18
B.3 Proof of Proposition 3
18
B.4 Proof of Lemma
19
Proof Theorem
20
B.6 Proof of Theorem
................................
24
B.7 Proof of Theorem 3
26
C Simulated Experiments
30
C.I Experiment Details
30
C.2 Additional Simulated Experiment Results
................................
32
Simulated Experiment more complicated structures ................................
D Image Counterfactual Experiments
41
D.1 Dataset Descriptions
41
D.2 Metrics
41
D.3 Causal Interpretation our experiments
42
Experiment Details
43
Additional Results
44
E Limitations
49
F Expanded Related Work
49
A DISCUSSION OF INVERTIBLE LATENT DOMAIN CAUSAL MODEL
This section gives further discussion and details about our ILD model and serves an expanded version of Section 2. We first remind the reader of the definition of an SCM using our notation.
Definition (Structural Causal Model). A structural causal model (SCM) considers m endogenous (causal) variables 2, and m exogenous noises jj j E [m], where each variable is deterministic function of its parents and independent exogenous noise. Formally, each endogenous variable has form A f(ej, 2Pn,), for all j E [m].
Note that the SCM could be multivariate and even infinite dimensional.
equations for each endogenous variables, where the exogenous noises
A. INVERTIBLE SCM AS A GLOBAL INVERTIBLE AUTOREGRESSIVE FUNCTION
For theoretic analysis, our main SCM assumption that the exogenous variables can be uniquely recovered from the endogenous variables, i.c., the SCM is invertible. This invertibility assump- tion will mean that domain counterfactuals are unique rather than being distributions over possible counterfactuals.
14

---

In the following, we show how to determine the intervention set using the SCM global functions f and fl directly instead of having to convert to the corresponding individual SCM mechanisms as in Published as a conference paper at ICLR 2024
Definition 7 (Invertible SCM), We say that SCM is invertible if the exogenous noise values can be uniquely recovered from the endogenous random variables, i.e., there a one-to-one mapping between exogenous variables and endogenous variables.
This invertibility assumption implies that the exogenous noises must be scalars? unlike standard SCMs, which can have multivariate exogenous noise variables.
We will now prove that all the SCM mechanisms can be represented by an vector to vector invertible autoregressive function (up to a relabeling), denoted as f E FIA, and we call this the SCM global function. We first define an autoregressive function below.
Definition 8 (Autoregressive Function). A functioon f : 12m + Rm is autoregressive, denoted by f FA. if for all i, the i-th output can be written function of its corresponding input predecessors, i.e,,
/EFA s.t. [rO)] - (ess)) where E R.
(4)
Given this definition, we can now state our proposition that an invertible SCM can be represented by a single global invertible autoregressive function.
Proposition 1 (SCM Global Funetion Representation). An invertible SCM { that is topo- logically ordered, i.e,, the parents have smaller index than the children, can be uniquely represented by an autoregressive invertible function f & FIA and vice versa.
See Appendix B.1 for proof. From here on, we will simply use f e FIA to represent invertible SCM.
While invertible SCMs do not subsume generic SCMs, we note that for any observed distribution of endogenous variables, there exist an invertible SCM that matches the observed distribution formal- ized as follows.
Proposition 2 (Existence of Invertible SCM for Any Distribution). Given any observed continu- ous distribution, there exists an invertible SCM with continuous exogenous noise whose observed distribution matches the given observed distribution.
The full proof is in Appendix B.2. This means that invertible SCMs can model any continuous distribution, but they are not as general as generic SCMs. In practice, invertibility can be relaxed using pseudo-invertille or approximately invertible functions, as seen with VAE in Section 5.2. We assume the exogenous noise distribution is standard Gaussian, which is made mostly for conve- nience and can be made without loss of generality due to the invertible Rosenblatt transformation (Rosenblatt, 1952; Melchers and Beck, 2018, Chapter B).
A.2 INTERVENTIONS FOR INVERTIBLE SCMS
If two SCMs are defined on the same space, then they could be regarded soft interventions of each other, i.e., one SCM can be viewed as the observational SCM and the other as the intervened SCM, or vice versa. Thus, using the standard notion of intervention in which the causal mechanism is different, we can define the intervention set between two invertible SCMs in a symmetric way.
Definition (Intervention Set). The intervention set between J'J' E FIA defined on the same sam- ple space is the indices of the intervened variables of the corresponding unique SCMs derived from Proposition 1 represented by the equivalent individual SCM mechanisms f(u) and f(6)) respectively, i...
I(S & (70)) 7700.
(5)
"The proof simple by contradiction. Suppose one mechanism had non- non-scalar exogenous noise. If the random variables are not perfectly dependent, then it would be impossible to recover more than one exogenous noise variable from a single endogenous noise variable and the parents. If the random variables were determin- istic functions of each other (i.c., perfectly dependent). then the exogenous noise variables could be collapsed into single exogenous noise without loss of generality.
15

---

Published as a conference paper at ICLR 2024
the definition above. This will aid in the theoretic analysis and simplifies the analysis of intervention sparsity. See Appendix B.3 for proof.
Proposition 3 (Intervention set characterized by SCM global function). The intervention set be- tween two SCM f, f FIa A equivalent to the set of variables where the inverse sub functions are different, ie., I f') = (j [ +
A.3 INVERTIBLE LATENT DOMAIN CAUSAL MODEL (ILD)
In this section, we propose the invertible latent domain causal model to capture multiple latent SCMs that are emerged through intervention. The data generated by SCM forms domain.
Definition 1 (Invertible Latent Domain Causal Model). An invertible latent domain causal model (ILD), denoted by F), combines a shared invertible mixing function Z - X with a set of Na domain- speniffc-peciicccccc latent SCMs F 2 {fa: : R + y) where fa are invertible and autoregressive. The exogenous noise is assumed to have a standard normal distribution, i.e., NO, Im).
ILD induces the following data generating process: for the d-th domain, z fa(E) and g(z). Because fa and y are invertible, we can write the observed distribution using the change of variables formula as: Pa( = Pv (fa g (x))
(z)). We now note that assuming topological ordering the latent variables does not restrict the ILD model class.
Remark 1. The latent SCMs in ILD can be assumed to be topologically ordered without loss of generality.
Because the latent variables are all unobserved, the labeling is arbitrary, thus could relabel them in way that preserves topological order and add a permutation to the observation function g. Es- sentially, given non-autoregressive ILD, we could convert to equivalent autoregressive ILD.
We now remember the distribution equivalence between two I.Ds. The distributional equivalence defines true equivalence relation because the equation in (2) has the properties of reflexivity, sym- metry, and transitivity by the properties of the equality of measure.
Definition 2 (Distribution Equivalence). Two ILDs (g,F) and (g',F) are distributionally equiv- alent, denoted by F) =D F", the induced domain distributions are equal, i.e, A d, PN (Jā o 1 tog (x)) PN o'"' 1J -
B PROOFS AND AUXILIARY RESULTS
In this section, prove propositions in Appendix A about the IL.D model and the other results in the main paper. Before proving Proposition 2, we first introduce another lemma that is useful later in proving Proposition
Lemma 1 (Invertible Upper Subfunctions). The upper subfunctions of f E F1 FA are also an invertible function of E
invertible, i.e., [z(ess) A [f(e
Proof. We will prove this by induction & where femm = E Fj. We will prove the inductive step by contradiction. Suppose f'sm-k is not invertible. This would mean is not injective and/or not surjective.
k. For l 0, it is trivial because
f, not injective, then 3cgj + de, such that fsy(c<)) = -Fs(((eg) We would then have for some Enj (e.g., all zeros):
=
-[sslel')) Mess)
(6)
but this would contradict the fact that J5j+1 is invertible by the inductive hypothesis.
16

---

Published as a conference paper at ICLR 2024
If fey is not surjective, then 3 æzz such that Vegy, (e)(Ee)) + æzj. We would then have that Exj
[5+1(es)+(51) ([)e)))) [[(es))s++]]' * [
but this would contradict the fact inductive hypothesis that fJ+1 is surjective. Therefore, fy must be invertible for all j E [m].
B.1 PROOF OF PROPOSITION
Proposition 1 (SCM Global Function Representation). An invertible SCM ( that is topo- logically ordered, i.e., the parents have smaller index than the children, can be uniquely represented by an autoregressive invertible function f E FIA and vice versa.
We first note that because of topologically ordering, using different notation w.l.o.g. as: 2j (eg,ze)) fl) Zsn,) with f() R x R--1 + IR The topological ordering ensures that the parents are earlier indices, i... Paj S (1,2, j 1}. so that this rewriting is possible w.l.o.g. Given this new notation, the unique representation is given by:
write the causal mechanisms f
13
f(e) = (()())) (ez, (es (())), (es, (e,, ..."
"..
(8)
recover
recuver 2c3
where for all j.
recover E.) from
Proof. We first prove one direction. Given an invertible SCM defined by it's causal mechanisms (e,) the observed variables are given recursively as:
2g = (e,,
(10)
We now define the corresponding as in the lemma:
f(e) a (V((E)) (es, (e)). (es. (1)) (e2,
(11)
rooover #c3 We need to prove that the observed variables are equivalent to the given SCM. Formally, we will prove by induction on j E [m]] the hypothesis that [f(e)], = (e) zzÿ) = 2j, Ve E R". The base case is trivial from the definition in (11), i.e. Ve E I [{()); = (e) = zj. For the inductive step, we have the following:
recover #3
[f(]++1= F0+1 (G)), (12 (c))...... f(tt)(((+), = 2j+1 (12)
21
where the first equals is by (11), the second is by the inductive hypothesis, and the last is by definition of the SCM.
Now prove the other direction. Given invertible autoregressive function f Fr FA, define the following recursive set of mechanism functions:
Yj, fe, xzソ) & (x<)) ("s),))))<< Eg,
(13)
Again, we will prove that these functional forms are equivalent via induction on j for the hypothesis
that (U)) =20) [f(e)], 2j. The base case is trivial based on (13):
(() Lf( Men -]1
[f(er, )]] 21
(14)
For the inductive step, we use the definition of Tej and its inverse from Lemma 1 and derive the final result:
fu+!)) (ef+1. = [f([f (x Mes,, = (x [f(Ee) ), 2j.
(15)
17

---

Published as a conference paper at ICLR 2024
B.2 PROOF OF PROPOSITION 2
Proposition 2 (Existence of Invertible SCM for Any Distribution). Given any observed continu- ous distribution, there exists an invertible SCM with continuous exogenous noise whose observed distribution matches the given observed distribution.
The proof leverages the invertible Rosenblatt transformation (Rosenblatt, 1952; Melchers and Beck, 2018, Chapter B) that transform any distribution the uniform distribution vice versa using its inverse. Given an ordering of set random variables, i.e., X = [X1, X2, .. Xm T the Rosenblatt transformation is defined follows:
== Fi (11)
U2 F2 (2(2(211)
U3 == F3 (1 x2)
(16)
Um == Fm (Tm 11 Ø2, .. Im-1);
where F, (z is the conditional CDF of X3 given Xzj = æzj i.e., the CDF corresponding to the distribution pl Ty Xxj F.)). It's inverse can be written follows:
11 = FT (u1)
F2
(uyl Fi (u1))
43 F 1 (u1). Fr (u2/z) PT- (ui)
(17)
Im Fr Fum (uml21) (x)) Fuglaz (uzla1 (uz)), -m ....)
where F ())) is the conditional inverse CDF corresponding to the conditional CDF F3 (1zg((
Let (x) denote the Rosenblatt transformation for distribution and let F- (u) denote its inverse as defined above. Assuming the random variables are continuous, the Rosenblatt transformation transforms the samples from any distribution to samples from the Uniform distribution (i.e., the push-forward of the Rosenblatt transformation is the uniform distribution and the mush-forward of uniform distribution through the inverse Rosenblatt is the distribution p).
Proof. Given any continuous target distribution we can construct invertible SCM whose ob- served distribution p. Specifically, we let q denote the exogenous noise distribution, then the following invertible and autoregressive function f-which defines an invertible SCM via Proposi- tion can be used to match the SCM distribution to p.
f(c) Fp F (c),
(18)
where F-1 maps to the uniform distribution and then Fp maps to the target distribution per the properties of the Rosenblatt transformation. The function invertible since both functions are in- vertible. Additionally, both functions are autoregressive and thus the composition is autoregressive. Therefore, f represents a valid invertible SCM whose observed distribution is p.
B.3 PROOF OF PROPOSITION 3
Proposition (Intervention set characterized by SCM global function). The intervention set be- tween two SCM f. f' E FiA is equivalent to the set of variables where the inverse sub functions are different, i.e., I(F, = ( | 1
Proof. Step 1: Prove (i [ + [r- c (I.F).
18

---

Published as a conference paper at ICLR 2024
For all j E [i * [,- there exists some 2, such that
[f (x)l, [f (z))
(19)
given that f' are auto-regressive function, we conclude there exists some (zzg) 22) such that <j [ (zej, 2g. ) # (zej; 23. M ey.
(20)
we have, for Cj eg and such وزح there holds
f - (20) piole, zei)
-5'( (z Mej e0,
* II (zZgi Mss ']
= (c) zcÿ)-
(21)
where (a) comes from the f' E F1. Thus implies I
[f. (f.f)
Step Prove I (i.F) S {i ( 13-
For all j E I (F.P) there exists some (cj. ماحح, such that
2j FUl =xح) # pullee zzz) 2 z/y.
(22)
Define
zgg a لزل and E// - زلز
(23)
then we have
(r )1 = 15 - Mi, [ (2SH) )1s # 15 (zzj إb
(24)
given that f.f' E Fr, we conclude.
(25)
which implies j E [f # [1]
B.4 PROOF OF LEMMA 2
Lemma 2 (Equivalence relation of counterfactual equivalence). Domain counterfactually equiva- lent, denoted by F) =e (g',)) is an equivalence relation, i.e., the relation satisfies reflexivity, symmetry, and transitivity.
Proof. We only need to prove that it satisfies reflexivity, symmetry, and transitivity.
Reflexivity Letting g' = and F F in the definition, it is trivial to see that Vd, d'
g fa fa -o------- go fa fa o-g
and thus F) =e (''
2. Symmetry - Similarly, it is trivial to see that Vd, d',
go far f og = g' fa fa - og' g g fa fa og" 1-= go o fa o fa 1g
and thus Fg NC (g' F) + (g F') ~ (g.F).
3. Transitivity For (g, F), F') and (g", ,",, we can derive the transitive property by applying the property twice to the first two and the last two pairs Vd, d':
go far fa g- "=g' goo o fa fa og g" f4 fd 9"
which means that (g. F) zo
19

---

Published as a conference paper at ICLR 2024
B.5 PROOF OF THEOREM 1
Theorem (Characterizatioon of Counterfactual Equivalence), Two ILDs are domain counterfactu- ally equivalent, i.e., (g, F) =c (g', F²) if and only if:
Bh2 E FI s.t. g' = go o E Fi and faa = h1 o fa o h2 E FA, Vd. and moreover, counterfactually equivalent models share the same intervention set size, i.e., if (g, =c F"), then [[F] = [IF')
Theorem 1 contains two parts. The first part characterizes the domain counterfactual equivalence model with two invertible functions. The second part proves that all counterfactual equivalent mod- els share the same intervention set size.
B.5.1 PROOF THE REPRESENTATION DCF EQUIVALENCE
The proof of the domain counterfactual equivalence representation. relies heavily on one the fol- lowing two lemmas that provides a necessary and sufficient condition for the composition of two invertible functions be equal.
Lemma 3 (Invertible Composition Equivalence). For two pairs of invertible functions (fi, f2) and (fi, f2), the following two conditions are equivalent:
1. The compositions are equal:
f2-2= f2
2. There exists an intermediate invertible function
snveriiblefunction s.I. f1 = fi S2 = h f2.
(26)
Proof of Lemma 3. For notational simplicity in this proof, we will let g 2 fi. f2. & fi and f' & f/ note that and are just arbitrary invertible functions in this proof. Furthermore, without loss of generality, we will prove for the property 3h : g' = g h, f = hh- f which is equivalent to 3h g g h-1 S h f. Thus, in the notation, we seeking to prove:
go f g' o f + 3h: g h. h
(27)
= h- f. then it is easy to show that g f g' - f': 1r gloh:ggggohh.. goh,,
g' - f' go h l go
(28)
For the other direction, we will prove by contradiction. First, using Lemma 4, we can first rewrite g' and f' using the two uniquely determined invertible functions hj and h2:
g' hi f = h2 f-
(29)
(30)
Now, suppose that f = g'o o f' but $ h such that g' ho and the facts above, we can derive the following:
h, I = oo f. By the first assumption
go g = g' o f'= = go h1 of
(31)
+ f = h o
(32)
+ 4 o = h2 f
(33)
From the second assumption, i.e., 3h: : g = go h = h- o f, we have the following:
4 s.t. g' = g o h, it holds that f + h h- f
(34)
= f * h of
(35)
+ h1 0 + h of
(36)
h12 /
(37)
20

---

Published as a conference paper at ICLR 2024
h h1.
(38)
where (34) is by assumption, (35) follows from (29) because hı is one particular h, (36) is by our
rewrite of f' in (30), (37) is by the invertibility of f. and (38) is by invertibility of hı and h2. Thus, there exists y, such that hğ) + h2()). Let choose I - f f")()) for the that satisfies the condition. For this ã. we then know that:
4 o f()) = h (0) + h2 h2 f(Ø)
(39)
+ 1 N f f.
(40)
But this leads to direct contradiction of (33). Therefore, gf = g'o then 3h :h:g' /== goh, h- f.
Lemma 4 (Invertible function rewrite). Given any two invertible functions f X + y and f' X' + V, f can be decomposed into the composition of and another invertible function. Specifically, I' can be decomposed the following two ways:
f' f ffoox
(41)
f' f
(42)
where lx - f-1 o f' X + X and hy ff y + y are both invertible functions.
Proof of Lemma 4. The proof is straightforward. We first note that xx and by are invertible because they are compositions of invertible functions. Then, we have that:
fo hx fo f of f= f
(43)
hyo ʃ 1l o S - os= S
(44)
Proof of Theorem 1: Part 1. The basic idea is to use repeated application of Lemma 3 under the constraint that fı and h2 must be shared across for all d and and g must be inverses of each other.
For one direction as Lemma 3, if (1) holds, it is nearly trivial to prove the equation in (3) holds, ie., for all d,d':
goo fl a = bTT))) ) o (h1 o fa h2) a (h₂ o fā oh (h1 o
= go fa o fā og
To prove the other direction, let us define the following functions for a specific (d, d') (we will treat the case of all (d, d') afterwards): fi & g- & f & fa, and fa & and similarly fi, f/ f2, and f/ for the other side. Given these definitions, we can write the property as:
fa fa o f2 f1 fa o fi.
By recursively applying Lemma 3 for each of the three function compositions, we arrive at the following fact:
f = h o fi and f4 f o f2 = f4 fa f2 hT
1a b2 ha, s.t.
fg h2 fz h and f f'S f4 fa h
L ha fa hr and f f4 o
By using the definitions of f1, f2, etc., we can now derive the following:
g' o hrz
f' ha fx h22
fd h2 fa о h
g hı og" og-1
21

---

Published as a conference paper at ICLR 2024
We can connect the first and the last equality to derive that hy h1:
g'² h1 gg-
+ g = goh hgohg
+ h = liz
+ h1 lt3.
Thus, there are only two free functions. Specifically, for any fixed pair of (d, d') there exist h d,dd d (= his,d,d) and h2zd,d'' such that
d : fa = h1,d,' fa ,2 and fa = hi,d, o før o d2 g = = h d
Finally, we tackle the case of all (d, d²) by assuming that there could be unique functions h,,d and d2 for all pairs of (d, d') and show that they are in fact equal. Because the condition holds for all (d, d'), we know that for any particular (d, d') and (d", d), we have the following two things based on the proof above:
go f ff og
g fa o fa og
g = g go go
+
3 3iddd 12,d0
I' hı,d,d' fa h 2,d,'
S1 = hiddd' o far o h 22,d,d'
g 8/ "
go fa fzd og
g = g o hi
+ 3h de d2 a S.t.
Se. = h1.d" A sa h 2
f/ = hi,d" a o fa h 2,8"1
By equating the RHS for the g' equations, we can thus derive that:
g o ho go - h h1,d,d' h1.d"'d.
Using this fact and similarly by equating the RHS for the fa equations, we can derive:
= h1,d," o fa 22, h1,d" a o fa h2 = hi,d, o fa o d2
h = -1 ,d h2,d,d 12.."
By applying these facts to all possible triples of (d, d', d'). we can conclude that Vd, d', hi,d,d' his hig,,d' fug, i.c., these intermediate functions must be independent of d and d'. Finally, we can adjust notation that Yd, f/ h fa h2 and g' go where hy h and h2 h which matches the result in the theorem.
B.5.2 PROOF THE SHARING INTERVENTION SET SIZE BETWEEN DCF EOUIVALENCE MODELS
Now we aim to prove the second part of Theorem 1, which states that all DCF equivalence models share the same intervention set size. The proof requires the concept of canonical form, please refer Definition 5 for the definition of canonical form and Theorem 3 for the existence of special kind of canonical form we refer as Idendity Canonical, where all the un-internvend nodes are independent standard Gaussian.
For two ILDS (g.F) FCDD (g'FF), we apply Theorem to get Identity Canonical form (gc.Fe) Fe) acDD F) and similarly Fc) 2CDD F"), Then use the following Propo- sition 4, we show they must have the same intervention set size. Lastly, notice that the DCF equiv- alence is equivalence relation Lemma 2, we show all the DCF equivalence models share the same intervention set size.
Before proving Proposition first introduce lemma that will be used in the main proof.
22

---

Published as a conference paper at ICLR 2024
Rm E FIA then [f(x)]ı must be non-constant functiooo of Ik- Lemma 5. f R
Proof. We prove this by contradiction. Suppose is the first index that [f()]] fx11.... 26 xk-1) Since & is the smallest index, [f(x)]< is uniquely determined by (k) The remaining dimension outputs could not be bijective to m + inputs.
Proposition (Identity Canonical ILD Shares Intervention Sparsity), Given an ILD Fg, and 9. la E for all d E are continuous, then all Identity Canonical ILDs that are distributionally and counterfuctually equivalent to (g. F) have the same intervention set, i.e,
((d.F)) I(F)=I(F),
(o. (o.
(45)
Proof of Proposition In the proof, we denote as non constant function without specifying the expression.
Step 1: Characterization counterfactual equivalence for Identity Canonical forms. rem states that there exists hi, h2 E Fr, such that for all d,
Theo-
fa = h1 fa h2- Furthermore, by the definition Identity Canonical form, we have
(46)
f Id, fi ld.
(47)
Plugging this into (46), we have
Id = h1 o Id o h2.
Thus,
h h2 h.
Plugging this into (46), for all d, we have
f = fa h.
(48)
Step 2: Counterfactual equivalence between Identity Canonical forms maintain the interven- tion set. The goal of this step is prove that h is bridge satisfying the following property: for any & I(f fa). for all x; there exists an unique j, such that [h [/>(( addition,
(z)] only depends on 2z. In
prove such j satisfies j & Ja)
We start with writing the i-th output of f as the following
U1 (x)))
(48) [A (5a
(M(x)))
(49)
= [h -1 (1 (a
(50)
@ h
(M())) Fant (I(())....... [h(z)]m))
(51)
where in step (a), we used autoregresiveness of fā and Tar is defined as a function from R to R.
According to Lemma 5, Far (x: is non-constant function of Ig-
Step 2.1: We show i and j must be one-to-one mapping of h. We proof this by contradiction.
Suppose -1 maps more than one index to i-th index, w.l.o.g, could assume ji and jz. That is,
[h (u)); depends on un and Ujz- Take u fa (I(x)), then we have
( (x:)) = (Tzh ) ((())).. (MM(()))))))
(52)
Due to that f E FIA, from Lemma 5, we have
Fz (Ih()),. [h(x)),,) F(hh(=));.s)) =
(53)
Fz ((()))),...(((())) =(([h<))))))
(54)
23

---

Published as a conference paper at ICLR 2024
Plug (53), (54) into (52), we have
If (a)]; ((h(())); (
(55)
Given that h FIA, we conclude ([h(=))s+ [h(=)),, depend at least two distinct indices. That is, there exists i1, l2 such that
(((())) 1 (b()))))) F F Fig)-
(56)
That implies (a)); is nontrivial function of ( This leads to the contradiction that i & Iffi:J') where for all æ, there holds
UT = x;
(57)
Step 2.2: We show such not in the intervention set between fi and fa. We prove this by contradiction as well.
Step 2.1 implics
if (x)) = ( (MM))))........))))))) = F((((=))),))
(58)
Suppose j I(fi. fa), then fa ([h()))) Recall that fā E FA.
then [I (h(2))) must be non-constant function of [h(=)),, and [h(=)], for some j j, i.e.
(h())) - - (((())) [M=))) (M,)))
(59)
Similarly, we know that [b<))]} and [M(=)], must be nontrivial functions of Tig and Vip which i3 * ig. However, we know (x)], a function æ; exclusively, which leads to contradiction. This shows that the number of non-intervened node in f/ must not be greater than that in fa. i.c.
I(fi, fd) fa). Vd.
(60)
We further notice that the symmetric relationship between and f'd we could also have
I(f,, - I(fi, Vd.
(61)
Union among d, we have
=(f)==((1)
(62)
B.6
PROOF OF THEOREM 2
Lemma 6 (Counterfactual Pseudo-Metric for ILD Model is pseudo-metric).
Proof. It is trivial to check that it is always positive, symmetric, and equal to 0 if (g, F) = F). Finally, because RMSE satisfies the triangle inequality (Chai and Draxler, 2014), Definition also satisfies the triangle inequality.
Theorem (Counterfactual Error Bound Decomposition). Given a max intervention sparsity and letting M(k) A ((g,F) F) =D (9" FF),,I(()) S maxtk; [IFF)) the counterfac- tual error can be upper bounded as follows:
F) S
min (G'(F))EM(L)
Fc (g' (g',F'))
max (g.R)
F).
(A) Evror due lock of distribution equivalence
evror given distribution equivalence
Furthermore, if we assume that the ILD mixing functions Lipschitz continuous, we can bound the worst-cuse error (B) as follows:
1/2
(B) < max T2 max [[((a(e) - Ja (e) + Gg. (E()) (e(m)
L. max ([ fa (c)) iE[m]
Error depends on since S max
Ervor only depends ground truch model
where k = II(F) and K- [I(F*)), Lg is the Lipchitz constant of g. and the expectation is over p(d, d' E) e p(d))(d))p(e).
24

---

Published as a conference paper at ICLR 2024
f) and (o',F") both canonical forms (See Definition 5). According Theorem 3, any two ILD's counterfactual error are equivalent two their equivalent canonical models, thus this bound holds for all pairs of ILD models.
Proof of Theorem 2. We will prove this theorem when both (0.f)
(2) is by the triangle inequality for any intervening FF and in this case we choose to minimize the bound over all possible distributionally equivalent models with a bounded sparsity- we know that least the true ILD satisfies this, and thus there is least one feasible solution. (2) is by the fact that choosing the ILD model with the worst counterfactual error is larger than the error incurred by (ū.F), under the same cettrrintt-aaaiin by construction, (ū.F) satisfies the constraints the maximization problem and thus least one ILD model is feasible.
Now we prove the worst-case counterfactual misspecificatiion error bound.
max (E.))
max (G.))
E Ep(dr Bp(an,))
go fa Fal (x) 9' fá
(x)
(Definition)
27
= max (..))M(())
Ep(ar Bp(and)
-o fe fo
x x g o fa o fa o og
fa
g
a (Inflation)
E Ep(dr Bp(ard) (e.H)
max
9 fo fa og (x)
go Ja fa og (x)
+ 9" o fā fā og" (x)
-9 fa
(x)
(Invertibilityy of ILD)
|| f Fa 07 go fa 0 fz o (2x)
< max
2 Ep(z.d) (Ep()EEp((dd)
(eM()
21Epp()) Ep(e,d) [110" fā fā fa- (x) g" fg fa fa og g (x)]2
(AM-QM Inequality)
2E Ey(e) [ g Fa(c)
max (G.))((())
+ 2E 2 Epp Ep(er)
[110* o fz'(') g" o ra(6)11)
then we aim to bound the both term related to worse-case ILD term and ground-truth ILD term. Lemma 7. If g is Lipschitz continuous with constant Lg and [[F)], then the following holds:
E Eptelpter o fx( - g fa(e) < Lig k max Eptarpiarppce [[a(E)) - far (e) (63)
iE(m)
Proof.
Entanta fa((c) - g fa(c)l
(64)
< me fa(c)
(Lipschitz)
Li IIla =1[3] = 13 F), Id).
(65)
(Interpretation as counterfactual error)
m
fa (fz (z)) x|Hl
(66)
= -3
[f (ff (ffff(()) (2)) -2
(Canonical form)
25

---

Published as a conference paper at ICLR 2024
k-1
= 1 Σ
B (f - 2|m=-)
(67)
1=0
2 1 k max 13 () (fa (z)) -= =|m-
(68)
back
2 k max P [aar / (2)) - 2|{ iE[mi
(69)
L k max E Epte far (fa (z)))² iE[mi
(rearrange)
19 k- max Bp(dh(dr)pee) [fa(ee) for (e)
(change back to E)
iE[m]
where the distribution for de in this case is the one induced by J.
B.7 PROOF OF THEOREM 3
The proof of Theorem 3 relies on the Swapping Lemma (Lemma 9), and before proving the swap- ping lemma, first introduce a lemma which will be used in the swapping lemma to show that if one domain in an ILD is identity, then we could check intervention set using fa instead of fa
Lemma 8. For an ILD with fi Id. I(fa, fi) {i :fal,, + [flly)
Proof Lemma 8, Suppose (or) = x' where z¹j # Jyy then fa(æ') = becasue that fa bijective. Then
[fa(o')l, = I, # x = f1(').
For any j & I(F), for any fa(x'), we have 2 Ifā (z)], = =If (x)];())]; x. xy z'j, thus
2 = (fat")) = [fa(a'))) = s'g-
Lemma' 9 (Swapping Lemma). Given that the first canonical counterfactual property is satisfied, i.e., fi = Id, denote f' SCM constructed by f" = h2 fo hu((),, where h -h2 denote swapping the j-th feature with feature. Then there exists g' such that
Fg. = (g.F'), fi ld, I(F) (I(F) (7())
if the following conditions hold
j E I(F) and vj j < S IIF)
Proof of Lemma 9. First, note that because j' is not intervened, then can derive that it's corre- sponding conditional function is independent of all but the j'-th value:
[faly- [fily- (a = Ji, (#rz)) = Ty-
(70)
(71)
For the new model, we choose the invertible functions as swapping the j-th and j'-th feature values, i.e.
(xx)) [11, "
Fm]"
(72)
and similarly for h2, i.c., h2 A h1 Because h1 and h2 are invertible, we know that the new model will be in the same counterfactual equivalence class by Theorem Construct g' 2 o h and then for all d,
fz( =h fa hz(x))
12 [m I1) =h, Sa((v1, "/..".,.............. Uy. 1/-"""11""" -By-1 ",, 3ymm l1)
=h1 fa(1z1, x2,
=hn (Ja,(1
26

---

Published as a conference paper at ICLR 2024
- -Fai(u)): : /,
(ua sam(Usm))
fa.
where we define y - h2 (x).
We now need to check that the first canonical counterfactual property still holds.
fi = h1 o fi o h2 = h1 o Id h2 = h1 h2 = ld,
(73)
where the last equals is because swap operations are self-invertible.
We move check that the autoregressive property still holds for other domain SCMs. 1) For the j-th feature, we have that:
[f(())] = (asr) = (a , 13 #g+1,"" 11 = Ij where the last equals is because the fa. (ygr) = My = æz. This clearly satisfies the autoregressive property [fal only depends on zyj.
2) For the j'th feature, we have that:
[a(w)) (as( = Sa, (1,,"'",
where again this satisfies the autoregressive property because all input indices are less than j' be- cause j < j' Now we handle the cases for other variables. ] < j. then we have the following:
[a]j = (/1 o Sa a halj = [fa 114h22 helj Ja. (1(2(())))) = Saz(11,.... 13)
(74)
3) Similarly j < < jz:
Ij- where we use the fact that there are no intervening nodes in between j and j'. 4) Finally, for j j', have:
Fdlj = Sa, (1) 1,- Fg+1, ;
x4)) = L3.
(75)
[fa]
11 By+1, 2y+i." ;Z3))
(76)
which is still autoregressive because > j' anddd j > j. Thus, the new fa is autoregressive and is thus a valid model.
remains prove that
( (()
1) When l < j. we have for all d,
[d]] = (y = (2tx( = [Sa]a,
then for all l E I(F), there exists dn. such that
Isa, 11 = 1fd, # [fila = 11
Thus, E I(F)..
If l & I(F), we have for all d,
lfá 1f--11 = [falk = [fil = If 1
Thus &o & I(F").
2) When j < k < j', we have Vd. [[]( 2x + 1sa (x)] In. Thus we have Vd,
If4 1 [f1 1 which means for all j < kt < jj k $ I(F)..
3) When A: = j', we have Vd, [fals' = (a ... I ). Furthermore, since, j E I(F), we
have 3dg [Saul] * [Sil, by Lemma 8. Thus [/, [ # [rl] = [fl + E I(F') also
by Lemma 8.
When & j', if k E I(F), de, lfa, # [fa,le, Chaining with (76), we have [f # [agh, Thus, & E I(F") by Lemma 8. Similarly, if $ I(F), then $ I(F')
To summarize, I(F') (I(F) ())
27

---

Published as a conference paper at ICLR 2024
Built upon swapping Lemma, move our main result on the existence of equivalent Canonical ILD.
Theorem (Existence of Equivalent Canonical ILD). Given an ILD [g. F), there exists a canonical
ILD that is both counterfactually and distributionally equivalent to (),F) while maintaining the size of the intervention set, Le., F), (g',F E C s.t. Fg'FF') =CDD (g, and [I(F) = [I(F)).
Proof of Theorem 3. At high level the proof is organized in the following three steps.
(Step 1) we use Theorem to construct an equivalent counterfactual (g =o Fg. by choosing two invertible functions h1 = fi and h2 = Id. In this way, Theorem implies
1 h1 o fi o h2 = SJ¹ о fi o ld ld
vd>1 1, Sdd = h1 o Sa o h2 = SJ' o faold= 1 Sa, and go go = go Si.
Equipped with ;()), we can show that part I of Definition 5 is satisfied, i.e., ʃ = ld. Choosing h2 = ld, we could prove this operation could guarantee the distribution equivalence.
(Step 2) we can further construct series of equivalent counterfactuals iteratively applying Lemma 9 to gradually satisfy part II of Definition 5. Specifically, in this step, we recursively construct, for all iteration & E {1, (1,2..... last
F A (1( o F(--)) ()(0))0)))))>
and
gh a glk- o h (k)))))(()))))) ((k-1) h)())))(()))
where (()) denotes swapping the (k:)-th and j'(k:) feature values, i.e.,
(x)) 12,, Iy 1f+1 21. Im
(77)
and further define
j'(k) A max {i.j $ I [ } and j(k) A max {i {i< E (F())
(78)
In high level, at each iteration, seek the largest index jk(k) which does not lies in the previous intervention set I (F(+)), and swap it with the largest index j(k) which is smaller than jkkk.. We terminate when {j j((k).j I (F(*) =0.
By the definition of ('),),, (K)) in (78), can show that 1) for each swap step k there holds
j(k) E I (F((),, and Vj j(k) <} E (FW))
(79)
which implies Lemma 9 can be applied to ensure the counterfactual equivalence each step.
2) When meeting the stopping criterion at step last i.e..
{i (A e ( (I) 0,
(80)
there holds
Vj E (I) j 1> m - I (F(HM
i.e. (glttmm 1) =() is in canonical form. Chaining 1) and 2), we conclude
3 F²) 1 (getu -ull- E C s.t. (g',F)) ~e F).
Note that (k)) s(L)t = = p((-1) (() and linear operator (1) is orthogonal, then iteratively, we conclude (g', )') =n F).
28

---

Published as a conference paper at ICLR 2024
To prove 1), observe in (78), j(k) is the largest index in the intervention set which is smaller than jl()). This simply implies (79). To prove 2), suppose when meeting the stopping criterion step belust, there holds
3j I (x(( such that S I (I((M--1)
(81)
It implies that
Aj & I (F) and j E {m - - (F(lm) +l......m
Then we can choose j'(k) implying j {i j'c),, EI (F( * 0, contradict to (80).
(Step 3) We use the same techinique in Step 1, where instead we choose hı fi and h2 Id. This concludes the proof part I Theorem 3.
remains prove that the construction of f(0 the step does not change the intervention set. 1) For any j & I(F), for any pairs d, d', we have [fā [fa') = [f II, based on the construction of f(0), we have
[200 ", 1sa fils = =Iax Sili = [120] [1,0 -1
(82)
thus, I( (
I(fa: f2).
2) For any j E I(F), there exists d, d' and 2, such that (=)] * [far [f (=)]s- Note that fi is bijective function, there exists z' such that fi(), we have
[T (i)))) # (1('))
[ (t)) # [x1]
(50]
+ E I (0) (1
thus I (12), I(fa, fa). Combining 1) and 2), have I (120) 1 I(fa, fa) This
show that the construction of step 1 and step 3 do not change the intervention set, combining the fact step 1, we iteratively used swapping Lemma and swapping Lemma 9 does not change the inter-
vention set size, i.e., I(F') (I(F) \ (5'), conclude that [ (12) (0)
[[fa, fa)
This completes the proof.
To help understanding, we design simple linear ILD model to demonstrate the theorem construc- tion procedure.
Example 1. Suppose we have a 4-dimensional ILD model (g, F) containing 2 domains, where
0
0
0
0
1
0
0
2
2
0
f A
f2
4
9 invertible.
1
1
0
1
1
1
1
1
1
1
Following the proof of Theorem 3, we have Following Step 1 in the proof of Theorem 3, we have h = f-
fT!
f1,22
fi
(0)
fi
f2.
(0)
g
f1.
[ 0
0
07
1
0
0
(0)
1
0 0 1
0
0
-1
-1
0
0
0
1
0
0
0
29

---

Published as a conference paper at ICLR 2024
1
0
0
07
g
(
1
1
0
1
1
1
1
1
Notice that I( = =2, 4,
3). Following Step 2 in the proof of Theorem 3, we first swap j = 3 and
1
0
0
0
ha++1
0
0
1
19
(1)
go
0
1
1
0_
We have f( A h344 f(1) h3e44
0
1
0
2)
1
0
0
0
Notice that I(f(D) = (2, 4). Following Step 2 in the proof of Theorem 3, we first swap j = 2 and = 3,
1
0
0
0
h2603
0
0
0
19
(2)
go
1
1
1
1
0
1
1
1
We have f(3) h2443 f2) h2
1
0
0
(2)
0
0
0
122
(2)
0
1
0
1
2
0
0 0
1
0
-1
I
0 1
0
07
( go
1 0
1
1
1
1
0
Then we follow Step 3, i.e.,
(()
f1
(2)
(3)
fi
f2
(2)
g
0
0
0
0 0
1
0
0
((3)
1
0
1
2
2
1
1
1
1
1
1
Notice that is the Identity Canonical form, and ((3) is in general canonical form. They counterfactually equivalent each other by checking definition.
C SIMULATED EXPERIMENTS
C.1 EXPERIMENT DETAILS
Dataset The ground truth latent SCM få E FIA takes the form fī(e) F e+bū1ī where F) (I (I-La)-1, L E mxx is domain-specifer lower triangular matrix that satisfies sparsity constraint, b2 E IR is Hooaann-speciff bias and 1. is an indicator vector where any entries corresponding the intervention set are To be specific, [Lile ~ ~/ 1) and by Uniform(-22 -2 m/mmmm/[./////// 2 2 Z]). The observation function takes the form g (æ) = G LeakyReLU (æ) where G E mgmx and the slope of LeakyReLU is 0.5. To allow for similar scaling across problem settings,
set the de- terminant of G to be and standardize the intermediate output of the LeakyReLU. The generated b% G* all vary with random seeds and all experiments repeated for 10 different seeds. We generate 100,000 samples from each domain for the training set and 1,000 samples from each do- main in the validation and test set.
30

---

Published as a conference paper at ICLR 2024
We test with two ILD models: ILD-Can introduced in Section 3.3 and baseline model, ILD-Dense which has no sparsity restrictions on its latent SCM. To be specific, the latent SCM of ILD-Dense could be any model in Fra- We use I and I" represent the intervention set of the model and dataset, respectively. We note that for H.D-Densee, contains all nodes and for HInD-Can, I contains only the last few nodes. Both models follow similar structure the ground truth. To be specific, the latent SCM takes the form fa(e) = Fa E+ bb where Fd = (1- (1-La)) La) > SS, La E m
Model
Sa E Rmxm and b R The observation takes the form g(x) G LeakyReLU (x) + b where
G E Rmxm E R and the slope of LeakyReLU is 0.5.
Ld
Sa
b
(a) H.D-Dense
Ld
Sa
ba
(b) ILD-Can
1
1
1
Lg
Sa ba
(c) ILD-IIennttt--CC
Figure 3: An illustration of the matrices/vector used create fa across the three ILD models when and [ 2. These are used such that fa(E) Fa + ba where Fa (1 La) Sg.. The grey elements are O, the orange elements are parameters that are different for different domains, and the blue elements
parameters shared across domains. We speciffy the value if it is fixed number other than 0. Note that we don't implement IL.D-Identity-CC in our experiments. We include it here only for illustration of our theory.
In Figure 3a and Figure respectively. We emphasize few main differences between the dataset and models here: (1) For ILD-Can, I only contains the last few nodes while for the dataset while I could contain any node we specify. We note that ILD-Dense is equivalent to ILD-Can with all nodes in its intervention set. (2) There is no constraint on the determinant of and standardization g(x). (3) The bias added to all dimensions in the ground truth model is the same scalar value, but the bias in the model allowed to vary for each axis. (4) In the model, is allowed a learnable bias.
add an illustration of the latent SCM for ILD-Dense and ILD- Can
Metric
To evaluate the models, we compute the mean square error between the estimated coun- terfactual and ground truth counterfactual, i.c. Error Na(N2-1) Ev a Ea Il9" fa. (fā)
31

---

Published as a conference paper at ICLR 2024
(0')~"(xa) far o fa og-'(xx)ll" As in practice, only check data fitting instead of counterfactual estimation, and we report the counterfactual error computed with the test dataset when the likelihood computed with the validation set is highest.
Training details We use Adam optimizer for both f and g with a Ir - 0.001, = 0.5, 82 0.999, and batch size os 500. We run all experiments for 50,000 iterations and compute validation likelihood and test counterfactual error every 100 steps. f is randomly initialized. Regarding g, G initialized as an identity matrix and is initialized as
C.2 ADDITIONAL SIMULATED EXPERIMENT RESULTS
For better organization here, we split our experiment into three cases as introduced below. The first two cases point to the question: given the fact that we use the correct sparsity, does sparse canonical form model designing provide benefits in generating domain counterfactuals? The third case investigates the more practical scenario where we don't have any knowledge of the ground truth sparsity and we explore what would be better model design practice in this case.
Case O: Exact match between dataset and models In this section, we investigate the perfor- mance of LeD-eense and ILD-Can while assuming that the ground truth intervention set only con- tains the last few nodes and we choose the correct size of the intervention set.
To understand how the true intervention set affects the gap between ILD-Dense and ILD-Can, we varied the size of the ground truth intervention. In Figure 4, we observe that the performance gap tends to be largest when the true intervention set is the most sparse and the performance of ILD- Can approaches to the performance of ILD-Dense ILD-Can is subset of ILD-Dense and they are equivalent when when the ground truth model is relatively dense (when [Id] is close to m), ILD-Can is still better than ILD-Dense. Then we test how our algorithm scales with dimension when the number of domains is different. In Figure 5, notice that ILD-Can is significantly better than ILD- ILD-Dense in 9 out of 12 cases. In the next paragraphs, we further investigate the 3 cases that do not outperform I..D-Dense to understand if it seems to be theoretic algorithnic/optiiizatinn problem.
we increase the size. This makes sense as
{1, 3, 4, 5, 6). Additionally,
We take a further investigation on the three cases where ILD-Can is close to or worse than ILD- Dense As shown in Figure when the latent dimension is 10 and the number of domains is 2. iee. m = 10 and N₂ = 2, the validation likelihood of ILD-Can is much lower than ILD-Dense especially in comparison to that with m 4, 6. We conjecture that the performance drop in terms of counterfactual eтror could be result of the worse data fitting, i..., the model does not fit the data well in terms of log-likelihood. As further evidence, we show the counterfactual error and corresponding validation log-likelihoon in Table 3. We observe that the log-likelihood of HLDD-Dense tends to be much lower when it has a larger counterfactual error than that of ILDD Densee As for the relatively worse performance of ILD-Can when m 4, Na 2 and m 4, Na = 3, we report the counterfactual error corresponding to each seed in Table 4 and Table respectively. When the latent 4, Na 2, ILD-Can is better than ILD-
dimension is 4 and the number of domains is 2, i.e., Dense with 9 out of 10 seeds. However, it fails significantly with seed 0 and thus leads to a larger average of counterfactual error. When m 4, N₂ 3. IL.D-Can is better than II.D-Dense with out of 10 seeds but II.D-Can is not significantly better than II.D-Dense in terms of average error. We think this more likely an optimization issue with lower dimensions, which is not explored by our theory. We conjecture that larger models with smoother optimization landscapes will perform better as we see in the imaged-basee experiments. We also note that these models are not significantly overparametrized and thus may not benefit from the traditional overparameteriization that aids the performance of deep learning in many cases. Further investigation into overparameterized models may alleviate this algorithmic issue.
Despite some corner cases in which the optimization landscape may be difficult for these simple models, all the results point to the same trend that the sparse constraint and canonical form motivated by our theoretic derivation indeed aids in counterfactual performance- not explicitly training for counterfactual performance.
Case 1: Correct [Z] but mismatched intervention indices In this section, we include more re- sults in the more practical scenario where choose the correct number of the intervened nodes
32

---

Published as a conference paper at ICLR 2024
but they are not necessarily the last few nodes in the latent SCM. This experiment is related to canonical ILD theory, i.c., that there exists a canonical counterfactual model (where the intervened nodes are the last ones) corresponding to any true non-canonical ILD that has the same sparsity. As a starting point, we first illustrate the existence of a canonical model we try to find in Figure 10.
To investigate the effect of different indices of the intervened nodes, in Figure 7, we change the true intervention set I while keeping the number of intervened nodes [Tr] the same. We observe that ILD-Can is consistently better than ILD-Dense regardless of which nodes are intervened except for one case. When the number of domains is 2 and I {4, 5). we find the gap is much smaller mainly because ILD-Can fails fit the observed distribution in one case as shown in Table 6. We then test the effect of the number of domains with different latent dimensions in Figure 8. We observe that our model performs consistently well with different numbers of domains and latent dimensions. In Fig- ure 9, we visualize how IIDD-Can leads to lower counterfactual error in comparison to II.D-Dense. As shown Figure 9a and Figure 9b, ILDDCan clearly does better in counterfactual estimation. In Figure 9c and Figure 9d, both of them have a relatively larger error. However, ILD-Can tends to find a closer solution while ILD-Dense matches distribution more randomly. This could result from the large search space of ILD-Dense and it can easily encodes transformation such rotation which will not hurt distribution fitting but will lead to a significant counterfactual error.
Even though we do not know the specific nodes being intervened on, similar to Case 1, show that sparse constraint leads to better counterfactual estimation.
Case 2: Intervention set size mismatch In this section, we include more results in the most difficult cases where we have no knowledge of the dataset. To investigate what will happen if there mismatch of the number of intervened nodes between the true model and the approximation, i.e., |Z| * [ we first change I" while keeping the model unchanged, i.e., I is fixed. As shown in Figure 11, the performance gap between II.D-Can and IL.D-Dense become smaller as the dataset becomes less sparse while ILD-Can outperforms ILD-Dense in all cases. We then change I while keeping I' unchanged. As shown in Figure 12, the performance of ILD-Can approaches to that of ILD- as increase [Z]. A somewhat surprising result is that ILD-Can has the lowest counterfactual error when [ = 1. However, as we check data fitting in Figure 13, we can tell ILD-Can fails to fit the observed distribution in this case. We conjecture the main reason for this is that our theory does not guarantee the existence of a distributionally and counterfactually equivalent canonical model in those cases we using model that is more sparse than the ground truth dataset. Hence, we cannot rely on the counterfactual estimation when the observed distribution is not fitted.
In summary, we observe that ILD-Can always tends to get a lower counterfactual error even though we choose a wrong size of intervention set, i.e. [ * |Z"|. However, we also observe that in the cases where our model is more sparse than ground truth. the data fitting performance of ILD- Can would drop more significantly. We believe this could also good indicator of whether we find a reasonable [Z].
Case different [z'] (M,=2)
D: different [2'| (N,-3)
Case 0: different [2"] (M1=100)
15
ID Dense
15
I.D. Dense
35
IDD Dense
LD-Can
D.DCCan
D-CCan
10
10
30
2'
(6)
(5.6)
(4.5.6)
(3.4.5.6)
x--(6) (6)
(5.6)
(4.5.6)
(3.4.5.6)
[')(G)
(5.8)
(4.5.6)
(3.4.5.6)
(a) Na
(b) N₂
(c) No 10
Figure 4: Case 0: Test counterfactual error with different I", To understand how the true interven- tion affects the gap between ILD-Dense and ILD-Can, we varied the size of the ground truth intervention. It can be observed that the performance gap tends to be largest when the true interven- tion set the sparsest and the performance of ILD-Can approaches the performance of ILD-Dense as we increase the size.
33

---

Published as a conference paper at ICLR 2024
Case Latent dimerision (N)
Case Latent dimension (N
Case Latent dimension (Ng=10)
10
10
10
ILD-Dense ILD-Can
ILD-Dense
ILD-Dense
ILD-Can
IDDCCan
(a) Na
(b) Ng
(c) Na 10
Figure 5: Case O: Test counterfactual error with different dimension. We investigate how algo-
rithm scales with dimension. We observe that ILD-Can significantly better than ILD-Dense out of 12 cases, and we also notice that there 3 cases where their performance is close that of each other. Here the intervention set contains the last two nodes. For example, when m = 4, = (3, 4). and when
10, I (9, 10).
Case Latent dimersion (M,=2)
log llaelhhood
ID-CCa
mmd
mm6
m=8
10
Figure 6: Case O: Lowest validation log likelihood (same when report the test counterfactual error) when testing different dimension with NJ = 2. We observe that the likelihood gap between ILD-Can and I.D-Dense largest when m 10.
Table 3: Case O: Test counterfactual error and validation log likelihood for each seed when m
10, Na 2. We observe that the log likelihood of ILD-Dense tends to be much lower when it has larger counterfactual error than that of ILDD-ense.
Seed
6
Counterfactual enor
ILD
4,625
0.111
0.120
0.072
4.572
10.617
4.360
6,809
0.099
0.479
ILD-Dense
23.811
0.611
2.188
5.825
4.779
0.694
0.487
1.633
3.170
6.365
Log likelihood
1aD).Con
68833
-7066
-.672
.4.637
40.572
-3.261
66062
4.552
-1.367
-5.170
ILDsDenss
4.044
6.43
33,79
-4.99
0.711
-1.908
4..80
2413
1.483
-4.96
Table 4: Case O: Test counterfactual eπrоr for each seed when m = 4, 4, = 2. ILD-Can is better than ILD-Dense except when seed is O. However, there significant failure for ILD-Can with seed
O.
Seed 0
2
3
4
5
8
9
ILD-Can
23.790
2.309
1.747
3.180
1.265
0.864
0.779
0.227
3.325
6.362
ILD-Dense
3.321
3.435
2.838
4.209
5.356
6.456
1.615
2.165
5.195
7.937
Table 5: Case Test counterfactual error for each seed when than Dens- with seed 1, 2, 3, 5, 6, 7,8.
4,
ILD-Can is better
Seed
0
2
3
4
8
9
ILD-Can
23.821
0.611
2.178
5.823
4.779
0.694
0.487
1,653
3.170
6.365
ILD-Densee
24.472
3,658
2.925
5.785
3.260
5.795
3.878
4.560
4.009
5.965
34

---

Published as a conference paper at ICLR 2024
(Ny=2)
(Ny=3)
indices (N,=10)
ILD-Oense
IUD-Can
TUD-Can
IUD-Can
c.T. eneer
C.. e..enoor
(5.6)(4.5)
[3.4)
(2.3)
(1.2)
2" (5.6)(4:5)
(3.4) (2.3)
(1.2)
(4.5) (3.4) (2.3) (1.2)
(a) Na
Na
N₂ 10
Figure 7: Case 1: Test counterfactual error with different indices, Here we observe that ILD- Can performs consistently better than ILDD-eesse. When Na and I {4,5), the performance of ILD-Can gets relatively higher because it fails significantly in one case as shown in Table 6.
Case domeins (m=6, (1.2H
Case domains (m=8, (H
Case domains Im=10, (1.21
10.0
10.0
10.0
ILD-Con
ILD-Can
ILD Can
c.ferror
cf.
2.5
2.5
2.5
0.0
0.0
0.0
Ma=2
1,-3
Mg=5
N,-10
Ma=2
N4-3
N,-5
N,-10
Ny-3
14,55
N₂=10
(a) m
6
(b) m =
(c) m = 10
Figure 8: Case 1: Test counterfactual error with different number of domains when I {1,2).
IL.D-Can performs consistently well with different number of domains and latent dimension.
Table 6: Case 1: Test counterfactual error and validation log likelihood for each seed when N₂ = and I 54. When seed 5, the error of ILD-Can much larger than that of ILD-Dense. In the meanwhile, we notice that the log likelihood of ILD-Can is much lower than that of ILD-Dense which indicates ILD-Can fails to the observed distribution well. When seed is 6, there is also gap in log likelihood. But both models perform very badly in terms of counterfactual error in this case, and we conjecture this results from very hard dataset.
Seed
2
3
4
5
8
Counterfactual envor
Can
1.395
0.862
1.338
0.193
7.557
12.422
21.762
3,899
2.352
0.479
ILDs-Dense
8.610
3979
4.134
2.983
9,795
4.79
242232
5.327
8.497
8,500
Log likelihood
1LD-Can
-4,41
-5.37
4.448
-5,004
4.393
+3,376
-3.187
-3073
44033
4.102
11D--Dense
4.170
-3,632
44316
-3458
.4.174
-2.181
.4005
-3.10
-5.270
-4302
35

---

Published as a conference paper at ICLR 2024
EST.
EST.
Ground Truth
Ground Truth
(a) ILD-Can: Domain + 2
(b) ILD-Dense: Domain -
EST. Ground Truth
EST.
Ground Truth
(c) ILD-Con: Domain 3 +
(d) ILD-Dense: Domain ->
Figure 9: Visualization of counterfactual error when m 6, N₂ 3, 2, (1, In each plot, we find the first two principle components and project the data along that direction. We select 10 points, then find the corresponding ground truth counterfactual and estimated counterfactual. The black arrow points from ground truth to estimated counterfactual.
36

---

Published as a conference paper at ICLR 2024
4
domain
domain
21
20
2;
X1
X2
Xs
X4
(a) Step 0: f( =
fa
(b) Step 0: g (0)
domain
domain
x
2:
2+
X1
(c) Step 1: y0) = fi fa
(d) Step I: g (2) = g f
X
domain
demain
4
(e) Step ((9) haas f. fa hresa
( Step 2: (2) g o f hies3
Figure 10: An illustration of the existence of a distributionally and counterfactually equivalent model in canonical form when m = 4 and = {2). hr+>3 represents a swapping matrix. g(2) o f(2) is of the caonical model we try find. Note that the observed distributions the right column always the same while the latent distributions on the left change. In particular, the canonical ILD model on the bottom left has independent distributions for the first three variables and only the non-identity on the last node.
37

---

Published as a conference paper at ICLR 2024
Case 2: different [ (My=2)
Case different [2" (Mg-3)
Case 2: different [E" (Mg=10)
15
L.D-Dense
L.D.Dense
I.D-Dense
Car
ILD-Can
ILD-Can
error
10
c1. orrrr
10
eroor
10
1. (63
(5.6)
(4.5.6)
(3.4.5.6)
(''(6) (6) (5.0) (4.5.6)
(3.4.5.6)
1' (6)
(5,6)
(4.5.6)
(3,4,66))
(a) Na-
Na
Na 10
Test counterfactual error with different [I*| and fixed [ 2. The performance of ILD-Can gets worse as the dataset becomes less sparse. But still better than D..D-Dense. Note that when [ = and I {6}, the ground truth canonical model is still subset of the models we search over.
Figure 11: Case 2:
15
Case 2: different [r] 11"= (55.)=)55mmm22
15
Gase 2: different [r] Ir" ((5.)))))mmmm3
15
Case 2: different [+] (r' ((5.))"(_6))---00 Nom10
ILD--ense
ILD-Dense
ILD-Dense
c. amorr
10
C... onor
10
c. emor
10
[X]-1
[F|-1
(a) Na
(b) Na
(c) Na 10
Figure 12: Case 2: Test counterfactual error with different [Z] and fixed I". The performance of
ILD- Can approaches to that ILD- DDD-eeese as we increase [Z].
Case 2: differeett [z] (2" (5.6))
Case 2: different [Z] (2" ((33)))Mu=))
Case 2: different [I] (2" (u=6))
103 (ikalioood
-2
108 4
D-Denne
D-DDense
DD-Dense
-0
17|=
11|=2
m/
14=3
(8) Na =
Na
(c) N 10
Figure 13: Case 2: Lowest validation log likelihood with different [Z] and fixed I". When [ 1, there is more significant gap between ILD-Can and ILD-Dense with all N₂ which indicates ILD- Can might fail to fit the observed distribution.
38

---

Published as a conference paper at ICLR 2024
3.0
ILD-Dense
ILD-Dense
ILD-Can
2.5
c.f. error
Ü 1.0
0.5
0.0
1' (5.() (4.5)
(3.4)
(2.3)
(1.2)
[1|=1
14|=
[/]=3 |/|=4
With knowledge of I and I
(b) Without knowledge of [ and I"
2
(5.6)
6, number of domains Na 3) where observation function of both the model and ground truth are implemented based on RealNVP. The model consists of 4 affine coupling layers and the ground truth consists of affine coupling layers. Results are averaged over 10 runs with different ground truth SCMs and the error bar represents the standard error.
Figure 14: Simulated experiment results (latent dimension
C.3 SIMULATED EXPERIMENT WITH MORE COMPLICATED STRUCTURES
As an extended study of our simulated experiment, we implement a more complicated observa- tion funetion for the ground truth g". We build g" based on RealNVP (Dinh et al., 2016), where g" is composed of four sequences of affine coupling layers followed by permutation. Follow- ing the notation used in Appendix C.I, the ground truth ILD takes the form g o f(() Permutation( (AfinnAAffeeeeeee Afffineouullinnn..... Permutation( A CiineCoulinn((eeal LeakyReL.U (æ) + b))...).
We first investigate the setting where model g exactly matches the ground truth ILD, i.e. the same number of affine coupling layers for the ground truth g and our model g. In Figure 14a, observe that, similar Figure la, ILD-Cat outperforms ILD-Dense no matter which nodes intervened when [Z| = | We then test our model in the setting where [Z] * |Z*], | and as shown Figure 14b, the performance of ILD-Can drops as we [Z] grows larger than [I but better than #LD-Dense. This trend is similar to what we observed in Figure 1b. We then test when the IID-Can architecture does not match H.D-DDesse To do this, we increased the number of affine coupling layers in the model y to be 8, and in Figure 15a and Figure 15b, we observe similar trends when comparing I..D-Dense and ILD-Can in all cases. This gives evidence that WLD-Can helps with domain counterfactual estimation even when the structures of the model and ground truth do not exactly match.
always
To further our investigation with mismatching models, set to be VAE-based model composed of three fully connected layers while keeping the ground truth the same as above. To keep the latent dimension relatively comparable, sion of VAE to be 6. In Figure 16a and Figure 16b, we observe a similar trend as that with flow model. In Figure 17, only when [Z] < [I*), there is significant difference in the pursuit of distri- bution equivalence. This again supports our finding that the distribution equivalence performance could be a good indicator of choosing in practice.
set the ground truth dimension to be 10 and the latent dimen-
39

---

Published as a conference paper at ICLR 2024
3.0
ILD-Dense
ILD-Dense
ILD-Can
2.5
c.f. error
1.0
0.5
0.0
1" (5.6) (4.5)
(3.4)
(2.3)
(1.2)
[1|=1
[4|=
[/]=3
|/|=4
With knowledge of I* and I
(b) Without knowledge of I and I"
2
(5.6)
6, number of domains Na 3) where the observation function of both the model and ground truth are implemented based on RealNVP. The model consists of 8 affine coupling layers and the ground truth consists of affine coupling layers. Results are averaged over 10 runs with different ground truth SCMs and the error bar represents the standard error.
Figure 15: Simulated experiment results (latent dimension
ILD Dense
ILD-Dense ILD. LaD
ILD-Can
c.I. error
C.I. etror
1' (5.6) (4.5)
(3.4)
(2.3)
(1.2)
[2|-1
|X|=2
|2]-3
7//4
(a) With knowledge of I and I
(b) Without knowledge of [ and T
2
15.00 6}
Figure 16: Simulated experiment results (in the ground truth latent dimension m = 10 and in the VAE model 6, number of domains N&- 3) where the observation function of the ground truth consists of affine coupling layers and the observation function of the model is a VAE composed of three fully connected layers. Results averaged over 10 runs with different ground truth SCMs and the error bar represents the standard error.
1.0
ILD-Dense
Negative ELBO
0.6
0.4
0.2
0.0
|===1
|+|=2
[2]-3
[2]-4
Figure 17: Lowest validation negative ELBO (in the ground truth latent dimension m 10 and in the VAE model m 6, number of domains Na 3) without knowledge of [Z*| and I = {8, 9}. Results are averaged over 10 runs with different ground truth SCMs and the error bar represents the standard error. The number here corresponds to the validation negative ELBO of the checkpoints we use report test counterfactual error.
40

---

Published as a conference paper at ICLR 2024
D IMAGE COUNTERFACTUAL EXPERIMENTS
D.1 DATASET DESCRIPTIONS
Rotated MNIST and FashionMNIST We split the MNIST trainset into 90% training data, 10% validation, and for testing we use the MNIST test set. Within each dataset, we create the domain-specific data by replicating all samples and applying fixed 0g counterclockwise rota- tion to within that domain. Specifically we generate data from domains by applying rotation 0f, 15", 30 450. For Rotated FashionMNISTT, except we used the Fashion MNIST (Xiao et al., 2017) dataset. This dataset structured similar to the MNIST dataset but is designed to require more complex modeling (Xiao al., 2017).
use the same setup as the RMNIST setup,
3D Shapes This is a dataset of 3D shapes that are procedurally generated from independent latent factors: floor hue, wall huc, object huc, scale, shape, and orientation (Burgess and Kim, 2018). In our experiment, we only choose samples with one fixed scale. We then split the four object shapes into four separate domains and set the 10 object colors as the class label. The causal graph for this dataset can be seen in Fig. 18c, and following this, we should expect to see only the object shape change when the domain is changed. Similar to the RMNIST experiment, we use 90% of the samples for training and 10% the samples for validation.
Color Rotated MNIST (CRMNIST) This is an extension of the RMNIST dataset which intro- duces latent color variable whose parents are the latent domain-specific variable and latent class variable. Similar to RMNIST, the latent domain variable corresponds to the rotation of the given digit, except here dj 0 rotation, d2 90 rotation, and the class labels are restricted to digits E {0, 1, 2]. For each sample, there is a 50% chance the color is determined by the combination of class and digit label and a 50% chance the color is randomly chosen. For example if ~ NV 1),
red,
if y 0, d 1, > 0 if = 0, d = 2, 6 0 if - - 1, 0
green,
fz. (v, d)
blue,
Random(red, green, blue, yellow, cyan, pink), if <0
The causal graph for this dataset can be seen in Fig. 18b. Similar to the RMNIST experiment, we use 90% of the samples for training and 10% of the samples for validation.
Causal3DIdent Dataset This is a benchmark dataset from (Von Kügelgen et al., 2021) which contains rendered images of realistic 3d objects colored background that contain hallmarks of natural environments (e.g. shadows, different lighting conditions, etc.) which are generated via a causal graph imposed over latent variables (the causal graph can be seen in Figure 18d). Similar to (Von Kügelgen et al., 2021), we chose the shape of the 3D object to be the class label, and we defined the background color the domain label. In the original dataset, the range of the background hue | 7 and to convert it to binary domain variable, we binned the background hue values - into bins 2 -0.8] and T 0.8). These ranges were chosen to be distinct enough that we can easily distinguish between the domains yet large enough to keep the majority of original samples in this altered dataset. We split the 18k binned samples into 90% training data and 10% validation data for our experiment.
was
D.2 METRICS
Inspired by the work in Monteiro et al. (2023), we define four metrics (Effectiveness, Preservation, Composition, and Preservation) specifically for the image-based counterfactuals with latent SCMs. The key idea is to check if the correct latent information is changed when generating domain coun- terfactuals (e.g.. Noomain-speciff information is changed, while all else is preserved). Since we don't 't have direct access to the ground truth value of latent variables (nor their counterfactual values), use a domain classifier homaain and class classifier helss to measure if the intended change has taken place.
41

---

Published as a conference paper at ICLR 2024
Effectiveness:
The idea is to check if the Samainn-peciffe variables change wish in the counter- factual samples.
(hdomain (&d-+d) d')
This checks if the semantically meaningful content (i.e. the class information) that is independent of the domain is left unchanged while the domain is changed.
Preservation:
P (helass (ia-sd') = y)
Composition: We check our model is invertible on the image manifold, thus satisfying the pseu- doinvertibility criteria.
PP (heasss (fa-sd) = y)
Reversibility: This metric checks our model is cycle-consistent, or in other words, checking if the mapping between the observation and the counterfactual is deterministic.
P (heluss (d)
For the domain classifier Adomii and class classifier hetass . we used pretrained ResNet18 8 models (He ct al., 2016) that were fine-tuned by classifying clean samples (i.c. not counterfactuals) for 25 epochs with the Adam optimizer, learning rate of le-3, and a random data augmentation with probabilities: 50%: no augmentation, 17%: sharpness adjustment (factor=2), 17%: gaussian blur (kernel size=3), MNIST,
17%: gaussian blur (kernel size=5). A reminder that for MNIST/EMNNIST/CooorRe the domain is rotation and the label is the original label of images (digits/type of clothes), for 3D shapes, the domain is object shape and the label is object color, and for Causal3Dldent, the domain is hue of the background and the label is the object shape.
D.3 CAUSAL INTERPRETATION OF OUR EXPERIMENTS
In this section, we introduce the causal interpretation of our experiments. To evaluate the model's s capability of generating good domain counterfactuals, for each dataset, have domain latent variable and choose one class latent variable that are generated independently of the domain latent variable. As an example, for RMNIST, we choose rotation the domain latent variable and digit class as the class latent variable. As indicated in Figure 18a, for the counterfactual query "Given observed an image in this domain, what would have happened if it was generated by different domain?", we should expect the image to be rotated while the class remain unchanged. Specifically, we want to check
PiZootp X = x, == PZ ay XX xx D= dd,d'd'' d" DDd's d' 4 d' P( P(Z X xr, D= d) po X x. D= D= Vd,d' d" E E
D,
where D is the set of all domains. However, in practice cannot directly get the value of those latent variables. This motivates our choice of evaluation metric of training a domain classifier and class classifier to detect if the domain latent variable is changed and class latent variable (which call class) is preserved in the counterfactuals.
choose rotation the domain variable and digit/clothes class as the class variable. For 3D Shapes, we choose object shape as the domain variable and hue of objects as the class variable. For CRMNIST, we choose rotation as the domain variable and digit class as the class variable. For Causalldent, we choose the hue of the background as the domain variable and object class as the class variable. In the case of 3D Shapes, we can technically choose anything other than object shape as the class variable. However, for simplicity, we choose one of them. In the case of CRMNIST, we cannot choose Zuokoe because it will change after we change the domain. In the case of Causal3DIdent, is easier to check and can reduce error caused by classifier proxies.
For RMNIST/RFMNIST
choose anything but the hue of the object, though we figure Z,
We also want to note that other than observational image, access to domain information is also important for answering this query. For example, in the case of RMNSIT, given an image that looks like digit "9", for the question "what would have happened if it is in domain 900", the fact that the current digit is in domain 0 (which means it is indeed digit "9") or the current digit is in domain 180 domain 0 (which means it is digit "6") would lead different answer.
42

---

Published as a conference paper at ICLR 2024
d
1o
Z
yes
Zoot
Zy
Zes
Ze
X
(a) RMNIST/RFMNIST
(b) ColorRMNIST
d
shape
Zuoc obj
Zhme floor
Zhoe
Zorient
x
(c) 3D Shapes
Zpos.apl
Z
Zuoa.bg
Zuee-spl
Zpon.obj
Zior.oii
Zuuc.obj
(d) Causal3Dldent
Figure 18: (a) RMNIST/RFMNIST. Here 7 represents the rotation of the image, 7.y repre- sents the original RMNIST/REMNIST class, Zres contains other detail information such as writ- ing style, which is controlled by how MNIST dataset was originally created. (b) Z. represents the color of the digit while others are the same as (a). (c) 3D Shapes. Zahupe represents the object shape. Zhee ubj. Zhue floer: Zrue wall represent the hue of the object, floor and wall respec- tively. Zorient represents the orientation of the object. (d) Causal3Dident. Zy represents the object class. Zimezabji Zhue.hy; Zhme.sph represent the hue of the object, background and spotlight respectively. Zpos.obj. Zpos.spi represent the position of the object and spotlight respectively. Ziocobi represents the rotation of the object. is not shown in the graph but nodes should point to it.
D.4 EXPERIMENT DETAILS
Model setup The relaxation to pseudo invertibility allows us to modify the ILD models to fit VAE (Kingma and Welling, 2013) structure. The overall VAE structure be seen in Fig. 19,
43

---

Published as a conference paper at ICLR 2024
Input: (xd=cylinder, d' sphere)
Domain Specific
Across
Output: (2t=sphere)
Domains
d
g
E (Hla.Ga)
g
fa
fåe
Figure 19: The model structure for the pseudo-invertible ILD model used in the high- dimennsinii experiments. The overall structure matches that of a VAE where the encoder (left) first projects to the latent space via g+ (the pseudoinverse of the observation function g). This latent encoding is then passed to two domain-specifie autoregressive models Iī J which produce the mean and variance parameters (respectively) of the Gaussian posterior distribution. During training. the ex- ogenous noise variable which can be viewed as stochastic SCM, however, during inference the exogenous variable is set to the mean of the latent posterior distribution (i.e. 6 == jua) to reduce noise when producing counterfactuals. The decoder (right) follows the usual VAE decoder structure, with the exception that the initial linear layer is an autoregressive function of the e input. The structure of all the f models is determined by the type of ILD model used (e.g., dense, canonical, or identity canonical) and matches that seen in Fig. 3.
then found via sampling from the posterior distribution (c ~ ( oa))
where the variational encoder first projects the latent space via 9 to produce the latent encoding 2, which is then passed to two domain-specific autoregressive models fā. f which produce the mean and variance parameters (respectively) of the Gaussian posterior distribution. The decoder of the VAE follows the structure typical ILD structure: g o fa. Here, g can be viewed as the pseu- doinverse of the observation function and fa can be viewed as a pseudoinverse of ft, During training, the exogenous noise variable is then found via sampling from the posterior distribution (c (Nup, oz)) which can be viewed as a stochastic SCM, however, to reduce noise when pro- ducing counterfactuals, when performing inference the exogenous variable is set to the mean of the latent posterior distribution (i.e. = jaz). In all experiments, 9 and g+ follow the 3- VAE archi- tecture seen in Higgins et al. (2017) (with the exception that in the Causal3DIdent experiment, and g+ follow the base VQ-VAE architecture (Van Den Oord et al., 2017) without the quantizer), and the structure of the f models is determined by the type of ILD model used (e.g., independent, dense, or relaxed canonical) and matches that seen in the simulated experiments and visualized in Fig. 3. For the f models which enforce sparsity (i.e. ILD-Can), we use sparsity level, [Z], of 5. We also introduce an additional baseline, ILD- LLD-Innependent, which has an architecture similar to the ILDD-Desse baseline, with the exception that the g and g² functions are longer shared across domains. The ILD-Independerl baseline can be seen as training an independent 33 for each domain, where each 8- B-E an autoregressive fdense model as its last (first) layer for the encoder (decoder), respectively. For experiment with RMNSIT, RFMNIST, 3D Shapes and Causal3DIdent, we choose m and for CRMNIST, we choose m 10.
Training We train each ILD model for 300K, 300K, 300K, 500K, and 200K steps for RM- NIST, REMNIST, CRMNIST, 3D Shapes and Causal3 respectively using the Adam optimizer (Kingma and Ba. 2014) with 81 0.5, 32 0.999, and batch size of 1024. The learning rate for g and g+ is 10-4, and all f models use 10-3 During training, we calculate two loss terms: reconstruction loss frecon x . 3 where & is the reconstructed image of x and the latign alignment loss which measures the KL-divergence between the posterior distribution Qa(((a) and the prior P(e). Following the 3-VAE loss calculation in Higgins et al. (2017), we apply (KLD upscaling to the alignment loss such that lioral = Erecon + 9KLD * Eatign- For all MNIST-like experiments, we use 3KLD = 1000, which we found leads to the lowest counterfactual error on the validation datasets across all models; this also matches the 3KLD used in Burgess et al. (2018), and for 3DShape and Dldsal we found BKLD 10 leads to the lowest counterfactual error.
D.5 ADDITIONAL RESULTS
The quantitative results in Table 7, Table 8, and Table 9 match the visual result seen in Figure 20, Figure 21, Figure 22, where almost across all datasets the ILD-Can model seems to find a proper latent causal structure that disentangle the domain information from the class information
44

---

Published as a conference paper at ICLR 2024
KMNIST
REMNIST
Comp.
Rev.
Eit.
Pre.
Comp.
Rev.
Ed.
Pre.
ILD Independent
98.
0.44
32.56
0.20
84.97
4.71
52.09
0.22
68.75
1.86
22.36
0.76
98.82 0:37
1.19 20.38
22.54
ILD-Deese
0.28
22.80
80.32
32.64
2.51
0.98
2:18
ILD Can
99.85 1 0.27
79.34
17.54
96.72
1.89
64.99
9.88
71.79
1
4.55
70.44
3.54
98.82 0.78
62.15 1: 6.65
Table 7: Quantitative result with RMNIST and RFMNIST, where higher is better. They are both averaged over 20 runs.
CRMNIST ER
3D
TED Independent
57,24
R.D-Doose
02.29
10.51
92.72
0.28
90.2
2.21
32.04
92.10 14.21
85.44
13.33
34.85
10.71
7235 12.42
12.42
99.85 +
0.27
19.34
19.54
16.72 1,30
64.30 +
9.83
Table 8: Quantitative result with CRMNIST and 3D Shapes, where higher is better. CRMNIST are averaged over 20 runs and 3D Shapes are averaged over 5 runs.
unlike the baseline models which seem commonly change the class during counterfactual. We again note that the training process for all of the models only include the typical VAE invertibility loss (i.e. reconstruction loss) and latent alignment loss (i.e. the KL-divergence between the latent prior and posterior distributions) and do not specifically include any counterfactual training. Thus, we conjecture the enforcing of sparsity in the canonical models correctly biased these models in manner that preserved important non-domain-speciff information when performing counterfactuals. In Figure 25, Figure 26, Figure 27 and Figure 28, we track the change of our metrics w.r. [Z] (we did not do this investigation for Causal3DIdent because that the training of that model takes much longer time). We observe that as we increase [Z], the reversibility and preservation tends to decrease while the effectiveness tends to increase. We conjecture that this is because as [Z] increases, there is less constraint on the original optimization problem (fitting the observational distribution) which could potentially increase the performance. However, it leads to lower chance in finding proper latent causal structure for domain counterfactual generation, which results in the decrease in preservation. ILDD-Dense can be regarded our model design in the pseudoinvertible setting with extensive study on 5 image-based datasets.
extreme case of this. In summary, validate the practicality of
d2
d3
d
d;
d
d2
dg
d,
d5
d
d2
dg
d2
1 dg
d,
O
1
2
3
4
6
Y
A
0
O
|
7
3
4
5
4
A
8
|
7
4
6
4
A
8
0
dg d
0
2
6
7
0
9
6
6
9
3
4
5
6
7
8
4 9
4, d
6
3
3
4
u
6
1
6
6
0
9
2
3
9
9
6
0
6
0
2
3
4
5
6
7
8
2
6
0
8
5
4
8
2
2
00100
6
3
4
6
6
8
a
4, d
1
0
6
V
2 4
0
2
A
6
6
^
8
A
4,
A 0
6
R
A
8
Baseline: ILD ILD-ennnnnndtnt
Baseline: ILD-Dense
Ours: ILD-Can
Counterfactual plots for the three relaxed ILD models, where across the columns show examples of two clothing classes (c.g., "handbag" or "boot") from each domain and each row corresponds to the counterfactual recover the rotation for each domain counterfactual, the baseline models usually change the class label during counterfactual, while ILD-Cat tends to preserve the clothing label, despite not being privy to any label information during training.
Figure 20:
different domain. It can be seen that while all models correctly
Causal3DLdent ElI.
Comp.
W.D. InDD-eedeenn
88. 18 5.0
51431177
91.05 + 17.7
51.11100
49.17.1.5
92.17
13.8
48.8313.0
ILD
6.6
84.15
23.5
Table 9: Quantitative result with Causal3DIdent, where higher is better. Causal3DIdent are averaged over 10 runs.
45

---

Published as a conference paper at ICLR 2024
d1
d2
d3
44
d5
d1
22
d
4
dg
d1
22
d
44
ds
42
4, d:
4, d2
d, d,
d,
Baseline: ILD-Independent
Baseline: ILD-Dense
Ours: ILD-Can
Figure 21: Counterfactual plots for the three ILD models, where across the columns show exam- ples of two classes from each domain and each row corresponds to the counterfactual to a different RMNIST domain. can be that while all four models correctly recover the rotation for each domain counterfactual, the baseline models usually change the digit label during counterfactual, while ILD-Can tends preserve the digit label, despite being privy any label information during training.
d
22
d3
d4
d1
d2
d
4
d1
d2
d2
dg
d2
dg
Baseline: ILD-eenepemdent
Baseline: ILD-Dense
Ours: ILD-Can
Figure 22: Counterfactual plots for the three ILD models, where across the columns we show exam- ples of classes from each domain and each row corresponds the counterfactual to different object shape domain.
d1
d2
d1
d2
dj
d2
2
Y
2
K
2
d,
1 2
2
d, - d,
ON
d, 422
Baseline: ILD-Independent
Baseline: ILD-Dense
Ours: ILD-Can
Figure 23: Counterfactual plots for the three ILD models, where the columns show exam- ples of two classes from each domain and each row corresponds to the counterfactual to different rotation domain.
d
d2
d
d
Yey
d,
- d,
d, -d2
Baseline: ILD-Independent
Baseline: ILD-Dense
Ours: ILD-Can
Figure 24: Counterfactual plots for the three ILD models, where across the columns we show exam- ples of two classes from each domain and each corresponds the counterfactual to different background hue domain.
46

---

Published as a conference paper at ICLR 2024
REde endiperdets
Con LD Oorot rokppertd
19
model
mide iutreeedeed
Figure 25: Change of metrics w.r.t [Z] for RMNIST. Results are with 20 runs and we remove outliers when plotting.
mode 1UC38 Indeppddert
model DrCerr
75.3
87.
model 8D Indepandes
953
58.0
90.0
LD
LD LD-Dems
Figure 26: Change of metrics w.r.t [Z] for REMNIST. Results are with 20 runs and we remove outliers when plotting.
47

---

Published as a conference paper at ICLR 2024
nedel 6. L0-erripereet:
18
Mode ID Can
100
Cene
PICEN EIC00SRPPPAPNT
Figure 27: Change of metrics w.rt [Z] for CRMNIST. Results with 20 runs and remove outliers when plotting.
505
mcdel LDbieesse
90.3
01
made
LD.CH
Figure 28: Change metrics w.r.t [Z] for 3D Shapes. Results are with 5 runs and we remove outliers when plotting.
48

---

Published as a conference paper at ICLR 2024
E LIMITATIONS
A practical problem we noticed in our simulated experiments is that sometimes the sparse model is harder to fit, i.e., its log- log-likelihood is worse than the dense model, even if we only consider the cases where the true model is in the model class being optimized (e.g., the sparsity of the model is at least as large as the sparsity of the ground truth model). We conjecture that this results from a harder loss landscape
we add more constraints to the model. We believe more careful investigation of the model and algorithm could be an interesting and important future work. For example, if we use more significantly overparameterized model, there are chances that the training of ILDD-an would become easier. Additionally, the addition of further loss terms could aid in the training of these models, such as, assuming access to some ground truth domain counterfactuals (e.g., the same patient received imaging at multiple hospitals) could be used to penalize our model when it changes latent variables which do not change under the ground truth counterfactuals.
In our experiments, we aimed to test the effects of breaking some of our assumptions (c.g., "what if our model is not strictly invertible"), and while our models still performed better in these cases, there are likely cases where the breaking of our assumptions can cause our models to fail to produce faithful counterfactuals. For example, in a case where there is a very large difference between domains and there is no sparsity in the domain shifts, then it is likely that the constraints constituted by our sparsity assumption will make the sparse models struggle to fit the observed distributions.
F EXPANDED RELATED WORK
Causal Representation Learning Causal Representation Learning (CRL) is a rapidly developing field that aims to discover the underlying causal mechanisms that drive observed patterns in data and learn representations of data that are causally informative (Schölkopf et al., 2021). This is in contrast to traditional representation learning, which does not consider the causal relationships between vari- ables. An extensive review can be found in Schölkopf et al. (2021). As this is a highly diffieult task, most works make assumptions on the problem structure, such as access to atomic interventions, the graph structure (e.g.. pure children assumptions), or model structure (e.g.. linearity) (Xie et al., 2023; Yang et al., 2022; Huang et al., 2022; Liu et al., 2022b; Xie et al., 2022: Chen et al., 2022; Squires et al., 2023; Zhang et al., 2023; Sturma et al., 2023; Jiang and Aragam, 2023; Liu et al., 2022a; Varici et al., 2023). Other works such as (Brehmer et al., 2022; Ahuja et al., 2022; Von Kügelgen et al., 2021) 1) assume a weakly-supervvied setting where one can train on counterfactual pairs (xr, .) during training. Lachapelle et al. (2023) address the identifiablity of a disentangled representation leverag- ing multiple sparse task-speeific linear predictors. In our work, we aim to maximize the practicality of our assumptions while still maintaining our theoretical goal of equivalent domain counterfactuals (as seen in Table 1).
Counterfactual Generation Counterfactual examples are answers to hypothetical queries such as "What would the outcome have been if we were in setting B instead of A?". A line of works focus on the identifiability of counterfactual queries (Nasr-Esfahane et al., 2023; Shah et al., 2022). For example, given knowledge of the ground-truth causal structure, Nasr-Esfahany et al. (2023) are able to recover the structural causal models up to equivalence. However, they do not consider the latent causal setting and they assume some prior knowledge of underlying causal structures such as the backdoor criterion. There is weaker form of counterfactual generation which does not use causal reasoning but instead uses generative models to generate counterfactuals (Nemirovsky et al., 2022; Zhu et al., 2017; Zhou et al., 2022; Choi et al., 2018: 8; Zhou et al., 2023; Kulinski and Inouye, 2023). These typically involve training a generative model which has a meaningful latent representation that can be intervened on to guide counterfactual generation (Ilse et al., 2020). As these works do not directly incorporate causal learning in their frameworks, we consider them out of scope for this paper. Another branch of works try to estimate causal effect without trying to learn the underlying causal structure, which typically assume all variables are observable(Louize et al., 2017).
Causal Discovery and nonlinear ICA Causal discovery focus on identifying the causal relation-
ships from observational data. Peters et al. (2016); Heinze-Deml et al. (2018) achieve this via the invariant mechanism between certain variable and and its direct causes. Most of these works do not assume the setting of latent variables. Similar to CRL, nonlinear ICA typically aims at finding
49

---

Published as a conference paper at ICLR 2024
the mixing function. For example, some works try to identify it with access to auxiliary variables (Hyvarinen et al., 2019; Khemakhem et al., 2020), by adding constraint on the mixing functions (Grescle et al., 2021; Moran et al., 2021) under specific scenario such as bivariate setting (Wu and Fukumizu, 2020). Zheng et al. (2022) relax the constraint of auxiliary variable and impose structure sparsity achieve identifiability result, where structure sparsity is less general than the mechanism sparsity discussed in our work. In contrast with CRL, most nonlinear ICA works do not consider latent variables that causally related.
50


Reviews:
[{"review_id": "ba0PFjSPk6", "reviewer": "Reviewer_ji9w", "summary": "The paper focuses on the problem of domain counterfactuals in the context of latent causal models and proposes a practical yet theoretically grounded approach to address this problem, aiming to improve the estimation of domain counterfactuals while making minimal assumptions about the true model and available data.  Experiments on extensive simulated and image-based data show the advantages of domain counterfactual estimation both theoretically and practically.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem setting, which revolves around domain counterfactual estimation, is a novel and highly intriguing area of research. I believe it has the potential to make a valuable contribution to the research community.\n\n---------\n\nMy main concerns are as follows:\n\n1) I am experiencing confusion regarding domain counterfactual equivalence. While I agree that it may not be necessary to fully identify the true latent causal model for domain counterfactual estimation, I would appreciate a more intuitive explanation of Theorem 1 and its implications. For instance, it would be helpful to know which specific aspects of the true latent causal model need to be identified to enable domain counterfactual estimation. This could include considerations like the identifiability of latent noise variables, the size of the intervention set, or any other relevant factors. \n\n2) How should we precisely define the size of the intervention set, denoted as 'k'? In discussions about interventions or the number of variables subject to intervention, there is usually a reference to a latent causal model where no interventions have occurred. How do we accurately establish this reference latent causal model?\n\n3) One of the main contributions, in my view, pertains to the definition of the canonical domain counterfactual model. However, this definition might appear somewhat stringent, particularly with its requirement that only the last variables be intervened. Even though it's surprising that any Invertible Latent Domain (ILD) model can be transformed into an equivalent canonical ILD, could you provide some real-world applications to illustrate and justify the relevance and utility of the canonical domain counterfactual model?\n\n4) It appears that in order to enhance domain counterfactual estimation, one needs to have prior knowledge of the intervention sparsity, denoted as 'k.' However, in practical scenarios, obtaining this information can be challenging. While the experiments do offer some insights and analysis regarding the mismatch of sparsity between generative and inference models, could you provide further justification or reasoning for the selection of the appropriate value for 'k'?\n\n\nI would be willing to increase my rating if these concerns are addressed.", "weaknesses": "See above", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "70M2CzQQbR", "reviewer": "Reviewer_4MQs", "summary": "UPDATE: I thank the authors for their reply, and am happy to see that the case I described was in fact covered by the theory. I have raised my score accordingly.\n\nThis paper considers the problem of making counterfactual predictions, in a multi-domain setting of latent causal models: In each domain, the latent variables form an SCM, the observed variables are computed from the latent ones by a deterministic function that is shared by all domains, and the counterfactual query asks, given observed data from one domain, what that data would have been had it been generated in another domain.", "questions": "One thing that is unclear to me about the problem setting is the following. Suppose $g = Id$, for some domains $f_d = Id$, and for others $f_d = -Id$. Then in all domains, the observed variables are independently distributed as standard normals. (The same would be true for variations where e.g. each domain flips the signs of a subset of the variables.) For a given counterfactual query, how can your method know whether to predict according to $x_{d'} = x_d$, or according to $x_{d'} = -x_d$? There is no signal in the available data to determine this. Is there an (implicit?) assumption somewhere to rule out cases like this?\n\nOther comments / questions:\n\n* Abstract, \"all non-intervened variables have non-intervened ancestors\": I suggest \"... have no intervened ancestors\". The current sentence can be interpreted as \"have some intervened ancestors\".\n\n* Paragraph before C1-4, \"Given our assumption ... distribution equivalence\": This is apparently without assumptions (1) and (2), which were stated in the preceding sentence. Please make clearer in the text that you're now considering assuming *only* (3).\n\n* In corollary 3, \"Prop. 8\" should refer to definition 8.\n\n* Section 3.3: \"establish on\" should be something else. These sentences should emphasize that you'll now be assuming continuity of $f$ and $g$ (so remove \"the\" and rewrite). In corollary 5, no requirement of continuity is currently stated, but I think it is needed. Also, just above the corollary, what do you mean with \"and not ill-defined\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly structured, guiding the reader through the theoretical setup.\n\nCompared to existing similar work, this paper provides a significant and novel contribution, both in terms of theoretical results and their application in an algorithm.", "weaknesses": "I have a concern that there might be an unstated assumption; see my main question below.\n\nThe text could benefit from more proofreading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mCy2p4mOWl", "reviewer": "Reviewer_jRXL", "summary": "**Post rebuttal update** I maintain my score (and uncertainty) but add my final reply to the author(s). Overall, I think both the original submission and the rebuttal revision are too clustered; the submission has many defs and results, and the revision adds more. \n\n*Generalized causal mechanisms*\n\nI still cannot understand. I have no problems with “reordering wlog”, however, if there exists a non-invertible subsystem $s$ in an invertible SCM (now Def 1), the reordering only changes the labels of the input nodes of $s$, not the *functional form* of $s$. Did I miss anything? Do you change the functional form of $s$ via $g$? But $g$ is invertible, I do not think we can change the invertibility of $s$ via $g$.\n\n*Rosenblatt transformation*\n\nI still hold my original view. To be clear, I use $J$ to denote the univariate CDFs with fixed conditional values. For a sample $\\mathcal{D}$, we have\n\n$$u_2=J_2(x_2):=F_2(x_2|x_1),$$\n\nbut, for another sample $\\mathcal{D}’$, we have\n\n$$u_2’=J_2’(x_2'):=F_2(x_2'|x_1’).$$\n\n$J_i \\neq J_i'$ except for $i=1$.\n\nOr, we say there does not exist a single set of invertible CDFs for all possible values of RV X.\n\n**End update**\n\nThis work considers the situation that different domains have the same causal graph but different structural causal functions; the “intervention” in this paper is the “*domain difference of causal mechanisms*”, and the “counterfactual” is the “*domain adaptation of observations*” under a pre-defined domain difference. More specifically, the “intervention” means changing a structural function (from one domain to another) but leaving any other things untouched, and the “counterfactual” means simply the new observation generated by the “intervention”.\n\nAn Invertible Latent Domain Causal Model (ILD) is a set of SCMs whose structural functions are invertible and autoregressive plus an Invertible Observation Function which connects the latent and observable. The equivalence class of ILDs is characterized, and it is proved that, for any CLDs, there exist equivalent “canonical forms” whose domain difference is described by the last k structural functions under the topological ordering. Based on this, the paper makes the point that “if we know the number k of different structural functions between the domains, then we can improve the performance”.\n\nThere are experimental supports for the effectiveness of the above idea.", "questions": "*Do we have any ideas on the identifiability of the model*? This is an important question because we discuss causality. Although the theory in the paper converts IASCMs on a set of domains into an easy-to-deal-with form that is the canonical ILD. There are no discussions on the identifiability of canonical ILDs, that is, when we really try to learn the canonical ILDs, can we identify the single eq class of canonical ILDs, which contains the true one? The practical application of the proposed idea depends critically on this question. \n\n*How can we understand if a practical dataset satisfies the theoretical setting*? Both high-level discussions and examples are highly desirable. For example, for MNIST datasets, how can we understand that the observable (image) is related to the latent by an injective mapping? Why the latent variables in different domains are related by invertible mappings? And in the end, what are the latent variables? Note that I do not require any precise claims on the causal structure (it is latent anyway), but even reasonable “guesses” would be very helpful.\n\nI think the 2nd equality of eq2 is also by definition?\n\nIn Def 3, point 3, I am not sure Gaussian exogenous noises are without loss of generality, any references and/or reasonings?\n\nShared observation function g should be explained right after Def 3.\n\nThe claim of “both counterfactually and distributionally equivalent” in Theorem 2 looks strange to me. Can’t we prove that counterfactual equivalence implies distributional equivalence?\n\nThere is a related work [1] that is worth mentioning; it also considers invertible causal mechanisms, shared function between domains, and transferability between domains. See Sec 2.3, 2.4 and 3.1 in that paper.\n\n[1] Wu, Pengzhou, and Kenji Fukumizu. \"Causal mosaic: Cause-effect inference via nonlinear ICA and ensemble method.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.\n\nMinor\n\nThere are two different defs of autoregressive functions, at the end of page 1 and in Def 1. Def 1 seems to be correct.\n\nIn Prop 2, generalize → generalized\n\nIn Corollary 3, you mentioned “Prop 8”, but I think it should be just Def 8.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "It is an interesting idea to see the difference between domains as invertible transformations.\n\nThe theoretical analysis seems serious (but it is quite impossible to check the proof in detail as a conference submission). \n\nIf a real-world situation satisfies the theoretical setting, then it is possible to largely reduce the complexity of finding domain differences.", "weaknesses": "*The idea of “intervention” and “counterfactual” in this paper is nonstandard and confusing*. As standard concepts in causal inference, an “intervention” roughly means setting a variable to a specific value but leaving any other things untouched, and “counterfactual” means imagining an intervention with the *past values* of exogenous variables untouched. In the above, *the structural functions are unchanged*. However, as indicated in my summary, the “intervention” and “counterfactual” in this paper refer to totally different things, in particular, the “counterfactual” does *not* consider the past values of exogenous variables which are the gist of “counterfactual”. Explanations regarding these are necessary, and I strongly suggest *not* using the terms “intervention” and “counterfactual” in this paper.\n\n*The “generalized causal mechanisms” in Prop 2 are only a subclass of “invertible SCMs”*. There is no formal definition of “invertible SCMs” in this paper; I think the definition should be like: writing the whole system of SCMs as X=f(\\epsilon) and function f is invertible. In the proof of Prop 2, it assumed that each $\\hat{f}^{(j)}$ is invertible. But this is not implied by the general concept/definition of “invertible SCMs” I mentioned above, because there is no guarantee that the *subsystem* involving only $z_{<j}$ is invertible; information from other variables might be needed to invert the whole system, and missing any piece of information might render the subsystems non-invertible. Hereafter, I name this class of models *“invertible autoregressive SCM (IASCM)“*.\n\n*The claim on the generality of the model class (Prop 1) seems problematic*. First, here the model class is IASCM. In the proof of Prop 1, the Rosenblatt transformation should be constructed for a fixed *sample point* of p(X), because each conditional CDF F_j should depend on fixed x_{<j}. Thus, we need a Rosenblatt transformation for each sample point, and we cannot construct the F_p and F_q for the whole distributions p(X) and q(\\epsilon). \n\nI have spent 3 hours to get around the confusion in the listed Weaknesses, and I can only skim the rest of the paper and ask some questions as below. *I will read the rebuttal and the revised paper again and update my review accordingly*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j5GuwBvQNU", "reviewer": "Reviewer_KSip", "summary": "This work focuses on domain counterfactuals: *What a sample would have looked like if it had been generated in a different environment*.\nIt provides a characterization of *domain counterfactually equivalent* models. They show that sparsity of domain interventions as an inductive bias can help reduce the search space and generate more accurate counterfactuals.", "questions": "* How does one set the number of intervention (k)?\n* If we don't have the right k, or if the form of the ILD is not precisely known, how off could be our counterfactual estimates? Do we have identifiability in this case?\n* How does this work compare with a prior line of work in using sparsity for identifiable representations, e.g., [Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning](https://proceedings.mlr.press/v202/lachapelle23a.html) or [On the Identifiability of Nonlinear ICA: Sparsity and Beyond](https://proceedings.neurips.cc/paper_files/paper/2022/hash/6801fa3fd290229efc490ee0cf1c5687-Abstract-Conference.html), just to name a few?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem that this work considers has practical significance, e.g., in combining datasets in domains where data collection is expensive.", "weaknesses": "* I think the writing and structure of the paper can be improved. A large fraction of space is used for many theorems, some of which are not that important. I suggest moving these theorems to the appendix and only keeping Theorem 2 in section 3.2. Furthermore, the simulated experiments look very contrived to me. The data generation mechanism used for creating data in these experiments is too simplistic. It consists of linear transformations followed by a leaky relu (I had to dig into a very long appendix to find this). The model also seems to have knowledge of the precise form of data generation mechanism (linear + relu), and only fits that form.\nTo me, the interesting experiments come in section 5.2. However, I can't find an explanation of how sparsity is enforced in these experiments.\nIn summary, I think authors can restructure the paper to provide more room for experiments in 5.2., and explain the methodology in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)