Conditional Generative Models are Sufficient to Sample from Any Causal  Effect Estimand
Authors: Md Musfiqur Rahman
Matt Jordan
Murat Kocaoglu
Venue: NeurIPS 2024 poster (2024)
Keywords: causal inference
causal graphs
deep generative models
Primary area: causal_inference
TL;DR: We propose a conditional generative model based approach to sample from any identifiable interventional or conditional interventional distribution given an arbitrary causal graph containing latent confounders.

Abstract: Causal inference from observational data plays critical role in many applications in trustworthy machine learning.
While sound and complete algorithms exist to compute causal effects, many of them assume access to conditional likelihoods,
 which is difficult to estimate for high-dimensional (particularly image) data. Researchers have alleviated this issue by simulating causal relations with neural models. However, when we have high-dimensional variables in the causal graph along with some unobserved confounders, no existing work can effectively sample from the un/conditional interventional distributions. In this work, we show how to sample from any identifiable interventional distribution given an arbitrary causal graph through a sequence of push-forward computations of conditional generative models, such as diffusion models. Our proposed algorithm follows the recursive steps of the existing likelihood-based identification algorithms to train a set of feed-forward models, and connect them in a specific way to sample from the desired distribution. We conduct experiments on a Colored MNIST dataset having both the treatment ($X$) and the target variables ($Y$) as images and sample from $P(y|do(x))$. Our algorithm also enables us to conduct a causal analysis to evaluate spurious correlations among input features of generative models pre-trained on the CelebA dataset. Finally, we generate high-dimensional interventional samples from the MIMIC-CXR dataset involving text and image variables.

Conditional Generative Models are Sufficient to Sample from Any Causal Effect Estimand
Md Musfiqur Rahman Purdue University
Matt Jordan University of Texas at Austin
Murat Kocaoglu Purdue University
Abstract
Causal inference from observational data plays critical role in many applications in trustworthy machine learning. While sound and complete algorithms exist to compute causal effects, many of them assume access to conditional likelihoods, which is difficult to estimate for high-dimensional (particularly image) data. Re- searchers have alleviated this issue by simulating causal relations with neural models. However, when have high-dimensional variables in the causal graph along with some unobserved confounders, no existing work can effectively sample from the un/conditional interventional distributions. In this work, we show how to sample from any identifiable interventional distribution given an arbitrary causal graph through sequence of puww-forward computations of conditional generative models, such as diffusion models. Our proposed algorithm follows the recursive steps of the existing likelihood-bart forward models, and connect them in a specific way to sample from the desired
identification algorithms train set of feed- distribution. We conduct experiments treatment (X) and the target variables (Y) as images and sample from P(y (z)))) Our algorithm also enables us conduct causal analysis evaluate spurious correlations among input features of generative models pre-trained on the CelebA dataset. Finally, we generate high-dimensional interventional samples from the MIMIC-CXR dataset involving text and image variables.
Colored MNIST dataset having both the
Introduction
Causal inference has recently attracted significant attention in machine learning (MI. due to its
application in fairness, invariant prediction, and explainability [60 62 49 Even though existing ML models show notable predictive performance by optimizing the likelihood of the training data, they are prone to failure when the covariate distribution changes in the test domain. Consider the medical scenario Fig. la with the causal order: the true data- tittta-gnrrating mechanisms. Suppose practitioner observes only X to make a high-level intermediate diagnosis S that contains sufficient information about the patient. The prescription report (t) is written only based on the diagnosis (thus X //4 R). Since data are collected from different hospitals locations (H), H acts as an unobserved common cause for both X and R, i.e.,
(R) representing
X ++ R (ex: correlation between x-ray artifacts and report writing style). The task is "x-ray to report" generation. One might train an MI model to directly learn a mapping f X' -> R. with maximum likelihood estimation (MLE) 10 mimicking the conditional distribution P(r(z). However, since I is an unobserved common cause between and R: I has some influence on P(r(z) [9]. Thus, the model is deployed in new location, its MLE-based prediction accuracy may drop since P(r[æ) shifts in that location. On the other hand, if we can remove the location bias X < R with an intervention the x-ray variable (do(x)), the x-ray report generation would be invariant to domain shifts. Thus, to obtain such generalization, we need to perform causal interventions in high- high-dimension.
"Equal contribution. Correspondence rahman89 purdue.edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Hospital Location (M)
Mx²
Mx
xx
Xray Scan (X)
Diagnosis (S)
Prescription Report (R)
X
Ms
5
MR
8
do(x 1)
Ma
Ms
P(a|do(a)) P(aaa)
P(rldo(z, s)) Σ Pla ) ((r()))))
P(rldo(z)) = Σ. P(lldo((.)))
E. P(a Pf P(a)) P(rl))(
Age (A)
Volume (B)
do(V v)
Ventricle Volume (V) MRI (1)
MAn
[4.
[A.8]
MAB
M₁
P(a. 5)
P(äidofe b)) P(
P(j)do(r)) Ens Pla. ai
Ens Pla, P(a, (i a.
(a) ML model failure scenario
(b) Train conditional models.
(c) Merge and sample ancestrally.
Figure 1: (Top: x- x-ray to report generation task) (a) do(X 2) removes the X ++ R bias and makes the generation of I domain invariant. (x)) is factorized into c-factors and (b) conditional models ((Mv, }&)) are trained for each factor (shown as boxes). (c) The intervened value X is propagated through the merged network and samples from the ((r]do(z)) are generated.
Structural causal models (SCM) 38 enable a data-driven approach to estimate interventional dis- tributions 43 33). Given the qualitative causal relations, summarized in causal graph, now have a complete understanding of which causal effectss/ueries (ex: P(r (r()) can be uniquely identified from the observational distribution and which require further assumptions experimental data 2 23 28
46 55). More precisely, if all conditional probability tables are available, sound and complete identification algorithms [51 46 can perform exact inference to estimate causal effects or 5 25 can sample from the interventional distribution, using a combination of marginalization and product operators applied those conditional distributions. However, such approaches struggle to deal with high-dimensional variables. In Fig. Ta we could intervene on x-ray X and estimate its effect on the report I as, (x)) = Σ. ((s|aa) Ex P(rl')' s)), i.e., functions of the observational distribution (X,X'': X': independent instances of the same variable). However, the second and third terms in this expression require marginalization over the "X-ray" variable. Exact Bayesian inference methods used for calculating conditional distributions are infcasible for high-dimensional variables since marginalization over their non -onaammmtt distributions generally intractable (
37
Deep generative models with variational inference methods approximate the intractable marginal- ization and can sample from such nigh-dimensional distributions 22
14] Recent works such as Xia et al. [59]. Chao et al. [1], Rahman and Kocaoglu 44 employ deep generative models match joint distribution of the system by learning the conditional generation of each variable from its causal parents. Nonetheless, it is highly non-trivial for these works to mimic any arbitrary causal model with high-dimensional variables, specially when there are unobserved confounders in the are correlated through unobserved hospital location. To learn the joint distribution P(x, the above approaches need to synchronously train their generative models. For that purpose, Xia et al. (59, Rahman and Kocaoglu 40 train two GAN networks concurrently by feeding the same prior noise. However, it is nontrivial to design a loss function for the joint distribution balancing multiple nigh-dimensional variables making it challenging for the discriminator to detect true/false sampled pairs. Thus, the high-dimensional intervention problem still requires a more effective approach.
causal model. Consider the X ++ R relation in Fig. La where R and X
In this paper, propose novel algorithm ID-GEN that can utilize any (conditional) generative models (such as GANs or diffusion models) to perform high-ddmensional interventional sampling in the presence of latent confounders. For this purpose, we resort the sound and complete identification algorithm [50] [6) and design our algorithm on top of its structure to sample from any identifiable causal query which may have an arbitrarily complex probabilistic expression (ex: Eq. [] More precisely, given a causal graph, training data, and causal query, our algorithm i) follows the recursive trace of the ID algorithm to factorize the query ii) trains set of conditional models for each factor, iii) connects them to build neural network called sampling network and generate interventional samples from this network. For example, to sample from P(r(do(x)) for the frontdoor graph in Fig. (top), we i) utilize ID to obtain the factors: ((s|d((z) and ((r(do(,, s)). ii) train conditional models (Ms}; ( (Mx; Mn) for the two factors (Fig. Tb), iii) merge all models based on input-output (Fig Tc). Sampling according to this network's topological order would produce samples

---

from P(rldo(x)). Similarly, for the backdoor graph (bottom), we train conditional models (MA. and {Mr} to learn P(a. b) and P(ilu, ju a, b) respectively and merge them to sample from P( (d(e)) v)).
To the best of our knowledge, we are the first to show that conditional generative/feedforwar models are sufficient to sample from any identifiable causal effect estimand. Our contributions are as follows:
We propose recursive algorithm called ID-GEN that trains set of conditional generative models on observational data to sample from high-dimensional interventional distributions. SCMs.
We are the first to use diffusion models conditional models for
We show that ID-GEN is sound and complete, establishing that conditional generative mod- els are sufficient to sample from any identifiable interventional and conditional interventional query. The latter type are especially challenging for existing GAN-based causal models.
We demonstrate ID-GEN's performance on three datasets containing image and text vari- ables. First, we perform image intervention with diffusion models for the Colored-MNIST experiment. Next, we show our application in trustworthy AI through quantifying spurious correlations in pre-trained models for the CelebA dataset. Finally, we make the report to X-ray generation task interpretable and domain invariant based on the MIMIC-XRR dataset.
2 Background
Structural causal model, (SCM) 36 is tuple M (G = (V.8).N.U,F.P()). V (V1...,,V, 3 is set of observed variables in the system. N is a set of independent exogenous random variables where N, E N affects Vú and 2/ is a set of unobserved confounders each affecting any two observed variables (for > 2 check Appendix C.7 This refers to the semi-Markovial causal model. set of deterministic functions
fv, determines the value of each variable Vú from other observed and unobserved variables as V. (Pa,, N., Us.) ), where Pa; C V (parents), N₂ E N (randomness) and Us. C L (latent confounders) for some S. P()) a product probability distribution over N' and U/ and projects a joint distribution Pv over the set of actions V representing their likelihood.
An SCM M, induces an acyelic directed mixed graph (ADMG) G = (V, E") containing nodes for each variable Vú E V. For each V = (Pa,, N₂, Us.). Pa, C V, we add an edge V, + Vi E E, VV, E Pa.. Thus, (V; becomes the parent nodes in G. G' has bi-directed edge, V; + V, E between V; and V, if and only if they share latent confounder. If pathhhh V; + .... + Vi exists, then V; is an ancestor of V, i.e., V= An(V))g. An intervention do(z) replaces the structural function fz with X = x and in other structural functions where occurs. The distribution induced on the observed variables V after such an intervention is represented as (v) or P))))o((:). Graphically, it is represented by Gx where incoming edges to X are removed (marked red). With a slight abuse of notation, we will use P(y) for both the numerical value IP = y) and the probability distribution [P()) depending on the context. An example for the latter is: "Let Y be sampled from P(y)". " Also. Pz Pv)) refers to the interventional distribution for all z. y. Given an ADMG G. a maximal subset of nodes where any two nodes are connected by bidirected paths is called e-component C(G). For any S E C(G), P(S dV \ S)) is called c-factor. We assume that we have access to the ADMG through some causal structure learning algorithm and expert knowledge. Classifier-free diffusion guidance (20 Let (v.c) ~ P(v, c) be the data distribution and z = [zxlA= E [Amin: \maz]] for Amin < Xmaz E TR. We corrupt the data as 2, OxX + axe and optimize the denoising model by taking the gradient step on Vellca c) cl12. Given that variables V are connected as directed acyclic graph and we have diffusion models trained to learn the distributions P(v))p((o:)) we can perform ancestral sampling from the joint distribution, ( ((u; ((vslpa(v.) (v:)) by making one pass through each model in the topological order while sampling from the conditional distributions (6. We choose classifier-fres diffusion as our conditional model, but the choice changes based on the application. We use My(c) and a square node as notation.
3 ID-GEN: generative model-based interventional sampling
Given a causal graph G, dataset D ~ P(v), our objective is to generate high-dimensional interven- tional samples from a query [ly])o(x)) or a conditional query P(y]do(x), 2). ID-GEN builds upon the recursive structure of the identification algorithm [46) to train necessary conditional models. Thus, we first discuss its connection with and show the challenges it faces if deployed for sampling.
3

---

3.1 Identification algorithm (ID) and challenges with high-dimensional sampling
Shpitser and Pearl 46 propose a recursive algorithm (Algorithm (6) for estimating an interventional distribution (x(y) given access to all probability tables. At any recursion level, it enters one of its four recursive steps: 2, 3, 4. 7 and three base case steps: 1, 5, 6. Below, we discuss them in detail.
Step occurs when the intervention set X is empty in (y()). The effect of X = 0 on Y is its marginal P(y) which is returned as output. Step 2 checks if there exists any non-ancestor variable of Y in the intervention set X. Such variables in the graph do not have any causal effect on Y. Thus, is safe drop them. In Step 3, it searches for a set W in G. which does not effect Y assuming that X has already been intervened on. Thus, it can include W as an additional intervention set: X X U W. An intervention on W implies deleting its incoming edges, which simplifies the problem in the future. Step 4 is the most important line and is executed when there are multiple c-components in the subgraph G X. It factorizes (decomposes) the problem of estimating (y()) into estimating c-factors (subproblems) and performs recursive calls for each c-factor. Base case Step 5 returns fail for non-identifiable queries. Base case Step asserts that when X does not have a bi-directed edge with the rest of the nodes in S and S consists of single c-component, intervening on
X is equivalent to conditioning on X. Thus, ID can now solve Px(y) as Evty ILIyE P( | ))= and return as output. Step 7 occurs when the variables in X can be partitioned into two sets: one having bi-directed edges to other variables (5") in the graph and one (defined as X2) with no bi-directed edges to S In that case, evaluating Px(y) (y) from P(V) is equivalent to first obtaining P'(V) = Pxx (V) and then evaluating xxxxx, (Y) from P'(V). Hence, Pxz (V) is first calculated as HAMEE P(V ( inn,, Ur (fi-1) \ (S") and then passed to the next recursive call for do(x xx) to be applied. One major issue of ID is that it requires probability tables and thus cannot be applied for -igh-dimensoonal sampling. Suppose we naively design an algorithm that follows ID's recursive steps and trains generative model for every factor it encounters and samples from it. This algorithm would not know which of these factors to learn and sample first, leading to a deadlock as shown in Ex C.1 ID-GEN solves such issue by avoiding direct sampling and building a sampling network.
Definition 3.1 (Sampling network, 70.. A collection of feedforward models {My; Jv. for a set of variables V {Vi is said to form a sampling network, H. if the directed graph obtained by connecting each Mv; to M MAn(Vi) via incoming edges according to some conditional distribution, is acyclic. Two sampling networks H., H. can be merged into a larger network H.
3.2 Recursive training of ID-GEN and interventional sampling
Similar to ID's recursive structure, ID-GEN has 7 steps (Algorithm TT. However, to deal with high- dimensional variables, we call three new functions: i) Algorithm
(.) inside steps 1 and 6 where we train diffusion models or other conditional models to learn conditional distributions, ii) Algorithm 3 MergeNetwork (.) inside step merge the conditional models, and iii) Algorithm Aa [Update inside step 7 to train models that can apply part of the interventions and update the training dataset for next recursive calls. We initiate with ID-GEN (Y, X, G, D, X
0, á = G). Along with the given inputs Y, X, G, D, ID-GEN maintains two extra parameters X, Ô to keep track of the interventions performed. During the top-down phase, ID-GEN updates its parameters: by i) removing interventions from the intervention set X. and ii) updating the training dataset D, X and the causal graph G, G according to the interventions. At any level of recursion, an ID-GEN call returns a sampling network H (DAG a set of trained models) trained on the dataset D to learn conditional distributions according to X, G, G. After the recursion ends, we can generate samples from (V),,, Y C V, ancestral sampling on H. See recursion tree in Appendix C.4
Base Case: Step 1: ID-GEN enters step the intervention X is empty. For X 0, we have, Py()) = P(y) = -Evy P(v)= = Ev.y IIv,v P(uy ((f)) which is suitable for ances- tral sampling. To train models that can collectively sample from this distribution, we call Al- gorithm Conditionall (.). Here, we train each model My, VV, V using V (-(--)) (i.e., variables that are located earlier in the topological order тT) as inputs to match ((x;(u111)) Note that X contains the values that were intervened in previous recursion levels and & is the graph at the current level that contains X with its incoming edges cut. Since we want our conditional models to generate samples consistent with the values of X, we consider the topologiaal order of & while using

---

Vii-1) inputs so that X are also fed as input while training. After training, we connect the trained models according to their input-outputs to build a sampling network H. and return it (Alg 2 lines 6 Note that when all variables in Y are low dimensional, we can also learn single model M to sample P(Y). However, for high-dimensional variables, matching such joint distributions is non-trivia [40]
Step 2& & 3: We follow the same steps of the ID algorithm as discussed in Section 3.
Step 4 and Merge sampling Networks: Our goal is to train models that can sample from Px(y) which unfortunately is not straightfor- ward. This step allows us to decompose our problem into sub-problems and we can train models to sample from the c-factors of (x()'ss factorization. The next challenge to connect these models consistently to sample from Px (y)) More precisely, if we remove X from G and the graph splits into multiple c-components (vari- ables in each component connected with <+) (Alg [inee] [1]. we can apply e-component fac-
Wi
X
Mw,
Mx
do(a))
Mws
Mxx
Wi
W/2
Mr
W
Mas
Y
Mdy
Mis
Figure 2: +>:Unobserved. Left blue samples
from Pa. (w1,y) y) = P(wi (x) P(ylar, wT, 102) Right blue samples from Pa,uy (w2) Ex P(x') P(wela', w1). Joint network samples from Pa(
torization (Lemma D.7 (51) to factorize Px(y) as Evi(yux) (r)e( Pvi.. Nte,(en (sn) where each
{SaJa is the c-factor corresponding to each c-component. To obtain trained models for each of these c-factors, we perform the next recursive calls: ID-GEN (Y - St, X = =V V G. D. X, G). When these recursive calls return a sampling network H; for each Pvi., (s:), we can wire them based on their input-output to build single sampling network H. According to Theorem D.2T and D.22 H now can sample from Py()).
We call Algorithm 3 MergeNetworkk.) to connect all sampling networks { (H..r.. Here, each H. is of trained conditional models {M ,, connected to each other as a DAG. If sampling network Hi contains an empty node Mr, 0 without any conditional model and some other sampling network Hr generates this variable Vi with its node Mv, i.e., Vi = Vv. then we combine My, and My, into the same node build connection between Hi and H. (lines 3 6) Intuitively, due to the c-factorization at this step, the variables intervened in one sampling network might be generated from models in another network. We connect two networks to continue the ancestral sampling sequence. Fig. 2 shows an example of this step where Pa (y) is factorized into c-factors as Ps (v) = E 302
Pa P2., mn (W2) Pa,wo (WI,y). For the e-component {W1, YJ. ID-GEN first obtains
Pa Pa.wng (W), y) = P(wi (x) P(yla, W1, w2), and trains conditional models Mw, and My for these conditional distributions. Similarly, for (W2), we have Pz. (w2) = Σx' P(x')P(ws [W]]aa, l wi) and we train Mx and Mw,. Finally, we merge these networks based on inputs-outpur to build a single sampling network and perform ancestral sampling it to sample from Pz Pz(),, Vz.
Base Case: Step 5: We follow the step 5 of the ID algorithm as discussed in Section 3.1
Base Case: Step 6: We enter Step 6 if G \ X is a single c-component S, and X is located outside the c-component. This situation allows us to replace the intervention X by conditioning on X: Py(y) = Eny IIves P(us luf ")) This step is similar to step I, except that now we have a non-empty intervention set, i.e., X + 0. Here, we consider the topological order of & and (((-1) contains both X and X. call A Condiiioonaalln
(.) which trains multiple conditional models to learn the above distribution. More precisely, we utilize classifier-free diffusion guidance for conditional training of each My; by taking the gradient step on Vellea(2i, UT (8-1)) el|2. Here,
z the noisy version of VG time step during the forward process and off (1-1) is the condition (see Background). Finally, we connect the input-output of these diffusion models according to the topological order to build a sampling network and return it as output. Note that for any specific conditional distribution, if we have access to pre-trained models that can sample from it, we can directly plug in the network instead of training it from scratch (motivated from 440).
Step 7: Here, ID-GEN partitions X into two sets: one is applied in the current step to update the training dataset and other parameters, and the other is kept for future steps. It performs this step if i) G X is single c-component S and ii) S' is sub-graph of a larger e-component S' in the whole graph G. i.e. (S C(G X)) C (S' é C(G)). For example, in Fig. 3, for P (g)) we have In this step, we call Algorithm 4 Update(.)
S = G (W1, W2 (VV,
5

---

Algorithm 1 ID-GEN (Y, (Y,X,C,D,X,C)
Algorithm Conditioonaldes(.X...22
I: Input:
target Y, to he intervened X, intervened
1: for cach V2 E {X U } do
variables at step 7s X, causal graph G without X.
2: Add node (V:, 0) to H. (Initialized 1/ = 0)
causal graph G with X having no parents, training data D [G] sampled from observed distribution P(V).
3: for each V; E Y in the topologiaa order Te do
4: Let My; he model trained DIV, 1/4
such that My; (V= ~ P(uslut
2: Output: A sampling network trained models.
5: Add node (V,, Mv,) to H
3: if x = @ then (Step 1]
6: Add edge V -> Vi to H. for all V, E v
4: Return Conditional Y X = 0.C,D.8,6)
7:
Return H.
5: if V V\ \An(Y)o # 9 then (Step 2)
6: Return ID-GEN(Y, X A (Y), Gan(Y) X,
Algorithm MergeNetwork( (H,, (Hr)v)
GANLY), D' = ([An(Y)
1: Input: Set of sampling networks [H.Jvv-
7: Let W = (V) x) \ An )ax SStpp 3/
2: Output: connected DAG sampling network H.
8: W * then
3: for Hi E {Hi]ve do
9:
Return ID-GEN(Y, X = X G. X, D)
4:
for My, E Hi do
10: if C(G \ X)
(St..... Sa) then (Step
5: if Mv, = 0 and 3Mv, E Hr, Vr such that
11:
for each S. C(G) X) (S S do
Vi = VA and Mv. # 0 then
12:
H,=ID-GEN(S, X V S, G,D)
6:
My, = Mv.
13:
Return MergeNetworz
7: Return H [Hi)v: (All H. are connected.)
14:
if if X) = } then
15:
if C(G) = { then (Step 5}
Algorithm 4 Update (S' X. G,C,D,X,6' 6)
16:
throw FAIL
1: Xx = X
17:
i E C(G)) then (Step 6)
2: H = ConditionallNa(5 ,x,, G. D, G)
18:
Return Conditional GMa X, G, G, G)
19:
(ES' such that S c S' e C(G) then (S7)
3: D' ~ H(Xx, X): X= XUXz
4: Return Y,X Gs, D'[x, S"), X, (s)x)
20:
Return ID-GEN(Update(5" X.G.D.X,GG)
which utilizes the larger c-component S' to partition the intervention set X into one set contained
within 5", i.e., nnS' and another set not contained in S" i.e., X = X x\S S" Evaluating (y)
from P(v) is equivalent to evaluating P Xxn (y) from P'(v) where p' P'(v) == Px. (v) is the joint
distribution. Hence, we first perform do(Xz) ) to update the dataset as D'. Next, we shift our goal of
sampling from (y( in G' with training dataset D P(V) to sampling from Pano (y) in G X)
with training data D' ~ Px. (v) in the next recursive calls. To generate dataset D' ~ Px. (v), we call Conditi onao GMs (.) and use the returned network to sample D' (lines 2
Note that given access to probability tables, the ID can use any specific value X== xx to calculate Px, (v) to get the correct estimation of (y) (Verma constraint (54 46)). In our case, if we use a specific value x; to sample the training dataset D' ~ Px, (v), the models trained on this dataset in subsequent recursive steps will also depend on x.- However, during ancestral sampling in the returned network, a different value X = x² might come from other c-components (ex: My (W2..) in Fig. 3) Thus, to make our trained models suitable for any values, we pick X from a uniform distribution or from P(Xz) and generate D' accordingly. We save X2 in X. its values in D X, S and in & G () with incoming edges removed, to be considered during training in the next recursive
calls. Whenever ID-GEN visits Step again, X will be applied along with the new Xz. Finally, recursive call is performed with these updated parameters (line 4 which will return a network trained on dataset D' ~ Px. (v). It can sample from Pxns: (y) and equivalently from the original Px
Algorithm simulation: We apply ID-GEN to sample from P.,, (v) for the causal graph G in Fig. Since G (Wi) } has three c-components (W2), (X). (Y), we first call (i) step 4 of ID-GEN. Pa,, (V)
factorized as: Σz,ur Pw..z.g/ (22) Pwj w0 (Ir) Pw,, (y). Thus, step 4 will return the sampling
networks {H.., Hx. Hy) that can sample from each of these factors. Here, we focus only on Hy.
ID-GEN reaches (ii) step 7 for the query: Pu,, h.. w..( since we have S G \ (W1, X)-
(v) in G, with observational training dataset is equivalent to sampling from Pa,w, (y) in G = Gw with do(W)) interventional data. With W₂ P(wz), we generate D' Ping (v) by calling step 6 (base case). We pass D' as the dataset parameter for the next recursive call. This step implies that if the recursive call returns a network that is trained on D' ~ Paz (v) and can sample from Pa w1 (v) it can also be used to sample
(Y), = (W1.XC Y and C S'. Here, sampling from Pa.er,wz

---

Mar;
May
do(W; = mig)
Step
(ii)) Step 2
Mas
W2
Mx
My
Mws
Mx
My
W.
W2 X Y
Step
(iv) Step
Uni
Figure (Left: top-down) Pwy (v) is factorized into Pyy.),( Pw,, ww.g (2.g(() and Pwi, was (v) (Step 4). Steps 7, 2, 6 is shown for Pm (v) only. (Right: bottom-up) we combine the sampling networks of each e-factor. For any do(W) = wi ), we use to get samples from Pwy (v).
from Pw., ar,, (v). Next, since W1 & An(Y) G. (iii) step 2, drop W1 from all parameters before the next recursive call. We are at the base case (iv) step with G: W2 - ---4+ + Thus, we train on D'2,, X, Y] that can sample from P(y] w2) This would be returned as Hy at step 4 (Fig. 3 green). Similarly, we can obtain sampling network Hw, and Hx to sample from Pwn, .r, (22) = P (1))(1) and P Pur,wa.y (a) = Ew, P(x w2) P(w() (Fig. blue).
conditional model 2W, 2, X)
We connect these networks and perform ancestral sampling with fixed w1 for do [W w1).
(Un) conditiooll sampling and complexity: ID-GEN returns sampling network H when recursion ends. For unconditional query (y)),, we fix X joint samples. We pick the values in these joint samples (equivalent to marginalization in ID) and report as interventional samples. For conditional query Px(y (2), ID-GEN uses the sampling network to first generate samples D(X, Z, Y] ~ Px and then train new conditional model My(X, Z. on D sample from Y (y(y 5IC-5DDD-GEN
in H. and perform ancestral sampling generate
The sampling network has
O(IAn(Y) YY)l) number of models and requires ([IAm(Y Yel) time to sample from it. Please, see our complexity details in Appendix C.5 Appendix C.6
Theorem 3.2. Under Assumptions: i) the SCM is semi- Markovian, ii) have access to the ADMG, iii) P(V) is strictly positive and iv) trained generative models sample from correct distributions, ID-GEN and IDC-GEN sound and complete to sample from any identifiable Pa(y) and Pa Pz(y2).
Note that although existing work can sample from (low-dimensional) distributions, Theorem 3.2 makes ID-GEN, to our knowledge, the first method to use only feed-forward models to provably sample from identifiable high-dimensional interventional distributions.
Experiments
To illustrate ID-GEN's capabilities with high-dimensional image and text variables, we evaluate it on semi- semi-synthetic: Colored MNIST and real-world: CelebA and MIMIC-CXR datasets. We provide additional details in Appendix Codes are available at
4.1 ID-GEN performance on napKin-MNIST dataset and baseline comparison
We consider a semi- cemi-synthetic Colored-MNIST dataset for the napkin graph 39 in Fig. 4 with image variables W1, X, Y and paired discrete variable W,. Here, X and Y inherit the same digit value image W1 which is propagated through discrete W2.d [0- - 9]. X is either red or green which is also inherited from Wi through discrete W2.c, i.e., Wi.color {r. y, b, y, m. m,,y) cy) - W2.c: (0, 1) -> X.color (r.g) Unobserved Ucolor makes W, and Y correlated with the same color, and unobserved Uthickness makes W, and X correlated with the same thickness. Even though image (takes r and g) is a direct ancestor of image Y (takes all colors), Y only inherits the digit property from X but correlates the color property with image Wi. This color correlation between W1 and Y is created by Ucolor. All mechanisms include 10% noise. Our target is to sample from Pa Pz()).
Setup:
Training and evaluation: We follow ID-GEN steps: [3, 7, 6]. Step 3 implies Pz(v) Pa, an w. (v). ie.. intervention set =(X, W1. W2). Step suggests to generate do(W2) interventional dataset D'[W,, W2. X, ~ P (W1. 3) = P(w1) P(x, y W1, W2). To obtain D', we i) sample W1 ~ P(w1), and ii) train a conditional diffusion model to sampling from P(x, y (/ w2) with arbitrary W2 values. Next, Step 2 drops non-ancestor W, and Step 6 trains diffusion model My(æ, w2) on the new dataset D' to sample from P'(y|æ, w2). My (x. w2) is returned as output that can sample from P. (w). We compare performance with three baselines: i) classifier-fre diffusion
7

---

Wi
Cond
FID
50.83
X
3
PLY ())
3
3
3
3
3
Color
True
Pa(y)
Cond
DCK
Ours
Uthickooss
DCH
66.37
3
3
9
3
5
3
R
0.17
0.17
0.24
0.13
W2
NCM
61.11
3
3
3
3
3
3
3
3
5
G
0.17
0.45
0.27
0.24
Uoolor
Ours
25.66
3
3
3
3
3
3
3
3
3
3
3
3
0.17
0.15
0.28
0.21
Yw
0.17
0.19
0.05
Cond
41.35
5
5
5
5
Mg
0.17
0.02
0.07
6.11
DCM
60.61
5 5 5 5
0.17
0.10
0.09
0.18
Ours
NCM
71.50
22.67
5
5
5
TVD
0
0.54
0.58
0.25
Figure 4: (Left:) Causal graph with color and thickness as unobserved. (Center:) FID scores (lower the better) of each algorithm and images generated from them. (Right:) Likelihood calculated from the Pa (y) images generated by each algorithm. We closely reflect the true Px(x) with low TVD.
model that samples from the (Cona)itional distribution P(r|ær) ii) the DCM algorithm [ that uses diffusion models to samples from P. (v()) but without confounders, and iii) the NCM algorithm [58] that uses GANs and considers confounders. We performed do(x) intervention with two images, digit 3 and ii) digit 5, both colored red. In Fig. 4 we show interventional samples for each method alongside their FID scores representing the image quality (lower:hetter). The (Cond) model (row 1. 5), DCM (row 2, 6) and our algorithm (row 5, 8) all generate good quality images of digit 3 and digit 5 with a specific color. However, the NCM algorithm (row 3, 7) generates images with blended colors (such as green + red). We observe that ID-GEN achieves the lowest FID scores (25.66 66.66 and 22.67), showing the ability to generate high-quality images consistent with the dataset. Whereas, Cond and DCM generate almost the same structure for all digits lacking variety, which explains their high FID. Note that do(x:) removes the color bias between X and Y along the backdoor path. Thus, interventional samples should show all colors with uniform probability. Since Cond and DCM can not deal with confounders they show bias towards R, G. B colors of Y for red X. ID-GEN removes such bias and balances different colors (Fig. D For a more rigorous evaluation, we use the effectiveness metric proposed in 34 and employ a classifier to map all generated images to discrete analogues Digit, Color, Thickness) and compute exact likelihoods. We compare them with our ground truth P(Y .cooor do(x)) (uniform) and display these results for the color attribute in Fig. 4 right). We emulate the interventional distribution more closely with a low total variation distance: 0.25 compared to the baselines Cond (0.54) and DCM (0.58). We skip classifying colors of NCM as they are blended.
4.2 Evaluating CelebA image translation models with ID-GEN
Setup: We apply ID-GEN to evaluate multidomain image translation of some existing generative mod- els (ex: Male to Female domain translation). We examine whether they apply causal changes in (fa- cial) attributes or add unnecessary changes due to the spurious correlations among different attributes they picked up in the training data. Our applica- tion is motivated by Goyal et al. [15], who generate counterfactual images to explain pre-trained classi-
Young(1) (1)
M4
My
Fa
Male
P2
Mele B
Ms
EGSDE: PlAldo[Male = OJ
Attribute sppearance (96)
69.20%
StarGAN: PIA]do/Male 0)
StarGAN: PIAlToung, dolMaie
fier while examine pre- pre-trained image generative models. We employ two generative models that are trained on CelebA dataset (29): i) StarGAN [12] and
20.
40.3%
41.43%
37.82%g5 24%
33.1%, 3p 40.%
30
ii) EGSDE (64) (an approach that utilizes energy- guided stochastic differential equations). We assume
24.764
the graph in Fig. rig.5] where the original image I₁ causes its own attributes Male and Young. We consider an unobserved confounder between them, as in the dataset, men are more likely be old (correlation coeff=0.42, 44) and classifier might have some bias toward predicting young-mat male images as old- male. These attributes along with the original image are used to generate a translated image 12. Next, P1 and P2 are 40 CelebA attributes of 71 and Iz. A is the difference between P1, P2, i.e., the additional attributes (ex: makeup) that gets added to 12 but are absent in I during translation. We estimate P (A), i.e., the causal effect of changing the domain from Male to Female on the appearance of a new attribute.
Weeeinng Lipslek
Henry Mataup
arched Frabroes
face
A Atterrooe
Clentionnt
Eaging
Baaid
Hong
Figure 5: i) Graph and sampling network for Parate (I2). ii) For both causal and non-causal attributes, EGSDE shows high correlation.
8

---

discuss how Luck zffuaiin And Lung Opacity
Laft Ellasion And Atelectasts
Brseline: Piappt
Lett Eftusion
Baseline Causal graph
True
.ef Naural Fuld
Causal graph
Hospital Locaign (i))
Prompu/ Report(R)
Prompt/
Reporti R)
Pneumonia
Pleural
(N)
Fiffusion F
Picumonia and un
Loft EMtucion And Lung Cpac
Let: Effusion A Atstaclasts
Healthy
Left Paunal PLid Pertite
Atelectasis (1)
Xray Image (X)
Lung
Xray
Opacity (L.
Image (X)
Figure Left: Baseline vs our causal graph. Right: images for specific prompt w/ and w/o pneumonia.
Inferred attributes are shown with their likelihood. Blue indicates changes compared to healthy.
Training and Evaluation: For (A), We first generate I2 ~ (Male=0(I2)) and then use a classifier I2- ID-GEN vists steps: 4,6 and factorizes as PMale=u(I2) = Jy., P(I Male O,Y,T, [1) +P(Y, To sample from these factors, it first trains My, M₁ Next, instead of training Mr, for P(2 Male 0, Y,!1), we plug in the model that want to evaluate (StarGAN EGSDE) as M,,, We connect these models to build the sampling network in Fig. 5 We can now perform ancestral sampling in the network with Male
0 and generate samples of 1.. Next, use classifier to obtain 40 attributes of I and In as P1 and Pz. Finally, we obtain the added attributes, A by comparing P1 and P2 and report the proportion the estimate of (Male=o(A).. Similarly, also estimate the conditional query, PMale=o(A|{)) using StarGAN with Y = fixed.
In Fig. 5 we show top 9 most appeared attributes. We observe that EGSDE introduces both causal (ex: WearingL.ipstick to 82%, HcavyMakeup to 69.28% of all images) and non-causal attributes (ex: (ex.Attractive to 37.61 % and Young to 24.76%) with high probability. The model might assign high-probabilityy to non-causal attributes because they were spuriously correlated in the CelebA training dataset (ex: sex and age). On the other hand, StarGAN and conditional StarGAN introduce new attributes with a low probability (S 30%) even they are causal which is also not preferred.
These evaluations enabled by ID-GEN help us understand the fairness bias in the prediction of image translation models. Finally, for the conditional query P StarGAN translates 54.68% of all images Young, Female which is consistent. In Appendix F3 baselines deal poorly with these queries.
4.3 Invariant prediction with foundation models for chest X-ray generation
Setup: In this section, demonstrate ID-GEN's utility intervene with text variable and generate images from the corresponding interventional distribution. Due to the lack of high-quality, annotated
medical imaging datasets, the task of generating X-ray images (X) from the prescription reports
(R) is recently being investigated. Consider the report-to-Xraz generation task shown in Fig. 6 The shown causal graph is designed based on the MIMIC-CXR dataset (4] (see Appendix F.4 for details). Existing works [10] solve this task by using uisionllanuuag foundation models to directly generate CXR images conditioned
prompts (or report): P(X (F)(F........ left). These models
can generate images satisfying the prompt attributes (ex: effusion). However, they ignore the status of other attributes caused by prompt attributes -attiiitn-ttllecaaaiss that might implicitly contribute generate the image. Moreover, these models are trained on multiple datasets collected from different locations, which could introduce different types of bias (9. For example, in many scenarios, patients with severe pneumonia are transferred to a different hospital and receives critical reports. Thus, there exists potential confounding bias between report (R) writing style and pneumonia prevalence (N) in a specific location (H) i.e. R + H - N. The direct R + X prediction of the above mentioned conditional models might absorb the backdoor bias: t- H + N ... + X and get affected if P(R, N) shifts in a new location. We aim to make the image generation task of such models, interpretable and invariant the test domain. Therefore, we consider the same input (report variable) and output (X-ray image) as such these models. For this purpose, ID-GEN performs do( It) intervention remove the backdoor bias and infer attributes from the interventional distribution. Now these attributes used to generate the final images (motivated by [48 27]).

---

Training and Evaluation: We indicate attribute extraction with outgoing edges from report R and image generation with incoming edges X-ray X. We assume pneumonia infection N affects other attributes. For our target query Pn(N, (N, E, A, X), ID-GEN first visits step 4 and factorizes it as PR((N F,, A, Tu, X) = P(N) (F) PR.E) N.,,NE (.NNNE((PPP))) step 6 for each, converts them to conditional distributions and trains corresponding models or use pre-trained ones for My, Mg, MA. M,, Mx. We utilize an LLM labeler [66] that can only extract attributes E, A, L from the report It (thus R + N). If R explicitly mentions E, A. L then the models ME, MA, Mr. output 1. Otherwise, they probabilistically infer their output conditioned on parent attributes in the causal graph. The models we train are able to converge to their conditional distribution with a low total variation distance (Re 0.05). We utilize a stable diffusion based model [00] as Mx to sample X ~ L. (X). After ID-GEN connects all models, we intervene with prompt, infer all attributes along with their empirical distribution and generate corresponding X-rays. In Fig. 6 (right), we show our results for the prompt intervention do R =left pleural fluid persists). For better visualization, we also condition on N 0 and N = 1 separately. First, the LLM labeler extracts effusion (E = 1) from the prompt. Next, ancestral sampling with F: = 1 in the sampling network provides us with the top 3 most probable attribute combination (N = 0/1, E' 1, A, L) and their corresponding images. In Fig. 6 we provide (i) a healthy image, (ii) image generated by baseline (10), and (iii) (v) images generated with ID-GEN and their attributes with empirical probabilities. ID-GEN makes these predictions interpretable while facilitating them to be invariant with domain shifts.
( ID-GEN goes to
5 Related work
Shalit et al. [43, Louizos et al. (30] Zhang et al. 663, Vo et al. (55) propose novel approaches to solve the causal effect estimation problem using variational autoencoders. Their proposed solution and theoretical guarantees are tailored for specific causal graphs containing treatment, effect, and covariates (or observed proxy variables), where they can apply the backdoor adjustment formula. Sanchez and Tsaftaris [42] employ DDPMs [22] to generate high- high-dimensional interventional samples, but only for bivariate models. Kocaoglu et al. (26) perform adversarial training on a collection of conditional generative models following the topological order to sample from interventional distributions. Pawlowski et al. (35) employ a conditional normalizing flow-based approach to offer high-dimensional interventional sampling as part of their solution. Chao et al. [11] designs diffusion- based causal models for arbitrary causal graphs with classifier-free guidance 21. However, these works have limited applications due to their strong assumption of no latent confounders in the system.
Xia et al. 588 propose a similar training process as Kocaoglu et al. [26] the presence of hidden confounders. They show explicit connections with the Causal Hierarchy Theorem [3] and formalize the identification problem with neural models. However, it is difficult to match an arbitrary high- dimensional distribution, and their joint GAN training approach may suffer from convergence issues. Rahman and Kocaoglu [40) utilizes a modular algorithm to relax the joint training restriction for specific structures, but might face the convergence issue when the number of high-dimensional variables increases. Note that these methods are not suitable for efficient conditional sampling. Jung et al. 25 convert the expression returned by identification algorithm (51) into form where can be computed through a re-weighting function to allow sample-efficient estimation. Similarly Bhattacharyya et al. 4 5 utilize bounded number of samples from the observational distribution to construct an interventional sampler. However, computing these reweighting functions/conniitional distributions from data is still highly nontrivial with -igeeniiensional variables. Zečević et al. [61] models each probabilistic term in the expression obtained for ('y](oo()) with (i)SPNs. However, they require access the interventional data and do not address identification from only observations. Wang and Kwiatkowska (36) design a novel probabilistic circuit architecture to encode the target causal query and estimate causal effects but do not offer high-dimensional sampling.
Conclusion
We propose a sound and complete algorithm to sample from conditional unconditional high- dimensional interventional distributions. Our approach is able to leverage the state-of-the-an condi- tional generative models by showing that any identifiable causal effect estimand can be sampled from efficiently, only via feed-forward models. In future, we aim to relax our assumption on access to the ADMG and adapt our algorithm to deal with soft interventions.
10

---

Acknowledgments and Disclosure of Funding
This research has been supported in part by NSF CAREER 2239375, IIS 2348717, Amazon Research Award and Adobe Research.
References
[1] Vahid Balazadeh Meresht, Vasilis Syrgkanis, and Rahul G Krishnan. Partial identification of treatment effects with implicit generative models. Advances in Neural Information Processing Systems, 35:22816 22829, 2022.
[2] Elias Bareinboim and Judea Pearl. Causal inference by surrogate experiments: z-identifiability. In Proceedings of the Twenty-Fighth Conference Uncertainty in Artificial Intelligence, pages 113-120, 2012.
[3] Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas Ieard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and causal inference: the works of judea pearl, pages 507-556. 2022.
[4] Arnab Bhattacharyya, Sutanu Gayen, Saravanan Kandasamy, Ashwin Maran, and Vinod- chandran Variyam. Learning and sampling of atomic interventions from observations. In International Conference on Machine Learning, pages 842-853. PMLR, 2020.
[5] Arna Bhattacharyya, Sutanu Gayen, Saravanan Kandasamy, Vedant Raval, and Vinodchan- dran N Variyam. Efficient interventional distribution learning in the pac framework. In International Conference on Artificial Intelligence and Statistics, pages 7531-7549. PMLR, 2022.
[6] Christopher M Bishop. Pattern recognition and machine learning. Springer google schola, 645-678, 2006.
[7] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanic Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, al. Making the most of text semantics improve biomedical vision-language processing. In European conference on computer vision, pages 1-21. Springer, 2022.
[8] Daniel C Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker. Morpho- mnist: quantitative assessment and diagnostics for representation learning. Journal of Machine Learning Research, 1-29, 2019.
[9] Omar Del Tejo Catalá, Ismael Salvador Igual, Francisco Javier Pérez-Benito, David Millán Escrivá, Vicent Ortiz Castelló, Rafael Llobet, and Juan-Carlos Perez-Cortes. Bias analysis public x-ray image datasets of pneumonia and covid-19 patients. Jeee Access, 9:42370-42383, 2021.
[10] Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Mal- gorzata Polacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P Langlotz. and Akshay Chaudhari. Roentgen: Vasgun-language foundation model for chest x-ray generation. arXiv preprint arXiv:2211.1277 2022.
[11] Patrick Chao, Patrick Blöbaum, and Shiva Prasad Kasiviswanathan, Interventional and counter- factual inference with diffusion models. arXiv preprint arXiv:2302.00866, 2023.
[12] Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-imant translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[13] Cleveland Clinic. Atelectasis. httpss //my www.weaaaid...n...oogeeel///aaad ateeettstls Accessed: May 12, 2024.
[14] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.

---

[15] Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint 1xXiv:1907.07165. 2019.
[16] Jawook Gu, Han-Cheol Cho, Jiho Kim, Kihyun You, Eun Kyoung Hong, and Byungseok Roh. Chex-gpt: Harnessing large language models for enhanced chest x-ray report labeling. arXiv preprint arXiv:2401. 1505, 2024.
[17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks, 2017.
[18] Healthline. Lung opacity: Symptoms, causes, diagnosis, and treatment. https: h/ss://wwww www Accessed: May 12, 2024.
[19] Jouni Helske, Santtu Tikka, and Juha Karvanen. Estimation of causal effects with small data in the presence of trapdoor variables. Journal of the Royal Statistical Society Series A: Statistics in Sociery, 184(3): 1030-1051, 2021.
[20] Jonathan Ho and Tim Salimans. Classifier-free difflusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[21] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[23] Yimin Huang and Marco Valtorta. Identifiability in causal bayesian networks: A sound and complete algorithm. In Proceedings of the national conference on artificial intelligence, volume 21, page 1149. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2006.
[24] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger Mark. and Steven Horng Mimic-cxi, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
[25] Yonghan Jung, Jin Tian, and Elias Bareinboim. Learning causal effects via weighted empirical risk minimization. Advances in neural information processing systems, 33:12697-12709, 2020.
[26] Murat Kocaoglu, Christopher Snyder, Alexandros G Dimakis, and Sriram Vishwanath. Causal- gan: Learning causal implicit generative models with adversarial training. In International Conference on Learning Representations, 2018.
[27] Kenneth Lee, Md Musfiqur Rahman, and Murat Kocaoglu. Finding invariant predictors effi- ciently via causal structure. In Uncertainty in Artificial Intelligence, pages 1196-1206. PMLR, 2023.
[28) Sanghack Lee, Juan D Correa, and Elias Bareinboim. General identifiability with arbitrary surrogate experiments. In Uncertainty in artificial intelligence, pages 389-398. PMLR, 2020.
[29] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[30] Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30, 2017.
[31] Mayo Clinic. Pneumonia symptoms and causes. https://www.mayoclll org/ diseases- wwwdiiionn//peeeoooooooooooooooi ottpdt/tw.iooo/oooooooao//////oo causes sstreeesssscccuee 20352224
Accessed: May
12, 2024.
[32] Mayo Clinic Staff. Atelectasis - symptoms and causes.
20collapse, Accessed: May 12, 2024.
12

---

[33] Medscape. Hyponatremia.
https: httes:///mmeiiiine medscape.com//ruee 296468 wwwwviiewoormm-ope... Accessed: May 12, 2024.
[34] Miguel Monteiro, Fabio De Sousa Ribeiro, Nick Pawlowski, Daniel C Castro, and Ben Glocker. Measuring axiomatic soundness of counterfactual image models. arXiv preprint arXiv:2303.01274, 2023.
[35] Nick Pawlowski, Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems, 33: 857-869, 2020.
[36] Judea Pearl. Causality: models, reasoning, and inference, 1980.
[37] Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688 1995.
[38] Judea Pearl. Causality, Cambridge university press, 2009.
[39] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[40] Md Musfiqur Rahman and Murat Kocaoglu. Modular learning of deep causal generative models for high-dimensional causal inference. In Forty-first International Conference on Machine Learning, 2024. URL http::::
[41] Fabio De Sousa Ribeiro, Tian Xia, Miguel Monteiro, Nick Pawlowski, and Ben Glocker. High fi- delity image counterfactuals with probabilistic causal models. arXiv preprint arXiv:2306.15764, 2023.
[42] Pedro Sanchez and Sotirios Tsaftaris. Diffusion causal models for counterfactual estimation. arXiv preprint arXiv:2202.10166, 2022.
[43] Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International conference machine learning, pages 3076-3085. PMLR, 2017.
[44] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vision and pattemn recognition, pages 9243 9252, 2020.
[45] Ilya Shpitser and Judea Pearl. Identification of joint interventional distributions in recursive semi-Markovian causal models. eScholarship, University of California, 2006.
[46] llya Shpitser and Judea Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9:1941-1979, 2008
[47] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500: 2020.
[48] Adarsh Subbaswamy. Peter Schulam, and Suchi Saria. Preventing failures due dataset shift: Learning predictive models that transport. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3118-3127. PMLR, 2019.
[49] Adarsh Subbaswamy, Roy Adams, and Suchi Saria. Evaluating model robustness and stability dataset shift. In International Conference on Artificial Intelligence and Statistics, pages 2611-2619. PMLR, 2021.
[50] Jin Tian. Studies in causal reasoning and learning. University of California, Los Angeles, 2002.
[51] Jin Tian and Judea Pearl. A general identification condition for causal effects. eScholarship. University of California, 2002.
[52] Jin Tian and Judea Pearl. On the identification of causal effects. Technical Report R-290-I. UCLA C.S. Lab, 2002. httlhttp:://eet ucl.. http://aa.eee/ee//ttteteee//.... pdf
13

---

[53] Santtu Tikka and Juha Karvanen. Enhancing identification of causal effects by pruning, Journal of Machine Learning Research, 18(194):1-23, 2018.
[54] Thomas Verma and Judea Pearl. Equivalence and synthesis of causal models. Probabilis- lic and Causal Inference, 1990. URL https://api. semanticscholar org/CorpusID;
27807863
[55] Thanh Vinh Vo, Arnab Bhattacharyya, Young Lee, and Tze-Yur Leong. An adaptive kernel approach to federated learning of heterogeneous causal effects. Advances in Neural Information Processing Systems, 35:24459-24473, 2022.
[56] Benjie Wang and Marta Kwiatkowska. Compositional probabilistic and causal inference using tractable circuit models. International Conference on Artificial Intelligence and Statistics, pages 9488 9498. PMLR, 2023.
[57] Linda Wang, Zhong Qiu Lin, and Alexander Wong. Covid-net: tailored deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Scientific Reports, 10(1):19549, Nov 2020. ISSN 2045-2322. doi: (00388-41598-022-76550000 URL https://doi.. orgooo//// 00000///41990 -255-7650-
[58] Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal neurall connection: Expressiveness, learnability, and inference. Advances in Neural Information Processing Systems, 34:10823-10836, 2021.
[59] Kevin Muyuan Xia, Yushu Pan, and Elias Bareinboim. Neural causal models for counter- factual identification and estimation. In The Eleventh International Conference on Learning Representations, 2023. URL https:// wttm://oooommmddddduud
[60] Yanan Xin, Natasa Tagasovska, Fernando Perez-Cruz, and Martin Raubal. Vision paper: causal inference for interpretable and robust machine learning in mobility analysis. In Proceedings of the 30th International Conference on Advances in Geographic Information Systems, pages 1-4, 2022.
[61] Matej Zečević, Devendra Dhami, Athresh Karanam, Sriraam Natarajan, and Kristian Kersting. Interventional sum sum-product networks: Causal inference with tractable probabilistic models. Advances in neural information processing systems, 14:15019-1503,, 2021.
[62] Cheng Zhang, Kun Zhang, and Yingzhen Li. A causal view on robustness of neural networks. Advances in Neural Information Processing Systems, 33:289-301, 2020.
[63] Weijia Zhang, Lin Liu, and Jiuyong Li. Treatment effect estimation with disentangled latent factors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10923-10930, 2021.
[64] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde; Unpaired image-to-image translation via energy energy-guided stochastic differential equations. arXiv preprint arXiv:2207.06633, 2022.
14

---

A Limitations and future work
In this section, we list some future directions of our work. i) We assume that we have access to the fully specified causal graph (ADMG) and the causal model semi- Mani-MarroviaN. Although these are common assumptions in causal inference, aim to extend our algorithm for structures with more uncertainty (ex: PAGs). ii) In certain cases, the identification algorithm acts on particular variables, yet the expression for the causal effect does not depend on these variables. However, given that our sample size is limited, distinct values of these variables could affect the accuracy of the causal effect [19]. Since our algorithm follows the same recursive trace the identification algorithm, it might suffer from the same problem. In addition, some model training might be unnecessarily performed due to this issue. To avoid this scenario, the pruning algorithm proposed in (53) might be merged with us to reduce the redundant cost of model training. iii) Since we target a specific causal query, if the query is substantially changed (causal effect on new variables), we might have to re-train some models. iv) For causal effect estimation in high-dimension, we follow the ID algorithm [A5] to consider a hard intervention do(XX z) i.e., fixing specific value of the intervened variable, and assume that the rest of the conditional distribution stays unchanged. An interesting future direction would be consider imperfect/soft intervention where the underlying mechanism of intervention changes instead of being fixed to specific value.
B Broader impact
During the last few years, researchers have proposed many deep learning-based approaches to learn the unknown structural causal model from available data and employ the learned model to estimate the causal effect or sample from the high-dimensional interventional distribution. However, all these methods propose solutions tailored to specific neural architectures. As result, when better generative models appear (such as diffusion models), existing methods lack the flexibility to utilize them, and thus we are in need of new causal sampling method to get benefits of the new architecture. ID-GEN proposes a generic method that independent of any model architecture and can generate high-dimensiooal interventional samples with any model architecture (GANs, VAF, Normalizing flow, diffusion models, etc.) as long as that model has the ability to generate conditional samples. Thus, algorithm would allow the causal community to always use the latest generative model for high-dimensional causal sampling-bant
applications.
On different note, the ability to sample from interventional distributions may enhance deep- generative models obtain fake data, which can be exploited similarly to fake image generation. Since our algorithm considers causal relations among different variables, it has the ability to generate comparatively sensible and realistic fake images. Fake images can be source of distress or might influence public actions and opinion. Thus, careful deployment of should be considered for deep causal generative models and the same procedures to detect fake image should be taken.
C ID-GEN additional discussion
In this section, we provide more details about our algorithm ID-GEN.
C.1 Sampling from any interventional distribution with ID-GEN
Each term is expressed in terms of observational distribution. Note that it nonintuitive and nontrivial for existing algorithms to sample from this complicated expression. Our algorithm ID-GEN solves this problem by training a specific set of models in a recursive manner and building a sampling network combining them. Finally, ID-GEN uses this network to sample from the following interventional distribution.
Here we provide the full form of the causal query P(v (())))()) for the graph in Figure 7
P(v]do(r)) P(w2, wa (d)(d))))))))))))) ((w,)))) P(w)
(wi, (()):=)) \ {x}))
P(w2, walr) P(wa /w3)) * Σ,- P(wi/ Ir', w2, wa, w:a) P(r' ww3, w4)
Σ. P(wi, sol' w2, W3, w4) P(r (wa wa)
* Σ.. (w minlrr,,,, W2, Wy, wa P(r (W3, wa)
15

---

do(R
do m))
P(ay, wng (o(())
Picldofy (x,)1)))
Preseription Report (A0)
Hangial
Mwa
W
Miy
Wa
W2
Mx
Mys
Myy
W2a
Mx
+
Agingi (WG)
Symptoms (1G))
Xray Scan (X)
W.
P(w (do(eaa)))
P(en (do sn,, me)))
Ws
W
Wi
Past illoess
Detection & Diagnosis (WC)
W2
Wa
Mw
W
(Wa)
Wi
Mwa
Mws
(a) ML model failure scenario (b) Train and build sampling network. (c) Merge and sample P(yldo(x))
Figure do R= r) removes the R <> X bias and makes prediction of X domain invariant. ID-GEN factorizes P(v]do(r)) into four factors and trains conditional models ({ (Mr; 1) for each (blue shades). The intervened value R r propagated through the merged network to generate all other variables.
C.2 Cyclic dependency dealt with ID-GEN
Mass
Mx²
Mass
Mxx
Wi
Y
My
W
Mws
Y
My
MWs
Figure 8: +*Coonoundding Mw, My sample from Pa.,, (w1, y) = P(wrlz) (x) P(ylæ, W1, W2). Mx', Mw, sample from Pz,w, (W2) Σg P(x') P(wylæ, w1). Joint network samples from Pz(y).
Example C.1 (Cyclic dependency). For the graph Fig. (v) does not fit any familiar criterion such as backdoor front door. ID algorithm factorizes (1) into e-factors its step 4, as (v) = Ewy, 222 Pa.w, P (Wi, y) and estimates each of them recursively. Suppose we naively design an algorithm that follow ID step-by-step and trains a generative model and sample from every factor it encounters. This algorithm would not know which of these factors to learn and sample first with the generative models. sample W2 ~ Pa.wy (W2) we need W, as input, which has to be sampled from Pa. (W1, y). But sample {W1, Y} Pe (W1, y). we need W2 as input which has to be sampled from P2,,, (W2). Therefore, no order helps to sample all W1, W2, Y consistently.
ID-GEN follows ID's recursive trace to reach at the factorization: Pz Ew, Pa.wy (wz) ) Pa,wa (w1, y) but solves the deadlock issue by avoiding direct sampling from them. Rather, first trains the required models for e-components (W1,Y) (W2) individually, considering all possible input values, and then connects them to perform sampling. Thus, ID-GEN first obtains P2,w,, (W1, y) P(wi Pzylz, W1, W2). and trains conditional model for each of the conditional distributions (Fig. 8 left-bluee.. Similarly, for Pa.w, (W2) = Σx P(x') P (oz w1), we train models for each conditional distribution (right-blue). Note that X and X' sampled from the same P(x) but considered as different variables. Thus, we train Mx to sample from P(x') which is different from intervention dx) Finally, merge these networks trained for each c-component to build a single sampling network and can perform ancestral sampling on this network to sample from Ps (y), Va.
C.3 Related works
we show visual comparison among different implementations of existing works. Researchers have proposed deep neural networks for causal inference problems [43] 30 and neural causal methods Xia et al. [58]. Kocaoglu et al. [26] to deal with high-dimensional data. However, proposed approaches are not applicable to any general query due to the restrictions they employ or their algorithmic design. For example, some approaches perform well only for low-dimensional
In Figure 9
discrete and continuous data (group in Figure 9 [58 II while some propose modular training to
partially deal with high-dimensional variables (group 3: [40]. Other works with better generative
16

---

Goal: sample from P(mida(o))
Group Likelhood estmation
2: architauures
Group Nommlcng Diflusion
Moduler nlwss trained
Cillusion sumpling
"o you you spexsier A mos a mikes. M Pieii
Yew tiPoeiPio
Cenernteed
Dengutte
voleme(V)
ware
(Cenerand
Gotteraoe
Dilaccn
+
VI
M
Figure 9: Suppose we aim to sample from P(m do())), with ( as high-dimensional. Group I includes algorithms (ex: ID) that depend likelihood estimation such as P(m|u, 6). Group 2 algorithms with GAN architectures have issues with GAN convergence while Group 3 improves convergence with modularization but both struggle match the joint distribution. Group 4 utilizes normalizing flow, diffusion models, etc but cannot deal with confounders. Groups 2-4 only do unconditional interventions or costly rejection sampling. Finally, our method can employ classifier- free diffusion models to sample from P(m]do(v) or conditional P(m|a, do(u)). The causal graph is adapted from Ribeiro et al. [4].
performance depend on a strong structural assumption: no latent variable in the causal graph (group 4: Kocaoglu et al. [26], Pawlowski et al. [35], Chao et al.
C.4 ID-GEN recursion tree example
we show a possible recursive route of ID-GEN for a causal query P(y[do(x)). any recursion level, we check condition for 7 steps (S1-S7) and enter into one step based on the satisfied conditions. The edges indicate the top-down phase, and the green edges indicate the bottom- up phase. The rectangular gray boxes CConditionalCnt
In Figure 10
(.), MergeNetwork (ovoork().. (.), Update(.)) represent the functions that allow ID-GEN to sample from the high-dimensional interventional distribution. Pvies (51) (82) are obtained after performing e-factorization at step 4. We also indicate the recursive route with increasing indices. We hope that this figure helps the readers understand the recursion route in better way.
C.5 IDC-GEN: conditional interventional sampling
Existing works that use causal graph-based feedforward models [59 40 need update the posterior of Z's upstream variables which is not efficiently feasible in their architecture. We, given a causal query Ps (y]]). sample from this conditional interventional query by calling Algorithm 5 IDC-GEN, This function finds the maximal set C Z such that we can apply do-calculus rule-2 and move from conditioning set Z and add intervention set X. Precisely. (zly2) Paua (y)a \a) (v..(v.2)a) Next, Algorithm I: ID-GEN (.) is called to obtain the sampling network that can sample (F)a)
from the interventional joint distribution Paue (v: z ax.. We the sampling network generate samples D[X U a. Y %)ool, We train a new conditional model My on that takes Z \ a, X a input and samples Y (y, a)) i.e. Y Pz(y)z).
C.6 ID-GEN computational complexity
Bhattacharyya al. 5 their work discusses the sample complexity and time complexity of Shpitser and Pearl [46] 's ID algorithm. Since our algorithm ID-GEN is build upon the recursive structure of ID, we follow their approach to determine the computational complexity of our algorithm.
Suppose, in step 4 ID-GEN factorizes Px(y) as
Evi(yux) (s1)... (. Pv............. (sn)
where each S; is the c-factor of each c-component Cj. Let the number of variables located in each c-component be k. Suppose, the intervention X can be partitioned into multiple e-components and the c-components can be arranged in way such that X = U,_,X, and X; C C.,, i.e., all interventions are located in the first / c-components. Following 5.. we define two sets C2 = Uis( and Cs Ut.
17

---

Px
S1
S2
S3
S4
S5
56
S7
Fall
53
54
Factorization
(51)
P Pr1s2 (s2)
MergeNetwwrr
$2
ConditionaloMs
Update
51
S2
56
ConditionnloMM
CondinsesalOMs
Figure 10: ID-GEN Recursion Tree Example
For |C>i I, c-components that do not contain any X, ID-GEN will either go to stepl step In the base cases, they will train & models for each c-components. Thus, the number of models trained for these c-components will be ((n-1)k).
e-components, we assume that each e-component contains intervention subset X, of size X. Then the size of the remaining c-component is k X. For each Pvie. (8,). ID-GEN will eventually reach the base cases: Step or and will train X' models. Now, consider recursive steps. We assume that, in the worst case, X, will reduce one at time, by visiting step 7 and step alternately. Thus, ID-GEN will visit step 7, O(x/2 - 1)) times (except base case). Whenever ID-GEN visits step will apply subset of the intervention X; and atmost k models will be trained to sample the updated training dataset. Thus, till the base case, the total number of models trained in step 7s will be (22 1)). If we consider the whole recursive route for each c-component O E Cg. we will train Ok((2/2 1)) + O(kk - X)) O(kX/2) number of models. For all c-components in Cst, we will train ((kx2/2) number of models.
For the first |Ceal
Finally, if we consider c-components, we will train in total O(nn - 1)A + 11:X/2) models. the cost of training diffusion model to learn specific conditional distribution is O(T), then the total training cost is (T l)k- - TikX 22.. Note that, when there exists no confounders, we have k 1 and the trining cost is OT - () + T() O(Tn). Here AA(YY)a
Existing algorithms train V number of models for a causal graph of variables. Thus, their training cost is (T)
C.7 ID-GEN for Non-Markovian Causal Models
We assumed the underlying true causal model to be semi-Markovian our paper. This not a limitation as [52 showed that causal models with arbitrary latent variables can always be converted into Semii-arkovian causal models i.e., models which latent variables have no parent and only two children, while preserving the same independence relations between the observed variables.
18

---

D Theoretical analysis
Here we provide formal proofs for all theoretical claims made in the main paper, along with accom- panying definitions and lemmas.
Definition D.1. A conditional generative model for a random variable X E V relative to the distribution P(v) function Mx Pax + |X| such that Mx (x(aax ~ P(ælpax), Vpa Pa, where Pa is subset of observed variables in V.
Definition D.2 (Recursive call). For a function f(). when a subprocedure with the same name is called within f() itself, define it as recursive call. At Steps 2, 3, 4, 7 of Algorithm [ [D-GEN, the sub- peob-rdddedure (D-GEN(.) with updated parameters are recursive calls, but the aub-procedure Condit tionallMMMMM GMs ... Update (.) are not recursive calls.
Here, we restate the assumptions that are mentioned in the main paper.
Assumption D.3. The causal model is semi-Markovian.
Assumption D.4. We have access the true scyelic-directed mixed graph (ADMG) induced by the causal model.
Assumption D.5. Each conditional generative model trained by ID-GEN correctly samples from the corresponding conditional distribution.
Assumption D.6. The observational joint distribution is strictly positive and Markov relative the causal graph.
Lemma D.7 (c-component factorization (51).. Let M be an SCM that entails the causal graph G
and Pa(y) be the interventional distribution for arbitrary variables X and Y. Let C(G \ X)
{ Then we have Pz(y) Σ
Purer Pore Pors.
o((yJi))
Definition D.8. We say that a sampling network is valid for an interventional distribution Px(y) if the following conditions hold:
Every node V E H. has an associated conditional generative model My (.) except the variables in X.
If the values X =
specified in H. then the samples of Y obtained after dropping H. \ (XUY) YY from all generated samples are equivalent to samples from Px
Proposition D.9. The ConditionalCHs, turns 2 (S)).
X2, D,D, X, G) called inside Update(.), re-
Proof. Suppose that our goal is generate samples from Px (Y S'). S' a single c-component and the intervention set X Xz located outside of S'. This is the entering condition for step 6 of the ID-GEN algorithm. Thus, we can directly apply it as Algorithm Condit conditionalGMs (. ).,, instead of
another recursive ID-GEN (Y
Xz, G. D, G) call.
Proposition D.10. At any recursion level of Algorithm 7 ID-GEN (Y, G) and Algo- rithu o ID(Y, P.P.G) the execution step determined by the values of the set of observed variables Y, the set of intervened variables X and the causal graph G, at that recursion level.
Proof. To enter in steps [1-7] of ID-GEN and steps [1-7] of ID, specific graphical conditions are checked, which depend only on the values of the parameters Y, X and G. If that graphical condition satisfied, both algorithms enter in their corresponding steps. Thus, the execution step of each algorithm at any recursive level is determined by only Y, X and G.
Lemma D.11. (Recursive trace) At any recursion level R, recursive call ID-GEN (Y, D, enters Step i if and only if recursive call ID(Y, Y,YYYPPPP.
P, G) enters Step i, for any E [7]
Proof. Suppose, both algorithms start with an input causal query Px for a given graph G. For this query, the parameters Y, X, G of ID(Y, X, P. G) and ID-GEN (Y. X, X' DD G) represent the same objects: the set of observed variables Y, the of intervened variables X and the causal graph G. According to Proposition D.10 Algorithm T ID-GEN and Algorithm 6 ID only check these
19

---

parameters to enter into any steps and decide the next recursive step. Thus, we use proof by induction based on these 3 parameters to prove our statement.
Induction base case (recursion level R = 0): Both algorithms start with the same input causal query (y( in G, i.e., the same parameter set Y, X,X, G. We show that recursion level R = ID-GEN enters step if and only if ID enters step for any E [7].
Step 1: Both ID and ID-GEN check if the intervention set X is empty and go to step 1. Thus, ID-GEN enters step iff ID enters step 1.
Step 2: If the condition for step 1 is not satisfied, then both ID and ID-GEN check if there exist any non-ancestor variables of Y: V An(Y); in the graph to enter their corresponding step 2. Thus, with the same Y and G, ID-GEN enters step 2 iff ID enters step 2.
Step 3: If conditions for Steps [1-2] are not satisfied, both IDD and ID-GEN check if W is an empty set for W = (V)) X) \ An(Y)) to enter in their corresponding step 3. Since both algorithms have the same Y, X and G. ID-GEN enters step iff ID enters step 3.
Step 4: If conditions for Steps [I-3] are not satisfied, both ID and ID-GEN check if they can partition the variables set C(G \ X) into & (multiple) e-components. Since these condition checks depend on X and G and both have the same input X and G. ID-GEN enters step 4 iff ID enters step 4.
Step 5: If conditions for Steps [1-4] are not satisfied, both algorithms check if C(G X) = {{}} and C(G) = {G) to enter their corresponding step 5. Since both have the same X and G, ID-GEN enters step 5 iff ID) enters step 5.
Step 6: If the conditions of Steps [1-5] are not satisfied, ID and ID-GEN check if C(G \ X)= = (S) and S E C(G), to enter their corresponding step 6. Since both have the same X and G, ID-GEN enters step 6 iff ID enters step 6.
Step 7: If conditions for Steps [1-6] are not satisfied, both ID and ID-GEN check if C(G X) = {S} and (3.5") such that S C S' E C(G) to enter their corresponding step 7. Since both have the same X and G. both will satisfy this condition and enter this step. Thus, ID-GEN enters step 7 iff ID enters step 7.
(X)
Induction Hypothesis: We assume that for recursion levels R 1,..., ID-GEN and follow the same steps at each recursion level and maintain the same values for the parameter set Y, X and G.
Inductive steps: Both algorithms have the same set of parameters Y, X, G and they are at the same step i the current recursion level I = r. As inductive step, we show that with the same values of the parameter set Y, X and G at recursion level R = ID-GEN and ID will visit the same step at recursion level R = 11
Step 1: Step 1 is a base case of both algorithms. After entering into step with X = 0. ID estimates Evyy P(v) and ends the recursion. ID-GEN also ends the recursion after training a set of conditional models Mv, (V (V= ~ P(v: |ug 1)) Thus, ID-GEN goes to the same step at recursion level R = - + iff ID goes to the same step which in this case is Return.
Step 2: At step 2, both ID-GEN and ID update their intervention set X as X n An(YY)c) and the causal graph G as GAn(Y). The same values of the parameters X, Y and G will lead both algorithms to the same step the recursion level R = + 1. Thus, at recursion level R = r + 1, ID-GEN visits step i iff ID visits step i, Vi E [7].
Step 3: At step 3, both algorithms only update the intervention set parameter X as X = X U W and perform the next recursive call. Since they leave for the next recursive call with the same set of parameters, recursion level R R= r ID-GEN visits step iff ID visits step i, Vi E [7].
Step 4: At step 4, both ID-GEN and ID partition the variables set C(G X) into (multiple) c-components. Then they perform recursive call for each c-component with parameter Y = S and X = V \ S,. For each of these k recursive calls, both algorithms use the same values for the parameter set Y, X and G. Thus, at recursion level R + 1, ID-GEN visits step i iff ID visits step i, Vi E [7].
20

---

Lemma 19,12: Same gruph as ID
Lemma D44 ID and ID DDN'' 's relation with input P(V)
Lemma D-42- ID-GEN Termination
Lemma D
Lemma D
Lemma Same recursive trace as ID
Theorem D D:23 ID-GEN completeness
M() samples correctly from joint dist.
ID-GEN data sampled from same distribution ID
Lemma D-T7:
Lemma D Conrectness of Step 6 (Basc)
Lemma D.24 Sampling net follows G5
Lemma 1+
Correctness of Step (Base)
Correctness of Case
Theorem D.22 ID-GEN soundness
Figure 11: Flow Chart of Proofs
Step 5: Step 5 is a base case for both algorithms, and both algorithms are at step 5, they return FAIL. This step represents the non-identifiability case. Thus, ID-GEN goes to the same step at recursion level R = + iff ID goes to the same step, which in this case is Return.
Step 6: Step 6 is a base case for both algorithms. II) calculates the product of a set of conditional distributions P() P( and returns it. While ID-GEN trains conditional models My, (V7 (V=l--)) to learn those conditional distributions, ID-GEN returns a sampling network after connecting these models. Both algorithms end the recursion here and return different objects but that will not affect their future trace. Thus, ID-GEN goes to the same step at recursion level R = r + iff ID goes to the same step which this case Return.
Step 7: ID algorithm at step 7, updates its distribution parameter P with Px, (S"). On the other hand, ID-GEN generates (o() interventional samples. Both algorithms update their parameters set Y, X and G in the same way as Y y= X = X n S" (or X X since X = X Xz) and G Gsy Since they leave for the next recursive call with the same set of parameters Y, X, G. at recursion level R = r + 1, ID-GEN visits step i iff ID visits step i, Vi E [7].
Therefore, we have proved by induction that ID-GEN (Y, X, G, D,D,D.D X, G) enters Step if and only if ID(Y, X, P, G) enters Step for any [7] for any recursion level Rt
Lemma D.12. Termination: Let Px(Y) be a query for causal graph G = (V and D ~ P(V).
Then the recursions induced by ID-GEN (Y, G,G. X, G) terminate in either step 1, 5, or 6.
Proof. The result follows directly from Lemma D.11 Consider any causal query PY Since ID complete, the query- whether it is identifiable or not terminates in one of the steps 5, or By Lemma D.IT, ID-GEN follows the same steps as ID, and hence also terminates in one of these steps.
Lemma D.13. Consider the recursive calls ID(,, G1D) and ID-GEN(s, GID-GEN, X, G) at any level of the recursion. Then GID = GID-GEN and GID G X.
Proof. According to Lemma D.IT ID and ID-GEN follow the same recursive trace. Thus, at any recursion level, both algorithms will stay at step E [7].
Base case: At recursion level R 0, both ID and ID-GEN start with the same input causal graph
G. Thus, G GID-GEN G G. Also R the set of intervened variables X 0. Thus, GID Ō 0 G holds.
Induction hypothesis: At any recursion level R r, let GID be the graph parameter of the ID algorithm and [GiD-GEN: Xr, Gr) be the parameters for the ID-GEN algorithm. We assume that
G GEN and Gjp = 6" X".
21

---

Inductive step: The set of parameters {GI-GEN, X, G) of ID-GEN and the graph parameter G of ID are only updated at step 2 and step 7 for the next recursive calls. Thus, we prove the claim for these two steps separately.
At step 2: In both IDD and ID-GEN algorithms, we remove the non-ancestor variables V An(Y) from the graph. For ID, GTD Gin(An(Y)). For the ID-GEN algorithm, obtain GDDGGEN = G (iD.GE((A(())) Thus, GTD GIDGGNN ID-GEN, we also update Gr+1 as Gr+l = (' Since removing von-ancestor does not affect the intervened variables X anyway, the same relation between GTD and Gr+1 is maintained, i.., GTD Gr+1 (X.
At step Both ID and ID-GEN finds Xz x S' and apply do(X2) as intervention. Also, in ID-GEN, X set to XrUX, U'
For ID-GEN:
Since Xz is intervened on, G" for ID-GEN is updated as G G and G is updated as Gr+1 = = Xx
Now, Xz has all incoming edges are cut off in G and does not have any parents. As a result, removing Xz from G*+1 is valid. Thus, we obtain Gr+1 Gr+1 X OxZ Xx Xx G's--
However, for the other graph parameter, we keep Gr+1 = G as it is, since we would follow this structure in the base cases and consider X while training the conditional models. Note that, in &" the intervened variables X" (before recursion level r) had already incoming edges removed. Thus, Since by
after X being updated as X = X" U Uz, we can write Gr+1 = - Gr XriI
inductive assumption, G 8" X", thus
Gm Gr- (X' Gr+1 \ -
(8
(2)
For ID:
ID algorithm updates its parameters G as Gr+1 Grg = G Xx. Since ID-GEN and ID have the
same graph parameter at recursion level i.. GiD G GID.GEN GEN and they updated the parameter in the
same way, GTD GiD-GEN holds true. Since G GID.GEN = Gr+1 {X++1) according Equation 2
we also obtain GTD = G (X++1).
Therefore, the lemma holds for any recursion level R.
Lemma D.14. Let P(.) be the input observational distribution to the ID algorithm and D P(.) be the input observational dataset to the ID-GEN algorithm. Suppose, X is the set of variables that are intervened at step 7 of both the ID and ID-GEN algorithu from recursion level R = 0 to R
Consider recursive calls ID(,, (D(s,Pi(V))) PiD(v)) and ID-GEN( D, X, G) at recursion level r. Then PID = Pv(v) and D D[X,V) V] ~ v))
Proof. According to Lemma D.IT ID and ID-GEN follow the same recursive trace. Thus, at any recursion level, both algorithms will stay at step E [7].
At recursion level R = O, ID starts with the observational distribution P and ID-GEN starts with the observational training dataset D P. At R O, X 0 since no variables have yet been intervened on. Thus, for ID algorithm PıD = PV(V) holds. For ID-GEN, DjD. V] ~ Po(0, V) holds. Thus, the claim is true for R
Base case:
Induction hypothesis: Let the set of variables intervened at step 7s from the recursion level R = to R ==:- r. be X At R = r, let the distribution parameter of the ID algorithm be P Px (v). Let the dataset parameter of the ID-GEN algorithm be sampled from Px v), i.e, D P Px-(X", v).
Inductive step: The distribution parameter of ID and the dataset parameter D of ID-GEN only change step 2 and step 7 for the next recursive calls. Thus, we prove the claim for these two steps separately.
At step 2: At step of both ID and ID-GEN algorithms, there is no change in X, i.e. no additional variables are intervened to change the distribution P or the dataset D)
22

---

At this step, we marginalize over V \ AY(Y). Since X C An(Y)) marginalizing non-ancestors does not impact X. Thus, ( = EviAntr) (vn Av) Px Pv (An(Y)) Px- (e=((A(X))) This holds true since we can marginalize over (Yn)) in the joint interventional distribution.
In the ID-GEN algorithm, we drop the values of variables V \ An(Y)o from the data set, that is, D X V 7 D XX An())al. Since we assumed D X V] ~ (x v), therefore D X, An (Y ~ Px- (An(Y) holds.
At step 7: At step 7, both ID and ID-GEN algorithm intervened on variable set X where X X \ S". Thus, at level R + 1, we have X = X U2z. Note that this step, S' V Xg.
ID algorithm updates its current distribution parameter PTD by intervening on Xz. Thus, at level R r + 1, P (v x2) = PT (s') = PiD (v]d(x( (())(xx)) x2)) = Px (v]do(x2)) = Pxruxx (5') (s'). Thus the claim holds for ID algorithm at level R r 1.
In the ID-GEN algorithm, we generate an interventional dataset Dr+11 [x,, X2, by applying the do(X2) intervention on dataset D'[X.V] Vj. Since [" V] Px v), obtain D zx X S"] Perux, Xg, 5') = D D SS] Ps++i (xr+1, ,'). Thus, the claim holds true for ID-GEN at the recursion level R = 10=++1
Therefore, P(.) being the input distribution for ID and D P(.) being the input dataset for ID-GEN
we proved by induction that PID = Px(v) and D[X, V] Px(x, v) at any recursion level.
Lemma D.15. Consider recursive calls X,, any recursion level R. Df V] PID-GEN(&,, v), thenPrn(y) PIO-GEN (v (X) for fixed values of X
G. PID(V)) and ID-GEN (Y, (Y,X,C,D,X,C))
Proof. At any level R = r of the recursion, we have GIn = GID.GEN and GTn = = \.> X according to Lemma D.13 and PıD (v(v) and D V] ~ (x v) according to Lemma D.14 where P(v)
the input observational distribution.
Let D[X, VJ P& v) PID.GEN(&,, v), VX.
Now, since in & we have already intervened on X, for fixed X X, we have P& v) Px(x) Ps (x(v(x) = (v)). thus V will be sampled from PID-GEN (v|*) for fixed consistent X
On the other hand, in the ID algorithm, at any recursion level, PID(V) = Px(v) for fixed consistent X x and GID = X.
Since according to Lemma D.14 PID = P&(v) and DJX, VJ ~ Px(x, v) at any recursion level, i.e., starting from the same P(V). the same of interventions are performed in the same manner in both algorithms; thus for fixed X x, we obtain Pin(v) = (v)
Lemma D.16. If a non-identifiable query is passed to it, ID-GEN will return FAIL.
Proof. Suppose ID-GEN is given a non-identifiable query as input. Since the ID algorithm is complete, if ID is given this query, it will reach its step 5 and return FAIL. According to the lemma D.II] ID-GEN follows the same trace and the same sequence of steps as ID for a fixed input. Since for the non-identifiable query ID reaches step 5, ID-GEN will also reach step 5 and return FAIL.
Lemma D.17. ID-GNN Base case (step 1):
Let, any recursion level of ID-GEN, the input dataset DIX, V] is sampled from a specific joint distribution P(X, V), ie. D[X, V] ~ P(X, V) where X is the set of intervened variables at step 7s. Given a target interventional query Ply) over causal graph G (V,E). suppose ID-GEN
23

---

Lemma 19.13 Same gruph
Lemma D44 ID and ID DDD''''s 's relation with input P(V)
Lemma D-42: ID-GEN Termination
ID
Lemma D
Lemma D
Lemma Same recursive trace ID
Theorem D 1:23+ ID-GEN completeness
M() samples correctly from joint dist.
ID-GEN data sampled from same distribution ID
Lemma D-H-
Lemma D Conrectness of Step (Base)
Lemma D.24 Sampling net follows G5
Lemma 1+
Correctness of Step (Base)
Correctness of Case
Theorem D.22 ID-GEN soundness
Figure 12: Flow Chart of Proofs
(Y,X- D, X, G) enters step I and returns H. Then H. is valid sampling network for P(y) with fixed consistent X x
Proof. Since the intervention set X is empty, we are at the base case step of ID-GEN. Suppose that for the same query, the ID algorithm reaches step 1. Let PTp (v) be the current distribution of the ID algorithm after performing series of marginalizations in step 2 and intervention in step on the input observational distribution. D D[X, V] is the dataset parameter of ID-GEN algorithm that went through the same transformations as the ID algorithm the sample space according to Lemma D.15 X is the set interventions that are applied on the dataset at step 7s.
Since ID algorithm is sound it returns correct output for the query P(y). We prove the soundness of ID-GEN step by showing that the sampling network ID-GEN returns, is valid for the output of the ID algorithm.
ID algorithm is sound and factorizes PID(v) with respect to the graph G below.
Pin(v)= = II
P( P( (w-1) Na
v.EV
And obtains PiD(y) by:
PiD(y) Σ II c/y NEE
P(us|uu
Let D|X, V] ~ PID-GEN(&,,)) v) = P(x, v), V(x, v). For fixed X x, PID-GEN(V($) can be factorized with respect to G and G as following:
#IDGEN(V\x))
II wEV
P(a:oofi) x)
II P(r./o(t11))
weV
[Changed the graph from G to & since X
& and only affects the descendants of X.]
And PID-GEN(y)* ()) can be obtained by:
PID (y|x)
Σ II N/ WEV
P(o.lo(i-))
Since for fixed X x, PID(v) PID (v/x) according to Lemma D.15 Thus, the cor- responding conditional distributions in Equation 3 and 4 are equal, i.e. PId (v; luli ((f-")) = (5-1)) )) ViilV V). Therefore, PID (y) and PID-GEN(y(X) having the same factoriza- tion and the corresponding conditional distributions being equal implics that PID(y) PIO-GEN(y x).
PID-GEN (osloff
24

---

Now, based on Assumption D.5 ID-GEN learns to sample from each conditional distribution M
U (6-1)) of the product in Equation 4 by training conditional model My; on samples from the dataset D[X, v] P(x, v).
(tD.GEN(o:Juf-1 (1-1)
For each Vi, we add node containing V; and its associated conditional generative model My, to sampling network. This produces a sampling network H. that is a DAG, where each variable V, E V has sampling mechanism. By factorization and Assumption D.5, ancestral sampling from this sampling graph produces samples from PID-GEN( (v ( We can drop values of V \ to obtain the samples from PID-GEn(y ( Since (ID(v) PIN-HEN (v)*), both factorized in the same manner and Mv; learned conditional distribution of ID-GEN's factorization, the sampling network returned by ID-GEN, will correctly sample from the distribution PTD (y) returned by the ID algorithm for fixed
X X.
Thus, the sampling network returned by ID-GEN valid for P(y).
Lemma D.18. ID-GEN Base case (step 6): Let, at any recursion level of ID-GEN, the input dataset D[X, V| is sampled from a specific joint distribution P(X, V), i.e., D[X, VJ~ P(X, V) where X the set of intervened variables at step 7s. Given an identifiable interventional query Px(y) over a causal graph (V, suppose ID-GEN (Y, K.N.G. G.D.D. G) immediately enters step and returns H. Then is valid sampling network for Px(y) with fixed consistent X =.
Proof. By Lemma D.11 both ID-GEN (Y, X, G,D,K,G, G) and ID(Y, P. enter the same base case step 6. By the condition of step 6, G X has only one c-component {S}, where S C(G).
Let P(v) be the current distribution of the algorithm after performing series of marginalizations in step and intervention in step 7 on the input observational distribution. Let D V] ~ P'( v) be the dataset parameter of ID-GEN algorithm that went through the same transformations as the ID algorithm in the sample space. X is the set of interventions that are applied on the dataset step 7s. According to Lemma D.15 for fixed values of X X, P(v) P' (v(x) holds true at any recursion level.
Since ID algorithm sound it returns correct output for the query Py(y). We prove the soundness of ID-GEN step 6 by showing that the sampling network ID-GEN returns, is valid for the output of the ID algorithm.
With the joint distribution P(v), the soundness of ID implics that
Px(y)= = Σ II
P(e P(o)lo(t11) ((t1))
(5)
SVY (IVES))
where मh is topological ordering for G.
With the joint distribution p" ('(v(x), ID-GEN can factorize PK (v/xx) in the same manner:
(y(()) =
II
P' P'(walo(t- x)
SYY ((IEES)
Σ
II
('(w.loft-1))
(6)
SYY LIVES)
[Changed the graph from G to G since X E G and only affects the descendants of X.]
Since P(v)- = P'(v x), the corresponding conditional distributions in Equation 5 and 6 are
equal, i.e, P(u, ((u.lufii1))) p' (w.:loff-") v/j S}. Therefore, Px(y) and P (y(x) having the same factorization and the corresponding conditional distributions being equal implies that (y()) ()))).
ID-GEN operates in this case by training, from joint samples D X, VJ. model to correctly sample each P'(ez ox (6-1)) term, i.e., we learn a conditional generative model My, (vi(-)) which produces samples from P'(n: w/ ( which we can do according to Assumption D.5 Then we construct sampling network H by creating node V; with sampling mechanism My, for each 1/2 E S. We add
25

---

edges from V + Vi for each V E Va(-)) Since every vertex in G' is either in S or in X U X, every edge either connects to a previously constructed node or a variable in X. Since we already have fixed values for X = X, when we specify values for X and sample according to topologiaal order aG. this sampling graph provides samples from the distribution [[vves P' P'(u:|utt () ). i.. (s)x))
Since, as shown earlier, (s)x)) (s(s), samples from the sampling network H. is consistent with (s) as well. We obtain samples from Px(y) by dropping the values of S \ Y from the samples show that this sampling network is correct for (y)): certainly this graph is a DAG and every U E S has conditional generative model
obtained from (s). We assert the remaining conditions
0. Then every node in H is either in S or is in X U X: hence the only nodes without sampling mechanisms are those in X U X as desired. Therefore, when X fixed as x, H is a valid sampling network for Px(y).
in H. By the conditions to enter step 6, G S L UX and S (x(X
Proposition D.19. At any level of the recurison, the graph parameters G and G in ID-GEN G, X, G) have the same topological order excluding X.
Proof. Let He be the topological order of G and =e be the topological order of G. At the beginning of the algorithm G = G. Thus, TG = FG.- According to the lemma D.13 at any level of recursion,
GID = GID-GEN and GID G X. Thus, we have G GID-GEN GID = Ô \ X. Therefore, TG = _
Lemma D.20. Let H be sampling network produced by ID-GEN from an identifiable query Px(y) over a graph G. If has the topological ordering n, then every edge in the sampling graph of H adheres to the ordering
Proof. We consider factors: which edges are added, and with respect which graphs. Since the only base cases ID-GEN enters are steps 1 and 6, the only edges added are consistent with the topological ordering π for the graph that was supplied as an argument to these base case calls. The only graph modifications occur in steps 2 and 7, and these yield subgraphs of G. Thus the original topological ordering ı for graph G is a valid topological ordering for each restriction of G. Therefore any edge added to H is consistent with the global topological ordering II.
O
Lemma D.21. Let H be a sampling network for random variables {V1, V2,... Vn3 formed by a collection of conditional generative models Mv, relative to P(v) for all V,. Then the tuple (V1, V22... Vv) obtained by sequentially evaluating each conditional generative model relative the topological order of the sampling graph is sample from the joint distribution II,P,(v; (..)).).o.)
Proof. Without loss of generality, let (V1, V2,..., Vx) be a total order that is consistent with the topological ordering over the nodes in G. To attain a sample from the joint distribution, sample each V, in order. When sampling Vj. each V; for all i < j is already sampled, which is superset of Pay (the imputs to My,) ) by definition of topological orderings. Thus, all inputs to every conditional generative model My, are available during sampling. Since each V, is conditionally independent of L
V, & Paj. the joint distribution factorizes as given in the claim.
Theorem D.22. ID-GEN Soundness: Let Pxly) be an identifiable query given the causal graph G G= (V, and that we huve access to joint samples D P(v). Then the sampling network returned by ID-GEN (Y, G, X G) correctly samples from Px(y) under Assumption D.5
Proof sketch: Suppose that Py() is the input causal query and Assumptions D.3 D.4
D.5
D.6
hold. The soundness of ID-GEN implies that if the trained conditional models converge (near) optimality, ID-GEN returns the correct samples from (x(y. J. For each step of the ID algorithm that deals with probabilities of discrete variables, multiple actions are performed the corresponding step of ID-GEN to correctly train conditional models to sample from the corresponding distributions. ID-GEN merges these conditional models according to the topological order of G, build the final sampling network H. Therefore, according to structural induction, when intervened on X and
26

---

Lemma 19.13: Same gruph as ID
Lemma D44 ID and ID DDN'' 's relation with input P(V)
Lemma D-42- ID-GEN Termination
Lemma D
Lemma D
Lemma Same recursive trace as ID
Theorem D D:23 ID-GEN completeness
M() samples correctly from joint dist.
ID-GEN data sampled from same distribution ID
Lemma D-17:
Lemma D Conrectness of Step 6 (Base)
Lemma D.24 Sampling net follows G+
Lemma 1-46;-
Correctness of Step (Base)
Correctness of Fail Case
Theorem D.22 ID-GEN soundness
Figure 13: Flow Chart of Proofs
perform ancestral sampling in H each model in the sampling network will contribute correctly to generate samples from (v)..
Proof. We proceed by structural induction. We start from the base cases, i.e., the steps that do not call ID-GEN again. ID-GEN only has three base cases: step is the case when no variables are being intervened upon and is covered by Lemma D.17] step is the other base case and is covered by Lemma D.18 step 5 is the non-identifiable case and since we assumed that (x is identifiable, we can skip ID-GEN's step 5.
The structure of our proof is as follows. By the assumption that (y(y) is identifiable and due to Lemma D.12] its recursions must terminate in steps or Since we have already proven correctness for these cases, we use these as base cases for a structural induction. We prove that if ID-GEN enters any of step 2, 3, 4 or 7, under the inductive assumption that we have correct sampling network for the recursive calls, we can produce correct overall sampling network. The general flavor of these inductive steps adheres to the following recipe: determine the corresponding recursive call that ID algorithm makes; ii) argue that we can generate the correct dataset be analogous to the distribution that ID uses in the recursion; iii) rely on the inductive assumption that the generated DAG from ID-GEN's recursion is correct.
We consider each recursive case separately. We start with step 2. Suppose ID-GEN (Y,X, G, D, X, G) enters step 2, then according to Lemma D.11 ID(Y, P,G) enters step 2 as well. Hence the correct distribution to sample from is provided by ID step 2:
Px(y) ID(Y, X An(Y)a,
Σ P(v))
ViAn(Y)o
Now, according to Lemma D.15] at the current step, the dataset D of ID-GEN is sampled from a distribution ('(v)) such that for fixed values of X = &, P(v) = (<v\x) holds true. Following our recipe, we need update the dataset D such that
sampled from EV(An(Y) P(v)
EviAn(Y) ('(v[2), We do this by dropping all non-ane ancestoc variables of Y (in the graph G) from the dataset D, thereby attaining samples from the joint distribution EviAn(Y) P(v). Since G is used at the base case, we update it as Gan(Y), the same as Gan(Y) to propagate the correct graph at the next step. Therefore, we can generate the sampling network from ID-GEN(Y, (EEN(YXX) X An( ). G D[An(Y)], X, GAm(Y) by the inductive assumption and simply return it.
Next, we consider step 3. Suppose ID-GEN (Y, X, G. D, X, G) enters step 3. Then by Lemma D.11 ID(Y, X, P.G. G) enters step 3, and the correct distribution to sample from is provided from ID step
Py( = ID(Y, XUW,P,G)
where W == (V \ X) An(Y) Since the distribution passed to the recursive call is P. we can simply return the sampling graph generated by ID-GEN (Y, X, G. D. X, G), which we know is
27

---

Lemma 19.13: Same gruph as ID
Lemma D44 ID and ID DDN'' relation with input P(V)
Lemma D-42- ID-GEN Termination
Lemma D
Lemma D
Lemma Same recursive trace as ID
Theorem D 2:23 ID-GEN completeness
M() samples correctly from joint dist.
ID-GEN data sampled from same distribution ID
Lemma D-17:
Lemma D Conrectness of Step 6 (Base)
Lemma D.24 Sampling net follows G+
Lemma 1-46:-
Correctness of Step (Base)
Correctness of Case
Theorem D.22. ID-GEN soundness
Figure 14: Flow Chart of Proofs
correct for Pxuw( ( Y) by the inductive assumption. Thus, the returned sampling network by ID-GEN can sample from Px(y). While we do need to specify a sampling mechanism for W to satisfy our definition of valid sampling network, this can be chosen arbitrarily, say W P(w) uniform the distribution.
Next we consider step 4. Suppose ID-GEN (Y, X, D, X, G) enters step 4. Then by Lemma D.1 ID(Y, X. enters step 4 and the correct distribution to sample from is provided from ID step as:
Σ II ID S,,.G.) V((Ua)))
where S are the c-components of G X, i.e., elements of C(G) By the inductive assumption, we can sample from each term in the product with the sampling network returned by ID-GEN(S,, =
V \ Sc G, X, G). However, recall the output of ID-GEN: ID-GEN returns a 'headless' (no conditional models for X) sampling network follows:
ID-GEN (Y, XY,X, D, G) returns sampling network, i.e., collection of conditional generative models where for each variable in G and every variable except those in have a specified conditional generative model. To sample from this sampling network, values for X must first be specified. In the step case, the values v \ 8; need to be provided to sample values for S,, and similarly for * j. values \ Sy are needed to sample values for Sj. Since S, (V) \ S,) and S, S (V) \ S,),, it might lead to cycles (as shown in Example C.T) if we attempt to generate samples for each e-components sequentially. Thus, it does not suffice sample from each c-component sequentially separately.
Note that, Ht, is the correct sampling network corresponding to ID-GEN(S, X VVS,, D, G) by definition, for each node V, E S,, V; has a conditional generative model in H,, By Lemma D.20 each edge in HI, adheres to the topological ordering FG (at the current level). Hence, if we apply MergeNetwork (.) to construct a graph H. from { HH }s. it will also adhere to the original topological ordering TG. Thus, H. is DAG.
Since every node V; in G \ X has conditional generative model in some H; the only nodes in combined 7 without conditional generative models are those in X. Finally, since each node in H samples the correct conditional distribution by the inductive assumption, H. samples from the product distribution (y) corretly. The sum Ev(ryx) can be safely ignored now and can be applied later since the sample values of the marginalized variables (v \ (y U x)) be dropped from the joint the end of the algorithm to attain samples values of the remaining variables. Hence H. is correct for Py()).
Step 5 can never happen by the assumption that Px(y) is identifiable, and step 6 has already been covered base case. The only step remaining is step 7.
Lemma D.11 says, ID-GEN (Y, G, G) enters step 7 by the same conditions TD(Y, X, G. enters step 7, Then by assumption, C(G X) = {S} and there exists a con- founding component S' C(G) such that S c S'. The correct distribution to sample from is
28

---

provided from ID step 7 as
= Xns' Gs5)
where
:= II P ns' u4 WIIVES
1) S*).
Examining ID algorithm more closely, if we enter step 7 during ID, the interventional set X is partitioned into two components: X and Xz := X S'. From Lemmas 33 and 37 of Shpitser and Pearl [46]. in the event we enter step Px is equivalent to (y) where P'(v) )) = Pxz (v). ID estimates Pxz (v) with similar computation as the step 6 base case.
To sample correctly in ID-GEN, we consider two cases.
i) X = When ID-GEN visits step 7 for the first time, we have X = In that case, we first update our dataset D[V] ~ P(v) to samples from D' Pxz (v), and then we recurse on the query P (y) over the graph Gg Also, Xz is outside the c-component S". Therefore we generate a dataset from D' ~ Pxx (5") via running directly the Conditional@e (.) of the ID-GEN algorithm, Algorithm 2
MaatioonnaaMMaMSS Xz, G, D, X, G). This is attainable via the inductive assumption and Lemma D.12 The only divergence from ID during the generation of DD' is that ID presumes pre-specified fixed values for Xz, where we train a sampling mechanism that is agnostic priori the specific choice of Xz. To sidestep this issue, we generate a dataset with all possible values of X and be sure to record the values of X z in the dataset D' 22 S").
ii) X 0: When ID-GEN has visited step 7 already once and thus X & 0. We consider X along with Xz when generating D' this time. More precisely, we update our dataset D[X, V] ~ P(x, v) to samples from D ~ Px Pxzux [x,, &, v). Rest of the steps follow similarly as the above case. We record the new Xz in X and carry them in D' D'(X,S") and G.
Next, we need map the recursive call ID(Y,XNS)) P'. XOY, parameters Y, X n'S' and G as ID. Now, equivalent to passing the distribution P' of ID, we pass the dataset D[X, S"] sampled from this distribution, including the intervened values for X used to obtain this dataset. According to Lemma D.15 this dataset is sampled from p' if we fix to a specific value X x. Finally, ID algorithm uses specific value of Xz and then ignores those variables from X and G for rest of the recursion. On the other hand, ID-GEN saves X in as X XUXz and keeps X connected to & with incoming edges cut, i.e., G x for the next recursive calls since ID-GEN utilizes
ID-GEN. ID-GEN sends the same
X and the topological order in G at the base cases (step and 6). By the inductive assumption, can generate correct sampling network from the call ID- ID-GEN(Y, X Xz, Gg, D S1. X, G and hence the returned sampling graph is correct for Px(y).
{s' ( }))
Since have shown that every recursion of ID-GEN ultimately terminates in a base case, that all the base cases provide correct sampling graphs, and that correct sampling graphs can be constructed in each step assuming the recursive calls are correct, we conclude that ID-GEN returns the correct sampling graph for (y))
Theorem D.23. ID-GEN is complete.
Proof. Suppose we are given causal query Px(y) as input to sample from. We prove that if ID-GEN fails, then the query (y( is non identifiable, implying that it is not possible to train conditional models on observational data and correctly sample from the interventional distribution Px(y).
ID-GEN reaches step 5, it returns FAIL. According to the lemma D.II ID reaches at step and only if ID-GEN reaches at step 5. Step 5 is also the FAIL case of the ID algorithm. Since ID is complete and returns FAIL at Step 5, the query (y) is not identifiable.
29

---

E Conditional interventional sampling
Conditional sampling: Given a conditional causal query P. (u(22,, we sample from this conditional interventional query by calling Algorithm IDC-GEN. This function finds the maximal set C Z such that we can apply rule-2 and move a from conditioning set Z and add it to intervention set (aua(p.2ka) Next, Algorithm ID-GEN(() is called to obtain the sampling network that can sample from the interventional joint distribution Paua (y, 2 a). We use the sampling network to generate samples D' through weed-forward. A new conditional model My is trained on D' that takes \ a and X U a as input and outputs Y. Finally, we generate new samples with My by feeding input values such that Y ~ (U a)) i.e, Y ~ (=(u))).
X. Precisely, P= (ul=) =) = Paua (u|2 \ a)
P=
Theorem E.1 (Shpitser and Pearl 46 D. For any G and any conditional effect Px(YIW) there exists 2)} such that rule applies to Z in G for Px Px(YIW). In other words, Px Px = PX Z.)
a unique maximal set = (Z E W ( Px,z W
Theorem E.2 (Shpitser and Pearl [46]). Let Px(Y(W) be such that every W E W has back-door path to in G A X given W (.). Then (x(Y(W) is identifiable in G if and only if Px W) is identifiable in G.
Theorem E.3. IDC-GEN Soundness: Let PY( be an identifiable query given the causal graph G = =V, and that we have access to joint samples D ~ P(v). Then the sampling G) correctly samples from Px ZY under Assump-
network returned by IDC-GEN
tions] D.3 D.4 D.5 D.6
Proof. The IDC algorithm sound and complete based on Theorem E. and Theorem For sampling from the conditional interventional query, we follow the same steps the IDC algorithm in Algorithm IDC-GEN and call the sound and complete Algorithm AlgorithnIIDDGEN ID-GEN as sub-procedure. Therefore, IDC-GEN is sound and complete.
F
Experimental details
F1 Training details and compute
We performed some of our experiments on machine with an RTX-3090 GPU. We also performed some training on 2 A100 GPU's which took roughly hours for 1000 epochs. Training for baseline NCM took more than 50 hours to complete 1000 epochs. The baseline DCM took around 10 hours. When variables were www-dimeniional discrete, it was quite fast and took 10-20 minutes to finish all training and get convergence. We discuss more specifics about each experiment in their individual sections.
F111 Reproducibility
For reproducibility purposes, we provide our anonimized source codes with instructions. Besides, provided explanations of each experiment along with model settings and hyperparameters. We provide the code to generate the Colored MNIST dataset.
F22 NapKin-MNIST dataset
Data Generation: First we consider a synthetic dataset imbued over the napkin graph. We consider variables W1, W2, Y. where Wi, are images derived from MNIST and Wa is paired discrete variable. We introduce latent confounders C, T. denoting color and thickness, where C' can be any of greed, blue, yellow, magenta, cyan), and T can be any of (thin, regular, thick). Data generation proceeds as follows: first we sample latent C. T from the uniform distribution. We color and reweight random digit from MNIST form W1. W2 only keeps the digit value in {O... | of W1 and a restriction of its color: if the color of W1 is red, green, or blue, W2 's color value is O, and it is otherwise. then picks a random MNIST image of the same digit as W2 's digit value (0-9), is colored according to W2's s color value (0-1), and is reweighted according to the latent T. Then Y the same original MNIST image as X, with the same thickness but colored according to the latent C. Further, for every edge in the graph, we include random noising process: with
30

---

probability 0.1, the information passed along is chosen in uniformly random manner from the valid range.
Here we describe the data-generation procedure, and training setup for the Napkin-MNIST experiment in full detail.
F.2.1 Data generation procedure: discrete case
As a warm-up, we outline the generation for the NapKin-MNIST dataset a low-dimensional setting. When we consider in the next section the high-dimensional case, we simply replace some of these discrete variables with MNIST images which can be mapped back into this low-dimensional case,
We start by enumerating the joint distribution and the support of each marginal variable. First lets define the sets
COLORS := {red, green, blue, yellow, magenta, cyan].
RG COLORS- := (red, green).
THICKNESSES = (thin, regular, thick).
DIGITS == 9).
And then the definitions and support of each of the variables in distribution:
(Latent) Color COLORS.
(Latent) Thickness E THICKNESSES.
W1 DIGITS COLORS THICKNESSES
W2 DIGITS COLORS.
X E DIGITS x COLORS x THICKNESSES
. Y DIGITS x COLORS x THICKNESSES
Now we describe the full data generation procedure, key hyperparameter is noise-probability This defines the probability that any variable flips to uniform probability. To ease notation, we define the function (u, 5) defined as
with probability P otherwise
(v, S)
U(S)
and define the mapping COLORS WESTRICTED_CCOLOO
red i E (red, green, blue)
R(c) =
green otherwise
Where U(S) means a uniformly random choice of S. Then our data generation procedure follows the following steps:
Color = (CCOLORS)
Thickness = U(THICKNESSES)
W1 (U(DIGITS), 7p (Color, COLORS), 1pp (Thickness, THICKNESSES)
W2 = (1p DIGITS), 1pp (R((W 1-cooor Roo.oooo......CLORRRR RG_COLORS))
.
X (12 di.. DIGITS), 22 ROLOOL) (Thi THICKNESSES)
Y == (1Ip (X (X-iigit: DIGITS), (Color COLORS),
1/p (X ([XXthickess:: THICKNESSES))
It is easy to verify that this describes the Napkin graph, each only Color, Thickness latent and each variable only depends on its parents in the SCM.
Secondly, observe that this structural causal model is separable with respect to digits, colors, and thicknesses. Since each digit only depends on parent digits, each color only depends on parent colors, and each thickness depends only parent thicknesses, these can all be considered separately.
31

---

Color
Thickness
4 6
W
...
C 1 6 9
5 5
6
/
6
7 5
C y 6 9
6
/
6
9
8
3 7
Figure 15: Joint samples from the Napkin-MNIST dataset: Samples from the Napkin-MNIST dataset are visualized columns above. The first row indicates the latent variable color, the second row indicates the latent variable thickness, and the row labeled W2 is a discrete variable holding a (color, digit), where digit is represented as the number of dots. Notice that the noising process sometimes causes information to not be passed to children.
Further, because this distribution is only supported over discrete variables, exact likelihoods can be computed for any conditional query. This is much more easily done programmatically however, and we provide code in the attached codebase to do just that. We will claim without proof that in the case of thicknesses and digits, Py (X) PX) However in the case of colors, (X) X( Hence we consider this case in the evaluations in the experiments section.
F.2.2 Data generation procedure: high-dimensional case
The high-dimensional case follows the discrete case of the Napkin- -NIIST dataset, with a few key changes. Namely, Wi, X, and Y are MNIST images that have been colored and thickened. We explicitly outline these changes:
W1: A random MNIST image of the provided digit is used, then colored and thickened accordingly (noisy from latents).
W2 This is a discrete variable, only encoding the (noised) digit and (noised) restricted color of Wi.
X: This is random MNIST image of the (noised) digit obtained from W2, then colored with the (noised) restricted color from W2 and thickened according to the (noised) latent thickness.
Y: This is the same base image of X, unless the noising procedure calls for a change in digit, then a random MNIST image of the specified image is used. The (noisy) color is obtained from the latent distribution, and the (noisy) thickness is obtained from X.
To color the images, we convert each I-channel MNIST image into a 3-channel MNIST image, and populate the necessary channels to generate these colors. Note that in RGB images: if only the RG channels are active, the image is yellow; if only the RB channels are active, the image is magenta; if only the BG channels are active, the image is cyan. To thicken the images, we use the MorPhoMNIST Castro et al. X package2] Operationally, we generate base dataset for our experiments of size equivalent to the original MNIST dataset. That is, the training set has size of 60K, and the test set has size of 10K. Because we have access to the latents during the data generation procedure, we are able to train classifiers for each variable to identify their digit, color and thickness. We use simple convolutional network architecture for each of these cases and achieve accuracy upwards of 95% in each case.
F.2.3 Diffusion training details
We train diffusion models during our sampling procedure, and we discuss each of them in turn.
To train a model to sample from P.X, Y W1,, W2), we train a single diffusion model over the joint (x, Y) distribution, i.e., 6 channels. We train standard UNet architecture where we follow the
32

---

conditioning scheme of classifier free guidance. That is, we insert at every layer an embedding of the Wi (image) and W2 (2-dimensional discrete variable). To embed the W, image, we use the base of 2-layer convolutional neural network for MNIST images, and to embed the W1 we use standard one-hot embedding for each of the variables. All three embeddings are concatenated and mixed through 2-layer fully connected network reach final embedding dimension of 64. Batch sizes of 256 are used everywhere. Training is performed for 1000 epochs, which takes roughly 9 hours on A100 GPU's. Sampling is performed using DDIM over 100 timesteps, with a conditioning weight of W 1 (true conditional sampling) and noise o = 0.3.
To train a model to sample Y from the generated dataset (W2, XX, Y), we follow an identical scheme. An option is to train single diffusion model for each choice of W2 in our synthetic dataset, however, we argue our schema still produces correct samples because: 1) W2 can be arbitrarily chosen, and thus should not affect , 2) we argue that the model fidelity benefits from weight sharing across multiple choices of W2, 3) the model is only ever called with a specified value of W2 so we always condition on this W2.
F.2.4 Extra evaluations
As such, our evaluations include verifying the quality of the trained component neural networks and, in the case of MNIST, surrogate ground truth for discrete version of the dataset. In each experiment, for conditional sampling with high-dimensional data, we train diffusion models using classifier-free guidance 21]] For conditional sampling of categorical data, we train classifier using cross-entropy loss. addition to the evaluations presented in the main paper, we can further perform evaluations the component models necessary sample (Y)
P(X, Y W2): We can evaluate the model approximating samples from P(X, Y W2) on deeper level than just visual inspection as provided in the main paper. In particular, assuming access to good classifiers that can predict the digit, color, and thickness of an MNIST image, we can compare properties of the generated images with respect to the ground truth in the discrete case. For example, assuming we have hyperparameter random noise equal to p, we can compute the following quantities analytically the discrete dataset as:
P[Xa = W2a] = - p + 10
.
P[X.. W2el = =1 - p + 10
P[X, = Wid = (1 - p + )2 + (g)2 2
P[Ya = Xa] = + p
P\Y. Wid| (1 p 2) + big(g))**
. P(Y x,) = 1 +
where Va, Ve, V. refer to the digit, color, and thickness attributes respectively. These calculations follow from two formulas. In discrete distribution with support S and [S] = K:
P(yp(2, 2] p k
. P(pp(2, 12, S)] p R2+ + (R) (K -)
where in the second equation, is assumed that ()) are two independent noising procedures.
Then to evaluate, we can 1) consider a large corpus of joint data, 2) run each of W1. Y through classifier for digit, color, and thickness, 3) evaluate the empirical estimate of each desired probability. We present these results for the synthetic dataset Daynth sampled from the diffusion model approxi- mating P(X, YW W2), dataset Dorig generated according to the data generation procedure, and Ptrue the true analytical probabilities. These results are displayed in the Table
33

---

Table Color probability distribution the sampled images. Ply.c/x.c = red) is biased towards
[R. G, B] while Ply.c/do( (2. red)) is not.
Predicted color probabilities
Red
Green
Blue
Yellow
Magenta
Cyan
Conditional model: P(y.clr.c = red)
0.1889
0.4448
0.1612
0.1021
0.0232
0.0798
Ours: Ply.c do.... red))
0.1278
0.2288
0.2097
0.1445
0.1177
0.17I5
Original
Star44
ESDSE
Figure 16: Samples from P(Izldo(Male 0). Row 1: original I,, Row 2: translated I2 by StarGAN and Row3: translated I2 by EGSDE.
Table 1: Evaluations on the Napkin-MNNI MNIIST generated dataset. V.d, V.c, V.t refer to the digit, color and thickness respectively of variable V. The first column with respect to samples generated from diffusion model P(X, W1, W2), Image Data is the dataset used train P. Discrete Data is the empirical distribution according to discrete Napkin-MNIST and the ground truth is analytically computed. Ideally all values should be equal across row. While our synthetic dataset generated from p is not perfect representation, is quite close all attributes except thickness. This is because the classifier for thickness has some inherent error in it, as evidenced by the mismatch between the base data and ground truth in the thickness rows.
P(Y, X WT, W2)
Image Data
Discrete Data
Ground Truth
P[X.d W
0.931
0.895
0.909
0,910
P[X.c W2.c
0.964
0.950
0.950
0,950
P[X.t = Wi.
0.683
0.776
0.879
0.873
P(Y.d X.d
0.927
0.895
0.909
0,910
P[Y.c Wi.c
0.847
0.841
0.841
0,842
P[Y.: = X.I]
0.830
0.851
0.933
0,933
F.2.5 MNIST baseline comparison
Here we provide some additional information about the baseline comparison we showed in Section 4.1 Although the implementation of the baseline methods different, we considered the performance of each method after rumning them for 300 epochs (until good image quality is obtained). The NCM algorithm took around hours complete while our algorithm took approximately 16 hours to complete. We observe the probabilities in Table 2
In the generated samples from the conditional model, the color of digit image X and digit image Y are correlated due to confounding through backdoor paths. For example, for digit image X with color as red, Y takes value from [Red, Green, Blue] with high probability. Thus, the conditional model does not have the ability to generate interventional samples. On the other hand, our algorithm generates samples from the interventional distribution (zy)do(z)) and the generated samples choose different colors:[Red, Green, Blue, Yellow, Magenta, Cyan] with almost the same probability. Therefore, our algorithm shows superior performance compared baselines and illustrates the nigh-dimensional interventional sampling capability.
F.3
CelebA experiment
Here we provide some additional information about the CelebA image to image translation experiment showed in Section 4.2 We used pre-trained classifier from this repository: https: ///g/tttn github

---

Figure 17: Multi-domain image translation by EGSDE. Images in and 3 are the original images (male domain) and images in rows 2 and 4 are translated images (female domain). Table shows, 29.16% of the total images are translated as young person.
Orignna
Starbva
origona
Figure 18: Performance comparison of multi-domain image translation between StarGAN and EGSDE.
com/ c.lementapp/Ceeebaacces Attributtsstttttteeeeeeeeeeeeeet Atttributee__aaassfficatttt Classification We use the pre-trained model of EGSDE from this model of StarGAN from this Saaptt/ss//gtttbb.ooooo/a//AA/A 850 samples withe these models. Note that although EGSDE generates better quality (blur-free) images compared to StarGAN (blurry). it makes its samples more realistic by adding different non- causal attributes such Young or Attractive. As result, it will generate Female samples from specific part of the image manifold. As a result, it might lack the ability to generate a different variety correlation in the dataset
We use the pre-trained We generated
of images (for example: old females). The reason behind this possibly it is trained on. For example, in the CelebA dataset, we observe a high correlation between Female and Y l'ung while also high correlation between Male and Old. As result, when EGSDE converts the images whether the Male is young or old, it converts them to young female. On the other
35

---

Table 3: Additional attributes added (in percentage) in the translated images. ArchedEycbrows
WearingLipstie HcavyMakeup
OvalFFace
Attractive
Young
Added Attribute (%)
87.5
79.16
66.66
54.16
37,5
29.16
Figure 19: Samples from "(I2Youung 1, do Male 0)) generated by StarGAN.
Distrihution Matched
Models
4
MN
P(N)
0.0395
ME
P(E R.
0.0512
MA
P(A[N, E, R))
0.0451
ML
PULIN, E.A) A.
0,042
P(X E,
LFre-
trained
Figure 20: CXR conditional models in the sampling network.
hand, starGAN does not change noncausal attributes, but it also does not properly change the causal attributes as well.
Existing algorithms such as Xia et al. [59, Chao et al. [1] do not utilize the causal effect expression of (AA do(Male = 0)), rather they train neural models for each variable in the causal graphs which will be costly in this scenario. Even if they utilize pre-trained models, they have to train models for the remaining variables to match the whole joint and perform sampling-basee methods to evaluate P(A) Young 1, do(Male 0)). Note that the ground truth if our considered attributes are causally related with an image a Fermale unknown. We assume that they are causal in the CclebA dataset based expert knowledge (26].
Baselines:
F.3.1
Conditional image generation
In Figure we show images generated from conditional interventional distribution.
19
E4
Explaining foundation model output for chest X-ray generation
F.41
Causal graph
Below we shortly describe the reasoning for our causal graph assumption.
Nodes and Edges:
Prompt/Report (R) is variable that represents textual input. In CXR datasets such as MIMIC- CXR, it is generally doctor written text report indicating the existence of different attributes. In our experiment, we consider as an input prompt feed by some user who interested to understand how the prompt affects other attributes and the CXR image generation. We
36

---

assume that the LLM labeler can extract effusion (E) and atelectasis (A). Thus, we consider E and R. - A.
Pneumonia(N) is an infection that inflames the air sacs in or both lungs 31
Pleural effusion (E) happens when fluid builds up in the space between the lung and the chest wall. This can occur for reasons such as pneumonia or complications from heart, liver, kidney discase. Pleural effusion also involves fluid in the lung area. It is common in patients who develop pneumonia. At least 40-60% of patients with bacterial pneumonia will develop pleural effusion of varying severity. Thus, we consider N + E.
Atelectasis (A) is one of the most common breathing complications after surgery. Here, complication is a medical problem that occurs during a disease, or after a procedure or treatment 13 External pulmonary compression by pleural fluid or air (i.e, pleural effusion, pneumothorax, etc.) may cause atelectasis
[3 Thus, we consider the edge -> A. Also, Therefore,
various types of pneumonia, which is a lung infection, can cause atelectasis 32 we consider the edge N -> A.
Lung opacity (L) is lack of transparency, i.e., an opaque or non-transparent area on an x- x-ra//radiograhh [88]. We consider pneumonia, effusion and atelectasis to be causes for lung opacity. Thus we consider edges N + L. L, A L.
Xray Image (X) represents different attributes such atelectasis, effusion and lung opacity in visible radiography form. Thus, we consider edges E + - -X X.
N represents the presence of hidden confounder between pneumonia and the input prompt/report. We followed [9] to consider the hospital location as the unobserved variable from where the chest x-ray datasets where collected. They discuss that when we merge/mix different datasets and train models on that different types of biases example, in many scenarios, most patients are screened in certain health services and highly suspicious patients are derived to different area. Another example of introducing bias is, when we aim increase the number controls or cases, we expand the dataset with samples coming from significantly different origins and labeled with unbalanced class identifiers. Since we utilized foundation models such as [16 10] which are trained in multiple chest radiograph datasets, they are prone to above kind of bias. In some cases, to discriminate between cases and controls, these models might depend on the origins/location based details rather than finding actual features related the disease. Thus, consider that hospital locations have some effect on the text report on which the models were trained on and also how likely people were affected by pneumonia infection in those areas. Therefore, we consider the presence of a confounder between the text report (Rt and pneumonia (N).
introduced. For
Note that all our results are based on the MIMIC-CXR dataset and our assumption on the causal graph. Thus, our results should not be used to make medical inferences without an expert opinion. In the real world, the Nomain-specifii causal graph might change. Our algorithm can be applied on the domain specific causal graph and the dataset to obtain correct results.
F.5
Covid X-Ray dataset
F.5.1
Data preprocessing
Note that full pipeline of data preparation is contained in provided codebase. We start by downloading the corpus approximately 30K Covid X-Ray images" Then we download Covid-19 labels and Pneumonia labels") and attach labels to each image. Then we convert each image black-white one-channel images and rescale each to a size of (128 x 128) pixels. Finally, random split of the 30K images is performed: keeping 20K to be used during training, and 10K be used as validation images. Note that the labels come with set of 400 test images, but 400 images is too small to be an effective test set (say for FID computations). We will be explicit about where we use each data set.
in the
37

---

Pneumonia And Left Effusion And Lung Opacity 25.58 z: 1.37%
Pneumonia And Left Effusion And Ate lectasis 21.54 r 1.09%
Pneumonia And Left Effusion 44,68 1.4%
Baseline:
Healthy
Original Prompt
Left Pleural Fluid Persists
w/ do
Effusion And Left Atelectasis 22.35 1.23%
Left Atelectasis And Lung Opacity 22.24 1.42%
Baseline:
Left Atelectasis 43.17 1 1.29%
Healthy
Original Prompt
Atelectasis At The
Left Lung Base
w/ do(N- 0)
Figure 21: Two prompts, their CXR images, and inferred attributes with the likelihood to appear. Blue regions indicate changes compared to the healthy X-ray.
E55.2 Diffusion training details
We train difffusion model approximate (C) To do this, we use a standard UNet and classifier- free guidance scheme. We train for 100K training steps over the 20K-sized training set, using batch size of 16. This takes roughly 10 hours on a single A100. The same classifier-free guidance parameters are used as in the NapKinMNIST diffusion training.
F55.3 Calibrated classifier training details
To train a classifier to sample from P(NIC, X), we note that our inputs are (1, 128, 128) image and binary variable. Our architecture final linear layer of a ResNet18 to have output dimension 64 (vs 1000), and modify the input channels to be 1. We create an embedding for N by using a standard embedding layer for binary variables, with embedding dimension 64. These embeddings are then concatenated and pushed through 3 fully connected layers with ReLU nonlinearities and dimension 128. final fully-connected layer with outputs is used to generate logits.
follows: we create an embedding of X by modifying the
Training follows a standard supervised learning setup using cross entropy loss. We train for 100 epochs using a batch size of 256 and a standard warmup to decaying learning rate (see code for full details).
We note the deep literature suggesting that even though classifiers seek to learn P(N C), neural networks trained using cross entropy loss do not actually do very good job of estimating this distribution. Training attains an accuracy of 91.2% on the test set. Calibrated classification seeks to solve this problem by modifying the network in way such that it more accurately reflects this distribution. We follow the standard approach using temperature scaling of the logits, where temperature is learned over the validation set, using LBFGS with learning rate of 0.0001 and maximum of 10K iterations. This does not affect the test accuracy at all, but drastically improves the ECE and MCE reliability metrics. See Guo al. [7 for further discussion of temperature scaling.
F.6 Covid X-Ray dataset
Data generation:
Next we apply our algorithm real dataset using chest X-rays on COVID-19 patients. Specifically, we download a collection of chest X-rays (X) where each image has binary
labels for the presence/absence of COVID-19 (C), and pneumonia (N) 57 We imbue the causal
Labels are www.l//wwwwtttooooo.o........coe
38

---

PCD 1.05
Figure 22: Reliability plots for P(N[C,X): Reliability plots which overlay accuracy versus classifier confidence. (Left) Reliability plot for P(N|C, X) without calibration. (Right) Reliability plot for P(NIC, X) with temperature scaling calibration applied.
Deal
(C41)
(C41)
Figure 23: Generated Covid XRay Images: Generated chest XRay images from our diffusion model, separated by class and compared against real data.
Table 4: (Left) Class-conditional FID scores for generated Covid XRAY images (lower is better).
Generated = c, means we sample from the diffusion model conditioned on c. Real (C c) refers to held out test set of approximately 5k images, partitioned based on C-value. Low values on the diagonal and high values on the off-diagonal imply we are sampling correctly from conditional distributions. (Right) Evalution of Interventional Distribution Pn(n). We evaluate the distributions Po(n = 1) for three cases for the XRAY dataset. Diffusion uses a learned diffusion model for P(ælc), No Diffusion samples P(z|c) empirically from held out validation set, and no latent evaluates the conditional query assuming no latent confounders in the causal model.
FID
Real: C Real: C'
Pa P 1)
c=
Diffusion
0.622
0.834
Generated:
61.29
No Diffusion
0.623
0.860
Generated C
101.76
23.34
No Latent
0.406
0.951
structure of the backdoor graph, where C C' N, and there a latent confounder affecting both C and N but not X. This may capture patient location, which might affect the chance of getting COVID-19, and quality of healthcare affecting the right diagnosis. Since medical devices are standardized, location would not affect the X-ray image given COVID-19. We are interested in the interventional query (n): the treatment effect of COVID-19 on the presence pneumonia.
Component Models: Applying ID-GEN to this graph requires access to two conditional distributions: P(x(c) and P(n|æ,, c). Since is high-dimensional image, we train a conditional diffusion model to approximate the former. Since N is binary variable, we train classifier that accepts X, C and returns a Bernoulli distribution for N. The generated sampling network operates by sampling an X given the interventional C. and then sampling an auxiliary C" P(c') and feeding X, C' to the classifier for Pnn|., c). finally sampling from this distribution.
Again we do not have access to the ground truth. Instead, we focus on the evaluation of each component model, and we also perform an ablation on our diffusion model. We first evaluate the image quality of the diffusion model approximating P(x|e). We evaluate the FID of generated
Evaluation:
39

---

samples versus held-out validation set of 10K X-ray images. When samples are generated with C taken from the training distribution, we attain FID score 16.17. We then evaluate the conditional generation by comparing class-separated FID evaluations and display these results in Table 4 (left). The classifier estimating P(n) x., c) has an accuracy of 91.9% over validation set. We note that we apply temperature scaling [17] to calibrate our classifier, where the temperature parameter is trained over a random half the validation set. Temperature scaling does not change the accuracy, but does vastly improve the reliability metrics; see Appendix.
Finally we evaluate the query of interest (n( Since we cannot evaluate the ground truth, we consider our evaluated (2(n) versus an ablated version where we replace the diffusion sampling mechanism with P(z]c), where randomly select an X-ray image from the held-out validation set. We also consider the query (.(n) if there were no latent confounders in the graph, in which case, the interventional query (n) is equal to P(n c). We display the results in Table (right).
G Pseudo-codes
Algorithm 5 IDC-GEN X, G)
1: Input: target Y. intervention X conditioning set Z. training data D, G.
2: Output: DAG of trained model to sample from Px (Y)) (Z).
3a E Z such that (Y IL a X, Z {a))oz.. then
return IDC-GEN (Y, [a]: z (a).D,
5: else
6:
ID-GEN (Y U X,X, x 0, Ô C)
7:
D' ~ (X)
H. 0
9:
Add node (X, 0) and (z,0) to H3
10:
Let My he model trained on {V, X, 3 ~ D' such that My(X,2) ~ P'(VXX, My
(Y
11:
Add node (Y, My) H₂
Add edge X Y and Z Y to Hz
13:
return H2
Algorithm ID(y, x, P.C)
I: Input: y. x value assignments distribution P.
2: Output: Expression for (y) in terms of P or F").
3: x = 0 then (Step:1)
4: Return Σ Evyy P(v)
5: VV YY * / then (Step:2)
6: Return 1D(y, Xn An(Y)c. Jc Evaant P, AA()))
7: Let W W==( (VV X An(Y) Gx (Step:3)
8: if W then
9: Return UU P,
10: if (TC( \ X) =
then (Step44
Return Evxrrxx) II. ID(s,, s. G.G)
12: CCG X) (S) then
CCG) = (G) then (Step:5)
14:
Return FAIL GnS) S E C(G) then (Step:6)
15:
Return Eay I[a,es (1
ir'S' s.t. S S' CG) then (Step:7)
18:
Return x Il INIEE P(VIV ( ((-))) (()
40

---

Algorithm 7 IDCCY, XX, G,
Input: z, y, 2 value assignments, P probability distribution, causal diagram (an I-map of P).
2: Output: Expression for Px (Y 2) in terms of P or Fail(F, F*).
3: if 3a E Z such that (Y IL al X, Z {o))or.. then
4: return X U [a). Z \ (,)
5: else
let P' = -ID Uz, x, P.
return Ey P
41

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer:
[Yes]
Justification:
We clearly enlist our claims and contributions in the abstract and in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We properly acknowledged our limitations in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
42

---

Justification:
We provide all our theoretical claims and their proofs in Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[Yes
Justification:
We discuss reproducibility in Appendix F.I.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
43

---

Answer: [Yes]
Justification:
We provide our source code as supplementary material and also as github
repo.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public/guides otwp:///eeoooouuubbsssssoooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wttgg/udde//eosuuueeeeeeeee ssinoolicyy for more details.
//nips.co
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide the experimental details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provided appropriate information about the statistical significance of the MIMIC-CXR experiment Section 4.3 We did not provide such evaluations for the Colored-MNIST or CelebA datasets, as we obtained consistent results in multiple runs. We plan run our algorithm on larger datasets and will include error bars in those experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
44

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss about the computing resources for each experiment in Appendix F.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttgs///euriig..cc/pul/////t/t
Answer: [Yes]
Justification: Yes, our research in the paper conform, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact of our work in Appendix B
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
45

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cited papers of all models, datasets we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
46

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "8o7Su9m1hS", "reviewer": "Reviewer_6o9W", "summary": "This paper provides an algorithm for sampling from a causal interventional distribution using conditional generative models, building on Shpitser and Pearl's ID algorithm. They discuss how their algorithm, ID-GEN, can sample from any identifiable distribution given a causal graph, and handles the presence of unobserved confounders (when identifiable). Empirically, they demonstrate their method can work for measurement, evaluation and interpretability purposes in the challenging setting where both the target and treatment are high dimensional e.g. images.", "questions": "- would be great to see clarification throughout Section 3, particularly in the highlighted areas and around merging and training data sampling\n- experimental setups all need clarifications\n- generally assuming I understood what's happening better I'd be happy to increase my score", "limitations": "n/a", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- interesting work, the case of high-dimensional variables in a causal graph is a super important and under-discussed one\n- thorough theoretical treatment of the extension of the ID algorithm\n- experiments show a nice range of usages of the suggested ID-GEN approach", "weaknesses": "- I frankly got a bit lost in a few key parts of Section 3. I got the main ideas (I think) but missed a bunch of nuance. Some spots were: Example 4.1 (I don’t understand why ID fails in this case but ID-GEN succeeds - specifically the importance of merging is a bit lost on me), Step 4 of ID-GEN (again, merging), and Step 7 of ID-GEN (I think the logic around how training data is sampled, used and modified wrt the graph needs to be explained more clearly)\n- Step 1 of ID-GEN confuses me - I don’t see why we can’t just learn a model of P(y) directly in this case? Also the 2nd equality in 203 doesn’t make sense to me - how is the sum over values of v equal to P(y)?\n- in each experimental section I find I have at least a medium-sized point of confusion around the setup or evaluation - more care should be taken to explain empirical setup + results overall\n- In 5.1, the authors state that U_color makes W1 and Y correlated by color - however, X contains color information and is a direct ancestor of Y, so this unobserved confounding seems trivial\n- in 5.1, it seems like a better metric than a classifier (which may be unreliable and as you note isn’t useful for all possible images) would be something based specifically on the RGB values of the pixels themselves\n- in 5.2, I don’t quite see why Young & Male have unobserved confounding - are they not fully determined by the shared + observed parent I_1?\n- in 5.3, I don’t understand why the report is a causal ancestor in the graph - isn’t it generated upon viewing the X-ray?\n- in 5.3, I think the setup with the labeller can be made clearer - how good is this labeller? How is it structured? Additionally, is the bottom row intended to be a success or failure examples? (label says it should be right lung base but all inferences name the left lung)\n\n\nSmaller points:\n- L115: are unobserved confounders only allowed to affect 2 variables in this framework? Is that more limiting than general SCMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable efforts on our paper. \nWe are happy to receive their appreciation for our theoretical contribution and experimental setup.\nBelow we address their concerns.\n\n> Step 1 of ID-GEN ... why we can't just learn a model of P(y) directly? ..., how is the sum over values of v equal to\n> P(y)?\n\nThe reviewer is correct that if the intervention set is empty, we can learn a single model $M$ that can sample the\nvariable set $\\mathbf{Y}$ from\n$P(\\mathbf{Y})$, specially if all variables in $\\mathbf{Y}$ are low dimensional. However, we consider that $\\mathbf{Y}$\nis allowed to contain both low and high-dimensional variables.\nRahman and Kocaoglu [40] show that it is non-trivial to achieve convergence if we directly attempt to match such joint\ndistributions $P(\\mathbf{Y})$.\n\nIn line 203, we stated that $P(y)= \\sum_{\\mathbf{v} \\setminus y} P(\\mathbf{v})$. Here, $\\mathbf{V}$ is the set of all\nvariables and $Y$ is the target variable.\nPlease note that we do not sum over $\\mathbf{V}$ but rather over $\\mathbf{V} \\setminus Y$.\nIntuitively, we first consider all variables and then drop (sum) the unnecessary ones.\n\n\n> more care should be taken to explain empirical setup + results overall\n\nWe apologize for the confusion and will bring the Napkin-MNIST, CelebA, and Xray generation empirical details from the\nappendix to the main paper.\n\n\n> In 5.1, the authors state that U\\_makes W1 and Y correlated by color - however, X contains color information and\n> is a direct ancestor of Y, so this unobserved confounding seems trivial.\n\n\nWe humbly point out that even though image X (takes R and G) is a direct ancestor of image Y (takes all 6 colors),\n$Y$ **only inherits the digit property** from $X$ but correlates the color property with image $W_1$. This color\ncorrelation between $W_1$ and $Y$ is created by U\\_color.\n\n\n> It seems like a better metric than a classifier, ..., would be something based specifically on the RGB values of the\n> pixels themselves\n\nAs we are not doing counterfactual generation, we should not match against any specific image.\nOur goal is to generate images that are correct at the population level.\nFor example, in Fig 2 (napkin-MNIST), if we generate 1000 images from the interventional distribution,\nthere should be around 167 images for each of the 6 colors (Pr=0.17). Since the output is not deterministic, we can not\nevaluate them with pixel-wise RGB values.\n\n\n> in 5.2, ..., why Young & Male have unobserved confounding - are they not fully determined by the shared + observed\n> parent I_1?\n\nIn the CelebA dataset, men are more likely to be old (correlation coeff=0.4, (Shen et al [44])). As a result, any\nclassifier trained on the CelebA, will\nnot have 100% accuracy at the population level and might have some bias toward predicting young-male images\nas [Old, Male].\n\n\n> in 5.3, ..., why is the report a causal ancestor in the graph - isn't it generated upon viewing the X-ray?\n\nWe followed the setup discussed by Chambon et al [10] where they generate synthetic X-ray images from prescription\nreports\nwith a vision-language foundation model. We aim to make the model's predictions interpretable and invariant with domain\nshifts.\nThus, we follow the causal mechanisms of the environment the model will be deployed in to build the causal graph.\nTherefore, we consider the same initial input and final output as the model, having the report variable as an ancestor\nof the X-ray image.\n\n\n> ..., the setup with the labeller can be made clearer - how good is this labeller?\n\nWe skipped details on the X-ray report labeler as it is taken from Gu et al [16] and not our contribution.\nHowever, we will add more details. In short, it is a BERT architecture trained on pseudo-labels created by GPT-4\nto classify findings in CXR reports. It achieves the best or the second-best F1 score compared to its baselines.\n\n\n> The label says it should be the right lung base but all inferences name the left lung.\n\nWe apologize for the mistake and have fixed it. The label should have said, \"Atelectasis at the **left** lung base\". Top\nand bottom, both rows are simulations of how ID-GEN is more useful compared\nto the direct application of baseline LLM.\n\n> L115: are unobserved confounders only allowed to affect 2 variables in this framework? Is that more limiting than\n> general SCMs?\n\nWe consider the commonly used semi-Markovian SCM where unobserved confounders only affect two variables. A more relaxed\nassumption is when hidden confounders\ncan affect any number of variables, known as the non-Markovian SCM.\n\n\n> Step 4 of ID-GEN (again, merging), and Step 7 of ID-GEN,...,needs to be explained more clearly\n\nWe apologize some parts of our paper were hard to understand. We will elaborate the definitions in Section 3 and add\nmore discussion on the merging in Step 4 and the parameter\nupdates in Step 7.\n\n\n\n> Example 4.1 ... I don’t understand why ID fails ... but ID-GEN succeeds. ...\n\nID can't deal with high-dimensional variables. However, if we imagine an algorithm that follows ID step-by-step while\ntraining a generative model for every factor it encounters,\nthat approach might fail at step 4. Step 4 factorizes the query into multiple factors and this algorithm would not know\nwhich factor to learn and sample first with the generative models.\nIn some cases, there exists no such sampling order of the factors at all (cyclic issue).\n\nID-GEN avoids sampling factor by factor, and first trains required models for each of them, considering all possible\ninput values.\nAfter the recursion ends, it disintegrates the variables in the factors and treats their model as individual nodes.\nA sampling network is built connecting these model nodes considering their input-outputs. This network now enables\nconsistent interventional sampling.\n\nWe will be happy to provide further clarification for any concerns of the reviewer. We would highly appreciate if the reviewer reconsiders a stronger score for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NBUnsljdIS", "reviewer": "Reviewer_5UTo", "summary": "This paper studies the problem of sampling from an interventional distribution of high-dimensional data, where the causal relationships are described by a acyclic directed mixed graph (ADMG). Motivated by the ID algorithm that provides a recursive way of identifying any causal effect from a conditional probability table, the authors propose ID-GEN that follows the recursive steps of ID but instead trains deep generative models to fit the conditional distributions. The final sampling model is then obtained by connecting all the trained networks together in some suitable way. The authors prove that ID-GEN is sound and complete, and run extensive experiments on both synthetic and real dataset to demonstrate the effectiveness of their approach.", "questions": "1. In some identification formula e.g. Eq.(1) in the paper, the probability on the denominator might be small. To what extent would this affect the stablity of the proposed algorithm.\n\n2. Can your algorithm be straightforwardly adapted to the case of imperfect interventions i.e. some conditional probabilities in the structural causal model are modified, but no causal edges are removed?", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies the problem of sampling from an interventional distribution, an arguably important problem with broad applications. Current approaches, as the authors point out, are either restricted to simple causal graphs or face computational challenges.\n\n2. Most parts of the paper are clearly written, and sufficient explanations are provided for the key steps in the ID-GEN algorithm. Also, simple examples are provided that help with the understanding of the paper. The paper is also well-organized and the authors put most complicated details into the Appendix.\n\n3. The authors conduct extensive experiments to demonstrate the superior performance of their model, by comparing with other sampling models proposed by previous works.", "weaknesses": "I don't think this paper has obvious weaknesses. One thing that the authors may wish to improve is that the notations are a litle bit complex; and it would be better to more often remind the authors of their meanings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their effort on our paper. We are really happy that they found our work broadly applicable,\nour paper well-written and our experiments extensive. Below, we address their concerns.\n\n\n>In some identification formulas e.g. Eq.(1) in the paper, the probability on the denominator might be small. To what\n> extent would this affect the stability of the proposed algorithm.\n\n\nA fraction in ID-GEN is generally created when we traverse the recursion and reach any point where we have to estimate\na conditional distribution $P(Y|Z)$. This $P(Y|Z)$ can be written as $\\frac{P(Y,Z)}{P(Z)}$ and estimated from the joint distribution.\nThus, in such an expression if the denominator $P(Z=z)\\sim 0$, it implies that we have a small number of samples for $Z=z$. This will result in a high variance for the estimation of $P(Y|Z=z)$. \n\nID-GEN trains generative models to learn and sample from $P(Y|Z)$.\nThus, for $Z=z$, the trained models might have low prediction accuracy or low image generation quality when sampled from the final interventional distribution.\nHowever, for other values $Z=z'$ where $P(z')$ is not close to zero, the trained models in ID-GEN shows good empirical performance (for example: Figure 2)\nand does not affect the over all stability of the algorithm.\n\n\n\n\n> Can your algorithm be straightforwardly adapted to the case of imperfect interventions i.e. some conditional\n> probabilities in the structural causal model are modified, but no causal edges are removed?\n\nWe thank the reviewer for such an interesting question. \nFor causal effect estimation in high-dimension, we follow the ID algorithm [1] to consider hard intervention $do(X=x)$, i.e., fixing a specific value to the intervened variable,\nand assume that the rest of the conditional distribution stays unchanged.\n\nHowever, there are recent algorithms such as [2] that \ndeals with **imperfect/soft intervention** where the underlying mechanism of intervention $X$ changes instead of fixing to a specific value.\nNote that, these algorithms deal with low-dimensional variables. \n\nSince [2] also utilizes the identification algorithm, in principle our algorithm could be adapted for such cases. However, it might require\nmore careful handling, and we will address it as our future works.\n\n\n[1] Shpitser et al. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9:1941–1979, 2008.\\\n[2] Correa et al \"A calculus for stochastic interventions: Causal effect identification and surrogate experiments.\" AAAI, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5E1qQuqv9G", "reviewer": "Reviewer_fUEc", "summary": "This paper proposes an algorithm for sampling from intervention distributions under known causal models with hidden confounders, using conditional generative models.\nThis allows nonparametric distributional estimation of high-dimensional outcomes such as X-ray image generation, unlike existing methods.\nThe proposed method combines the existing ID algorithm with a generative model and inherits the ID algorithm's theoretical guarantees. That is, non-identifiable quantities are indicated to be non-identifiable, while all identifiable quantities can be estimated.", "questions": "[Q1] Related to W2, is the proposed method a non-trivial novelty in situations where there are no unobserved confounding variables, i.e. no bidirectional edges?\n\n[Q2] Related to W3, are there any non-trivial points in theoretical guarantees when the generative model is combined with the ID algorithm?", "limitations": "The method is limited to the cases where the causal direction is known.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "* They shed light on the new problem setting of causal simulation for outcomes with high-dimensional and multimodal distributions, such as image generation. This could open up new applications if well justified. Such a problem setting requires a very different approach to point estimation of expectations for low-dimensional, unimodal distributions.\n* The theoretical background is solid and well-explained. The method is proved to be able to estimate all identifiable quantities and otherwise outputs \"unidentifiable.\"", "weaknesses": "[W1] The motivation for high-dimensional distribution estimation is weak. For example, it does not seem very meaningful to me to generate synthetic X-ray images.\n\n[W2] In particular, is it important in cases where there are bidirectional edges due to the presence of hidden confounding factors, but where the causal orientations are all identified among variables? A clear comparison with similar methods would be beneficial for readers, e.g., comparison in assumptions and targets (e.g., parametric/nonparametric, latent confounder, distributional estimation, etc.).\n\n[W3] The base procedure seems to come from existing methods, such as the ID algorithm, and they just combine it with a generative model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their efforts and useful feedback. We are happy that they found our work important and our\ntheoretical background solid. Below we address their concerns.\n\n\n>Is it important in cases where there are bidirectional edges ..., but the causal orientations are all\n> identified ...?\n\nOur considered setup is well known as the semi-Markovian causal model in the literature.\nHowever, all causal orientations can not be identified in general, and we obtain Markov equivalent graphs. There exist\nsome recent works [1] that can estimate specific causal effects in such scenarios.\nThus, one could draw connections between [1] and ID-GEN to perform high-dimensional causal inference in the presence of\nuncertain causal orientation. \n\n[1] Jaber et al. \"Identification of conditional causal effects under Markov equivalence.\" NeurIPS 2019.\n\n> Is the proposed method a non-trivial novelty in situations where there are no ... bidirectional edges?\n\nEven though ID-GEN will work for graphs with no bidirectional edges, it is a much easier problem and can be solved by existing works such as Kocaoglu et al [26]. The problem becomes non-trivial when bi-directed edges are considered.\n\n\n>The motivation for high-dimensional distribution estimation is weak. ... not seem very meaningful to generate\n> synthetic X-ray images.\n\nWe would like to humbly point out that X-ray image generation is not the main goal of our algorithm but part of\nanswering a causal question.\nWe borrowed this application from Chambon et [10] who developed the generative imaging models to deal with the lack of\nhigh-quality, annotated medical imaging datasets.\nOur main purpose is to illustrate the type of high-dimensional causal query we can answer.\n\n\n\n> The base procedure seems to come from existing methods, such as the ID algorithm, and they just combine it with a\n> generative model.\n\nWe respectfully disagree with the reviewer that we just combine the ID algorithm with generative models.\nTo explain this point, let us imagine an algorithm, AlgX that follows ID step-by-step and trains a\ngenerative model for every conditional probability table it encounters.\nBelow we discuss the challenges it faces.\n\n$X \\rightarrow W_1 $\\\n$\\updownarrow~$ $\\swarrow$ $~~\\updownarrow$\\\n$W_2 \\rightarrow Y$\n\nGiven the query, $P_{x}(y)$, ID factorizes as:\n$$P_{x}(y) = \\sum_{w_1,w_2} P_{x,w_1}(w_2) \\cdot P_{x,w_2}(w_1,y)$$\n\nwhere each factor of this product is an interventional distribution.\n\ni) We have only observational samples as training data. If we train the generative models on our dataset, it will learn\nconditional distributions. However, if we want to sample from the above factors we need interventional samples for model\ntraining.\nAlgX doesn't know how to obtain such interventional samples. \\\n-We recursively solve each factor while generating interventional training data during the top-down recursion (**Alg 1:\nstep 4+Alg 1: step 7+Alg 4:Update**).\n\nii) Suppose, AlgX somehow can sample from each factor. In which order should it\nsample?\nIf it samples $\\\\{W_2\\\\}\n\\sim P_{x,w_1}(w_2)$ it needs $W_1$ as input, which has to be sampled from $P_{x,w_2}(w_1,y)$. But to sample $\\\\{W_1,\nY\\\\}\n\\sim P_{x,w_2}(w_1,y)$, it needs $W_2$ as input which has to be sampled from $P_{x,w_1}(w_2)$.\nTherefore, it will reach a deadlock situation and fail to sample all $W_1, W_2, Y$ consistently.\\\n-We disintegrate the factors after the recursion ends, into single variable models and build a sampling network\nconnecting them (**Alg 3: MergeNetwork**).\n\niii) It is unclear how AlgX will mimic the product and the marginalization of the factors for the generated\nhigh-dimensional samples (ex: image generation).\\\n-We perform ancestral sampling on our built sampling network and drop the marginalized ones at the end.\n\niv) ID goes to step 7 for specific queries such as $P_{w_1, w_2, x}(y)$. It iterates over all values of $X, W_2$ and\nperforms $do(X, W_2)$ to update its probability distribution parameter $P(V)$ as\n$$P'(V)= P(W_1|X)P(Y|W_1, x, w_2)$$ for the next recursive calls.\nAlgX equipped with generative models does not give any hint about how to mimic this step for high-dimensional sampling.\\\n-We generate $do(X, W_2)$ interventional training data with **Alg 4:Update**.\n\n\n\n\n> ...Any non-trivial points in theoretical guarantees when the generative model is combined with the ID\n> algorithm?\n\n\nWe believe that the following theoretical contributions are nontrivial, and allow us to establish our novelty.\n\n**Lemma D.11**: ID carries a probability table as a recursion parameter whereas we carry an interventional dataset.\nEven though we follow ID's recursive trace, to deal with the interventional dataset, we have two extra parameters:\nintervened variables at step 7s: $\\hat{X}$\nand causal graph $\\hat{G}$ containing $\\hat{X}$. This lemma ensures that our interaction with these extra parameters\nwill not deviate us from the trace of ID.\n\n**Lemma D.14**: We perform on our training dataset and update them to the interventional dataset at step 7s whereas ID\nmanipulates its probability table from observational distribution to interventional distribution.\nThus, we show a guarantee that our training dataset represents the corresponding probability distribution in\nthe ID algorithm at each recursion level.\n\n**Lemma D.20**. shows that every edge in the sampling graph adheres to the topological ordering $\\pi$ of the original\ngraph. This ensures that our proposed concept of\na sampling network containing the trained models is not invalid and samples consistently **in an acyclic manner.**\n\n**Lemma D.21**: After training the conditional models from our recursive steps, we connect them to build a sampling\nnetwork before performing the sampling. Here, we prove\nthat the sampling network consists of trained models can sample from the expected joint distribution.\n\n\nWe are more than willing to have further discussion. We would highly appreciate it if the reviewer reconsiders a stronger\nthe score for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lAhdmlRK5x", "reviewer": "Reviewer_xD7P", "summary": "The paper leverages state-of-the-art conditional generative models and algorithms from causal do calculus to perform \"approximately correct\" high-dimensional interventional sampling. Their contribution is ID-GEN, a recursive algorithm that uses diffusion models (among other generative models) to sample from any identifiable causal effect estimand in high-dimensional settings. The efficacy of the method is demonstrated on three diverse datasets: Colored MNIST, CelebA, and MIMIC-CXR.", "questions": "- Can you clarify whether the proposed ID-GEN algorithm is primarily an implementation of the existing ID algorithm with diffusion models substituting for probability tables, or does it introduce fundamentally new recursive methodologies? How does this distinction affect the perceived novelty of your contribution?\n- What are the potential extensions of your algorithm to areas like symbolic calculus or probabilistic programming?\n- Are there any specific obstacles that need to be overcome to advance towards automatic causal inference using high-dimensional data such as images? How feasible is this prospect in the near future?", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- The paper introduces ID-GEN, a novel algorithm that integrates diffusion models with causal inference for high-dimensional interventional sampling.\n- ID-GEN creatively exploits the causal structure via the known recursive algorithm for sampling in complex causal scenarios, particularly with latent confounders.\n- The algorithm is applied in three applications across diverse datasets (Colored MNIST, CelebA, MIMIC-CXR).", "weaknesses": "- The paper is objectively hard to read. Many important graphical elements that should be paired with the text in the main paper are delayed to supplements. This is notably problematic for Example 4.1, where one would expect the full example to be self-contained in the main text. \n- The contributions are stated in the introduction, but it still seems hard to understand if the proposed method is \"just\" an implementation of the ID algorithm, replacing probability tables by samples from diffusion models. I appreciate that this is hard already, but it has much lower novelty then proposing a new recursive algorithm. This should be well explained in the manuscript. \n- The paper does not discuss the implications of the proposed algorithm. Is there any way to extend this to symbolic calculus, or to probabilistic programming? What are the obstacles for moving towards automatic causal inference with images (which would be a super exciting prospect).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable effort. We are happy that they acknowledged our algorithm as novel and sound. Below we\naddress their concerns.\n\n\n> ..., Obstacles for moving towards automatic causal inference with images?.\n> How feasible is this prospect in the near future?\n\nWe interpret the reviewer's mention of \"automatic causal inference\" as the process of\nautomating every step of causal inference where the inputs are raw data and assumptions/constraints and the outputs are\nhigh-dimensional\ninterventional samples/images. Below we share the steps and challenges for each step.\n\nStep 1, causal variables: We detect the causal variables in the raw data (challenging in high-dimension).\n\nStep 2, causal graph: We discover the causal relationships among the causal variables obtained from Step 1.\nIt is hard to obtain fully specified causal graphs specially when high-dim variables are involved.\nGenerally, we obtain partial graphs with uncertain edges and/or uncertain directions.\n\nStep 3, causal inference: We learn the underlying causal mechanisms that generate each variable from\nthe causal graph parents obtained from Step 2. Finally, we generate interventional samples using the learned mechanisms\nand report samples as output. Even though some promising results have been shown for fully-specified graphs,\nhigh-dimensional causal inference is\nstill under-explored in presence of unobserved shared parents (confounders). Our proposed ID-GEN offers a solution for\nthis problem. \n\nWe hope our contribution becomes a useful piece in this challenging process.\n\n\n> ... extend this to symbolic calculus\n\nIn Pearl's causality, we found do-calculus [1] closest to symbolic calculus [2].\nIn [2], Pearl mentioned that \"Polynomial tests are now available for deciding when $P(x_i|do(x_j)$ is identifiable,...,\nin terms of observed quantities.\nThese tests and derivations are based on a **symbolic calculus**,..., in which interventions, side by side with\nobservations, are given explicit notation,\nand are permitted to transform probability expressions.\"\n\nThe 3 do-calculus rules express an interventional distribution in terms of observational distribution. The ID\nalgorithm (Shpitser et al [45]) we considered\nactually applies these 3 rules systematically along its recursive steps. Since we follow ID's recursive steps, we are\nimplicitly following the do-calculus rules.\n\n[1] Pearl, J. 2000. Causality. Cambridge University Press, 2nd edition\\\n[2] Pearl, Judea. \"A causal calculus for statistical research.\" AISTAT (1996): 23-33\n\n\n> or to probabilistic programming?\n\nWe found a relevant work [3] that implements Bayesian causal inference based on probabilistic programming using MiniStan\nand Gen programming languages.\nAccording to [3], one can use these programs to estimate the parameters of a model from data that represents the\nconditional distributions of variables. As ID-GEN is model agnostic,\none could explore the connection between ID-GEN and the probabilistic programming approach.\n\n[3] Witty, Sam, et al. \"Bayesian causal inference via probabilistic program synthesis.\" arXiv (2019).\n\n\n> ..., for Example 4.1, ... ,expect the full example to be self-contained in the main text.\n\nWe will to utilize the additional space of the camera-ready version to accommodate Example 4.1 if our paper gets accepted.\n\n\n> ...if the proposed method is \"just\" an implementation of the ID algorithm, replacing probability tables by\n> samples from diffusion models.\n\nWe apologize for such an impression. Our algorithm definitely offers novel methodologies compared to the mentioned\nimplementation. Let us imagine an algorithm, AlgX that follows ID step-by-step and trains a\ndiffusion model for every conditional probability table it encounters.\nBelow we discuss the challenges it faces.\n\n$X \\rightarrow W_1 $\\\n$\\updownarrow~$ $\\swarrow$ $~~\\updownarrow$\\\n$W_2 \\rightarrow Y$\n\nGiven the query, $P_{x}(y)$, ID factorizes as:\n$$P_{x}(y) = \\sum_{w_1,w_2} P_{x,w_1}(w_2) \\cdot P_{x,w_2}(w_1,y)$$\nwhere each factor of this product is an interventional distribution.\n\ni) We have only observational samples as training data. If we train the diffusion models on our dataset, it will learn\nconditional distributions. However, if we want to sample from the above factors we need interventional samples for model\ntraining.\nAlgX doesn't know how to obtain such interventional samples. \\\n-We recursively solve each factor while generating interventional training data during the top-down recursion (**Alg 1:\nstep 4+Alg 1: step 7+Alg 4:Update**).\n\nii) Suppose, AlgX somehow can sample from each factor. In which order should it\nsample?\nIf it samples $\\\\{W_2\\\\}\n\\sim P_{x,w_1}(w_2)$ it needs $W_1$ as input, which has to be sampled from $P_{x,w_2}(w_1,y)$. But to sample $\\\\{W_1,\nY\\\\}\n\\sim P_{x,w_2}(w_1,y)$, it needs $W_2$ as input which has to be sampled from $P_{x,w_1}(w_2)$.\nTherefore, it will reach a deadlock situation and fail to sample all $W_1, W_2, Y$ consistently.\\\n-We disintegrate the factors after the recursion ends, into single variable models and build a sampling network\nconnecting them (**Alg 3: MergeNetwork**).\n\niii) It is unclear how AlgX will mimic the product and the marginalization of the factors for the generated\nhigh-dimensional samples (ex: image generation).\\\n-We perform ancestral sampling on our built sampling network and drop the marginalized ones at the end.\n\niv) ID goes to step 7 for specific queries such as $P_{w_1, w_2, x}(y)$. It iterates over all values of $X, W_2$ and\nperforms $do(X, W_2)$ to update its probability distribution parameter $P(V)$ as\n$$P'(V)= P(W_1|X)P(Y|W_1, x, w_2)$$ for the next recursive calls.\nAlgX equipped with diffusion models does not give any hint about how to mimic this step for high-dimensional sampling.\\\n-We generate $do(X, W_2)$ interventional training data with **Alg 4:Update**.\n\nWe are more than willing to have further discussion. We would highly appreciate it if the reviewer reconsiders a stronger\nthe score for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission provides an algorithm for sampling from a causal interventional distribution using conditional generative models, building on Shpitser and Pearl's ID algorithm.

While most reviewers consider the contribution to be sound and valuable for the field, there has been concerns that the manuscript is hard to parse. Overall, the contribution seems technically sound, but highly inaccessible to a majority of the audience, in addition to the significance of the contribution being highly challenging to assess given the complexity of the assumptions and statement of the results. The AC recommends to make an effort towards the general readership and recommends acceptance.
Decision: Accept (poster)