Counterfactual Image Editing with Disentangled Causal Latent Space
Authors: Yushu Pan
Elias Bareinboim
Venue: NeurIPS 2025 poster ()
Keywords: Causal Inference
Generative model
Causal representation learning
Image editing
Primary area: probabilistic_methods
TL;DR: We bridge the gap between causal image editing and large-scale text-to-image generation.

Abstract: The process of editing an image can be naturally modeled as evaluating a counterfactual query: “What would an image look like if a particular feature had changed?” 
While recent advances in text-guided image editing leverage powerful pre-trained models to produce visually appealing images, they often lack counterfactual consistency -- ignoring how features are causally related and how changing one may affect others.
In contrast, existing causal-based editing approaches offer solid theoretical foundations and perform well in specific settings, but remain limited in scalability and often rely on labeled data.
In this work, we aim to bridge the gap between causal editing and large-scale text-to-image generation through two main contributions. First, we introduce Backdoor Disentangled Causal Latent Space (BD-CLS), a new class of latent spaces that allows for the encoding of causal inductive biases. One desirable property of this latent space is that, even under weak supervision, it can be shown to exhibit counterfactual consistency.
Second, and building on this result, we develop BD-CLS-Edit, an algorithm capable of learning a BD-CLS from a (non-causal)  pre-trained Stable Diffusion model. This enables counterfactual image editing without retraining. Our method ensures that edits respect the causal relationships among features, even when some features are unlabeled or unprompted and the original latent space is oblivious to the environment’s underlying cause-and-effect relationships.

Counterfactual Image Editing with Disentangled Causal Latent Space
Yushu Pan and Elias Bareinboim
Causal Artificial Intelligence Lab
Columbia University
(yushupan, eb}cccs. columbia. edu
Abstract
The process of editing an image can be naturally modeled evaluating counter- factual query: "What would an image look like if a particular feature had changed?" While recent advances in text-guided image editing leverage powerful pre- pre-trained models to produce visually appealing images, they often lack counterfactual con- sistency ignoring how features are causally related and how changing one may affect others. In contrast, existing causal-based editing approaches offer solid theoretical foundations and perform well in specific settings, but remain limited in scalability and often rely on labeled data. In this work, we aim to bridge the gap between causal editing and large-scale text-to-image generation through two main contributions. First, we introduce Backdoor Disentangled Causal Latent Space (BD-CLS), a new class of latent spaces that allows for the encoding of causal in- ductive biases. One desirable property of this latent space is that, even under weak supervision, it can be shown to exhibit counterfactual consistency. Second, and building on this result, we develop BD-CLS-Eddi an algorithm capable of learning a BD-CLS from (non-causal) pre-trained Stable Diffusion model. This enables counterfactual image editing without retraining. Our method ensures that edits respect the causal relationships among features, even when some features are unla- beled or unprompted and the original latent space is oblivious to the environment's underlying cause-and-effect relationships.
Introduction
Image editing is important task in computer vision, which enables counterfactual question: "What would a given image be had feature X changed from x to x'?" Addressing such questions benefits generative models by realism, interpretability, fairnsss, generalizability, and transportability
63 23 31. Earlier approaches to solving this problem typically consider inverting images into Latent Space (LS) and manipulating latent vectors leveraging correlations between the labels of intervened features and images
6 42
5
51
Recently, text-guided image editing methods leverage large-scale pre-trained models, such CLIP, Stable Diffusion, and Rectified Flows 45 48 [5], to enable edits that align with general human common sense concepts, without 49). These methods prioritize keeping the edited image close possible to the original and maintaining non-edited features unchanged (called as semantic invariance), but are often oblivious to the effects of the edited features on other features.
[53 17
26 8
requiring model retraining [2]
33
More recently, causal generative models have been proposed to capture causal effects in data (59 60 40 57 58). By integrating Structural Causal Models (SCMs) with modern deep generative architectures, these models can practically achieve high-quality causal image editing given observa- tional image samples, corresponding feature labels, and a specified causal diagram under restricted From a theoretical standpoint, these methods often (point-)estimant
assumptions [47 46 B
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Semantic Invariance
Causal
Inversion
Change weather
from "sunny" to "rainy"
Age
Weather
Pose
Age Levariatt
Age Channe
Age IIvariant
Umbrella- Change
Umbrella Change
Umbrella Invariant
Umbrella
Pose Invariant
Pose Change
Pose Invariant
Figure 1: Editing results from Example I
causal editings (blue); non-causal editings (yellow).
counterfactual queries under the assumption of identifiability, without formal guarantees on the validity of their outputs. [36] addresses this gap and introduces the notation of counterfactual con- sistency, criterion that offers formal guarantees for the causal effect of editing specific features. Despite these recent advances, existing causal editing methods fall short in terms of scalability compared large-language vision models and struggle to produce state-of-the-ar image gener- ation results in broader, more diverse real real-world contexts. In addition, these methods generally require explicit annotations of features perform counterfactual edits. However, in many real- world scenarios, obtaining these labels can be challenging due to time, cost, or feasibility constraints.
In this work, aim is combine the best of both worlds, having methods that are counterfactually consistent while generating high-quality images with only partial annotated data. Figure 2 summarizes our contributions along three key axes, (1) realism, (2) efficiency and scale, and (3) weak supervision. Realism. In contrast to current large- scale text-to-image editing methods, our method enables causal editing, which aligns more closely with the goal of realistic image manipulation by respecting underlying causal relationships among features; Efficiency and Scale. Unlike prior causal editing approaches, which typically re- quire full model retraining and operate on narrowly scoped datasets, our method leverages pre-trained language-vision models to enable efficient editing without retraining. Weak Supervision. We address the weakly supervised setting, where only a subset of generative factors is labeled/prompted. The next example illustrates these challenges.
Realism
Causal
Weskly superctiion
LS inversion
labeledi Partially labeled/ procpeed
Traic foon scrach
prompted
Pre-truined
Efflelency and Scale
Figure 2: Three axes of contributions.
Example 1. Consider images describing "a lady is standing in garden in sunny day". Human common sense suggests that weather and age are not causally related; however, spurious correlations exist in the training dataset, e.g., young women appear more often in rainy scenes. Beyond weather and age, the images include other generative factors, such as the presence of an umbrella (causally influenced by both age and weather) and the pose of the lady (independent of weather, age, and umbrella use). The causal relationships are shown in the diagram Fig. T (bottom left), Age and weather are labeled prompted; other features, such as umbrella and pose, unlabeled (gray).
Now, we consider an image editing task "change the weather from sunny to rainy". one naively edits the initial image by LS inversion and alters the weather based purely on correlations, features like the age or pose may also change undesirably, despite the nonexistence of a causal link from weather to them. Furthermore, while such methods may raise umbrella in the edited image, there is no causal guarantee the probability of it being raised. On the other hand, approaches that prioritize semantic invariance aim to keep features unchanged, e.g., age and pose in this case. However, this

---

may also result in the umbrella never being raised. Fig. [ (yellow) illustrates the edited images following non-casual methods. Editing images with causality. the effects of target interventions on the other features in the image are guaranteed to be carried over from the factual to the proper corresponding counterfactual world. To illustrate, the age and pose of the lady should be invariant and the effect of the rainy weather should be correctly reflected such that an umbrella is likely to be raised. See the editing results shown in the figure(blue).
To enable causal image editing under weak supervision and at scale, we formalize the image generation process using an Augmented Structural Causal Model (ASCM) [Def[] that allows for both labeled and unlabeled generative factors with proper causal semantics. We then introduce the Backdoor Disentangled Causal Latent Space (BD-CLS) (Def 4 as modified latent space that serves as a proxy for the true generative processes. Building on ASCM and BD-CLS, our main contributions are:
1. We formally study which features should be invariant and which should change, and how these features change when causally editing images (Thm. We then show that BD-CLS provides causal guarantees for both changed and invariant features, even when they are unlabeled (Thm. 2
2. We develop BD-CLS-Edit (Alg. D post-training algorithm that learns a BD-CLS from a pre-trained Stable Diffusion model and enables counterfactual image editing. Extensive experiments are conducted to demonstrate the effectiveness of the proposed framework.
All supplementary material (including proofs) is provided in the full technical report 377
Prelininaries. An uppercase letter X indicates a random variable and lowercase letter indicates its corresponding value; bold uppercase X denotes a set of random variables, lowercase letter x is its corresponding values and Xx denote its domain. We denote P(X) as probability distribution over a set of random variables X and P(x) as the probability of X being equal to the value of x under the distribution P(X). Our work uses Structural Causal Models (SCM) as the underlying semantic framework [41 Ch. 7], and we follow the presentation provided in 4 A Structure Causal Model (for short, SCM) is a 4-tuple < U, F, P(U)= >, where (1) U set of background variables, also called exogenous variables, that are determined by factors outside the model; (2)
V V= V2...., Vaj is the set of endogenous variables that are determined by other variables the model; (3) F is the set of functions {Sv SV, Jv.) mapping Uv, U Pay, to Vj. where Uv, S U and Pay; E Viy;; (4) P(U) is probability function over the domain of U.
Each M induces causal diagram G, directed acyelic graph (DAG) where each V2 E V corresponds to a vertex. There is a directed arrow (V, - V2) for every V; E V and V, E Pay, and there is a dashed-
arrow (V, a----+ V/) for every pair V,, V2 V such that Uv, and Uv, not independent. We denote Gr as the causal diagram G over V after V is marginalized out, where V C V. For example, for G = Lz + X, Z + Y.X + Y}, 9x.x = [X 4----+ Y.X < Y}. set of variables B said to be backdoor ser relative to the pair (X, Y) if no node in Z is a descendant of X, and B blocks every path between X and Y that contains an arrow into X.
The counterfactual quantities induced by an SCM M are defined as 4 Def. 7]:
p.M (y [V) Y2ixa).....
1 Y (u) = y1, Yx (u) = Y .......
(1)
Xu where Yx (u) is evaluated under Fx, := [fv, : VV E V ( + x: X.). Each Y, corresponds to a set of variables where the original mechanisms fx are replaced with constants x; for each X E X,, This procedure corresponds to interventions, and we use subscripts to denote the intervening variables (e.g. Yx) or subscripts with brackets when the variables are indexed (e.g.
Y ((xx)) We refer to App. A for more background on causal models and diffusion models.
2 Augmented SCMs and Causal Consistency
We begin by defining a class of SCMs that models the ground- -ruth image generation process, incorporating both labeled and unlabeled generative factors.
Definition 1 (Augmented Structure Causal Model). An Augmented Structure Causal Model (for
short, ASCM) over generative level SCM Mo = -fU,. Vo. Fo. (U))) is tuple M =
(U, (V, L, F, P(U)) such that (1) U Uo: (2) (V. L} is a partition of all generative factors Vo. where V are labeled factors; L == Vo V are unlabeled factors; I is an image variable; (3)
F= (Fo Ji), where fi maps from (the respective domains of) V U to I, which is an invertible function. Namely, there exists function h such that (V, L) h(I). (4) P(U,) (U.).
3

---

In words, an ASCM is "larger" SCM describing a two-stage generative process of the image variable I. First, high-level factors Vo are generated the generative level MM- Labels (e.g.. annotations, text descriptions, etc.) are given only on the part of factors V E Vo- The remaining factors L are unlabeled. Second, all factors are mapped into pixel spaces to form the image. This mapping is invertible, which means that, given an image instance x, all factors can be recognized. That is, there exists a function / that maps from I to L). See further discussion in App. E.1
Equipped with ASCMs (Def. D. our task to edit the concepts X in an original image i from X = x to X x can be formalized as querying an Image counterfactual distribution (I-ctf) (L = the counterfactual effect in editing between generative factors. Formally, given factual factors W1 = W1 (W1 S V), the probability that factors W2 will be w2 after the edit do(X = x') is formalized by a counterfactual quantity P* P*(W2[X== 2[X=xx] = W2 w1) at the generative level Mo.
i) induced by the true underlying model M In addition, ASCMs can formalize
2.1 Proxy model for ground-truth ASCM
Consider an underlying ground-truth ASCM M and image I generated from M Generative models such as Stable Diffusion, VAEs, and GANs learn a latent space Z along with a mapping function f Z + I, which enables the generation of synthetic images I. Formally, this synthetic generation process can be regarded as a proxy SCM M over variable Z, I such that M approximates the image distribution induced by MM",.. i... PM' (I) = (I). Thus, editing a given image i by alternating latent vectors (do t'), S Z) in proxy models can be modeled as counterfactual query (IT=' i) (See Ex. for illustration). Then the following quantity is defined to capture the counterfactuals over features when editing images in such ways.
Definition 2 (Feature Counterfactual Query). Consider an underlying true ASCM over generative factors V and L, proxy model M over {Z. I), set of factual features We S {V. L}. and a set of counterfactual features W1 S (V, L}. A feature counterfactual (F-ctf) query is defined as:
M.M W W wi W2 = w2) = Suvex 1 [hw, (1', W1, (i) w2] dP (T
(i, (i
SIEA 1 |/w, (i) = W2 diM((i)
where hw,, and hw,, are the mappings from I to W1 and W2.
In words, PM W 1[TT=t] W W2 = W2) the probability that the feature Wi would take value Wi had latent vector T = t', given that the features W2 are currently equal to w2- The denominator integrates all images i such that i, has features w1 in factual worlds; the numerator integrates over counterfactual worlds P(i, (e)= such that fi] has features {W1, w2]. Def. provides way to describe counterfactual quantities over features W1 I and W2 even when Wi and W₂ are not necessarily endogenous variables in the proxy model M. See more discussion in Ex. and App. E.3
Next, we establish concept to evaluate whether an F-etf query PM PM[jee W2) induced by the proxy model constitutes reliable approximation to a counterfactual quantity P* ("(W1[xe] w2) induced by the ground truth model.
Definition 3 (Cif-consistency). Consider an ASCM M over factors [V.L) and a proxy model
M. An F-ctf query, [WI[te]] W2), is said to be counterfactually consistent with the ground
truth P* (Wi[pe] W2) under (2,9), if p.M (Wite) w2) E [1, r). where [I,r] is the union of optimal bounds of P* (w [xx]] w2) given the observational quantities Q = {P* (v" (vve)))vx and the
causal diagram GR. where 23, Cv. X C Xt. and R C I.
This definition offers principled way to evaluate the estimate produced by a proxy model against the ground truth counterfactual quantity. It extends the formulation of [36] Def. 4.4). This is needed since given the observational distribution and the causal diagram, the target counterfactual query is not always uniquely computable but some possibly informative bound Def. can be obtained and serve as a natural measure of distance from the data and the true, yet unobserved, counterfactual
We assume * x' throughout this work, implying that the concepts indeed change.

---

Initial image
Ctf image
Initial image
Ctf image
Initial image
Ctf image
Change
Change
Change
X
X =
X -
X =
Change
Preserve
Presrrve
ND nd
ND nd'
ND nd
ND = nd
ND nd
ND = nd'
de
DE de'
DE Preserve
DE = de
Sample
de
PIDE& xx
(3)
(b)
(C)
Figure 3: Invariance relationships between features in original image and counterfactual images (images after editing) cross noncausal methods (a-b) and the newly proposed causal method (c).
distribution. Def. 3 says that any value that out this bound regarded as invalid estimations, and any value within this bound is acceptable. See Ex for illustration.
3 Causal Estimator for Image Editing
In this section, we present our main theoretical results. We first factorize the target I-ctf query PM (iz i), identifying which generative factors should change or remain invariant (Sec 3.1 across the factual and counterfactual worlds (Sec. 3.1) Then we introduce Backdoor Disentangled Causal Latent Space, and establish the causal guarantees provides for evaluating I-ctf queries (Sec. 3.2)
3.1 Factorization of I-ctf query
We begin by expressing the I-ctf query in terms of generative factors and factorizing it based on their descendant relationships.
Theorem Consider the true underlying ASCM M over {V, Ler ND denote nx, Dx,N((X,) deseendncssssnts of X. in GV. and DE denote Ux, DE())) (descendants of X.) in GV L. The target I-clf query pa (Ix = I= can be factorized as
P' I= = (1) =x]
1[nd'= = nd
P (Dex = de' v,
(3)
Intervention Consistency Non-descendonzs Invariacee
Amount of Descendant Changing
where nd hinD (1), nd' (1')(1))) de hi and [v,1] h
This result circumscribes which features should remain invariant and which should change through the editing process. Specifically, the first term in the r.h.s. of Eq. corresponds to the notation of interventional consistency the edit should effectively change the features X in the counterfactual image such that these features equal to x'; the second term corresponds to non-descendants invariance- the non-descendant features ND must remain invariant across factual and counterfactual images; the third term corresponds to descendant delta the descendants of X are possibly affected and the probability of changes should be consistent with the counterfactual distribution P* ("Dexx between pre- and post-intervention worlds shown in Fig. 3 c) (see also Ex. 6 for further grounding).
v. .).. These feature invariance/variant
Using this result, we identify key limitations in the current evaluation of image editing methods. A common approach, LS inversion, edits images by inverting images into an LS and sampling from P(I x'), enforeing the target feature value z'. While this ensures interventional consistency, it often violates non-descendant invariance and descendant changes. Specifically, editing X can lead to unintended consequences in correlated non non-descendants ND. Furthermore, the amount of change in descendant features DE change lacks a proper causal guarantee, e.g., counterfactual consistency (see Fig. 3 a) and Ex(7). Modern text-to-image editing methods typically pursue: (1) editing effectiveness: removing original features x and incorporating the target x', and (2) semantic invariance: preserving other content. These correspond to the first two terms in Eq However, the third term, the descendant change, is often violated. As illustrated in Fig. 3)) the De are forced to be unchanged. while it should follow counterfactual distribution P (DE vv, 1). See more discussion in App. E.4
3.2 Backdoor Disentangled Causal Latent Space
In this section, we develop a class of generative models that ensures editing behavior consistent with both non-descendant and descendant requirements, depicted by Thm.
5

---

Definition 4 (Backdoor Disentangled Causal Latent Space). Consider a true ASCM M with diagram
Iv,r, and target I-ctf distribution query P(Ix' i). Let B S V be a backdoor set w.n.t X in
Iv.I- Denote the mapping from I to ND as hiND- A Backdoor Disentangled Causal Latent Space
(BD+CLS) is an SCM MBD.CIS = (Ü, V (x, B, Z, I, F,
such that fi(X B Z). (I (,) p.M (I X, B); (2) (disentanglement) OTND/OX = 0.
and (1) (generation) pM
where TND iiD Jr: (3) (structure) 9 is compatible Fig.
In words, BD-CLS regarding to an editing task (I-ctf query P(Tx 1i) proxy MBD-CLS with endogenous variables (X, B, Z}. where B is a backdoor set for X to image I in Iv In addition, this proxy model should satisfy three requirements. First, MBD.CLS induces the same P(I X, as ground truth M Second, BD-CLS MBDDCII requires TND be disentangled to X, which means changing the value of X will not change the value of TND (, B, Z), where
B
Figure 4: G in Def.
ND TND X. B, Z) IID (X
(4)
Third, G induced by MBD-CIS should be compatible with Fig. 4 It is worth noting that X are independent of Z given B, which implies that B inherits its backdoor property in Gv.1- See Ex. X for more details. The next result discusses the validity of generating samples from a BD-CLS.
Theorem 2 (Causal validity of BD-CLS). Consider an MaD.CIS for M and the target query
(' i). Let pMmns (I', i) be an estimator for P(K, i). Then, (a) (intervention) P.Mao (xx v, = 1[x= =x), where & = ii'(');; (b) (non-descendants) p.Mpacus (nd's v, Indd nd], (c) (descendants) p.tpecesss (de's v,1) is etf-consistent with P* (de I, whh (w)(i), w' hww (1') for any W VUL
under (P(X, nd, De), GX.ND.I), where w =
In words, (a) implies that the query P. (Fx=x/ |i) first achieves intervention consistency. The value of feature X is exactly equal to the intervened value x'. (b) implies that the estimation p.Maacss (ix=x ii satisfies non-descendant invariance. To illustrate, the feature value nd' of the counterfactual image will be the same as the feature value nd of the initial image. (c) says that a
(de's v, is ensured to be within the bound of pa (doxx v.1.. Thm. 2 powerful since BD-CLS can achieve the causal editing principles of Thm. D it precisely captures the first two principles and softly satisfies the third by providing an in-bound estimation. Thus, BD-CLS enables causal image editing, improving upon prior non-causal methods. Furthermore, BD-CLS necessitates supervision only for X and B, whereas ND and DE can remain unlabeled. See Ex. 9 for illustration.
BD-CLS can guarantee the descendant delta. To illustrate, p.M.ass
4 Learning Backdoor Disentangled Casual Latent Space
Now, we show how to obtain a BD-CLS from a pre-trained text-to-image diffusion model for sampling counterfactual images. Given a target distribution P* (Ix i) induced by the true model M the goal is to generate ~ pu (Ix i) induced from a BD-CLS BDDD-CL using initial image's label (text prompt), the diagram Gv I. and a pre-trained Stable Diffusion (SD) model.
SD models are capable of sampling images from P(I c) with classifier-freence 221. where c is given p The (reverse) generation process (Fig. S1I starts from a noise vector (T)) ~ (O 1) and iteratively denoises it to produce a clean image with recursion (!!-1)
((1()) c, 1) + o,Z(() where I(0) = II fi is the mean predictor; ( are gaussian random vectors, and of are variance terms. This generation process can be regarded as proxy model MD over (C,N (I(T), TT_ ..... ((1)) and I <- (SD N) (Fig. ().. Setting the prompt C as the observed variable [X, B), an SD model will directly satisfy condition (1) of a BD-CLS (Def.
"The "compatible" here does not exactly means M induce the same graph in Fig. There can be less edge than in Fig. 4 hut there cannot he more edges. The definition of "compatible" is formally defined by Def, Mooe details about diffuusion models can be found in App.AA

---

PM [ (I), Then an SD model can constitute BD-CLS if condition (2-3), i... disentanglement and structure, are satisfied. The following proposition formally grounds this.
Proposition (Sampling I-ctf instances through SD model). Consider a ground-truth ASCM M over (V, the targer (' 1) and a pre-trained SD model MMD Suppose there exists a pair of transformationn satisfying Z 1/1 B,N) and N th [X, B. Z). Denotes 1/ x'; b. n) = fSD (x', b, 222(x', b, 2 (x. b. n))) Then there exists a BD-CLS M MBn-cts over (X, B, 2} such that
the causal validity in Thm. are also offered by En PMM (n i, b)1l1' = (xx x', b, n)] if
(xx,,,,,
| and (i x', b, n)) = hiND(() (i).
(5)
To illustrate, Prop. 1 says that sampling counterfactual image i' from a BD-CLS involves two steps using the SD model after setting the prompt variable as {X, B). First, invert i to sample noise p.reeas (n i, b) Then the sampled n, the initial prompt {x, b), and the target prompt {x', b} are fed into the compose transformation ve to generate This vl should sat- isfy two constraints presented in Eq. S The first ensures that vé transforms Gaussian noise N into following P(I x). This constraint implicitly encodes the structural relationship be- tween X and B, since b remains unchanged in the counterfactual world, indicating that X has no effect on B. Moreover, because N is independent of C in the SD model, Z is not bidirec- tionally connected to B.B. The second constraint enforces that the non-descendants ND re- main unchanged, reflecting the disentanglement and ensuring that Z is not a descendant of X.
Building on this, we develop BD-CLS-Edit (Alg. D learn and generate i' simultane- ously. The algorithm begins by identifying the largest backdoor set B from X I in I. then matches the prompt variables C with the observed values of XUB from the label v. Next, it samples noise n given the SD model. In this work, we follow the DDPM inversion [22 and details are provided in Appendix step, we search the transformation the in candi- date space to ensure that counterfactual samples come from a BD-CLS. We detail this optimization procedure in the next section.
Algorithm I: BD-CLS-Edit
Input Initial image i: Tnitial label/prompt v; SD model; Causal diagram Gv. Initialized transfommaionn tha
Output: Cif-consistency counterfactual image for
P (I (1)
B Backdoor (X. Iv.1)
Backdoor eet // Initiaa Prompt
(x c- (X B. v)
n pS |i,
find any the s.t. vollx, X ,b N)
P(I xx and
(NDDVVoX (Vo((, n]) = (ND(()
B In the fourth
e- Val(,, x', ,',
return
4.1 Implementation of searching transformation vb
We now describe the implementation of the 4th-step in Alg. I first focusing on the candidate space of the and the optimized parameters O. We extend the original denoising process. Formally, with a specific sample n (i(T), 2(T), ...... (1)),,,))))),, }. the new transformation vhe is the iterative process
(((-1) = (1)))) c + 0,(c' - c),t) + o,
(6)
where 1((T) = -.)) To illustrate, the new prompt is linearly mixed between e and c' at each time step t by 0, This prompt mixing technique leverages the coarse-to-fi to- nature of the denoising process and demonstrates an ability to disentangle features [56 39 2 9
Next, illustrate how search 0 = {01, 0z -0r to satisfy two constraints in Eq. First, it is necessary for the (x, x', b, n) to exhibit prompts c' = fx', b) to guarantee the first constraint. It is demonstrated in [44 that updating the parameters with the direction
0i
Vo, (sn (1,, c, e) (7(1), c, a0,"
will motivate i' to exhibit features c', where is the noise added in the forward process and ē the noise predictor in diffusion (App- (A.2). This gradient update can be interpreted as asking counterfactuals: "Given that μe, generates Y, how should 6g be updated had the resulting i' resembles feature than c?". However, updating
this direction does not guarantee the second constraint. due to the entanglement of po, (See Ex. 1O). The key challenge is that the true mapping WND is unknown, making it hard to keep non-descendants unchanged without influence other descendants. To address this, we propose an alternative optimization direction inspired by the following proposition.
7

---

Image
Image
What would the image look like if the digit had been "?"?
Change the dight
D =T DE =0 IC O AW =
Coustefaciaa Initaal
Causal Edit Principles
T.DC BC
7
7
7
7
7
Intervention Condstency: =
with prbbbiitt (873
Nan-deseendants Invartance DCC
7
Changes
BD-CLS (Ours. HW unibced)
CDiiluton
CGN
NOM [+iS (iblls
probitilyy (.1)
(a)
(b)
Figure 5: Edit red "I" with thin red bar to digit "7" (a) Expectation of counterfactual consistent editing; (b) Edit results. Top- initial image. Bottom counterfactual images.
Proposition (Toy entanglement between binary X, Y and R). Consider binary X, non- descendant R and descendant Y. Suppose (y pay) + P" pay). Suppose R and Y are both entangled with {X, B. N in MSD, and = TA TR B,N)) and = Ty (X,B,N2) where N1 NN 1 and NI. N2 N, then PMMD (12, b,r,)) p.MDD (ra b,r, v) and pM Wgr12 b.r.y) * pMM (V/l2', b,
In words, this proposition suggests that even a descendant Y and non-descendant R, both entangled with X, counterfactual behaviors can be leveraged to distinguish and See Ex. [1 To use Prop. 2 for image editing tasks, let and have different feature but the same other features. Then
p.M (r'2 Jrr b, r. r,y) approximates the gradient of R when i' moves to have feature x' with Eq. Similarly, pM (r/laz,, b, y approximates the gradient when reverting x' to x in and these two gradients are equal. In contrast, the gradients of when toggling X between x and x' and i contrast SDS losses update gradients similar to DDS [18] 4 to encourage the new direction is orthogonal to ND. Formally, this direction is
under the same toggling not equivalent. Thus,
0i,
Co. c', c,t) (2(1, Ti, c, t)) 00,
(8)
and this idea is visualized in Fig. 6 Specifically, when guiding i' with the SDS direction and prompt c' {x', 6} (top-left panel in Fig. (). the weather feature (e.g., the appearance of rain the ground) changes from x (sunny) to x'' (rainy). However, due entanglement, the denceddecnndnt feature, such as trees, also tends to change from to r'. Mcanwhile, the descendant feature (e.g., the umbrella) correctly changes to y'. contrast, when guiding with the SDS direction and
i'
24t prt..p.
V mm
K.Wasther' t ACPree) = 1000))
PV; Iuhon
Final i'
{x, b) (bottom left panel of Fig. 6). the weather changes from x' (rainy) to (sunny). Yet, the non-descendant features again change in the same direction from r to r', and the umbrella no longer tends to change y/. After com- bining these contrasting directions (right panel in Fig. 6 the weather reliably changes from x', the umbrella (a descendant) appropriately changes as well, and the non-descendant feature (e.g., the tree) remains invariant, correcting the entanglement artifacts through cancellation.
XLWralbery - RCPree) - 1000)) wyy
(AGAA Pis Lxidungh
Weather WiDree) - Y(Ule w
Figure 6: Optimization direction Vo, LCat-
To obtain i, we move i, in the DDS direction, which only changes feature X but preserves others as the same. Specifically, a subset of time steps T is randomly selected, i.e., T C (1...., every / E T. we follow the DDS
......,T. T). And for
F(0 - (()) + Xe c), ci.,,,,))
where X is hyperparamter controlling the intensity of the change of X. More details on this optimization are given in App. B
'See the details of DDS in App.
A.2
Notice that the key improvement is that leverage the Prop.

---

5
Experiments
In this section, we empirically validate our theoretical results (Thm 2 and demonstrate the effective- ness of BD-CLS-Edit (Alg Additional experimental details are App. D
5.1 Colored MNIST and Bars
We first evaluate the guarantees provided by BD-CLS (Thm 2
D
DC
modified MNIST dataset [4 [6) featuring colored digits and bars.
The ground truth ASCM includes factors: Digit (0-9 D), Digit
BW
BC
Color (red DC = 0; green DC = 1), Bar Width (thin BW = 0;
thick BW = 1). Bar Color (red BC = O; green BC 1), and other latent factors such as handwriting style. The causal relationships are shown in Fig. 7 To illustrate, the digit (D) and digit color (DC) are
confounded: larger digits tend to he red, and smaller digits tend to be green, but they do not directly affect each other. The digit color (DC) has positive effect on bar color (BC), as red digits often have red
Figure 7: G in Sec. 5.1 bars. The digit (D) has a positive effect on bar width (BW); larger digits are more likely to be with thick bars. However, this effect reverses when the digit color is green.
We first consider editing the digit. Suppose that we are editing a red "1" with thin red bar and wonder what would happen had the digit "1" been a "7". According to the editing principles in Thm. [ the edit should achieve (1) interventional consistency. The digit should be "7"; (2) non-descendants invariance. digit color (DC) and bar color (BC) remain red;. (3) Descendant Delta. The BW as a descendant, may change thicker and the probability should be Q P (BW D=7 = 1 1, DC BC = 0, BW 0). To guarantee counterfactual consistency, the estimation of Q should be within the bound according to Def. 3 These edit expectations are shown in Fig. ((a).
We evaluate both causal and non-causal methods on the digit editing task. According to Theorem - our proposed BD- CLS enables counterfactual consistent editing, even with un- labeled features. We obtain BD-CLS using a Neural Causal Model (59 60, 36) trained without labels for BW. also train NCM with full supervision. For compari- son, we include two non-causal baselines: (1) Conditional Diffusion (CDiffusion), which relies correlations, and (2) CGN [50]. which preserves original semantics. The editing results (Fig. (() show that all models change the digits to "7". However, CDiffusion alters non-descendants (e.g., color), and CGN fails to change descendants (bar width). In contrast, BD-CLS and fully supervised NCM preserve non-descendants and correctly update the BW, despite BD-CLS not using BW labels. To quantify descendant changes, we estimate the query Q (8ig by measuring how often the bar becomes thicker after editing. Both BD-CLS and fully supervised NCM stay within theoretical bounds, while CDiffusion and CGN do not. Additional tasks are in App. D.
Pc 1 D = 1.DC 0L0C = 0.80 0)
00
Figure 8: The estimated F-ctf query by our BD-CLS and baselines.
5.2 Real World Scenarios Counterfactual Editing
In this section, we validate BD-CLS-Edit for counterfactual image editing in real-world scenarios.
We compare it against three non-causal SOTAs: (1) DDPM Inversion (22) and (2) SDEdit representing LS inversion, and (3) DDS [22). which emphasizes semantic invariance. Qualitative evaluations are shown here. Quantitative evaluations and additional details are provided in App. D.3
We begin with the setting from Example I where the goal is to change the weather from sunny to rainy in an image of a young (or old) lady in a garden (or street). The causal relationships between the generative factors are shown in Fig(Ja) (a)). According to Thm. [ non-descendants (e.g.. scene, age, pose, person's 's height) should be preserved, while descendants (e.g., umbrella, shadows) should change accordingly regardless of whether they are prompted. For example, an umbrella may appear, and shadows should become fuzzier on wet ground due to the weather change. As shown in Fig. (d)), all methods achieve interventional consistency. However, DDPM inversion and SDEdit alter non-
A bar in an image refers complete row of pixels with the same color.

---

Westher
Soate
Seeeaa
Scrme
Clotbes
Celing
Causal Diagrams
Joat
Pose
Presscc laight
teight
Gruosy beg
Socae layout
(a)
[b)
(c)
hat
image like the w weather rainy?
W hat would the image look like the season had been fall?
Initial Image
DDPM
SDE
DDS
BD-CLS (ours)
Initial Image
Inversion
DDPM
SDEdit
DDS
BD-CT. (nurs)
hat would the image look like the scene had been garden?
Initial Image
DDPM
SDBai
(6)
(E)
Figure 9: The causal diagrams and editing results for Sec. 5.2 descendants, changing the lady pose and part of the scene. DDS maintains visual similarity to the original image, but does not reflect downstream effects. To illustrate, the umbrella does not appear and the shadows on sunny days are preserved. In contrast, BD-CLS preserves non-descendants and correctly reflects the causal effects on descendants, such as the umbrella and shadow.
Next, we edit an image of person in forest by changing the season from summer fall (Fig Figgb) Non-descendants (e.g., person's is pose and heights, forest layout) should be preserved, while descen- dants, e.g., clothing and leaf color, should change since people wear warmer clothes in the fall. As shown in Fig. ()), DDPM inversion and SDEdit fail to preserve the person's 's identity. DDS preserves the person's 's features but produces unrealistic clothing by retaining too much from the original image. BD-CLS-EaR accurately reflects warmer clothing while preserving non-descendants. Third, we edit an image a person in a grocery store by changing the scene to a garden (Fig. ([c) Non-descendants (e.g., person's pose) should remain unchanged, while descendants, like grocery bag. should be removed, as a person is unlikely to bring a grocery bag into a garden. In addition, the ceiling is also likely to be removed since garden usually is not indoors. As shown in Fig. It), DDPM and SDEdit inversion noticeably alter the person. DDS keeps the grocery bag. In contrast, BD-CLS-Edii preserves non-descendants and removes the grocery bag.
6 Conclusions
We develop a counterfactual image editing framework that works with pre-trateeee pre-trained diffusion models under weak supervision, without retraining. We introduce data structure called Backdoor Causal Latent Space (BD-CLS), which ensures counterfactual consistency (Thm. [ and 2). and then develop an BD-CLS-Edit (Alg. [ extract it from Stable Diffusion model. Our approach advances image editing in terms of causal realism, scalability, and weak supervision.
10

---

Acknowledgments
This research is supported in part by the NSF, ONR, AFOSR, DoE, Amazon, JP Morgan, and The Alfred P. Sloan Foundation.
References
[1] Ba, J. L., Kiros, J. R., and Hinton, G. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.
[2] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Zhang, Q. Kreis, K., Aittala, M., Aila, T., Laine, S., et (2022). ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01322
[3] Bareinboim, E. (2025). Causal Artificial Intelligence: A Roadmap for Building Causally Intelligent Systems. https:/ccausalai-bookee
[4] Bareinboim, E., Correa, D., lbeling, D., and Icard, (2022). On pearl's hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl, page 507-556. Association for Computing Machinery, New York, NY, USA.
[5] Bareinboim, E.. Forney, A., and Pearl, (2015). Bandits with unobserved confounders: causal approach. In Advances in Neural Information Processing Systems, pages 1342-1350.
[6] Bareinboim, E. and Pearl, J. (2016). Causal inference and the data-fusion problem. In Shiffrin, M., editor, Proceedings of the National Academy of Sciences, volume 113, pages 7345-7352. National Academy of Sciences.
[7] Brooks, T., Holynski, A., and Efros, A. A. (2023). Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18922-18402.
[8] Chai, I.., Wulff, J., and Isola, P. (2021). Using latent space regression to analyze and leverage compositionality in [gan}s. In International Conference on Learning Representations.
[9] Chefer, H., Alaluf, Y.. Vinker, Y., Wolf, L., and Cohen-Or, D. (2023). Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM transactions on Graphics (TOG), 42(4):1-10.
[10] Correa, J. and Bareinboim, E. (2024). Counterfactual graphical models: Constraints and inference. Technical report, Technical Report R-115, Causal Artificial Intelligence Lab, Columbia University.
[11] Couairon, G., Verbeek, J., Schwenk, H., and Cord, M. (2022). Diffedit: semantic image editing with mask guidance. arXiv preprint arxiv:2210.11422
Diffusion-based
[12] Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., and Raff, E. (2022). Vggan-clip: Open domain image generation and editing with natural language guidance. In European Conference on Computer Vision, pages 88-105. Springer.
[13] Dash. S., Balasubramanian, V. N., and Sharma, A. (2022). Evaluating and mitigating bias in image classifiers: A causal perspective using counterfactuals. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 915-924.
[14] Deng, L. (2012). The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6): 141-142.
[15] Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. (2024). Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning.
[16] Gandikota, R., Orgad, H., Belinkov, Y., Materzyúska, J., and Bau, D. (2024). Unified concept editing in diffusion models. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 5111-5120.

---

[17] Härkönen, E., Herizmann, A., Lehtinen, J.. and Paris, S. (2020). Ganspace: Discovering interpretable gan controls. Advances Neural Information Processing Systems, 33:9841-9850.
[18] Hertz, A., Aberman, K., and Cohen-Or, (2023). Delta denoising score. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2328 2337.
[19] Hertz, A., Mokady, R., Tenenbaum, J. M., Aberman, K., Pritch, Y., and Cohen-Or, D. (2022). Prompt-to-promy image editing with cross attention control. ArXiv, abx/2208.01626.
[20] Ho, J., Jain, A., and Abbeel, (2020). Denoising diffusion probabillistic models. Advances in neural information processing systems, 33:6840-6855 6851.
[21] Ho, J. and Salimans, T. (2022). Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.
[22] Huberman- Sppeeelllp,igelllas,, I,, Kulikov, V., and Michaeli. T. (2024). An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12469-12478.
[23] Hwang. I., Pan, Y., and Bareinboim, E. (2025). From black-box to causal-box: Towards building more interpretable models. In Mechanistic Interpretability Workshop at NeurlPS 2025.
[24] Hyvärinen, A. and Pajunen, P. (1999). Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-433
[25] Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. (2020). Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR.
[26] Khorram, S. and Fuxin, L. (2022). Cycle-consistent counterfactuals by latent transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10203-10212.
[27] Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv: 141.:644..6990.
[28] Kulikov, V., Kleiner, M., Hubereaa--Speeeeeeeeee Spiegelglas, I., and Michaeli, T. (2024). Flowedit: Inversion- free text-based editing using pre- pre-trained flow models. ArXiv, abs/2412.08629.
[29] Lachapelle, S., Rodriguez, P., Sharma, Y., Everett, K. E., Le Priol, R., Lacoste, A., and Lacoste- Julien, S. (2021). Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In First Conference on Causal Learning and Reasoning.
[30] Lee, H., Choi, S., Zhang, B.-T., and Hwang, I. (2025). Towards spatially consistent image generation: On incorporating intrinsic scene properties into diffusion models. arXiv preprint arXiv:2508.10335
[31] Li, A., Pan, Y., and Bareinboim, (2024). Disentangled representation learning in non- markovian causal systems. Advances in Neural Information Processing Systems, 37: 37:104843- 104903.
[32] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. (2019). Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pages 41 14-4124. PMLR.
[33] Meng, C., Song, Y., Song, J., Wu, J., Zhu, J.-Y,, and Ermon, (2021). Sdedit: Image synthesis and editing with stochastic differential equations. ArXiv, abs/2
[34] Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen-Or, D. (2023). Null-text inversion for editing real images using guided diffusion models. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 6038-6047.
[35] Orgad, H., Kawar, B., and Belinkov, Y. (2023). Editing implicit assumptions in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7053-7061.
12

---

[36] Pan, Y. and Bareinboim, E. (2024). Counterfactual image editing. In Salakhutdinov. R.. Kolter, Z. Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. F., editors, Proceedings of the s1 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 39087-39101. PMLR.
[37] Pan, Y. and Bareinboim, E. (2025). Counterfactual Image Editing with Disentangled Causal Latent Space. Technical Report R- R-137, Columbia University, Department of Computer Science, New York.
[38] Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., and Zhu, J.-Y. (2023). Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 conference proceedings, pages 1-11.
[39] Patashnik, O., Garibi, D., Azuri, I., Averbuch Elor, H., and Cohen-Or, D. (2023). Localizing object-level shape variations with text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 23051-2300 230611.
[40] Pawlowski, N., Coelho de Castro, D., and Glocker, B. (2020). Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems, 33:857-869.
[41] Pearl, J. (2000). Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, NY, USA, 2nd edition.
[42] Plečko, D. and Bareinboim, (2024). Causal fairness analysis: causal toolkit for fair machine learning. Foundations and Trends@ in Machine Learning, 17(3):304-589
[43] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T. Müller, J., Penna, J., and Rombach, R. (2023). Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint ar 23x.:2337.0195:
[44] Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. (2022). Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprins ar XrXiv:2209. 149888.0..4488.
[45] Radford, A., Kim, J. W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR.
[46] Rahman, M. M., Jordan, M., and Kocaoglu, M. (2024). Conditional generative models are sufficient to sample from any causal effect estimand. ArXiv, abs/2402.07419.
[47] Ribeiro, F. D. S., Xia, T., Monteiro, M., Pawlowski, N., and Glocker, B. (2023). High fidelity image counterfactuals with probabilistic causal models. In International Conference on Machine Learning, pages 7390-7425. PMLR.
[48] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695.
[49] Rout, L., Chen, Y., Ruiz, N., Caramanis, C., Shakkottai, S., and Chu, W.-S. (2024). Semantic im- age inversion and editing using rectified stochastic differential equations. ArXiv, abs/2410.10792.
[50] Sauer, A. and Geiger, A. (2021). Counterfactual generative networks. In International Confer- ence Learning Representations.
[51] Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. (2021). Toward causal representation learning. Proceedings of the IEEE, 109(5):612-634.
[52] Schramowski, P., Brack, M., Deiseroth, B., and Kersting, K. (2023). Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22522-22531.
[53] Shen, Y., Gu, J., Tang, X., and Zhou, B. (2020). Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9243-9252.
13

---

[54] Sohn, K., Lee, H., and Yan, X. (2015). Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28.
[55] Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. (2023). Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1921-1930.
[56] Wu, Q., Liu, Y., Zhao, H., Kale, A., Bui, T., Yu, T., Lin, Z., Zhang, Y., and Chang, S. (2023). Uncovering the disentanglement capability in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1900-1910.
[57] Xia, K. and Bareinboim, E. (2024). Neural causal abstractions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 20585-20595.
[58] Xia, and Bareinboim, E. (2025). Causal abstraction inference under lossy representations. Technical Report R-124, Causal Artificial Intelligence Lab, Columbia University, USA.
[59] Xia, K., Lee, K.-Z., Bengio, Y., and Bareinboim. E. (2021). The causal-neural connection: Expressiveness, learnability, and inference. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 10823-10836. Curran Associates, Inc.
[60] Xia, K., Pan, Y., and Bareinboim, E. (2022). Neural causal models for counterfactual identifica- tion and estimation. In International Conference Learning Reprexentations.
[61] Zhang. J., Jin, T., and Bareinboim, (2022). Partial counterfactual identification from observa- tional and experimental data. In Proceedings of the 39th International Conference on Machine Learning (ICML-22).
[62] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. (2018). The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595.
[63] Zhao, S., Ren, H., Yuan, A., Song, J., Goodman, N., and Ermon, S. (2018). Bias and generaliza- tion in deep generative models: An empirical study. Advances in Neural Information Processing Systems, 31.
14

---

Appendix
Contents
A
Background
15
A.
Causal Models
15
A.2 Denoising Diftusion Probabilistic Model and Score Distillation Sampling
17
Algorithm Details
18
Omitted Proofs
20
Proof of Thm.
20
Proof of Thm
Proof of Prop.
24
C.4 Proof of Prop.
25
Experiments
26
D.
Colored MNIST and Bars
26
D.2
Implementation Details for Colored MNIST and Bars Experiments
29
D.3
Real World Scenarios Editing
30
D.4
Implementation Details for Real World Scenarios Experiments
33
D.5
Sensitivity and Robustness for BD-CLS
34
D.6
Editing with BD-CLS for Other Datasets
35
Further Discussions and Examples
35
E.1
Augmented Structural Causal Models (Def.)
35
E.2
Proxy Models and Latent Space
37
................................
Feature Counterfactual Query (Def.
2)
38
Counterfactually Editing principles Thm.
39
E.5
Backdoor Disentangled Causal Latent Space Def. 4 and Thm.
40
E.
Difficulties satisfies constraint Eq. 5 Alg.
41
7
Limitation
42
E.8
Broader Impact
43
E.9
Safeguards
43
Background
A.1
Causal Models
Our work relies on the basic semantical framework structural causal models (SCMs) 41 Ch. 7], and follow the presentation in 4
Definition 5 (Structure Causal MSde) A Structure Causal Model (for short, SCM) is 4-tuple < U, V, P(U) >, where (1) U is set of background variables, also called exogenous variables, that are determined by factors outside the model; (2) V = (V1. V2...., ., Va is the set of endogenous variables that determined by other variables in the model; (3) F iis the set of functions fv. .......... fvv. mapping Uy, U Pay, to V,, where Uv, C U and Pay, C VVV; (4) P(U) is probability function over the domain of U.
We bring forth the longer and more formal definition causal diagrams induced by the SCMs. Definition (Causal Diagram 4 Def. 13]). Consider an SCM M (U, V, P(U)). We construct graph & using M follows:
15

---

(1) add vertex for every variable in V,
(2) add directed edge (V, + Vi) for every V,, V, E V if V appears as an argument of fv. F.
(3) add a bidirected edge (V, Vi) for every Vi, V3 E V if the corresponding Uv. Uv. U are not independent or if fv; and JV, share some U E U as an argument.
We refer to G as the causal diagram induced by M (or "causal diagram of M for short).
Then structure can be defined with the bi-drected edges in a causal diagram. Definition 7 (C-component 44 Def. 14]). Let 9 be a causal diagram. Let C1, C2...., . C₁ be partition over the set of variables V. where C, is said to be confounded component (C-component for short) of G if for every Va. V,, E C; there exists path made entirely of bidirected edges between V. and V,, in G. and C; is maximal. We denote C(V.) as the C-component containing Va-
An intervention on a subset of X C V, denoted by do(x). is an operation where X takes value x, regardless how X are originally defined. For an SCM M, let MMx be the submodel of M induced by do(x). For any subset Y S V. the potential outcome Y (u) is defined as the solution of Y after feeding U u into the submodel MMx. Then Yx is called a counterfactual variable induced by M. Specifically, the event Yx = y represent "Y would be y had X been x". The counterfactual quantities induced by an SCM M are defined as Eq.
P.M (y1(x) Y2 ..................... /
1
Yx (u) Y2. IdP(u).
(10)
Au Each Y, corresponds set of variables in a world where the original mechanisms fx are replaced with constants x; for each X E X,; this is also known as the mutilation procedure. This procedure corresponds to interventions, and we use subscripts denote the intervening variables (e.g. Yx) or subscripts with brackets when the variables are indexed (e.g. (({))) ). For instance, P(y= y22) is the probability of the joint counterfactual event Y = had X been and Y = y/ had X been x'.
After describing causal model in SCM semantics, can also define graphical model independent of a particular generative process and instead based on a set of constraints. Counterfactual Bayesian Netwrok B 10], similarly to Bayesian Network
Causal Bayesian Network [4], which are graphical models that relate a graph and a (set of) counterfactual distribution(s) is defined as follows. Definition 8 (CTFBN Semi-Markovian 3 Def. 13.2.1]). Let P** be the collection of all distributions of the form P(W 1[=1], W Wefr2)....). where W; E V. X; C V. A directed acyclic graph G over V is Counterfactual Bayesian Network for p if:
Independence Restrictions] Let W. be a set of counterfactuals of the form W Waa.. Cı,..., C the c-components of G(V(W.)) .)), and Ci....., Ct. the corresponding partition over W.. Then P(W.) factorizes as
1 P[ P( f=1 Wpag, EC2,
P( A
W Waa.
A
W )pe..)
(11)
Wpag EW.
2. Exclusion Restrictions] For every variable Y V with parents Pay. for every set Z C (Pa,,, U (Y)), and any counterfactual set W.. we have
P(Ypa,,. pay,.: W.) = P(Y pay: W.)
(12)
3. [Local Consisteney] For every variable Y V with parents Pay, let X C Pag. then for every set Z C (XX U (Y)), and any counterfactual set W., we have
P(Y= y, X, W.) P(Yxa y, X2 x, W.)
(13)
Then the compatibility of graph and an SCM can be defined as follows.
Definition 9 (Compatible with G on counterfactuals). Consider an SCM M over V and a DAG g
over V. The SCM M (or the graph g.M induced by M) is said to be compatible with g if is CTFBN for p** where p the collection of all distributions of the form P(W1x)) W Wexx))....) where W, E V. X, C V.
16

---

Prompt/ Label
Prompt/ Label
I
((-1)
10
N
7
Pule
z
X Gy
X
X di
7
7
7
(a)
(b)
Figure S1: The generation process of a diffusion model. (a) recursion version; (b) proxy model version.
Informally speaking, we say that an SCM M (or the graph G induced by MM is compatible with a given graph if G. imposes constraints that are at least as strong those in G. Since the absence of an edge in causal diagram represents a constraint, this means that if G.M has strictly fewer edges than G, then M is compatible with G. Thus, not mean that SCM induces G exactly.
discussed in 4 an SCM compatible with Ç does
Given the observed distribution P(V) and the causal diagram 9, the optimal counterfactual bounds are closed intervals based on the optimization problem following [61.
Definition 10 (Optimal Counterfactual Bounds). For a causal diagram G and observed distributions P(V), the optimal bound [1, r over counterfactual probability PM (.M (yx..... Zw) is defined as, respectively, the minimum and maximum of the following optimization problem:
max / min (yM,........ ,2w S. P. (V) = P(V) MEn(g)
(14)
where (GG) is the space of all SCMs that agree with the diagram G, i..., (2G) MBMGGM G).
By the formulation of Eq. 14 all possible values of counterfactual query induced by SCMs that agree
with the observational distributions and causal diagram are contained in the closed interval [1, r).
A.2 Denoising Diffusion Probabilistic Model and Score Distillation Sampling
A Denoising Diffusion Probabilistic Model (DDPM) model [54] 20 are deep generative models that consists of forward process and reverse process with T time-steps. The forward process gradually perturbs [((-1) (the image at step 1) with gaussian noise to ( (the image at step l), where [(0]
(image at step 0) is the original image. Formally,
(t) V (0) V1 -.
(15)
where 0e is the noise scheduler and E is the standard gaussian noise. the reverse process, diffusion model predict noise each time step using neural network taking It() and text prompt or label input. Specifically, the reverse starts from random Gaussian noise vector (T)) ~ NN(0 1)
and iteratively predicts noise with using recursion
(t-1) (1)) c, a
(16)
where e is the text prompt/label and z(t) are gaussian random vectors; and a, are pre-specified variance terms.
at â. - , c. 0-- (
a0()) c, t))
(17)
This process is illustrated in Fig. Saaa).
Text conditioned diffusion models use classifier-fre free guidance 21 to sample images from conditional distribution P(I c). Specifically, the reverse process does not only involve noise precitor ?(1(0 with prompt e but also a non-conditional term. Formally, the denoise term
(1 + () c. -,, 00
(18)
17

---

where w is hyperparameter and ( is the null text. In this work, we fix the parameter w and simply denote the denoise term in the generation process as ē(i(), c,t). Similarly, the corresponding mean predictor Then, as we discussed in Sec. 4 this recursion process can be seen as a function that takes } and generates the image I as illustrated in Fig. [1).). In Sec. [ we will demonstrate how to transform N) to our proposed BD-CLS and use it for image editing.
input N (I(T), TTT
In the DDPM training process, the network is trained to predict the noise & scheduled in the forward process. The training objective can be expressed as:
Diif [|e(1()) c,t) E E
(19)
Recently, proposed Score Distillation Sampling (SDS) and showed that given an arbitrary differentiable generator go that can generate f'()) (the noise image i' at the time step f), updating the parameters of the go in the following direction can render features in the image:
VoLsns(", c'. .e, (T(1), , - 00
(20)
Later, 18 proposes the Delta Denosing Score (DDS) and shows that by updating the parameters in the following direction, the generator produces the image i' that is the closest image to i, where i' matches the text c' and i' matches the text c,
H
Vo, LDDs c,e,
- (1e. c,
(21)
00
In other words, DDS is one of the semantic invariance image editing approaches that preserves the features except as close as possible to the features in the initial image.
B Algorithm Details
Here we illustrate more details of our proposed BD-CLS-Edit.
First, we justify the necessity of the second step in BD-CLS-Edit, constructing the prompt using X, B. The key idea is that the prompt must not include descendants of X. If the target prompt involves descendant of X, then after the intervention, that descendant will have a fixed value that is aligned with the prompt. This contradicts one of the editing principles, descendant delta, demonstrated by Thm
In the third step of BD-CLS-Edit, N is sampled from the observational distribution PSD (N i, x, b). This sampling process is related to in- version process that aim to find a noise sample given the prompt
Algorithm 2: DDPM Sampling [22]
Input :Iiitial image i; Prompt c; SD
model;
Output: n from 0 for t + To 1 do
x. b, such that the dif- fusion model MD reproduces the source real 38). In other words, this is related to the inference process corresponding to P(n i, c). However, exiting methods often focus on finding single valid noise sample n that can reproduce the initial image, rather than sampling from the distribution PSD NN c). This is incorrect according to Thm. For in- stance, in an extreme case, some methods de- terministically compute specific n given the initial image. If such deterministic is used during generation after an intervention, the descendants become fixed, then there is no randomness
image [11 19 34
55
<~ 10,
j(0) ~ (a,i(1)) + v1 aze
fort + to do
20) (i-t-1) (((((), c, t,t)
j(t-1) + ()(())) c,,) 1) + (1)
return n = (i(T), (T) (1)
left in the process. However, Thm [ implies that the descendants should follow a counterfactual
distribution pa (DE v. v.),, and therefore should vary accordingly (see Ex. 77). In this work, we
leverage the DDPM inversion [22] to sample from PSD i, c) and this full sampling algorithm
is shown in Alg. 2
For the forth step of BD-CLS-Edit, we first elaborate more on Pag The denoising process is modified to take as input a mixing prompt of c = (x, b} and c' = fx', b). Formally, with a specific sample (i(7), z (T) z(( {). the transformation the is the iterative process
(((1)) 1) (+)) + 0,(c' - c) t) + oz
(22)
18

---

Algorithm 3: Search v in the forth step of Alg 1
Input :n {i z (2)) ..... }; Initial 0 {0,,.... 0r): Selected T; Adjustment
parameters AO = {>i..... Ar Target prompt embedding c' = {x', , b): Initial
prompt embedding
(x. b): Optimization iteration number 7maxx Noise
predictor é and mean predictor fi in SD model; Variance scheduler {ox) in SD model: learning rate y; Clip value 6max
Output: 0 (01, (01...., ,0}
p(0) <- i(2)
fort t T to do
// get i(-11)
{(-1)) p(110), c,t) + oz
ift T then
L 1(0 - j(0) Xe c',)) - c,
for it mmxx do // get p(l-1)
Cmix + - 0-(c(('))) c)
1((-1) t - Cmix, t) + 012
// Update 0
0 0 0 Cotr (it, c' c,t)
10
x- clip(0, maxx)
1)- + ((11(), Cmix, 1) + (12
11
12 return :(1.....,0)
where j((2) (()) To illustrate, the new prompt is linearly mixed between e and c' each time step l by a different parameter O,. When 0, 0, the input prompt at step / is the initial labels = {x. b), which encourages the image output to have features prompt step t is the target labels c' = {x, b), which encourages the image output to have features X x' and B b. Formally,
x and = b; when 0: = 1, the input
vo(c, N) ~ P(I c) 0 = =y. 0r
(23)
vh (o N) ~ P(IC) If = 02 -..- 0r =
Then, we illustrate the new counterfactual updating direction designed based on Prop. 2 Here we explain more about this process leveraging the visualization. Specifically, when guiding i' with the SDS direction and prompt c' {2',b} (top-left panel in Fig. S2). the weather feature (e.g., the appearance rain on the ground) changes from æ (sunny) to x' (rainy). However, due to entanglement, the non-descendant feature, such as trees, also tends to change from r to. Meanwhile, the descendant feature (e.g.. the umbrella) correctly changes to y'. In contrast, when guiding with the SDS direction and c = {z, b) (bottom left panel of Fig. S2 the weather changes from x' (rainy) to (sunny). Yet, the non-descendant features again change the same direction from to r', and and the umbrella no longer tends to change to y'. After combining these contrasting directions (right panel in Fig. ( the weather reliably changes from to x', the umbrella (a descendant) appropriately changes as well, and the non-descendant feature (e.g., the tree) remains invariant, correcting the entanglement artifacts through cancellation.
To obtain 1, we move i, in the DDS direction, which only changes feature X but preserves others as the same. Specifically, a subset of time steps T is selected, i.e., T C (1...., T}. And for every t T, we follow the DDS
100 1(0) + Xe( c', ci,,,,
(24)
where A, is hyperparamter controlling the intensity of the change of X. Finally, to prevent 0, from being too large, 0, is cut to fixed maximum value 0max- This step is needed since valid solution
to satisfy the first constraint of Eq. 5 is 0₃ = 02 = Or as illustrated in Eq. 23 Thus, 0, should be encouraged to be around
The complete procedure for searching zi is shown Alg
19

---

V L SDS ē( {x', b), f)
S1 (x. 0-
V L CTF &
ı( (x,
Tree
Um y
Tree
Weather
Um =
Weather
X(Weather) =X R(Tree) = (YUU)) y
Piry Ix,,,,,,,,,,,,, 1
0.P(% | b,,,,,, (x,b,r,))
V. SDS 2 (x,b).
Final i'
Tree
y
Weather =
X(Weather) = x', R(Tree) = Y(Um)
Weather(X) = x,, R(Tree) = Y(Um) y
P(r, v,)), Ply. x., b,,,),
Figure S2: The contracting updating direction VeTFF. The entanglement -endeeseendants are
canceled by contrasting while the intervention and effect on descendants are reflected in results.
Omitted Proof's
C.1 Proof of Thm.
Theorem 1. Consider the true underlying ASCM M over (V,I) Let ND denote nx. ND(X;))
(non-descendants of X) in Gy and DE denote Ux, DE(X.) (descendants of X) in GV.L. The largel I-clf query pa (Ix = I can be factorized as
Pd (T
i) 11xx()=x')
=
P (Dexx de' v,
(3)
Intervention Consistency Non-descendoozs Invariance
Amount Descendant Changing
where nd 1iND (i), nd' 1iD (1), de (i), and [v,1] hvv.
Proof. We will use etf-calculus 10 Thm 3.1] for solving this counterfactual. Specifically, we will rule the exclusion rule
P(yxa, w*) = P(ix, w+)
(25)
if X n Anc(y) = @ in I5. here Anc(y) Anc(Y) Also, Recall lww is the inverse mapping from to W, for any W VUL.
I= Σ P(t
(26)
sum over v", 1"
(27)
v"
Σ P(2 Vi,"" 1")/[""""""" 1"" (i)
invertibility
(28)
Σ P(L x nd de" 1xx""">.................>> nd" de" |v,
(29)
nd 'e
=P(Xx- = nd de | v,1)
x
invertibility
(30)
=i()x((') = P(nd' de v,
intervention definition
(31)
-1()x((') = [x)]==>ppppmmm P(nd', de v.
Rule 3, X. Anc(ND) = in 9x
(32)
=1(hx(1') = xx]][nnd = nd P(de v,
ND C VUL
(33)
20

---

C.2 Proof of Thm.
We first introduce the following lemma to map F-ctf query to the generative level.
B
Lemma 1. Consider ground truth M For every BD-CLS M there exists another M' over (X, B. NB,, Y,I, that in- duces the same 1-ctf distribution P(Ix [i] and is compatible with G' shown Fig. 7 where Y E Dee
X
ND
Proof. Consider the Def. for any MBD-CIS, the mixing mecha- nism is expressed as:
I (X
(34)
Since J invertible DE ADE (1 where aDE is the mapping from I to DE. According to structure condition Def. 4
Z (B, Uz). According the disentanglement conditions
Figure S3: The structure in
Def. state that function TND iND fi such that
Lemmal
ND TND B)
(35)
Notice that
I f = fi hv, f(X,B,2
(36)
Thus, we construct an M' exact the same as MBBD-CLS except
ND TND (Bz Uz), B) Y hy ffz Uz), I fX B, Z) = (TX, B,N B, Y, NEYY o f(X, B, /z Uz)))
(37)
(38)
(39)
This will lead the same P i) since the mechanism is the same over (X, B, Z} and the diagram is shown in Fig. S3 according to the mechanisms.
Before formally proving Thm. We first state important assumption and clarify notation here. We assume that the domains of X, ND are discrete and finite. P(X, nd, De) denotes all observational quantities that ND takes nd.
Theorem 2 (Causal validity of BD-CLS). Consider an MBD-CLS for M and the target query
(1'2 i) be an estimator for P(ik 1). Then, (a) (intervention) PM (Xx) 1) = 1|x = =x1, where & = lx(i'); (b) (non-descendants) pM (nd',
("
i). Let p.Nauaes
v.1) 1/d nd], (c) (descendants) p.MRo-sss (de' v,1) is etf-consistent with P (dc |v11)
under (P(X, nd. De), fX.ND.)) where w hww (i), w = (i') (1') for any W VUL.
Proof. (a) intervention. According to Def. Sef..4cmm econdition (1), Mi
(I (I
(40)
Due to invertibility from I X in M and the conditional distribution match, the invertibility also exists from to in bid-cls Then p.MaaaLs
(xx² v, = 1[x = x'), where x = hi'))).
(b) non-descendant. Then use Lem. to map F-ctf query to the generative level. According to Def. and ND IND,
MM (ndk vv
(41)
= piipiss (nd' |i)
(42)
-f
[ (i") ndd ap (1)
(43)
..p''nd'e (nd'
purthee (Ix (1) pMe pMx(I [1), Lem.
(44)
(45)
21

---

B
X
ND X
ND
(a) G1
(b) G2
Figure S4: Diagrams for two proxy SCMs in Proof for Thm.
Since ND in M' is also non-descendant of X, according to proof of Thm.
p.M (nd i]) = [[nd' = nd]
(46)
(47)
(c) descendant. According to Def. 3 now the proof goal is to show the bound of F-ctf query P.te (de' x, .n,dd) (given g. and P(V, I)) is in the bound of query p (ddes nd, de) (given GXNND.I and P(X, nd, DE)). Leveraging Lem[] we can map F-ctf query the generative level.
Following the same procedure of the mapping of an F-etf query above using the fact Y = hy
p.tpi (de's x, nd, de)
(48)
-pittuss (de'x |i)
(49)
JrEx, [ (1') v/ (i) nd] dpM. (1,1%)
(50)
Six 1 [NND (i) nd] d.p.iww. (i)
=P.M -dd'' x, de)
Lem. and Eq.
(51)
(52)
Now the goal is to proof PMM (de'> x, nd, de) (given gM and P(X, nd, I)) is in the bound of query Pa (dee x, nd, de) (given IX.ND.I and P(X, nd, I)). Since PM' (I x, b) p x, b) = PM x.,),
p.M' ND X,B) PME(DEE
(DE, ND X,B)
(53)
To illustrate, this means that the observational distributions over (DE, ND, X, B} are equivalent between M and M'. However the graph between them are different, i.e., G G Thus, the proof goal now is show the bound of the same query given same observational distribution but different graph. We will prove that the bound the same as given G1 and the diagram G2 shown in Fig. S44 the observational distribution can be matched between M(G2) and M(G1). ). To simplify the notation, we simplify DE as Y since GX,ND,I does not restrict any structure among DE.
Denote the two bounds as Bg, and B5G- BBg S Be, since G2 is a subgraph of Gi thus M(G2) satisfy all constraints induced in MMGG)).
We prove Be, C Bg, b proving that for every M, that induces GI, we can find another M2 that induces G2 such that pMi (y/4 b, nd, v) p.Ma x, b, nd, y). Formally, We will use the spirit of canonical SCM (61 Def. 2.2] for expressing the P.M (t'= y. nd, x, b) and P.Ma (v'e y. nd, x, b). Denote the domain of X as (x1,X2, X2,...; xd, J. the domain of as (bi, ba...., b.. the domain of ND as {ndj, nd2.... ndd,a }. the domain of Y as {v1, V2. ....... yd,
22

---

Consider the function class FY
Fz {(y + {x1, b1, ndj } 3111 + [x1, b1, n..)....... 31 t {xdz, bid, ndddaa) }
{u (I,, ndl J. y1 + {x1, b1, nd2)..................... ................................ .22 t {xdxx bud: nddinal ".,
(54)
Mdd - (xi, b1, ...)....... (Xddx bāg ndddad (yd, - (x,,b,, ndi,i,,dd,:... Bdd,
and the function class FND:
Fzy
Undj {bi: ndd {b2). ndl + {bat} {ndj f- {bi. Indd f- nd2 + {bor}
(55)
(ndona + {bi, hdnddaaa f {b2)..... nddad t {bot)
Consider f and M, in the canonical type for Mi
Y f+ franonical XX B, R)
(56)
ND JND ND
rcanonical (B,
(57)
where the domain of R are discrete values (ryND, fr INNEFND,FEEFY
Let,
femmomical (x, b, TYND = 1) = f (X, B) and IND framonical (x, b, SN = 1)
JND
(B).
For every M', the functions fr and I'ND can be expressed in the above way. Then:
of = PM (skr x. Ey' b.d) (d= P(ryr,
1)
P(nd, y x)
(58)
and the conditional observational distribution can be expressed as:
Q =PM
.MM'(y, (y, nd x, b)
Σ
P(rsy,JND)
(59)
y= (x, /DD
and
Q3 pM (y', nd x),b)=
Σ
P(ryr,,ND)
(60)
S (x', bdl. nd (b)
For M2, consider the same function class
Fz ((n {x1, bi, nd, J. < {x1, bi, an2).d........ < {xdx bd nddda)
(2) - {xi, bi, nd J.y1 < {xi, bi,
-- (d2x bag ndd.aa
(61)
(va, < {xi, bi, ndd) Ud, + (x1, b1, nd2).d)...... Vd, {xdx. bur nddda)
But with a different canonical model
Y JMa peanonical (X, B,S)
(62)
(63)
where the domain of R are discrete values (S brr For given canonical M1, the counterfac-
tual quantity
PM= y, nd, x, b) Ey=r((xeb.dd) (xt.../bnddd////////./
P(agr 1)
(64)
P(y x, b, nd)
and the condition observational distribution:
O = =PM (y nd, x, b) =
Σ
P(sy= =1)
(65)
y= (x.
23

---

and
03 = PMa nd,x'.b)
E
P(s 1)
(66)
y' ff= (x).
We set Qf to be equivalent to Qi namely,
Σ
Plapr =)
(67)
y'=f" b.ndd, (x.b.nd)
P(ryr, =1) P(y x, b, nd)
Σ
(68)
P(nd, x. b)
8= (x',b)d Sr ndd
P(rgr,gND =1
(d
(69)
P(nd] x)
(70)
This set is feasible due to the following reason. First, the observation constrain P(y, x, b, nd)
(Q), and P(v',1 x', b, nd) (Q) Q3) are satisfied. The reason is that all summed term P(s) in Q
are strictly subsets of P(s) in Q2 and (()) in Q3: all summed term P(r) in Ql are strictly subsets of P(r) in Q2 and P(r) in Q3: Q = Q Q = Q3- Setting these sub-terms will not violate the sum. Second, this will not violate the observational constraints for any P(y", x" b, nd), where (v"" + y, x" 4 x or {v5" + y/, x" x'). For all other observation quantity P(y", x",b, nd), there no P(S) in Qi and P(R) in Q belongs to them. Third, this will not violate the observational constraints for any P(y",1x' b" nd") for b" * b and nd" o nd. Any terms in Q and Qi are partially summed into these quantities. To construct all terms in Qi P(y", x", b", ndd) must be sum for all v" E Domain(Y). Then Q satisfies
o S Σ P(y", 3
x" P b", nd") =1
(71)
From this construction, we know the bound is the same as given G1 and the diagram G2 shown in Fig. S5 And similarly, if any edge from B and ND into Y is missing in G2 compared to the true diagram G", but M(g") and M(92) are capable of inducing the same observational distribution, the bound will be the same. Then we conclude P.M P.M'''s y,, nd) (given gM and P(X, nd, Y)) is in the bound of query (2 y, n. since the bound of P.M (u'x y. nd, x) (given GM' is the same with the bound of P* (B/r y, nd)
C.3 Proof of Prop. 1
We first list the important assumption about the pretrained model. We assume the pretrained model MD matches perfectly the conditional distribution P*[I] X, B), i.e.,
(I x|x" nM
P P* x" b²)
(72)
for every x" and b".
Proposition (Sampling I-ctf instances through SD model). Consider a ground-truth ASCM M over IJ. the target P* (Ix i) and a pre-trained SD model MS Suppose there exists pair of transformations satisfying Z 2/ (X, N) and N 22 B. Z). Denotes v/(x, x', n) JSD((', b, 122('', b, th (x. b. n))). Then there exists a BD-CLS MBD-CLS over (X, B, Z} such that the causal validity in Thm. are also offered by En PM (n i, x, b)1(1' = 1x x', b, n)), if
v(x, x b. N) P(I x',,), and hinD x,, b, n)) = NiND
(5)
Proof. let lx (i) and b (i). First, all samples i' from >, p. (n i, b) (xx x', b, n)] satisfied the interventional consistency validity, which means x' = hxx()) (1). This is
because p.rS (I x bz) = P*[ x", b") and
(1 x", b²) 0, if + hxx''), P' b²) =1 x" = hxx
(73)
24

---

Second, all samples i' from En PM i, validity since
ix b)1 = (xxx x', b, n)] satisfied the non-descendant
(1' = (DND(VE2) x,, b, n)) = (i)
(74)
Then we will prove that all samples i' from En P. (n b), = (xx x', b, n)] also satisfies
descendant validity. since MSSD matches the observational distribution P(T x, x.b). p.u (x, b I) = 1 due to the invertibility. Then,
pm 50 (n,x,b I) = pMd (n 1,x, ()x,,) b II)
(75)
= -PM (n I,x, b)
(76)
Let M be an SCM over (X, B, 2} and compatible with 9 in Fig.
Σ M (n i,x, b) (xxx b)
(77)
Σ PMd (n i)p ( (x b)) = i')
(78)
p. (n i))PMM () (x by
b)) = i')
(79)
Σ P.vi (n, b" i) pM b, b, m). x,, b)) = 1')
(80)
nxx" b"
pM psp (x, b, (z, x', b)) = |
(81)
pM (z, x, x, ii pyp (ix xx b,)
(82)
= -P. (Ix=x' = =i'li
(83)
Since (xxx x', b2 N) ~ (I | x', x) and (x) b, vo( b),,,,,,))) = (SD N), which
implies P. [I X, b). Then M is BD-CLS offering escendant validity.
C.4 Proof of Prop.
Proposition 2 (Toy entanglement between binary X, Y and R). Consider binary X, non- descendant R and descendant Y. Suppose (y paa) # (" pay). ). Suppose R and Y are both entangled with {X, B, } in MSD and R = (R(, B,N)) and Y = B,N, where N₁ N2 1 and N N2 S N. then pMrD (22, b, b,) pM (12 b,r, y) and p (V'r (icl,,,,r,000 b,r,v) + p (V/l2', b,r.y).
Proof. Construct an SCM M over R. with fn Tn. fy = ry and Un N₁. Un N2.
Then
MM (w'r Ix, br,,)))= = PM(ws. I. (r,b,r,))
(84)
for any W& E (R.Y). With fact N, 1 N2,
PM (1' (x,,,,,,) = p.t( |x, b,r)
(85)
and
PM x, b) = p.. x. b.b.
(86)
Since Wx 1 X for any W E (R,Y) in M
pM (ww,, Wa b) MM (W L.,b) b) = p. (w. wx 1 b)
(87)
25

---

Red Digit (DCC 0)
Green Digit (DC 1)
Small Digits
DC
3
(D 5)
BW
<
BC
Big Digits
9
G
7
7
(D 5)
0
(a)
(b)
Figure S5: The causal diagram and samples from the ground truth generation process in Colored MNIST and Bars experiments.
When P(w] 2. b) P(w x b).
PM (wyr,
x. b) P.M (w. Wz b),b)
(88)
P(w x, b)
P(w x2,b)
p (war
Iy b, p.
pm (w2 ae, b,
(89)
PMuwe x. (,b,w) w) 1 p.M (w> b,,b,,")
(90)
PM(w'rr x. b, w) pM x', b, w)
(91)
Since B is the backdoor set that X 1 R B in M we have P(r b) P(r x1,b), then
P.M (r's x, b, r) = p.M n', r)r.
On the other hand, since Ply x. b) + Ply b).
(M (,,,,) y) + pM (2's | byy,,))
(92)
D Experiments
D.1 Colored MNIST and Bars
We first evaluate the guarantees provided by BD-CLS (Thm on a modified MNIST dataset [4] 36
featuring colored digits and bars. [ The ground truth ASCM includes generative factors: Digit (0-9
D), Digit Color (red: DC = 0: green: DC = 1), Bar Width (thin: BW = 0; thick: BW = 1), Bar Color (red: BC = 0; green: BC 1), and other latent factors such handwriting style. The causal
relationships are shown in Fig. S5 a) Other factors (e.g., writing style S) are considered independent
factors and ignored in the diagram.
To illustrate, digit (D) and digit color (DC) are confounded, showing negative correlation: the larger digits (2 5) tend to be red, and the smaller digits (< 5) tend be green, but do not directly affect each other. The digit color (DC) has a positive effect on bar color (BC); for example, red digits more likely have red bars. The digit (D) has positive effect bar width (BW); larger digits are more likely to be with thick bars. However, when the digit color is green, this causal relationship is flipped, and the digit negatively affects the bar width. Formally, the ground truth generation process M given by
D UD DC 1(D 2 5] o Une
BC DC UBC
(93)
BW (1I 2 5] AU) @ (1[D < A @ DC
S (Us)
I fi(D, DC, BC,S),
'A bar in image refers complete of pixels with the color.
26

---

Image
Image
What would the image look like if the digit had been "?"?
Change the dight
D =T DE =0 IC O AW =
Countefacial Initaa
Causal Edit Principles
T.DC BC
7 7
7
7
Intervention Condstency: D =
with prbbbiitt (873
Nan san-drsoendants Invartance DCC
7
Changes
BD-CLS (Ours. AW unibced)
CDiiluton
CGN
NOM [wiS (iblls
probitilly (.1)
(a)
Figure S6: Replot of Fig. 5
Edit red "I" with thin red bar to digit "7". (a) Expectation of
counterfactual consistent editing; (b) Edit results. Top initial image. Bottom counterfactual images.
where the exogenous variable distributions are:
Un ~ Uniform(0, 9]
UD Bernoulli(O.75)
Una Bernoulli(O.4))
(94)
U1 Bernoull
U2 ~ Bernoulli(0.))
Fig. S5
b) shows 50 random samples in the data set.
Task 1: Counterfactually editing digits
We first consider editing the digit 1) in an image. Suppose that editing red "1" with thin red bar (D = 1, DC = 0, BC = 0, BW = 0) and wonder what would happen had the digit "1" been "7". According the data generation model M and the counterfactual behavior delivered by Thm. [ the digit should be a "7", which implies interventional consistency achieved; (2) the non-descendants should be invariant. So digit color (DC) and bar color (BC) remain red;, (3) The descendant BW (bar width) should change, and the probability of being thicker is
Q1 = P(BW = = 1, DC = 0, BC = 0, BW = 0)
(95)
To guarantee counterfactual consistency (Def. 3) the estimation of Qı should be within the bound [0.73, 0.82] according to Def. 3 These edit expectations are summarized in Fig. S6(a).
The editing results shown in Fig. (G) All models achieve interventional consistency, that is, all edited images depict the digit '7' However, CDiffusion fails to preserve non-descendant invariance: both the digit color and bar color sometimes change to green. CGN, on the other hand, fails reflect descendant delta: the bar width remains unchanged, even when the digit changes. In contrast, both the BD-CLS (without BW labels) and the NCM (with full labels) achieve counterfactual consistency. They preserve the color of both the digit and the bar and successfully induce increase width in bar width when editing the digit. Notably, while the fully supervised NCM requires labeled data for BW, BD-CLS achieves the same behavior without requiring those labels, demonstrating its ability to provide counterfactual consistency for unlabeled features.
To quantify descendant delta, we report the results of estimating the query Q in Fig. S7 a). Specifically, we repeat each method four times and measure the probability that the bar becomes thicker after changing the digit to "7". The numerical results show that both BD-CLS and the NCM with full labels maintain the estimate within the theoretical bounds, whereas the CDiffusion and CGN do not.
Task 2: Counterfactually Edit Digit Color
We next consider editing a digit's s color. Suppose that we are editing green "O" with thick green and wonder what would happen had the digit color been red. According to the data generation model M and the counterfactual behavior established by Thm. T the digit should be green, which implies interventional consistency achieved; (2) the non-descendants should be invariant. So the digit (D) remain a "0"; (3) The descendant BC (bar color) should change, and the probability of
27

---

PLAMicci 1,DC 0,8C -
ON
(a)
(c)
Figure S7: Numerical evaluations of F-ctf queries in Colored MNIST and Bars experiments. (a) coun- terfactually edit digits; (b, c) counterfactually edit digit color. To achieve counterfactual consistency, the estimation should fall within the bounds (between the green line and the red line).
Initial Image
Counterfactual Image
Change digit color to red
What would the image look like if the digit color had been red?
D O=0,0C 8C 1.BW
Lantd Imaggs
BC
Causal Principles
with protaabilty in (1L32A]]
Interveation DC
0
Nos-doccendants Invariance: D =
D DL
BMM
Doscondants Delta:
with protaabkitt (105.4)
BC PuBCicaal 1D - DC - L. BC 1,BW
BD-CLS 8W; BC
CDiitation
OGN
NOM (wth labda
PLRlE a O, DC I. BC 1.8
unldelld)
(a)
(b)
Figure S8: Edit green "O" with green thick bar to red digit. (a) Expectation of counterfactual consistent editing; (b) Edit results. Top initial image. Bottom counterfactual images.
being red is
Q2 = PBBCccco== -
1,BC - BW =)
DC
(96)
To guarantee counterfactual consistency, the estimation of Q1 should be within the bound [0.33, 1]. Another descendant BW (bar width) should also change and the probability of being thin
Q3 = P(BW = = 0, DC = 1, BC = 1, BW 1)
(97)
To guarantee counterfactual consistency, the estimation of Q3 should be within the bound [0.89, 1] according to Def. 3 These edit expectations are shown in Fig. S6 a). Unlike editing digits (Task 1), BD-CLS are obtained in this task with only labels of and DC.
The editing results are shown in Fig. S6(b). b). All models achieve interventional consistency, that is, all edited images depict the red digit. However, CDiffusion fails to preserve con-descendant invariance: the digit almost always changes. CGN, on the other hand, does not reflect the change in descendant: the bar color and width remain unchanged, even when the digit changes. In contrast, both the BD-CLS and the NCM with full labels achieve counterfactually consistent results. They preserve the digit, and successfully change the bar color to red and reduce the bar width. Notably, while the fully supervised NCM requires labeled data for BC and BW, BD-CLS achieves the same behavior without requiring those labels, demonstrating its ability to provide counterfactual consistency for unlabeled features.
To quantify descendant delta, we report the results of estimating the query Q in Fig. S7 b. c). Specifically, we repeat each method four times and measure (1) the probability that the bar becomes red after changing the digit to red (Fig. ()); (2) the probability that the bar becomes thicker after changing the digit to red (Fig. S7 c)). The numerical results show that both BD-CLS and the NCM with full labels maintain the estimate within the theoretical bounds, whereas the CDiffusion and CGN do not.
28

---

DC
D
D
BC and N
DC
N
I
(a)
(b)
Figure S9: (a) BD-CLS for Task (b) BD-CLS for Task 2.
D.2 Implementation Details for Colored MNIST and Bars Experiments
We first provide more details on the architectures of the BD-CLS and other baselines: conditional diffusion, CGN, and NCM with supervision. We first present the formal definition of NCM (59 60)
Definition 11 (G-Constrained Neural Causal Model (G-NCM)). Given causal diagram 9, a G- constrained Neural Causal Model (for short, G-NCM) M(0) over variables with parameters
0 {0v, V E VJ is an SCM (Ū, V, F, P(Ū)) such that - = {Üc C C V), where (1) each Ü is associated with some subset of variables C S V, and X3 = [0. 1] for all Ū e Ü;
(2) F = -f.v V E V}. where each fv; is a feed forward neural network parameterized by
0y. E mapping values of Uv. U Pay, to values of V3 for Uv, = {0c ÜC U.t ViE CJ and
Pay, (V,);
(3) P(Ū) is defined s.t. Ū ~ (0, 1) for each Ū E Ü.
BD-CLS. As illustrated in Sec. 5.1 the implementation is based on NCM. The architecture designed has two stages, which mimics the ASCM generation process. In the first stage, we train a GAN-NCM 600 on observed generative factors the generative level. Specifically, the observed generative factors are {D, DC, BC} and BW does not belong to V in the NCM for task 1 (editing digits). In the second stage, we train conditional diffusion model ft taking conditions {D, DC, BC} and noise N as input to generate image I.
NCMs ensure that the resulting model satisfies the definition of BD-CLS, For example, in our setting, Digit Color (DC) serves as a backdoor set for Digit (D) based on the ground-truth causal graph G shown in Fig.7 According to Def this augmented NCM model satisfies the generation condition, as the conditional diffusion model is trained to approximate P(I D, DC). Second, taking Z (D,N). N). the non-descendants (DC, BC) are directly modeled in the NCM and remain disentangled from the intervention variable D. The structure of this augmented NCM, shown in Fig. (99a) a), aligns with the structural condition in Def[4] confirming its compatibility with the BD- CLS framework. For task 2 (editing digit's s color), the observed generative factors are (D, DC} and [BC, BW) do not belong to V in the NCM. The corresponding NCM structure is shown in Fig.
$9 b).
For detailed implementation, at the generative level, each function fv Ff in M is a feedforward neural network with hidden layers of width 64 with layer normalization applied (l. Each exogenous
variable Ū E Ô a standard normal four-dimensional distribution. The generator and discriminator are trained with learning rate of 10-1, and are optimized with Adam optimizer
(27) All training processes are performed with batch size of 100. The model architecture of conditional diffusion follows the implementation in (20). Specifically,
four feature map resolutions (32 x 32 to 4 x 4). Two residual blocks per feature map and self-attention blocks at 16 x 16 are implemented. The total step size 7 is set as 1000. We train the model on a single NVIDIA H100 GPU epoch for 100 epoch. In addition, we generate a pair of initial image and counterfactual image from the model in this experiment. In other words, we do not take real image input, but generate the initial image and edit it at the same time.
Conditional Diffusion. The first non-causal baseline is chosen as the conditional diffusion model that approximates P(I] X). To have a better comparison, we use the exact same architecture of fr for BD-CLS.
29

---

Initial Image
DDPM Inversion
SDEdit
DDS
BD-CLS (ours)
Edit task 3:
What would the image look like if the weather had been rainy?
Age
Weather
Scene
Shadow
Pose
Umbrella
Person
Height
(a)
(b)
Figure $10: The causal diagrams and image editing results for Task 3 Edit the weather (umbrella).
CGN [50]. The second baseline is CGN. We follow the implementation in (36. CGN proposes to encode an SCM over variables Shape, Texture, Background, and Label into the proxy generative model. Given the label of the image, Shape, Texture, Background are independent. Formally, the mechanism of this SCM designed as follows:
Label + fi(Ui)
Shape + Fa(Label, Ua)
Texture + Leabel,, U.)
(98)
Background t Fo (Label, Ub)
I I (R(Shape,TTee Texture, Background),
where mechanism fs, ftr fo designed learn the conditional distribution P(V Label) with prior knowledge, where V e {Shape, T Texture, Background]. The composition mechanism fi is not learned but is defined analytically. After fitting the given observational distribution P(Label, I). the intervention can be performed by changing Label. task 1, the digit and the writing style are regarded Shape; the color is regarded Texture and the colored bar is regarded Background. In task 2, the color the digit and the writing style are considered as Shape; the digit is considered background and the colored bar regarded T Tezture and the colored bar is regarded as Background.
We use the same conditional diffusion model learn mechanism Sa. ft, Jú. are directly hand designed in task while fi is learned through conditional diffusion task 2. Theoretically, CGN learns the independent mechanism from Shape, Terture, Background to the image. After performing interventions on one variable, others should be preserved in the image.
Full supervised NCM. The third baseline is chosen as fully supervised NCMs. The implementation of this casual basline exactly the same BD-CLS but with all the labels over (D, DC, BC, BW).
D.3 Real World Scenarios Editing
In this section, we validate BD-CLS-Edit for sampling counterfactual images in more open scenarios. We compare it against three non-causal SOTAs: (1) DDPM Inversion and (2) SDEdit 33] representing L. inversion, and (3) 2DS which emphasizes semantic invariance.
Task 3: Counterfactually editing the weather
30

---

Initial Image
DDPM Inversion
SDEdit
DDS
BD-CLS (ours)
Edit task 3:
W hat would the image look like if the weather had been rainy?
Age
Weather
Scene
Shadow
Pose
Umbrella
Person
Height
Figure S11: The causal diagrams and image editing results for Task 3 Edit the weather (shadow).
We begin with the setting from Example 1 where the goal is to change the weather from sunny to rainy in an image of a young (or old) lady in garden (or street). The causal relationships between the generative factors are shown in Fig (10(a)]
According to Theoren[] non-descendants (e.g., scene layout, age, person's 's height) should be pre- served, while descendants (e.g., umbrella, shadows) should change accordingly regardless of whether they are prompted. For example, an umbrella may appear and shadows should become fuzzier on wet ground due to the weather change. As shown in Fig. STO and Fig. SII all methods achieve interventional consistency. However, DDPM inversion alters non-descendants, changing the scene and person's 's height. SDEdit also changes the layout and person's pose. DDS maintains visual similarity to the original image but fails to reflect downstream effects. To illustrate, the umbrella does not appear, and the shadows non-descendants and correctly reflects the causal effects on descendants like the umbrella and shadow.
sunny day preserved. In contrast, BD-CLS preserves
Task 4: Counterfactually editing season
Next, consider editing an image described 'a person in forest' by changing the season from summer to fall. The corresponding causal diagram is shown in Fig a). According to Theoren [ non-descendants, for example, the person's gender and height, forest layout, should be preserved, even if not prompted, while descendants, such as clothing and leaf color, should change according to the causal effect of season. To illustrate, person the fall intends wear more clothes. Fig. S12 b) shows the editing results of our BD-CLS method compared to the baselines. DDPM inversion fails to generate details for the person and the person's 's height changes. SDEdit also changes the person's ''s identity. DDS preserves personal details, but the resulting clothing appears unrealistic since it keeps too many original details in the clothes. In contrast, BD-CLS produces appropriate generate the season and realistic warmer clothing while preserving non-descendant features.
Task 5: Counterfactually editing the scene
Third, we consider editing an image described as "a person grocery store". Specifically, we intervene the scene and aim to change the background to garden. The causal diagram is shown in S13 a). According to the causal structure, non-descendants, for instance, background layout, person's pose, should remain unchanged, while descendants, such as grocery bag, should be
'It is worth noting that even though diagram shows fewer than 10 variables the demonstrated graph, many additional variables are implicitly present in the causal graph, and BD-CLS-EA also provides counter- factual guarantees for these implicit variables. For example, other factors, such as the hair color, and scene lightness, are also part of generative factors in images but not drawn explicitly for the clean presentation.
31

---

Initial Image
DDPM Inversion
SDEdit
DDS
BD-CLS (ours)
Edit task 4:
What would the image look like if the season had been fall?
Season
Leaf colors
Clothes
Pose
layout
Person height
(a)
(b)
Figure S12: The causal diagrams and image editing results for Task 4 Edit the season.
Initial Image
DDPM Inversion
SDEdit
DDS
BD-CLS (ours)
Edit task 5:
What would the image look like if the scene had been garden?
Scone
Gmicer Bag
Ceiling
Pose
Figure S13: The causal diagrams and image editing results for Task 5 Edit the scene.
removed, as it is unlikely to appear in a garden setting. Figure ST3 shows the editing results. DDPM inversion and SDEdit alters the person significantly, failing to preserve non-descendants. DDS retains most personal details, but incorrectly preserves the grocery bag. In contrast, BD-CLS maintains interventional consistency while correctly removing the grocery bag, reflecting the expected causal effect.
Task 6: Counterfactually editing the place and the sport Fourth, we consider editing an image de- scribed as a person is skiing in the snow' by intervening in the place and the sport. The corresponding causal diagram shown in S14 a). According to Thm. non-descendants, such as the position of the person in the image, should remain unchanged, while descendants, including surrounding details and sports equipment, should change accordingly. Figure[S14[b shows the editing results. DDPM inversion alters the person's 's gesture and location, failing to preserve non-descendants. DDS retains most visual details from the original image, but this leads to unrealistic edits: for instance, snowy mountains and trees remain, and skiing gear (e.g., ski poles and clothing) are preserved, which would not usually appear surfing scene. contrast, SDEdit and BD-CLS preserves the person's gesture and location while transforming the background into occan-lik waves and replacing skiing
32

---

Initial Image
DDPM Inversion
SDEdit
DDS
BD-CLS (ours)
Edit task 6:
What would the image look like if the place been sea and the sport been surfing?
Place
Sport
Surrousding details
Sport goods
A
(a)
(b)
Figure S14: The causal diagrams and image editing results for Task 6 Edit the place and the sport.
Task #
DDPM Inversion
SDEdit
DDS
BD-CLS-Edit
3
0.25
0.36
0.13
0.16
0.33
0.15
0.14
0.14
0.71
0.69
0.66
0.61
0.58
0.55
0.57
0.47
Figure S15: LPIPS comparison across editing methods. Lower values indicate stronger structure preservation. BD-CLS-EA maintains low LPIPS across diverse tasks while preserving causal consistency.
gear with appropriate surfing equipment. The reason that SDEdit performs well in this task that there are not too many non-descendants needed to be preserved.
To qualitatively assess the generation results, we edit 120 images for each real-world task and report LPIPS [62] to quantify the degree of structure preservation. A lower LPIPS value indicates that the edited image more similar the original one. According to Thm. 1 lower LPIPS suggests better preservation of non-descendant features. However, a low LPIPS does not necessarily imply better editing quality, as it may conflict with the third principle descendant delta. The LPIPS results shown in Fig. S15 As illustrated, BD-CLS-EAL achieves lower LPIPS values than DDPM Inversion and SDEdit, but higher LPIPS than DDS. In addition, when tasks primarily require preserving non-descendants rather than changing descendants (Tasks and 2), the LPIPS of BD-CLS-EAi close to that of LS inversion methods. Conversely, when tasks demand more modification of descendants than preservation of non-descendants (Tasks 3 and 4), the LPIPS of BD-CLS-Ead again aligns
closely with those methods. This observation aligns with the expected behavior of counterfactual editing: latent-space inversion methods may inadvertently alter non-descendants. while ramantic-ivvarince approaches can mistakenly preserve descendants. contrast, the proposed ED-CLS-Edii generates more causally consistent and realistic results.
D.4 Implementation Details for Real World Scenarios Experiments
We first illustrate the dataset construction process for the real-world scenario. For each scenario, the underlying ASCM can be illustrated by specific text. To illustrate,
Task 3. An image of a (Age) woman standing in a (Scene) during (Weather) day.
Task 4. An image of person standing forest during (Season).
Task 5. An image of lady standing in (Scene).
33

---

Task 6. An image of person (Sport) in (Place).
construct data from the observed distribution P(V,1) and initial images for editing tasks in three steps:
1. Design the generative SCM.
Define structural causal model (SCM) over the labeled generative factors V that is compatible with the causal diagram Gv. and sample concept data v ~ P(V)
2. Generate images.
For each sampled concept datapoint generate an image ~ pI v), where pGan (I
v) denotes text-to-image generator (DiffusionXI. is employed in this work).
3. Select editing inputs.
Collect a set of images that all contain a specific feature x to serve as initial inputs.
Following this procedure, the resulting observed distribution P(V, I) guaranteed be compatible with Iv.1. In addition, all initial images share the same original feature values, which differ from the intervened ones (x F x') assumed in this work. Specifically, a generative SCM over (Seene, Age, W Wather) for Task 3 is designed as follows:
Scene + Us
Age + UAi (Uw + UA22)
(99)
Weather + Uw
where the exogenous variable distributions are:
Ug- Bernoulli (0.5)
UA, Bernouli (0.5)
(100)
U A2 ~ Bernoulli
Uw Bernoulli(0.4)
where Scene = 0 means the scene garden; Age = 0 means the age is young; Weather = 1 means the weather is rainy. Then, data for Scene, Age, Weather be sampled from this generative SCM. In Step 2, for each sampled datapoint-such as Scene 0, Age = 1, Weather = 0-an image i' is generated by providing the prompt "A woman standing in garden text-to-image generator. Finally, in Step 3, all images corresponding to the sunny condition are collected as the initial set for editing.
sunny day" to the
We use Stable Diffusion [3),3), and all editing is performed the latent space, after encoding input image. other words, j(0) in Alg. 3 refers to the latent representation obtained via pre-trained SDXL autoencoder. The input image size is 1024 x 1024 x3, and the image after encoding 128 x 128 x 4. For classifier- free guidance, we fix the parameter w (Eq. I8 is fixed as 7.5. Other hyperparameters in Alg. 3 are given as follows. The total inference steps are set to 200. T of length 40 is randomly sampled from [1, We manually tune the hyperparamters for BD-CLS-EAN S-Edit, including learning rate y, optimization iteration number Nmaas and clipping value max- Specifically,
we compare the combination of y {le le 2, 3, le 4}. mmmax [2, 10, 20), and O,,ax E (1.2, 1.5, 2.0). We found that 7 = Ie and Témax = 10 lead to the best BD-CLS-Èdit's performance considering both effectiveness and optimization time. The BD-CLS-Edii is relatively robust to a. The initial 0 is set to 0 for 0r through 0r-50 and the others are initialized as 1. are optimized individually for each input image. The adjusted parameters follow the coefficients in DDS [8]. The learning rate y is set as 0.1 and the optimization performed with SGD. The experiments also conducted single NVIDIA H100 GPU.
D.5 Sensitivity and Robustness for BD-CLS
In this section, we examine the sensitivity and robustness of BD-CLS to the input causal graph. Recall that Thm. 2 requires the labeled or prompted feature set B to serve a valid backdoor set in the true causal diagram g. Following this requirement, BD-CLS-EAR set the prompt to include a description
'We exclude unlabeled generative factors from the design, there can be hundreds or thousands of them in image, many of which cannot be easily described in text, such subtle lighting details shadow textures.
34

---

D
DC
D
DC
BW
BW
BC
BC
(a)
(b)
Figure S16: Non-complete misspecified causal graphs used for sensitivity analysis experiments. (a) g¹: (b) g"
involving the feature b at step 2. Here, we theoretically and empirically analyze scenarios where the provided causal graph is mis-specified.
Let g' denote a non- non-complete or specified causal graph. the backdoor set B' derived from g' still satisfies the backdoor criterion in the true graph G, then BD-CLS continues to provide causal guarantees. For example, consider the Colored MNIST and Bars experiments (Appl) D.I)
In the ground-truth graph G (F(g)) the digit (D) is confounded with the digit color (DC), and the har color (BC) is a descendant of DC. Even if one mistakenly assumes that both DC and BC are ancestors of D (g' in Fig. [16). the set DC, BC still satisfies the backdoor criterion from D to I in both the incorrect and true graphs. Thus, BD-CLS-EN mis-specification, and counterfactual editing performance unaffected. To ground this, practically evaluate F-etf query P(BWD=7 = 1|D = 1, DC 0, BC = O, BW = 0), the same query as in Fig. a)). using the misspecified graph g'. This query measures the probability that the bar becomes thicker after editing, capturing the causal effect from D to BW. The results (Fig. [S17] third column) show that the F-etf estimate remains within the theoretical bounds, indicating that BD-CLS provides robust and reliable counterfactual edits even with a misspecified input graph.
remains valid under this
Conversely, if the backdoor set derived from the input graph is not valid in the true causal graph, counterfactual editing may become unreliable. For instance, suppose the misspecified graph g" (Fig. b1)(b) incorrectly models DC as descendant of D, leading to its exclusion from the backdoor set B. In this case, interventions on D can inadvertently affect DC, violating the dencenscnnd invariance editing principle in Thm. [ In addition, such incorrect adjustment results in biased estimation of the causal effect from D to BW. Evaluating the same F-ctf query under this G" (column 4 in Fig. $17 confirms this: the estimated effect from D BW is significantly underestimated.
r
BD-CLS (g') 0.73 + 0.03
BD-CLS (G")) 0.45 - 0.06
0.73
0.82
Figure S17 Sensitivity analysis results.
D.6 Editing with BD-CLS for Other Datasets
The experiments for other datasets are provided in the full technical report [37
E Further Discussions and Examples
E.1 Augmented Structural Causal Models (Def. T
We begin with several remarks regarding this ASCM generative process.
Remark (Unlabeled factors L). The unlabeled factors L are the key difference compared to the ASCM [36. An image often contains rich concepts that cannot be fully captured by humans. Thus, the labeled information cannot be given to all of them. For example, annotations of an image are only given several user care features; a text description of an image usually focuses on main concepts and ignores details.
Remark (Unobserved endogenous variable L and unobserved exogenous variable U in ASCM). There can two standard confusions related to the difference between U and L as they are not all unobserved/aaeleed. First, generative factors L Pa(I) are directly reflected in the image, while
35

---

young/old lady is standing in the garden during a rainy/sunny day
We
Ag
We
Ag
4
Um(Ly)
i
i2
(a)
(b)
(c)
Figure S19: (a) The causal diagram over V and L at generative level; (b) The causal diagram over V and I; (c) observational image samples Example.
U is not. Specifically, even if the unsupervised concept is not described in the annotation or text, it exists the image and can mapped by from I. See 2xample more details.
Remark 3 (No exogenous variable U1 for image I). L unobserved parents of image variable I. While one might surmise that L can be treated as the exogenous variable Uą associated with I that is, denote L as U1- U-tthi is not the case. In the SCM, the variables in L are endogenous and may be descendants of V, whereas UI, by definition, must not be descendants of any observed variables.
Remark (The invertibility of fi). The fi is assumed invertible in the generative process since these generative factors are present directly in given image, regardless of features being labeled or not. This assumption is standardly used in non- non-linear ICA and representation learning literature
32
29
24
25
37V.
The ASCM induces causal diagram Gv L.I over all gen- erative factors V, L, and image I. This full diagram can
We
Ag
Um
P(We, Ag, Um)
0
0
0.4416
be projected onto causal diagram involving only the ob- served variables, denoted Gv.1. In this work, we assume that prior knowledge of Gv,r- sometimes abbreviated G for simplicity is available, from the human common sense or from experts the domain and is used as an in- ductive bias. However, it is not assumed that the complete generative graph Gv LLI is known.
0
0.0384
1
0.3136
0
1
0.0064
0
0.0224
0
0.0576
1
0
0.0984
0.0216
Example (continued Example. [D. Consider image describing "a young/old lady is standing in the garden during rainy/non-rainy day". We consider the augmented generative process, ASCM
Figure S18:
P(V)
induced by the
ASCM in Ex
M (U (U1, U3, U4. UL) (V eWe, Ag). ((m La 1. p (U))
(101)
where the mechanisms
We + U1
Ag + U1 + U2
Um + ((-Ag) @ U2) A (We
F
(102)
La + (ig Ut,), + fi, (Uiz)
I eWe,, Ag, Um, Lz, Ta....)
and exogenous variables U1, U2 Us. U4 are independent binary variables, and P(U; = 1) = P(U, = 1) 0.4, P(Us 1) = 0.2, P(U, 1) 0.1. UL (UL,, UL..... are also independent of U2, Ug. U.}.
At the generative level, the labeled variables V contain two variables (We, Ag); We represents if the weather rainy (rainy We 1; non-rainy We 0); Ag represents the age the lady (Young Ag 1; Old Ag = 0:). L represents unlabeled factors that do not appear in the text description,
36

---

including whether the person has an umbrella Um( (with umbrella Um 1; without umbrella Um 0), the hair color of the person (L2), pose (Lz), etc. As discussed in Remark [ although these factors are not labeled, they are parents of image variable I, and play different roles with U1, U2 U3, U4, Ur. the generative process. The causal diagram 9 over V, L induced by Ma at the generative level is shown in Fig. S19(a), a). The distribution P(We, Ag. Um, T22 I.3, ...) induced by M is displayed in Fig. [ (only one unlabeled factor L1 (U') is shown explicitly for simplicity). This distribution suggests that there is positive correlation between rainy (We 1) and young age (Ag = 1); a negative correlation between the umbrella (U' = 1) and young age (Ag = 1);
positive correlation between rainy We 1 and umbrella Um +mm=1. =1 1.
In the second stage of the generative process, all V and are mixed by function fi and generate the corresponding pixels. Some image samples {ir i2] from the observational distribution are shown in Fig. (TT)c). The causal diagram 9 over the observed variables V, (projected from the whole
diagram) is shown in Fig. (ST9 b).
Equipped with ASCMs (Def. D. task to edit the concepts X in original image i from = x to X = x' can be formalized as querying an Image counterfactual distribution (1-ctf) P' i' I = i) induced by the true underlying model .M" In addition, ASCMs can formalize the counterfactual effect in editing between generative factors. Formally, given factual factors W, Wi (W1 S V), the probability that factors W2 will be W2 after the edit do(X x') is formalized by a counterfactual quantity P" P"(W x[X=x]] = W2 wi) at the generative level Mo- For example, the task "edit the original image i (shown in Fig. [ST)())
rainy day" can be written as counterfactual distribution "what would the image be had the weather been rainy?". corresponding to the I-ctf distribution (I I= counterfactual probability over features "given an old lady without an umbrella on a sunny day, would the age of the person still old and the umbrella be added?". This probability corresponds to P" (Agwe=1 = 1, Umwe=1 = 1 We 0, Ag = 0, Um = 0). where W, as (We, Ag, Um} (weather, age, umbrella) and the counterfactual set W₂ as {Ag, Um].
i,). One may be interested in
The I-etf query can be mapped back to the generative level by the following result.
Lemma 2. Consider true generative process described by ASCM M" Then,
P.M (1'r 11)
Pm (V (V' 1// v,
(103)
images
generative factors
where v, v, v,
To illustrate, Lemma. 2 states that I-ctf query is equivalent asking "What would all generative factors be had a concept change to x'?". For example, P" (Iwe=1 I= = i2) in Ex. 2 equivalent to asking what age, umbrella, shadow, the pose of the lady, and other unlabeled Tactors would be had the weather changed to raining day. However, it is reasonable that users may only care about counterfactual reasoning about a subset of the generative factors. For example, a user may specifically care what age and umbrella would be had the weather changed to rainy (do(We = 1)) given an old lady without an umbrella on a sunny day, namely counterfactual distribution PM' (Agwe=1, Umwe=1
We Ag 0, Um 0).
E.2 Proxy Models and Latent Space
As illustrated in Sec. 2 editing a given image by alternating latent vectors (do(T =tt), T C Z) in proxy models can be modeled as a counterfactual query P.M (I I)).
Example 3 (continued Ex. 2). Consider the ASCM image generation process M illustrated in Ex. 2 Consider an SCM,
M= (Ū 10, Uz), Z2, 21 (2,,
2. (U
(104)
37

---

Zz
Z1
Z2
Z1
7
Z3
1
Z3
Z1
(a)
Figure $20: (a) Causal proxy model. (b) A standard LSS
where the mechanism,
Z1 + 01
Z2 02 ez
Zz (-Z2) + 0z Z + Z, 00
F
(105)
+ U,, Z6 t ÜL,
I f-)) (i [2, 1 Zz A Za. fī. (Zz, Zs), fig (Ze)....
and Ö follows the same distribution as U of M namely, P(Ū) = Pm (U). It is verifiable that M defines the same mapping from U to I as the true M thus M induces the same observational image distribution, i.e., PM (I) (I). Obtaining synthetic generative models is process to get such an SCM by training models on collected data. Since P.M (I) approximates the true (.M(I.. one can sample from U generate latent factors Z. such latent representations intermediate variables (e.g., hidden units in a neural network), and subsequently generate the corresponding image samples.
Then, editing image through Z be interpreted as evaluating counterfactual distribution, P. (IT=t' i), where C Z. For example, one might intervene the latent representation Z1 as value of to change the weather to rainy day, and the counterfactual distribution is P. (M (Iz)= i).
E.3 Feature Counterfactual Query (Def.
The following example gives illustration about the counterfactuals over features when editing images through a proxy model M
Example (continued Example. 3). Consider the proxy model M in Example 3 Suppose that the
F-etf query interested is the probability that "given old lady without umbrella on sunny day, the
age of the person would still be old and the umbrella would be added if Z₁ = 1". According to Def. 2 the factual set W, is chosen (We, Um} (weather, age, umbrella) and the counterfactual
set W2 is chosen as (Ag, Um) (age, umbrella). Then the F-etf query is P.M (Agz,=1,
We = 0, Ag 0, Um 0). Since (Ag, Um, We) are not endogenous variables in M, the F-etf
query cannot be calculated directly through M and should be computed from Def. 2 To illustrate, the denominator of Eq. evaluates the factual part: the probability of generated images describing "an old lady without an umbrella in sunny day", which is
/
0, 0, JuE (1) 0, hig (i) 0, hūm (i) dP(in) 0, 0,(23s) A Z1 0).
(106)
The numerator evaluates the counterfactual part, integrating over counterfactual worlds P(i, ( such that i describing "an old lady without umbrella in sunny day" and i' ddeccribing "an old lady
38

---

with an umbrella in a rainy day".
J 1 [bwv (1) = 1, (1) = 0, him (i) = 0. (2g (i)) = 0, hiri (i)) = 1, 1. (i) = 1] di,i (2, (21=1))
J.VE.
P(Z1 0, 72 0, (- Z4 0, 2| 0, (Z Za)(z =1)
(107)
Then we have
[Mg (A We 0, = 0, 0,0m = =P(22|z==1)=00 = (Z3 A [2,11 = 1| 21 = Z2 = 0, (3Z3) A Z4 0)
(108)
This quantity implies that applying intervention do(Z1 = 1) on proxy model M for editing would never generate an old woman with an umbrella.
Next, we present an example to illustrate how to evaluate an F-ctf query is a valid estimation for the ground truth using ctf-consistency Def. 3 even when the query not identifiable.
Example 5 (continued Example Consider the true ASCM introduced in Example 2 in M',
the proxy model .M with the F-ctf query (MMgZ Umz =1 We 0, Ag 0, Um = 0) in Example 4 Ctf-consistency provides way to evaluate whether the F-ctf query is ctf-consistent query for the query P [Agwe=1 1, Umme=1 = We 0, Ag 0, Um 0). According to mechanism F (Eq. 102 and P* (U),
We 0, Ag 0, Um = 0) P((-US) A A (-Us) 0) 0.78
P*
P((-Us) A AU = 0)
(109)
This is the ground truth and not immediately obtainable. On the other hand, the bound [1, of this query given P(V) and Gv L can be derived as (see [41] Thm. 9.2.
max (0, 1 P(Um We = 1, Ag 0)
P(Um = = 0 We = 0, Ag = =0) 0.70
(110)
P(Um We 1, 0) = 0.78
r r= min(1, P(Um = 0 We = 0.A9
Def. 3 Bis saying that any value within the bound [0.70, 0.78) is regarded as counterfactual consistent We 0, Ag Um = 0) and
estimation for the ground truth P (Agwe=1 any value out of this bound will be regarded as invalid from a causal stand point. Specifically, since the F-etf query PM (Agz,=1+ Un, We 0, Ag = 0, Um 0) induced by the proxy model is always 0 (Eq. 108 M not a considered as a counterfactual consistent estimator for this query.
Umw e=1
E.4 Counterfactually Editing principles Thm.
We first apply Thm. t to the raining and umbrella setting introduce in Ex. 2
Example (continued Ex. 2 Consider the ASCM introduced in Ex. and the task of editing the weather in image i, (describing old lady standing in sunny day without an umbrella, shown in
Fig. (19(c) to rainy. The target query is written as P(i [1), where i' E Ag. Following Thm. we can have
P* (i
1' (i = 1hi 1]
IM (1') = 0, Tu[e==o] = hi (t',, (1), (1),
Intervention Consistency
Non-deseendants Invariance
p (Umweeo = liūm
Descendant Delia
(111)
The first term ascertains that the weather in the result i' must be indeed rainy. The second term says that non-descendants, such as age, should be invariant after the edit. The third says that the weather's descendants, such the umbrella, should change following Pa (Umwe=o v, 1).
We then present a concrete example to clarify the notion of the "amount of factor may change during editing, yet still fail to reflect valid counterfactual.
why
39

---

Nd nd, DE de
Nd ndd, DE = de
Nd nd, DE de'
Nd nd DE de
Causal
causal Semantic Invariance
Non-causal LS Inversion.
Initial image space
Edited image space
P( = x)
P(I = i' x')
Figure $21: The comparison of non-causal editing methods, and the causal editing methods.
Example (continued Ex. and 5). Consider the true ASCM M introduced in Ex. 2 and the image editing task "change the weather to rainy", which corresponds to I-ctf query Pa (Iwem) i,),
where i, is shown in Fig. S19(c). Since 21 exactly represents We, e, one may use pzza (Iz,=1 in) to
estimate P(Iwe=1 i.).
Consider interesting probability that "given old lady without umbrella a sumny day, an umbrella would be added, if the weather changed to rainy". which corresponds to Q = P" P"
We 0, Ag 0, m 0). According to Def.
estimate is etf consistent with Q if the
estimation is within the bound [0.70, 0.78) Ex..
Consider the proxy model M introduced in Ex. The F-ctf query P. pMM(Umzz1 We = 0, Ag
0, Um = 0) induced by M can be calculated as:
MM (UmmZ =1 We = Ag = 0, Um 0) = =(333 A 24 21 0, Z2 = 0, ^ Z4 0) 0.02
(112)
Thus, the umbrella would be raised with probability 0.02 due to the statistical correlation between Um and (Ag, We). However, the umbrella would be raised at least 0.70 to be etf-consistent. In other words, naively using the correlation between the intervened feature Um and the descendant Um, the amount of descendant change is not guaranteed.
In addition to the discussion between change and invariance in Fig. 3 We provide another graph to illustrate the editing path I between causal methods and non-causal methods shown in Fig. S21
E.5 Backdoor Disentangled Causal Latent Space Def. 4 and Thm. 2
We first give an example of BD-CLS.
in) in Ex. ggiven P(V, , I)) and the causal diagram Gv. shown in Fig. S19(b). Note that {Ag} serves as backdoor set B in Gv. for the intervened variable set (We).
Example 8 (continued Ex. 6). Suppose our goal is the edit task formalized as P* p* (T
Consider the proxy model introduced in Example
M is not an BD-CLS according to Def. To
witness, condition (1) is satisfied setting We = Z₁ Ag Zn and Z BDSCLS = Z.M, ((,, 2z) To
illustrate, according to Eq. 102 and 105
We h111 fī(Z) = Zi,
(113)
Ag = 1111 fī(Z) Z22
(114)
P(I We, = P(I Z Zz).
(115)
40

---

Condition (2) is satisfied since
Ln TLa (Z) [2, 7s)
(116)
L3 TL3 (Z) fī,g (Zs)
(117)
(118)
which implies no L3 is function of 21 (which is in this context). However, condition (3) is
not satisfied since B = {22} is a descendant of X } in this proxy SCM M, which is not compatible with graph shown in Fig. 4
Now we consider another SCM MDD with endogenous variables Z exactly the same with M but different fz, as follows:
Z2 +- 01.
(119)
Then Z (X= = -2) (Zg. Zz....> satisfies the disentanglement requirement
(similar to M illustrated above) and also satisfies the structural requirements since Zg is not descendant of 1 but now confounded with Z1.
Then, the next example shows how the estimation provided by the above BD-CLS satisfies the editing principles in Thm. [
Example 9 (continued Example 8). Consider M introduced in Ex. Notice that {21} = X (We} and {Zz) B {Ag]. Consider the image editing task "change the weather rainy". which is formalized as the target I-ctf query P* (f... i1) in Ex. 8 where i contains the feature (We e we 0, Ag ag 0, Um um 0} and contains the feature {We we' 1, Ag ag' = 0, Um um }}.
First, BD-CLS guarantees interventional consistency.
MM (we'e==1 v,1) = 1[we' = we] =
(120)
Since Ag is a non-descendant of We, Thm. 2 suggests that
MM (agwe=t v, [ag' ag] =
(121)
In other words, BD-CLS guarantees that the feature Age is invariant after editing.
Next, consider the descendant Um. Thm suggests that the estimation PM (um we, ag, um) induced by BD-CLS is etf-consistent with ground truth P* P*(um' v, 1), which means pz (um we, ag. um) is in the optimal bound of P* (um'wezt v,1). According to Example 5 this bound is (0.70, 0.78).
E.6 Difficulties to satisfies constraint Eq. Alg.
In Scc.(4.] we argued that if one naively updateS learnable parameter 0g in the direction shown in Eq. 7 it cannot guarantee the second constraint in the optimization problem (Eq. 5). The following example gives details about how this happens.
Example 10. Consider the underlying true ASCM Mün labeled generative factors X and unlabeled generative factors Y and R with the following generation mechanism:
X U1 Y XAU2
F²
(122)
R + U3
11 + X, + Y,13 +
where U1, U22 Uy are binary variables and P(U, 1) 0.5, 1)) 1) = 0.2.
To illustrate, Y is the descendant X and R is non-descendant. The generative factors {X, Y, 2) mapped 3-bit image I (/1, 22, Ia). The inverse mappings from image feature
= h (I) = 11
Y hy (I) I
(123)
R = (1) = =13,
41

---

which implies that feature X, Y and R are shown in pixel I1 Iz. Is. respectively. Given an image {I 0, 12 = 0, I3 0}, the intervention do(X 1) is possible to change the factor Y but will never change the non-descendant R.
Now consider an SD model May over prompt X and N = (N1, N2, Na), with mechanism
X + U1
J&D N1 t Uni N2 Uny N3 Ns + Una
(124)
11 + X, + X A N1. 13 + (-X @ (X A Na)
where all U are binary variables, and P(U 1) 0.5, P(U,, =1 0.4, P(Un; = 1) 0.2. P(Uns = 1) 0.2, P(Un, = 1) = 0.6. It is verified that MS induce the same conditional observational distribution P(I x) as Many According to the mapping of the image I to features Y, R in Eq. 123. the transformation from X and N to features are shown as,
[Y -ANN { ( ( A N4,
(125)
This indicates that the non- de-deeseeddant feature R is entangled with X. The intervention on X is likely to change R. In the process of searching for the transformation & based on the given SD models, our aim is to alternate 0 to intervene on X and also intervene on I3 to keep R the same. However, the mapping from image to features Eq. 123 is unknown to SI model (Y and R are unobserved), thus the model itself cannot know to keep 13 the same but allow 12 to change.
Prop. 2 shows that even the unobserved mapping TR (from {X, B to non-descendant R) and Ty (from XX B,N to descendant Y) are functions with imput X. counterfactual behaviors can be used to distinguish R and Y. See the next example for illustration.
Example 11 (continued Example TO). First, notice that an empty set B will serve backdoor set for X and image I in the true Misy and M Consider the given image is an 3-bit {41 = 0, 12 = 0,13 = 0}
with feature X 0, Y 0, R 0. Based on the mapping from Eq. 125 we have
puu pu (Mx-1 = 1 X 0, R = 0, = 0) = 0.2
(126)
PM = 1 X 0, - = 0) = 0.4
(127)
(128)
This implies that when X changes from to 1, feature R will change to 1 with probability 0.2. And Y will change to with probability 0.4. Consider another given image {T1 = 1, I2 = 0, = 0} that only I is different. Similarly,
p. (Rx-0 = 1 X 1, R= = 0, = 0) = 0.2
(129)
PM = 1 =
1 0 = 0) = 0
(130)
(131)
This implies that when X changes from back to 0, feature will change to 1 with probability 0.2. And Y will change to 1 with probability 0. This gives us the opportunity to reduce the change of R but keep the change of Y by comparing these two interventions.
E.7 Limitation
We discuss several limitations of our approach. First, since our method relies on pre-trained diffusion models, its performance bound by the capabilities of these models. For example, if a model does not understand the input prompt, interventional consistency may not be achieved, and the edited image may not reflect the intended features.
Second, while our theoretical results demonstrate the soundness of BD-CLS-Edit, the practical implementation, particularly Step 4, which searches for Jb, relies on the expressiveness of the candidate class μ4 (Sect. I.). There is no guarantee that this class can disentangle all non- non-descendants ND, especially in cases involving complex causal relationships with object moving, sizes changing, etc.
For example, editing an image of "a rabbit looking at carrot the forest" to "a rabbit looking at a wolf" fails to reflect the expected size differences: the wolf should appear much larger than the rabbit,
42

---

Change the carrot to wolf
Change the white cat to rat
Category (cllrot/wolf)
Species (Cat/Rt)
Size
Action (look/chase)
(small/big)
(a)
(b)
Figure S22: Failure cases of BD-CLS-Edit.
but this is not captured by the current edit (Fig S22 a)). Similarly, replacing white cat with rat in an image of "a black cat and a white cat room does not correctly reflect the causal effect of chasing behavior: the black cat should be chasing the rat, and both should be in motion, which is not the case in the result shown in (22 In other words, there are some more complex dynamics in the relationships of these objects that not captured in single image.
E.8 Broader Impact
This paper aims to bridge the gap between causal image editing and the capabilities of large-scale pre-trained text-to-image models. Our work contributes the growing need for more principled and reliable generative models by introducing causal framework that respects the underlying structure of the data, rather than relying correlation-driven editing strategies. A key motivation for this work to challenge the common practice in current editing methods that prioritize semantic invariance, ie., preserving much of the original image possible, while ignoring the causal effect of the edit on other semantics. This often leads to unrealistic results, particularly when editing should naturally induce downstream changes. By incorporating causal principles into the editing process, our method enables generative models to produce more realistic, consistent, especially in cases involving complex dependencies between visual features, which is beneficial for downstream tasks related reliability, interpretability, and fairness generation.
E.9
Safeguards
Similar to previous generative methods, our framework could be misused, for example, to manipulate visual content in ways that appear causally plausible but are misleading, such as in the spread of misinformation the generation of unsafe content. Since our method builds on pre-trained models rather than creating new one, existing safety mechanisms developed for diffusion- based models can be applied to enhance the safety of our approach [16 35 52 30
43

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that we theoretically show that the proposed BD-CLS can give counterfactual guarantee when editing images and this is shown in Sec. 3 We claim
design BD-CLS-Edit (Alg. I that can generate counterfactual consistent samples and this is illustrated Sec. 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the assumptions made in this paper in Appendix Section E.7 and how future work can potentially improve upon our limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
44

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: each proposition and Theorem Thm. and Prop. 1 2 stated in the main text are numbered. And provide corresponding proof and assumptions in Appendix Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes)
Justification:
We provide details of our experiments in Appendix Section
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
45

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code is released https://githu..com/rree 1LBDD-CS--EBBB
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips cc/ wtc/publiic// cubbiic/guides gwt.///ieee//eee// Codeeoomiiieeeee duse//uuuude//o SanSSuubissss solbbmmsssioo ssisnnoiccyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide details of our experiments Appendix Section D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we report the error bar for numerical experiments in Sec. 5.1 show error bar for image generation results.
We do not
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
46

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details of our experiments Appendix Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ttt.itttrrrpp/.oooorr...ooo///tt
Answer: [Yes]
Justification:
We provide details of our experiments in Appendix Section
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in Appendix Section E.8
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
47

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss safeguards in Appendix Section E.9
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
(22. which we leveraged their code to produce experimental results shown in the paper. We do not repackage any datasets, code.
Justification: We cited 36
18
and
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
48

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Our paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
49

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method is not involved with LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur//: httpp/////////oofeeeeee///////// for what should or should not be described.
50


Reviews:
[{"review_id": "0BQeXmHG05", "reviewer": "Reviewer_NNpN", "summary": "This paper introduces Backdoor Disentangled Causal Latent Space (BD-CLS), a method that enables edits to reflect causal effects without altering unrelated features, even under weak supervision. It proposes BD-CLS-Edit, an algorithm that extracts this structure from pre-trained diffusion models like Stable Diffusion without retraining. The approach ensures edits are causally consistent, scalable, and effective even when only partial labels are available. Results on synthetic and real-world tasks show improved causal fidelity compared to existing baselines.", "questions": "While results on real-world images are promising, most examples involve relatively small-scale causal graphs (3–5 variables). How does BD-CLS-Edit scale when editing in scenes with richer semantics?", "limitations": "Yes, but only in the supplementary materials.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper presents a solid theoretical framework for counterfactual image editing. It integrates well with pre-trained diffusion models and requires no retraining. The structure is clear however the paper relies heavily on the supplementary material which make reading harder. The work addresses a critical gap in generative modeling. The ability to edit images in a causally grounded way under weak supervision is valuable. The formulation is original and timely.", "was_revised": true, "final_justification": "Based on the overall quality of the submission and the authors’ clarification regarding the scale of the causal graph, in the rebuttal, which addressed my key comments, I’m inclined to recommend acceptance for this paper.", "rebuttal": "We thank the reviewer for the positive evaluation of our work and the thoughtful comments. Below, we address your feedback and questions in detail.\n\n> ***W1. the paper relies heavily on the supplementary material which make reading harder.***\n\nThanks for pointing this out. We have plenty of examples and experimental results to validate our theorem and algorithmic contributions. Due to space constraints, we had to move some of them to the supplementary material. If the paper is accepted, we will utilize the additional page to relocate some content from the supplementary material into the main paper, hopefully translating into improved readability.\n\n> ***Q1. While results on real-world images are promising, most examples involve relatively small-scale causal graphs (3–5 variables). How does BD-CLS-Edit scale when editing in scenes with richer semantics?***\n\nWe appreciate this important question and would like to provide further elaboration. \n\nFirst, we note that even though our real-world experiments (e.g., Fig. 9, Fig. S10–S14, p. 9, 29-32) explicitly show only 3–5 variables in the demonstrated graph, **many additional variables are implicitly present in the causal graph, and BD-CLS-Edit also provides counterfactual guarantees for these implicit variables** (as long as they obey certain topological requirements as delineated by Thm. 2). This is an essential, compelling feature of the proposed framework.  To illustrate, generative factors are divided into two buckets (1) observed ones and (2) unobserved ones. Real-world images often involve numerous concepts and fine-grained details, many of which are either unlabeled or not explicitly described in the prompts. As a result, unobserved factors can involve a wide range of latent concepts.\n**We drew several unobserved factors in the graph for discussion. However, our theoretical results and algorithm are not limited to the specific unobserved factors shown; they remain valid for the other unobserved factors.** \n\nFor concreteness, consider the weather-editing task (Fig. 9 (a)), where we explicitly draw variables such as weather, age, umbrella, shadow, and scene. However, other factors, such as the person’s pose, scene layout, hair color, and scene lightness, are also part of generative factors in images but are not drawn explicitly for the clean presentation. \n\nBD-CLS-Edit also gives causal guarantees for these variables. As shown in Fig. 9, edits to the “weather” do not affect the pose, scene layout, and hair color (non-descendants of weather), while descendant features such as the lightness changes appropriately. We will clarify this point to make the scope of the settings more explicit in the next revision.\n\nSecond, we note that as the number of variables grows, the performance of BD-CLS-Edit may be increasingly constrained by the capacity of the pre-trained backbone model. Theoretically, BD-CLS-Edit assumes access to a pretrained model that can fit the conditional distribution $P(\\mathbf{I} \\mid \\mathbf{x, b})$. In practice, modeling this distribution is more challenging as the size of the set $\\{\\mathbf{X}, \\mathbf{B})$ increases, and the concepts become more complex. Improving the causal guarantees when involving more complex graph structures and more variables could be an interesting question for future work. \n\nIn terms of the larger context and history, we note that, in a novel way, the contributions of this paper constitute an important step toward enabling large-scale generative models to exhibit causal capabilities and perform counterfactual reasoning. To the best of our knowledge, the theoretical contributions presented here tie these very different modalities together in a fundamental way. We hope this connection, now formally established, can be leveraged to build not only larger, but also more realistic models. In other words, we view this paper as a first step in initiating a conversation with the community about the necessary developments on the causal-generative front, including further scalability challenges, as noted by the reviewer."}, {"review_id": "jf3zaAHqj0", "reviewer": "Reviewer_hSEs", "summary": "This paper aims to study the counterfactual image editing problem with pretrained diffusion models. Current image editing works often ignore causal consistency, where some works change non-relevant parts and others fail to edit logically relevant parts. The authors claim there lack of scalability and existing methods rely on full supervision. On the contrary, the authors propose a theoretically grounded (based on Structural Causal Models) image editing method, which (1) considers the causal relationship between different factors; (2) is efficient without training the full model; (3) can work with weak supervision. The proposed method, BD-CLS-Edit, learns to optimize the linear combination parameters of conditional variables $c$ and $c'$ during the denoising process, under the constraints grounded by SCM. The method is demonstrated on Colored MNIST and Bars dataset and stable diffusion, where qualitative examples show better performance than baselines: LS inversion and DDS.", "questions": "See weakness. Major points are 1 and 2. It would be better if the paper could provide quantitative measurements and compare the proposed method with the baseline. It would be better if the evaluation dataset could practically cover diverse editing scenarios. It would also be better to provide robustness analysis on the scene graphs. Other points are minor and would be better clarified.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The method is highly theoretically grounded and novel.\n2. The motivation is clear and interesting, counterfactual image editing.\n3. The method is shown to be working on both toy models trained on MNIST and the stable diffusion model.\n4. The actual implementation of BD-CLS-Edit by optimizing linear combination coefficients guided by the gradients is novel.\n\nWeaknesses\n1. Although some qualitative examples are shown, the paper shows no quantitative measurements (LPIPS, FID, human evaluation, etc) with large-scale evaluation datasets. This is a key limitation since the method's effectiveness may not be valid.\n2. Although novel, the method requires a predefined causal graph as input. In practice, such a graph may not be easy to obtain. The paper does not provide a practical solution to get such graphs. Furthermore, in practice, the gained graphs may be incomplete or noisy, and it is worth analyzing whether the method is robust to such causal graphs.\n3. The paper lacks some experimental details, such as how $\\theta_i$ is regularized, whether it is learned per edited image, or can be shared across certain groups of images.\n4. Current illustrations of the method are still constrained within simple causal relationships and do not seem scalable to more complex editing scenarios. For example, whether emotional relationships can be reflected as well?\n5. Some writings are not clear, for example, the term LS inversion are used several times before defining it. The introduction of SCM related parts is also not clear. For example, the notion $\\mathbf{P a}_{V_j}$ (Parents of $V_j$) is never introduced but used as granted.", "was_revised": true, "final_justification": "The authors' response has addressed most of my concerns, especially for the \"assumed scene graph\".", "rebuttal": "We thank the reviewer for recognizing the theoretical strength and novelty of our work. We feel that a few misreadings of our paper make the evaluation overly harsh, and hope you can reconsider the paper based on the clarifications provided below.\n\n> ***W1. the paper shows no quantitative measurements with large-scale evaluation datasets.***\n\nWe appreciate the suggestions of using metrics such as **LPIPS, FID**. However, we found that these metrics are potentially misleading in the context of counterfactual editing; we elaborate a bit on these.\n\nWe first highlight the counterfactual editing principles proposed by the paper. Our evaluation framework is theoretically grounded in Section 3 (lines 159–184): (1) **Interventional consistency** – the edited variable must match its intervened value; (2) **Non-descendant invariance** – non-descendants of $\\mathbf{X}$ should remain unchanged; (3) **Descendant change** – descendants should change according to the counterfactual distribution $P(\\mathbf{De}_{\\mathbf{x}’} \\mid \\mathbf{v, l})$.\nSatisfying these three principles is essential for faithfully modeling the causal effect from X to the other variables, which leads to more causally-realistic image edits. However, LPIPS and FID are not aligned with the causal editing principles. \n\n**LPIPS.** This metric is typically used to assess the perceptual distance between the source and edited images. The lower distance implies better content preservation. However, this contradicts the descendant change principle described above (lines 180-184).\nLPIPS measures perceptual similarity between source and edited images, with lower scores indicating better content preservation. However, this can conflict with the descendant change principle (lines 180–184). For example, in Fig. 9(a, d), editing “weather” should causally introduce an umbrella. Semantic-invariance-based editing that achieves high LPIPS may suppress such changes, producing perceptually similar but causally inaccurate images.\n\nWe evaluate LPIPS of baselines and BD-CLS-Edit through tasks in text-to-image settings. We add another baseline, DDIM inversion, which is widely used in Diffusion-based methods. DDIM and DDPM Inversion alter non-descendants, resulting in higher LPIPS. BD-CLS-Edit correctly modifies descendants, leading to slightly higher LPIPS than DDS, but this reflects faithful causal editing, not inaccuracy. We will add this clarification in the next revision.\n\n| DDIM Inversion| DDPM Inversion | DDS | BD-CLS-Edit |\n|-----------------|-----------------|-----------------|-----------------|\n| 0.32 $\\\\pm$ 0.09| 0.25 $\\\\pm$ 0.08     | 0.13 $\\\\pm$ 0.03    | 0.16 $\\\\pm$ 0.03     |\n\n**FID**. FID measures the distance between the distributions of the original observational dataset and the edited results of this dataset. However, in causal modeling, counterfactual images over the dataset follow the distribution $\\sum_i P(i’_{x’} \\mid i)P(i)$, which is theoretically different from the observational distribution $P(i)$. Therefore, a lower FID score does not indicate a more accurate counterfactual distribution and may misrepresent the quality of the generated edits.\n\n**To quantitatively validate the effectiveness of counterfactually editing provided by BD-CLS, we report estimation results of the F-ctf query as proposed in [1]**. This metric directly evaluates whether the edited outcomes align with the expected counterfactual effects (Fig. 8 and Fig. S7). For instance, in Fig. 8, we measure the probability that the bar becomes thicker after editing the digit. Our method estimates this probability as 0.73, consistent with the theoretical range [0.73, 0.82], reflecting a valid causal effect from “digit → barwidth.” The table version of Fig.8 is as follows.\n\n| Gound truth       | lower bound       | upper bound       | BD-CLS       | CDiffusion       | CGN       |  CGN       |\n|----------------|----------------|----------------|----------------|----------------|----------------|----------------|\n| 0.75    | 0.73    | 0.82    | 0.73 $\\\\pm$ 0.02    | 0.40 $\\\\pm$ 0.02    | 0.0    | 0.75 $\\\\pm$ 0.02    |\n\n[1] Pan, Yushu, and Elias Bareinboim. \"Counterfactual Image Editing.\".\n\n> ***W2. Although novel, the method requires a predefined causal graph as input. ... it is worth analyzing whether the method is robust to such causal graphs.***\n\nWe appreciate raising this point. This concern touches on foundational principles in causal inference – why **assume a given causal graph?**\n\nThroughout the literature of causal inference, causal diagram assumptions are made not by choice, but out of necessity when inferring counterfactuals from observational data. To illustrate, any SCM induces families of different distributions related to the activities of seeing (observational), doing (interventional), and imagining (counterfactual), which together are known as Pearl Causal Hierarchy [2-3]. In this context, the passive collected images with labels are observational data, and the image editing query is a counterfactual quantity. According to the Causal Hierarchy Theorem [3, Thm. 1], distributions from the lower layers almost always underdetermine higher layers. In other words, in a measure-theoretic sense, it is not possible to make a statement about a higher layer (counterfactuals) without more assumptions; we refer to Sec. 1.3 in [3] for a formal discussion. We mention this since it also applies to our setting in computer vision – the graph assumption is made out of necessity and is somewhat expected in the broader causal inference literature. \n\n**Why not learn the causal diagram from the data?**\nThe complete true underlying causal diagrams cannot be learned only from the observational data in general. Causal discovery methods (e.g., PC, FCI algorithms) can recover up to an equivalence class of diagrams. With higher layer (such as interventional) or multi-domain distributions, it is possible to prune the equivalence class further. However, these distributions are often not available in image editing tasks in practice, especially considering large-scale text-to-image generation. In contrast, the diagram itself can be more easily obtained from human knowledge in some common-sense tasks. For example, one knows that rain will cause the use of an umbrella. LLMs also offer a venue for learning causal facts about the world, which depicts common-sense knowledge. \n\n**Next, we elaborate on the sensitivity and robustness of the input graph and provide additional experimental results.**\nBD-CLS-Edit demonstrates robustness in many practical scenarios. According to Thm 2, the set $B$ is required to be a backdoor set in the true graph $G$. If a misspecified or incomplete graph $G’$ is used, BD-CLS still provides causal guarantees as long as the chosen backdoor set $B’$ remains valid in $G$.\n\nFor example, consider the true $G$ in Fig. 7 (p. 8), where D is confounded with DC, and BC is a descendant of DC. Even if one mistakenly assumes both DC and BC are parents of D, DC plus BC still satisfies the backdoor criterion from D to image in both the incorrect and true graphs. Therefore, BD-CLS-Edit remains valid under this mis-specification and counterfactual editing is unaffected.\n\nTo ground this, we practically evaluate F-ctf query $P(BW_{D=7}= 1 | D = 1, DC = 0, BC = 0, BW = 0)$ (same query in Fig.8) when the misspecified graph $G’$ is given. \nThe results, as shown in the first column, suggest that the F-ctf estimation remains within the bounds, indicating that the counterfactual editing provided by BD-CLS is still effective and robust to the misspecified graph.\n\n\n| BD-CLS (correct B but wrong graph)       | BD-CLS (wrong graph and wrong B)      |\n|----------------|----------------|\n| 0.73 $\\\\pm$ 0.03    | 0.45 $\\\\pm$ 0.06    |\n\nIf the backdoor set from the input graph is invalid in the true graph, causal editing may fail. For instance, if the misspecified graph wrongly treats DC as a descendant of D and excludes it from the backdoor set, an intervention on D may incorrectly influence DC, violating non-descendant invariance. This leads to biased estimation of D’s effect on BW, compromising counterfactual validity.\nTo illustrate, we evaluate the same F-ctf query using this misspecified graph (results in column 2), and observe that the effect from D to BW is underestimated.\n\n**[2] Pearl, Judea, and Dana Mackenzie. The book of why: the new science of cause and effect. 2018.**\n\n**[3] Bareinboim, Elias, et al. \"On Pearl’s hierarchy and the foundations of causal inference.\"**\n\n> ***W3. experimental details***\n\nDuring optimization, we apply a penalty when $\\theta_t$ becomes too large to avoid unstable updates (Step 10 of Algorithm 3 on p. 24). This regularization, along with other algorithmic details and experimental parameters, is described in App. C and D. $\\theta_t$ is optimized individually for each input image. We will make this design choice explicit in the next revision.\n\n> ***W4.*** \n\nWhile our real-world results (e.g., Fig. 9, Fig. S10–S14) show only a few variables for clarity, many additional factors (e.g., pose, hair color, lightness) are implicitly modeled. BD-CLS-Edit still provides causal guarantees for these, e.g., editing “weather” changes lightness (a descendant) while keeping pose and hair color (non-descendants) unchanged. (see [Q1](https://openreview.net/forum?id=u2Lgi4NIe7&noteId=WttAWh3uHg) here for more details)\n\nWe agree that handling richer, more abstract concepts (e.g., emotion) is challenging. As $X, B$ grows in size and complexity, performance may be limited by the pre-trained model (see App. E.6 limitation). Improving the causal guarantees when involving more complex graph structures and more variables could be an interesting future work.\n\n\n> ***W5.*** \n\nThanks for raising this. We add an illustration for LS inversion before using it and rephrase the preliminary for better notation illustration and explicitly add “the parents of $V_i$ are denoted as $Pa_{v_i}$ in SCMs.” into the manuscript."}, {"review_id": "QF9DSohh5W", "reviewer": "Reviewer_VBDQ", "summary": "This paper tackles the problem of counterfactual image editing—\"What would this image look like if feature $X$ were changed from $x$ to $x'$?\"—by marrying causal inference with pre-trained diffusion models. The key steps are: \n\n1. Augmented Structural Causal Models (ASCM) formalize the true generative process, partitioning factors into labeled (V) and unlabeled (L) sets (Def.1, p.3).\n\n2. Backdoor Disentangled Causal Latent Space (BD-CLS) is defined (Def.4, p.5) as a proxy SCM over $\\{X, B, Z\\}$ that (i) matches the true conditional distribution $P(I|X, B)$, (ii) disentangles non-descendant features from interventions, and (iii) respects the causal graph. Theorem 2 (p.6) proves that sampling from any BD-CLS yields intervention consistency, non-descendant invariance, and counterfactual consistency on descendants.\n\n3. BD-CLS-Edit (Alg. 1, p.7) shows how to extract a BD-CLS from a pre-trained Stable Diffusion model without any retraining. By identifying a backdoor set B, inverting the image to noise $n$, and optimizing a prompt-mixing schedule $\\theta_t$ via gradient updates (Eqs. 6–8), one can sample counterfactual images $i'\\sim P_{\\rm{BD-CLS}}(I_{X=x'}|i)$. \n\n4. Experiments:\n\n* Colored MNIST & Bars: BD-CLS yields the correct flip probabilities for descendant features (bar width) within theoretical bounds, while non-causal baselines (CDiffusion, CGN) violate invariance or descendant behaviour (Figs. 6–8).\n\n* Text-to-Image Editing: On real-world prompts (weather → rainy, season → fall, scene → garden), BD-CLS-Edit preserves non-descendants (pose, layout) and induces realistic descendant changes (umbrella, clothing, removal of grocery bag), outperforming DDPM inversion and DDS (Fig. 9).", "questions": "1. Robustness: How sensitive is BD-CLS-Edit when the assumed backdoor set B omits a confounder or includes an extraneous node?\n\n2. Inference efficiency: What is the average runtime and GPU/memory footprint for a single 512×512 edit?\n\n3. Generalization: Can the method handle continuous interventions (e.g. brightness levels) or multiple simultaneous edits (changing both season and weather)?\n\n4. Automated graph discovery: Is it possible to infer or verify the necessary causal graph and backdoor sets from data, reducing manual effort?", "limitations": "1. Manual causal input: The need for a domain-expert causal graph and feature labels may hinder adoption in uncontrolled settings.\n\n2 .Optimization overhead: Per-instance gradient searches for $\\theta_t$ may not suit real-time applications.\n\n3. Evaluation breadth: The paper does not demonstrate performance on large-scale or highly-diverse image datasets beyond a few case studies.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe ASCM → BD-CLS framework gives formal guarantees on varying and invariant concepts under intervention without costly retraining. Moreover, only target feature $X$ and backdoor set $B$ need labels. All other factors remain label-free. Comprehensive quantitive and qualitative demonstrations via experiments convincingly demonstrate causal consistency where baselines fail. \n\n**Weaknesses**\n1. The approach presented assume a fully specified SCM over image factors $V$ and backdoor set $B$, which can be difficult to obtain in practice. The authors should consider ways of automatically extracting or at least validating the causal graph from data.\n\n2. The setup assumes an exact and invertible mapping between image space and diffusion latents, which is not always available. \n\n3. Each edits requires a gradient search over the per-timestep mixing weights which can be quite expensive. There is a lack of discussion in terms of the overhead for inference time. \n\n4. The prompt-mixing schedule $\\theta_t$ and its learning rates (Eqs. 6–8) require manual tuning. The paper lacks an extensive ablation on how robust the method is to these settings.\n\n5. Validated mostly on synthetic datasets (Colored MNIST, Bars) and a handful of text-to-image edits. It remains unclear how well the approach scales to complex, real-world scenes with many entangled factors.\n\n6.  Although only weak labels for $X$ and $B$ are needed, it still demands annotated examples of each intervention and backdoor factor. For richly structured images, obtaining such labels can be laborious.\n\n7. Disentanglement emerges indirectly via $\\theta_t$ optimization rather than explicit architectural constraints. There’s no guarantee that unrelated factors won’t bleed into each other under challenging edits.", "was_revised": true, "final_justification": "I believe the contributions and claims of the paper are novel and useful. I will keep my original rating and recommend acceptance.", "rebuttal": "Thanks for the valuable suggestions and detailed comments. We appreciate the opportunity to clarify some of the concerns.\n\n> ***W1, L1. The approach presented assume a fully specified SCM over image factors V and backdoor set B, ...The authors should consider ways of automatically extracting or at least validating the causal graph from data.***\n\n**Fundamental point – our approach does not require a fully specified SCM.**  \n\nWe want to clarify that we do not require the fully specified SCM (a tuple $<U, V, F, P(U)>$), but we do need a causal diagram over V and I. The graph is substantially weaker than (in reality, a coarsening of) the SCM. \n\n**The necessity of diagrams.**\n\nThroughout the literature of causal inference, causal diagram assumptions are made not by choice, but out of necessity when inferring counterfactuals from observational data. To illustrate, any SCM induces families of different distributions related to seeing (observational), doing (interventional), and imagining (counterfactual), which together are known as Pearl Causal Hierarchy [1-2].  According to the Causal Hierarchy Theorem [2, Thm. 1], it is not possible to make a statement about a higher layer (counterfactuals) without more assumptions. We mention this since it also applies to our setting in CV – the graph assumption is made out of necessity and is somewhat expected in the broader causal inference literature. Please see [W2](https://openreview.net/forum?id=u2Lgi4NIe7&noteId=6AEm7uDxbd) for more details. \n\n**Why not learn the causal diagram from the data?**\n\nThe complete true diagrams cannot be learned only from the observational data generally. Causal discovery (e.g., PC, FCI algorithms) can recover up to an equivalence class of diagrams. With higher-layer or multi-domain distributions, it is possible to prune the equivalence class further. However, they are often not available in many image editing tasks in practice, especially considering large-scale text-to-image generation. In contrast, causal diagrams can often be specified using human knowledge or common sense (e.g., rain causes umbrella use). LLMs also offer a venue for learning causal facts about the world, which depicts common-sense knowledge.\n\n*[1] Pearl, Judea, and Dana Mackenzie. The book of why: the new science of cause and effect. 2018.*\n\n*[2] Bareinboim, Elias, et al. \"On Pearl’s hierarchy and the foundations of causal inference.\"*\n\n> ***W2. The setup assumes an exact and invertible mapping between image space and diffusion latents, which is not always available.***\n\nWe want to clarify that the invertibility assumption in Def 1 refers to the underlying ground truth generative process $f^*_{\\mathbf{I}}$, but not to the mapping between the image and the diffusion model’s latent space (e.g., $\\mathbf{N}$ or $\\mathbf{Z}$). This assumption is standard and widely adopted in the disentangled representation learning literature (line 1033 Remark 4, pp. 32), where it is used to model that true semantic factors can be recovered from images.\nWe will make this distinction clearer in the revised version.\n\n> ***W3, Q2, L2. Inference efficiency: What is the average runtime and GPU/memory footprint for a single 512×512 edit?***\n\nThanks for raising this issue. We agree that optimizing $\\theta_t$ for all time steps in [1, 1000] for a stable diffusion training scheduler is costly.  In practice, we set the inference step to 200 for all text-to-image editing in this work (App. D.4, line 1011). The GPU cost of editing one 512*512 image is 28.9G, and the cost of editing a 1024*1024 image is 58.4G (results in the manuscript have a resolution 1024*1024). We add experiments about execution time for BD-CLS-Edit. We compare the excusion time of BD-CLS-Edit and DDPM inversion here for 512*512 edits for the request.\n\n| DDPM inversion 100 steps| DDPM inversion 200 steps | BD-CLS-Edit 100 steps | BD-CLS-Edit  200 steps |\n|-----------------|-----------------|-----------------|-----------------|\n| 0.36 $\\\\pm$ 0.01  min   | 0.79 $\\\\pm$ 0.02  min  | 4.88 $\\\\pm$ 0.06 min     | 9.87 $\\\\pm$ 0.08 min     |\n\nAs shown, BD-CLS-Edit is slower than existing diffusion-based editors (~1 min). However, it remains significantly faster than retraining or fine-tuning full models, which can take hours to days. This was indeed one of the primary motivations of this work: how to leverage pre-existing, non-causal pre-trained models, which are already refined in terms of image quality, towards causally-aligned models. As the first to incorporate counterfactual editing into pre-trained text-to-image models, we prioritize accuracy (i.e., counterfactual consistency) over speed. We will add this discussion to the limitations section and leave inference acceleration to future work.\n\n> ***W4.  \\theta and its learning rates (Eqs. 6–8) require manual tuning. The paper lacks an extensive ablation on how robust the method is to these settings.***\n\nThanks for the helpful suggestions! The hyperparameters in our optimization procedure (Sec. 4.1, Alg. 3), including learning rate $\\gamma$, the optimization steps $n_{\\max}$, and the clipping value $\\theta_{\\max}$, require manual tuning. We have added ablation studies over $\\gamma \\in \\{10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}\\}$, $n_{\\max} \\in \\{2, 10, 20\\}$, and $\\theta_{\\max} \\in \\{1.2, 1.5, 2.0\\}$.  \nWe find that $\\gamma = 10^{-2}$ and $n_{\\max} = 10$ yield the best balance between performance and efficiency, and BD-CLS-Edit is relatively robust to  $\\theta_{\\max}$. \n\n> ***W5, L3. Datasets for Evaluation breadth***\n\nThanks for the suggestions. We agree that including additional experiments can strengthen our work and will add results on CelebA in the next revision. However, we want to clarify that our real-world experiments (e.g., Fig. 9, Fig. S10–S14) do involve many entangled factors even though only 3–5 variables are explicitly shown. There are many implicit factors that are omitted from the graph but are still modeled. BD-CLS-Edit provides causal guarantees for these as well, for example, editing “weather” affects lightness (a descendant) while leaving pose and hair color (non-descendants) unchanged.\n\n> ***W6, L1. ..., it still demands annotated examples of each intervention and backdoor factor. For richly structured images, obtaining such labels can be laborious.***\n\nWhile our method does require labels for the intervention variable (X) and backdoor variables (B), we would like to emphasize that the labeling requirement is significantly lighter than in existing counterfactual editing methods. Specifically, BD-CLS-Edit only requires labels for X and B in the image to be edited. In contrast, prior methods typically require dense annotations of all generative factors across the entire training dataset, which is substantially more expensive.\n\n> ***W7. Disentanglement emerges indirectly via optimization rather than explicit architectural constraints...***\n\nThanks for mentioning this. Whenever the edited concept sets X and backdoor sets B become complex and larger, BD-CLS-Edit performance will be limited by the given pre-trained model. We show failure cases in the limitation discussion (App. E.6). We believe encoding explicit architectural constraints in the model is also valuable, especially when concepts are more complex and the pre-trained model is not given. Encoding explicit architectural constraints for text-to-image generation could be an interesting research direction. \n\n> ***Q1.Robustness: How sensitive is BD-CLS-Edit when the assumed backdoor set B omits a confounder or includes an extraneous node?***\n\nWe clarify that BD-CLS-Edit exhibits some robustness. First, note that the backdoor set B is not unique. For example, in Fig. 7 (p.8) for MNIST, both DC and {DC, BC} are valid, thus BC is redundant. If a mis-specified graph (e.g., both BC and DC are regarded as parents of D) lead to extra variables like BC in the backdoor set, BD-CLS still yields causal editing (see column 1; effect from D to BW remains within ground truth optimal [0.73, 0.82]). However, if a true confounder like DC is mistakenly excluded, e.g., treated as a descendant of D, editing may fail. The resulting bias violates non-descendant invariance and leads to incorrect causal estimates (column 2, not within bound).\n\n| BD-CLS (correct B but wrong graph)       | BD-CLS (wrong graph and wrong B)      |\n|----------------|----------------|\n| 0.73 $\\pm$ 0.03    | 0.45 $\\pm$ 0.06    |\n\n> ***Q3. Can the method handle continuous interventions (e.g. brightness levels) or multiple simultaneous edits (changing both season and weather)?***\n\nThis is a good question, thanks. This work assumes discrete generative factors (not necessarily binary). For continuous variables, non-descendant invariance extends naturally, and they should be invariant. However, for descendants, in order to provide theoretical guarantees, the theory should be extended carefully, and this requires some non-trivial work since new machinery based on measure theory should be evoked.\nOn the practical side, the algorithmic machinery proposed in the paper can be leveraged, and it should work in continuous settings. \n\nAdditionally, BD-CLS-Edit is built to perform multiple simultaneous edits. The intervention set $\\mathbf{X}$ can be a set including several variables. In experiments, we show multiple simultaneous edit results in Fig. S14 (App. E, pp. 32). \n\n> ***Q4. Automated graph discovery: Is it possible to infer or verify the causal graph and backdoor sets from data, reducing manual effort?***\n\nWe agree that reducing reliance on manually specified graphs is important for practical use. However, as discussed earlier, with only observational data, the true causal graph is generally unidentifiable.  \nSome methods aim to learn backdoor sets under assumptions, but they often require prior causal knowledge. Formalizing counterfactual editing with learned backdoor sets remains an open challenge, especially with unobserved factors, and it's an interesting research direction."}, {"review_id": "Mw4v1VKBE2", "reviewer": "Reviewer_UHq7", "summary": "The paper introduces a framework for performing causally consistent image editing by leveraging pre-trained text-to-image diffusion models, without requiring further training or fine-tuning. The authors model image editing as the estimation of counterfactual queries. They propose a new class of Structural Causal Models (SCMs), called Backdoor Disentangled Causal Latent Space (BD-CLS), which satisfy key properties of causal image editing: intervention consistency, invariance of non-descendants, and controlled change of descendants. The authors also present an algorithm for obtaining a BD-CLS from a pre-trained diffusion model and sampling causally faithful counterfactual images. Lastly, the proposed method is assessed through experiments on both synthetic and real-world image data.", "questions": "- In Algorithm 1, Step 1, how do you identify the largest backdoor set?\n- In Figures 6 and 9, which present qualitative comparisons, could you clarify which are the source and target prompts for performing the causal edits?\n- What diffusion backbone model do you use for the ColorMNIST-Bars dataset? Is it a latent diffusion model or a pixel-based diffusion model?\n- I believe the paper would benefit from some form of quantitative evaluation. If this is not feasible, it would still strengthen the work to include some qualitative results on additional known synthetic datasets (e.g., MorphoMNIST) or other real-world datasets such as CelebHQ, where causal graphs can be assumed as in [1] and [2].\n\n**References:**\n\n[1] Yang, Mengyue, et al. *CausalVAE: Disentangled representation learning via neural structural causal models.* Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.\n\n[2] Melistas, Thomas, et al. *Benchmarking counterfactual image generation.* Advances in Neural Information Processing Systems, 37 (2024): 133207–133230.", "limitations": "The limitations are adequately described in the appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper tackles a significant gap in the literature on how to leverage text-to-image foundation models for performing causally consistent counterfactual image generation.\n- The authors provide a solid theoretical framework regarding how to design a BD-CLS SCM. They clearly justify why the proposed BD-CLS achieves the desired properties for faithful causal editing.\n- The proposed algorithm 1 for extracting a BD-CLS from a pre-trained diffusion model offers a compelling, causally-aware alternative to existing diffusion-based image editing methods, which may not respect assumed causal relationships when edits are performed.\n- The algorithm 1 is efficient, as it does not require updating the diffusion model parameters.\n\n\n**Weaknesses:**\n\n- The proposed method's qualitative evaluation lacks depth.\n  - More qualitative comparisons should have been included in the main paper.\n  - The authors qualitatively compare their method with only two baseline image editing techniques.\n- The paper lacks a thorough quantitative evaluation.\n  - The authors do not include tables with quantitative results or use metrics that are commonly applied in counterfactual image generation [1, 2, 3].\n- For real-world images, the authors do not evaluate their approach on a standard benchmark dataset (e.g., the datasets used by compared baselines such as DDPM inversion [4], or alternatives datasets with text annotations like Multi-Modal-CelebA-HQ [5]).\n- Although the paper is well-structured, in some parts the notation becomes complex and difficult to follow.\n\n**References:**\n\n[1] Monteiro, Miguel, et al. *Measuring axiomatic soundness of counterfactual image models.* arXiv preprint arXiv:2303.01274, 2023.  \n[2] Ribeiro, Fabio De Sousa, et al. *High fidelity image counterfactuals with probabilistic causal models.* arXiv preprint arXiv:2306.15764, 2023.  \n[3] Melistas, Thomas, et al. *Benchmarking counterfactual image generation.* Advances in Neural Information Processing Systems, 37 (2024): 133207–133230.  \n[4] Huberman-Spiegelglas, Inbar, Vladimir Kulikov, and Tomer Michaeli. *An edit-friendly DDPM noise space: Inversion and manipulations.* Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.  \n[5] Xia, Weihao, et al. *TediGAN: Text-guided diverse face image generation and manipulation.* Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.", "was_revised": true, "final_justification": "The authors adequately addressed my concerns regarding the number of real-world images used in the evaluation of their method. That’s why I decided to raise my initial score.", "rebuttal": "Thank you for recognizing the motivation behind our work and for finding its theoretical and practical contributions compelling. We address your concerns in the sequel.\n\n> ***W1. The proposed method's qualitative evaluation lacks depth.***\n\nWe appreciate this suggestion and agree that more qualitative comparisons in the main paper would improve clarity. In the revision, we will move additional examples from the appendix (Figs. S12, S13, S14) into the main body.\nWe also add a new baseline, **DDIM Inversion**, a popular method for diffusion-based editing. While DDIM Inversion is often strong in perceptual fidelity, it fails to ensure non-descendant invariance (the second principle in Thm. 1, p. 4). For example, changing weather alters the garden layout  (Fig. 9(d)); changing season affects the person’s size  (Fig. 9(e)); Changing scene causes layout shifts (Fig. 9f). We will include these qualitative comparisons in the next revision.\n\n*[6] Mokady, Ron, et al. \"Null-text inversion for editing real images using guided diffusion models.\"* \n\n> ***W2. The paper lacks a thorough quantitative evaluation.***\n\nWe appreciate the suggestions of metrics mentioned in [1-3]. However, we found that some metrics are potentially misleading in the context of counterfactual editing; we elaborate a bit on these.\n\nWe first highlight the counterfactual editing principles proposed by the paper. Our evaluation framework is theoretically grounded in Section 3 (lines 159–184): (1) **Interventional consistency** – the edited variable must match its intervened value; (2) **Non-descendant invariance** – non-descendants of \\mathbf{X} should remain unchanged; (3) **Descendant change** – descendants should change according to the counterfactual distribution $P(\\mathbf{De}_{\\mathbf{x}’} \\mid \\mathbf{v, l})$.\nSatisfying these three principles is essential for faithfully modeling the causal effect from $X$ to the other variables, which leads to more causally-realistic image edits. However, metrics in [1-3] are not aligned with our goal. \n\n**LPIPS[3], minimality[3].** \nLPIPS measures perceptual similarity between source and edited images, with lower scores indicating better content preservation. However, this can conflict with the descendant change principle (lines 180–184). For example, in Fig. 9(a, d), editing “weather” should causally introduce an umbrella. Semantic-invariance-based editing that achieves high LPIPS may suppress such changes, producing perceptually similar but causally inaccurate images.\n\nWe add evaluation of baselines and BD-CLS-Edit through tasks in large-scale text-to-image settings. DDIM and DDPM Inversion alter non-descendants, resulting in higher LPIPS. BD-CLS-Edit correctly modifies descendants, leading to slightly higher LPIPS than DDS, but this reflects faithful causal editing, not inaccuracy. We will add this clarification in the next revision.\n\nDDIM Inversion| DDPM Inversion | DDS | BD-CLS-Edit |\n|-----------------|-----------------|-----------------|-----------------|\n| 0.32 $\\\\pm$ 0.09| 0.25 $\\\\pm$ 0.08     | 0.13 $\\\\pm$ 0.03    | 0.16 $\\\\pm$ 0.03     |\n\n**Effectiveness [1-3]**. This metric aligns with our principle of interventional consistency. We add this evaluation to this rebuttal later. \n\n**Composition [1-3]**. This metric evaluates similarity when the intervention matches the factual value $\\mathbf{x}’ = \\mathbf{x} $ by measuring P(i’_{\\mathbf{x}} \\mid \\mathbf{i}). However, our focus is on counterfactual edits, where $\\mathbf{x}’ \\ne \\mathbf{x}$. Therefore, this metric is orthogonal to our goal.\n\n**Reversibility [1-2]**. This metric is saying that the intervention should be cycle-consistent and relies on the assumption that there exists a function $g$ such that endogenous variables  $\\mathbf{u} = g(\\mathbf{i}, \\mathbf{x})$. However, this assumption does not hold in our non-parametric underlying SCM. Moreover, as discussed in [3], cycle consistency is not generally guaranteed in causal models and may lead to incorrect assumptions about editability.\n\n**To quantitatively validate the ability of BD-CLS to edit images counterfactually, we report estimation results of F-ctf queries as proposed in [7]**. This metric directly evaluates whether the edited outcomes align with the expected counterfactual effects (Fig. 8 and Fig. S7). For instance, in Fig. 8, we measure the probability that the bar becomes thicker after do(D=7). Our method estimates this probability as 0.73, consistent with the theoretical range [0.73, 0.82], reflecting a valid causal effect from “digit → barwidth.” The table version of Fig.8 is as follows.\n\n| Gound truth       | lower bound       | upper bound       | BD-CLS       | CDiffusion       | CGN       |  CGN       |\n|----------------|----------------|----------------|----------------|----------------|----------------|----------------|\n| 0.75    | 0.73    | 0.82    | 0.73 $\\pm$ 0.02    | 0.40 $\\pm$ 0.02    | 0.0    | 0.75 $\\pm$ 0.02    |\n\nWe report two additional F-ctf evaluations to demonstrate our method’s ability to edit counterfactually. First, we assess the probability that digit “7” appears in the edited image after the intervention do(D = 7). The ground-truth value is 1, reflecting interventional consistency, similar to effectiveness metrics in [1–3]. Results show that all methods satisfy this property.\n\n| Gound truth       | lower bound       | upper bound       | BD-CLS       | CDiffusion       | CGN       |  CGN       |\n|----------------|----------------|----------------|----------------|----------------|----------------|----------------|\n| 1.000    | 1.000    | 1.000    | 0.980 $\\pm$ 0.004    | 0.978 $\\pm$ 0.004    | 0.995 $\\pm$ 0.001   | 0.988 $\\pm$ 0.003    |\n\nSecond, we evaluate the probability that the bar color becomes green after the digit changes. This quantity reflects non-descendant invariance. The ground truth should be 0 (line 318-328). The results suggest that BD-CLS is capable of keeping non-descendants invariant after editing (as expected).\n\n| Gound truth       | lower bound       | upper bound       | BD-CLS       | CDiffusion       | CGN       |  CGN       |\n|----------------|----------------|----------------|----------------|----------------|----------------|----------------|\n| 0.0    | 0.0    | 0.0    | 0.0    | 0.15 $\\pm$ 0.04    | 0.0  | 0.0    |\n\nWe conduct this F-ctf quantitative analysis only on the ColorMNIST datasets because we have full control over the generation process here, allowing us to evaluate the ground truth (and corresponding causal bound) and compare the proposed and baseline methods. In real-world dataset, the ground truth mechanisms for generating data are unknown and we provide a more qualitative type of discussion. \n\n[7] Pan, Yushu, and Elias Bareinboim. \"Counterfactual Image Editing.\" ICML24.\n\n> ***W3. Experiments on other datasets***\n\nThank you for the suggestion. We agree that additional results on other datasets can strengthen our work.  We will include results on CelebA to highlight causal effects on facial features in the next revision. As for the datasets in [4], while they are suitable for traditional image editing tasks, they lack clear and unified causal structures to demonstrate causal effects. For example, in tasks like editing a cat to a dog, the goal is often to change the animal while keeping the background unchanged and no other causal effects are considered.\n\n> ***W4** Although the paper is well-structured, in some parts the notation becomes complex and difficult to follow.***\n\nThank you for pointing this out. We have revised the preliminaries for clarity and moved illustrative examples from the appendix into the main paper to better explain the notations used in the theorems.\n\n> ***Q1. In Algorithm 1, Step 1, how do you identify the largest backdoor set?***\n\nBy “largest backdoor set,” we mean a maximal backdoor set—one that blocks all backdoor paths from X to the image I and cannot be extended further without violating the criterion. We compute such sets using a polynomial-delay algorithm [8].\n\nWhile BD-CLS-Edit does not require the backdoor set to be maximal (any valid set suffices theoretically), we use the maximal set in practice to better preserve non-descendant invariance, ensuring that more observed non-descendant variables in B remain unchanged during editing.\n\n*[8] Van der Zander, Benito, Maciej Liśkiewicz, and Johannes Textor. \"Efficiently finding conditional instruments for causal inference.\"*\n\n> ***Q2. In Figures 6 and 9, which present qualitative comparisons, could you clarify which are the source and target prompts for performing the causal edits?***\n\nIn Fig. 6 (BarMNIST), we use explicit labels for \\mathbf{X} and \\mathbf{B}, not text prompts. To illustrate, the task “edit a red ‘1’ with a thin red bar to digit “7”, we use the source labels {D=1, DC=0, BC=0} and target labels {D=7, DC=0, BC=0}.\n\nIn Fig. 9 (text-to-image tasks), we use prompts such as: \n\n**Task: Weather edit (Fig. 9a,d; Fig. S10–S11):**\n\nSource prompt: “an old/young woman standing in a garden/street on a sunny day\". \n\nTarget prompt: “an old/young woman standing in a garden/street on a rainy day\". \n\nBD-CLS-Edit edits umbrella/shadow/pose appropriately, unlike baselines.\n\n**Task: Season edit (Fig. 9b,e; Fig. S12):**\n\nSource prompt: “a person in the forest during the summer\". \n\nTarget prompt: “a person in the forest during the fall\". \n\nBD-CLS-Edit correctly adjusts unprompted features like clothing and the shape of the person.\n\n**Task: Scene edit (Fig. 9c,f; Fig. S13):**\n\nSource prompt: “a person standing in the grocery store\". \n\nTarget prompt: “a person standing in the garden\".\n\nBD-CLS-Edit provides causal guarantees for unprompted features such as person details and the grocery bag.\n\n> ***Q3. What diffusion backbone model do you use for the ColorMNIST-Bars dataset? Is it a latent diffusion model or a pixel-based diffusion model?***\n\nIt is a pixel-based diffusion model trained to map observed generative factors to image pixels directly."}]

Author rebuttal:

Meta-review: This paper proposes a counterfactual image editing method using a pre-trained diffusion model. Specifically, they are interested in causal consistency, i.e., only edit the logically relevant parts. All the reviewers are supportive of this paper. The main discussion points during the discussion period are the empirical results and many of the implementation details. The authors have successfully addressed the reviewers’ concerns. The AC does not see reasons to overrule the reviewer’s recommendation.
Decision: Accept (poster)