Diffusion Counterfactual Generation with Semantic Abduction
Authors: Rajat R Rasal
Avinash Kori
Fabio De Sousa Ribeiro
Tian Xia
Ben Glocker
Venue: ICML 2025 poster (2025)
Keywords: counterfactuals
causality
diffusion
image editing
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: A general framework that integrates semantic representations into diffusion models through the lens of Pearlian causality for image editing via counterfactual reasoning.

Abstract: Counterfactual image generation presents significant challenges, including preserving identity, maintaining perceptual quality, and ensuring faithfulness to an underlying causal model. While existing auto-encoding frameworks admit semantic latent spaces which can be manipulated for causal control, they struggle with scalability and fidelity. Advancements in diffusion models present opportunities for improving counterfactual image editing, having demonstrated state-of-the-art visual quality, human-aligned perception and representation learning capabilities. Here, we present a suite of diffusion-based causal mechanisms, introducing the notions of spatial, semantic and dynamic abduction. We propose a general framework that integrates semantic representations into diffusion models through the lens of Pearlian causality to edit images via a counterfactual reasoning process. To the best of our knowledge, ours is the first work to consider high-level semantic identity preservation for diffusion counterfactuals and to demonstrate how semantic control enables principled trade-offs between faithful causal control and identity preservation.

Diffusion Counterfactual Generation with Semantic Abduction
Rajat Rasal Avinash Kori!
Fabio De Sousa Ribeiro
Tian Xia Ben Glocker'
Abstract
looked like had they undergone a different treatment?") or enabling fairer Al by disentangling attributes in face mod- elling (e.g., "what would this person look like if their age, hairstyle or expression changed?"). Despite methodological advancements, high-fidelity counterfactual generation faces challenges with perceptual quality, identity preservation, and faithfulness to an underlying causal model (Monteiro et al., 2023; Melistas et al., 2024).
Counterfactual image generation presents signif- icant challenges, including preserving identity, maintaining perceptual quality, and ensuring faith- fulness to an underlying causal model. While ex- isting auto-encoding frameworks admit semantic latent spaces which can be manipulated for causal control, they struggle with scalability and fidelity. Advancements in diffusion models present oppor- tunities for improving counterfactual image edit- ing, having demonstrated oftte-of-the-ar visual quality, numan-aligned perception and representa- tion learning capabilities. Here, we present a suite of diffusion-basee causal mechanisms, introduc- ing the notions of spatial, semantic and dynamic abduction. We propose a general framework that integrates semantic representations into diffusion models through the lens of Pearlian causality to edit images via a counterfactual reasoning pro- cess. To our knowledge, this is the first work to consider high-level semantic identity preservation for diffusion counterfactuals and to demonstrate how semantic control enables principled trade- offs between faithful causal control and identity preservation.
Diffusion models (Sohl-Dickstein et al., 2015; Song & Er- mon, 2019; Ho al., 2020) have achieved state-of-the- -frth-rr- performance in image synthesis (Dhariwal & Nichol, 2021; Podell et al., 2023) and exhibit strong alignment with human perception on recognition tasks (Jaini et al., 2023; Clark & Jaini, 2024), making them ideal candidates for improving automated creative reasoning through causal representation learning. Their popularity has grown in tasks related to counterfactual inference, such as image-to-image transla- tion (Meng et al., 2021; Yang et al., 2023a) and image editing (Couairon et al., 2022: Hertz et al.. 2022; Epstein et al., 2023), gradually supplanting traditional GAN and VAE-based approaches (Goodfellow et al., 2014; Karras, 2019; Karras al., 2020; Kingma, 2013; Sohn et al., 2015). However, unlike closely related hierarchical generative mod- els (Luo, 2022; Ribeiro & Glocker, 2024), where compact latent projections encode high-level semantics (Child, 2020; Vahdat & Kautz, 2020; Shu & Ermon, 2022), vanilla dif- fusion models lack a controllable semantic representation space (Turner et al., 2024). This limitation prohibits their applicability with existing counterfactual generation frame- works. Current methods introduce semantics into diffusion by interpreting low-rank projections of intermediate images (Park et al., 2023; Haas et al., 2024; Wang et al., 2024) or by employing diffusion models as decoders within VAEs (Preechakul et al., 2022; Pandey et al., 2022; Zhang et al., 2022; Batzolis et al., 2023; Abstreiter et al., 2021), where encoders capture controllable semantics. For explicit con- trol, essential for counterfactual reasoning, diffusion models rely guidance using discriminative score functions, ei- ther amortised within the diffusion model (Ho & Salimans, 2022; Karras et al., 2024; Chung et al., 2024; Dinh et al., 2023) or trained independently (Dhariwal & Nichol, 2021). However, challenges with identity preservation and faithful causal control - critical for image edits to align with our understanding of the world still persist.
1. Introduction
Answering counterfactual queries is central to understanding cause-and-effect relationships between variables in a system (Pearl, 2009; Peters al., 2017; Bareinboim et al., 2022). Humans pose counterfactuals when reasoning creatively about what could have been under retrospective, hypothet- ical scenarios (Pearl, 2019). There has been increasing in- terest in counterfactual reasoning about images using deep generative models (Komanduri et al., 2023; Zečević et al., 2022). These advances are crucial for applications such as improving healthcare outcomes through personalised treat- ment (e.g.. "what would a patient's tumour progression have
Departmennt of Computing, Imperial College London, UK. Correspondence to: Rajat Rasal rasalt 7 @ @imperial.ac uk>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Diffusion Counterfactual Generation with Semantic Abduction
To address these challenges, the framework of structural causal models (SCMs) (Pearl, 2009; Peters et al., 2017; Bareinboim et al., 2022) has been extended with deep generative models, resulting in deep SCMs (DSCMs). These are used for structured counterfactual inference via a simple three-step procedure:
2. Background
2.1. Causality
Structural Causal Models. We assume Markovian SCMs, referred to simply as SCMs henceforth, defined via the tuple G (F, X) consisting of set of func- tions F = {[x] called mechanisms, a set of observed variables X = { and set of exogenous noise variables E = {eukk= which are jointly independent, p(ε) II p(Ex). Each mechanism models causal re- lationships acyclic, invertible assignments of the form
cecticciionaationnno prredicionn (Pawlowski et al., 2020; De Sousa Ribeiro et al., 2023; Xia et al.. 2023; Wu et al.. 2024; Dash et al., 2022: Sauer & Geiger, 2021). These methods typically parameterise their gener-gene components using normalising flows (Papamakarios et al., 2021; Winkler et al., 2019), VA (Kingma, 2013; Sohn et al., 2015), GANs (Goodfellow et al., 2014; Mirza & Osindero, 2014), and HVAES (Vahdat & Kautz, 2020; Child, 2020). More recently, diffusion models have been integrated into DSCMs leverage their superior perceptual quality and explore counterfactual identifiability (Sanchez & Tsaftaris, 2022; Komanduri et al., 2024; Pan & Bareinboim, 2024). However, these efforts often over- look key nuances of guidance and representation learning unique to diffusion models, and their faithfulness to causal assumptions. This is particularly limiting for real-world applications, where backgrounds, illumination changes, and artefacts complicate causal modelling (Anonymous, 2024; Mokady et al., 2023; Lin et al., 2024). Additionally, SCMs have been informally applied to text-conditional diffusion editing (Song et al., 2024; Gu et al., 2023; Prabhu et al., 2023), yet text control alone often fails to provide the preci- sion needed for real-world counterfactual reasoning.
X fx (pa ft), where pag S XX{x are X o's di- rect causes or parents and c₁ ~ p((x). This induces a conditional factorisation of the observational joint distri- bution satisfying the causal Markov condition, Pe(X) (X) =
K Ps (xg) pa,), whereby all observed variables are in- dependent of their ancestors given their parents. An SCM can, therefore, be represented using a Bayesian network, known as a causal graph, where mechanisms are represented by edges going from observed (pay) and exogenous causes (Et.) to effect (x&).. pay X₁: <- EA-
Counterfactuals. SCMs can be used to answer retrospec- tive, hypothetical questions, known as counterfactuals, of the form "Given observations X, what would X₃ be had pag been different", via the following three-step procedure:
Abduction: Sample the exogenous posterior E (e(E) = Ixxxxx Ps (Ex X,,, pay) by inverting the parent mechanisms E; f= (xx, pax)
We argue that diffusion models hold potential far beyond their established applications for recognition and image syn- thesis at the associative (C1) and interventional (C2) rungs on Pearl's 's ladder of causation (Pearl, 2009). Specifically, they can serve as foundations for counterfactual reasoning frameworks at La, encompassing applications at L1 and Lz. to enable principled decision-making. In this work, we take a significant step in this direction by revealing trade-offs inherent to using diffiusion for image counterfactuals. We do this alongside integrating semantic control and high-level identity preservation into diffusion models via the DSCM framework. Our main contributions are as follows:
2. Action: Perform intervention on x, pa by fixing the output of (-( to b, denoted do(x; := b). The set of modified mechanisms is denoted as F.
3. Prediction: Use I = (F, X) to infer counter- factuals of interest using their mechanisms, X fi(Ek, pag). where pa, denotes the counterfactual parents under interventions.
1. We present a causal generative modelling framework for counterfactual image generation using diffusion models (Section 3.1).
Counterfactual inference adheres to the axiomatic proper- ties of composition, reversibility, and effectiveness (Galles & Pearl, 1998; Halpern, 2000), framed as evaluation metrics by Monteiro et al. (2023), detailed in Appendix A. Com- position measures -errcoonstiutto error by comparing observed images with counterfactuals under null interven- tions. Reversibility assesses cycle-consst by measuring the L2 disttanee between observed images and their cycled- back counterfactuals. Effeectvvnness evaluates faithfulness to interventions using anti-causal predictors for classification or regression. To measure a model's 's identity preservation (IDP), we compute LPIPS (Zhang et al., 2018) between compositions and their corresponding counterfactuals.
2. We introduce semantic abduction to improve counter- factual fidelity and identity preservation (Section 3.2).
3. We improve pntervention-aaitffuune using efficient amortised anti-causal guidance (Section 3.3).
4. We propose dynamic semantic abduction to infer se- mantics at each step in the diffusion process, better preserving backgrounds and specific characteristics in real-world images (Section 3.3).
2

---

Diffusion Counterfactual Generation with Semantic Abduction
2.2. Diffusion Models
3. Methods
Denoising diffusion probabilistic models (DDPMs) are class of latent variable model (Ho et al., 2020; Sohl- Dickstein et al., 2015; Song & Ermon, 2019) trained to progressively remove noise from samples X ~ (V, p(XT) over 7 timesteps to reveal an image Xo- We focus on the deterministic variant of DDPMs, called denoising diffusion implicit models (DDIMs) (Song et al., 2020a). For conditional image generation, their generative model is
This section introduces our diffusion-based mechanisms, each distinguished by their exogenous noise decomposi- tion and associated abduction. For simplicity, we denote image in our SCM x, its parents as pa, and its ex- ogenous noise as 6. The image-generating mechanism f(-) is parameterised by a diffusion model with parame- ters 0: x := fa(e. The counterfactual conditions, under interventions, are denoted pa and used to gener- ate an image counterfactuals as: x := fo pa), where (x, (x, pa). We provide definitions for a variety of abduction procedures which implement c: fo (x, pa).
T
:= fg
Po (xx:r (c) == p(xT) II
po (x
(xx,, c),
1=1
Po Px(x,-11 xx, c) := (xxx-1 |x, dx( t)),
(1)
3.1. Spatial Mechanism
the inference distribution uses linear-Gaussian transition,
Our spatial mechanism is defined using the DDIM genera- tive transitions in Equation (6):
T
(x0) = g(xr (x)) II 1x,,
x == fo pa) x (u, pa)
1=
= (hije .... o (u))
(8)
(xx,-11x,, xo) = (N((x0 xx, 1), 0),
(2)
where C = (pa, 0). the observed variable x in our SCM is defined to be the DDIM generated image xo. he(-) samples the image marginal (x(paa and p(u) = p(xr) is the spatial exogenous prior. Spatial abduction is performed by sampling the spatial exogenous posterior:
g(xx|x0)= == (xxx vaixo, (1 (x)
where pxxo, Xt, 1 1,t-1)
X2- v Va
1x + VI a1-1
0z
Pe (u|x, pa) x 8(u - hg (x, pa)),
(9)
the denoiser da(-) is defined as
where 8(.) denotes a Dirac delta distribution 15 is im- plemented using inverse DDIM transitions (Equation (7)) such that:
1 (xt v (x c, f)), (4)
do(x;, c,
u u: fo (x, pa) ~ h pa) h (x))
c0(+) is the noise estimator parametrised by UNet (Ron- neberger et al., 2015) and ag:r E [0. 1] defines the noise schedule. The image marginal (x)/o)c) is learned by opti- mising an evidence-lower bound via surrogate objective: sample a timestep 1 ~ 4(1,7) T), create noisy images by + ++11-a 11--a,e arE with
(h77le
(10)
Here, the abducted u is a noisy and highly-editable version of (Mokady et al., 2023; Hertz et al., 2022; Parmar et al., 2023) which encodes low-level structural information in the image space (Wang & Vastola, 2023). This differs from existing VAE and HVAE-based mechanisms (Pawlowski et al., 2020; De Sousa Ribeiro et al., 2023), which infer spatial noise through linear reparamerisation for intensity/- contrast correction, whereas our formulation allows more flexibility by inferring complex structural information via DDIM inversion. After performing interventions, we infer counterfactual conditions pa, then seed the counterfactual prediction step with to generate image counterfactuals as x = = 1)), (6)
reparametrising ((xx as &₁- = Vaixo +
~ (N, I), then use (-) to predict
argmin ( Ex,, - (x,, 1)
(5)
We denote generative transitions
which can also be inverted
X == fe pa) ss he (u, pa).
(11)
x; hijt
depicted via the twin network representation in Figure la. Our generalised spatial mechanism enables modelling mul- tiple discrete or continuous parents of x, thereby improving upon the DiftSCM framework (Sanchez & Tsaftaris, 2022), which only allows for single discrete parent.
11) 1,t)
(7)
where we use C = (c, to denote the partial application set which we abuse to include model parameters.
3

---

Diffusion Counterfactual Generation with Semantic Abduction
z
pa
Ø
pa
Z
pa
pa
pa
"i hr
his
*i the
kz
x
Ny ho
X
x
(a) Spatial Mechanism ($3.1)
(b) Semantic Mechanism ($3.2)
(c) Amortised Guidance Dynamic Abduction ($3.3)
Figure 1. Twin network representations for our diffusion mechanisms. Black and white arrowheads refer resp. to the generative and abductive / inference directions. Edges ending in black circles depict conditions. Circular and diamond nodes refer resp. to depict random and deterministic variables. Boxes house the independent exogenous decomposition of e. In (a)-(b), generation and inference performed resp. with DDIM h+)+) and DDIM inversion ha ()). Optionally, amortised guidance be used by incorporating blue functions and conditions; hō .) for generation and Ø for conditioning. In (b)-(c), the probabilistic encoder is depicted via e-) ). In (c), amortised guidance for the semantic mechamism is compulsory, and the diamond arrowhead denotes dynamic abduction.
Recall that counterfactual functions should be sound in terms of their composition (reconstruction) and reversibil- ity (cycle-consistency), which also serve as indicators of identity preservation (Monteiro et al., 2023). We choose conditional DDIMs for abduction and prediction, as this forms bijection under the null intervention pa given a per- fect noise estimator (Song et al., 2020b; Chao et al., 2023), resulting in good composition and reversibility. Existing works have used unconditional DDIM inversion or the ag- gregate inference posterior (xx(Xx) for abduction followed by guidance for prediction (Komanduri et al., 2024; Sanchez et al., 2022a; Weng et al., 2023; Fang et al., 2024) (c.f. Sec- tion 3.3). In contrast, our spatial abduction enforces that spatial noise u will encode information pertaining to pa through conditioning, in line with Pearlian abduction.
codable high-level exogenous noise term z, which remains fixed during generation:
x fo(c,pa) P (u Czem)
(12)
where Csem = (z. pa), ha(-) samples po(x z, pa), and ex- ogenous noise is decomposed independently into spa- tial u and semantic terms p(E) = p(u)p(z) with p(z) = (O, I). To learn fa(-) such that z encodes high- level semantics, we marginalise f (x|z, pa)dz by intro- ducing a semantic variational posterior qu (x)ex(()))) =
N(z (x), ()) where the probabilistic encoder e.).) is implemented with a convolutional neural network (CNN), In practice, we use the surrogate objective:
argmin (zl (x)) + 0,.p
(13)
Our spatial mechanism is trained using the standard denois- ing objective in Equation (5), setting pa. Training is more stable than VAE/HVAE mechanisms, which attempt to learn complex latent posteriors from data. Furthermore, through spatial abduction enabled by DDIM, we can sig- nificantly mitigate posterior-prior mismatch caused by pro- jecting inputs into unsupported regions fow-dimensional latent space present in VAEs (Rezende & Viola, 201 8; Ho al., 2020; Ribeiro & Glocker, 2024). Instead, intermediate images X1-7-1 remain the data manifold through itera- tive noise refinement by the denoiser. As such, u provides an in-distribution input to the prediction step with ha
Ex....ezzl 6X(3,, Czemy 1)1
Here and parametrise a conditional diffusion based decoder and CNN-based encoder, respectively.
For semantic abduction, an approximation to the exogenous posterior can be defined using the probabilistic encoder in a manner similar to the amortised, explicit mechanism defined by Pawlowski et al. (2020):
(e pa) = Ps (2) x) (u|x, Csem) hg" (x, Csem)), (14)
s qu (zle,
3.2. Semantic Mechanism
noting the key benefit of our model in that low-level noise is inferred via DDIM inversion hg (.) as opposed to sim- ple linear function (Section 3.1.. Preechakul et al. (2022) presents a similar model (DiffAE) which trains semantic encoder in unconstrained space, as such an additional diffu- sion model is trained post-hoc to sample z unconditionally for random image sampling. In contrast, our mechanism
Recall that the spatial noise u is iteratively refined by the denoiser during the counterfactual prediction process. As such, u alone cannot explicitly preserve the high-level se- mantics of x at each timestep (Preechakul et al., 2022). To address this, we introduce into our spatial mechanism a de-

---

Diffusion Counterfactual Generation with Semantic Abduction
trained end-to-end with a regulariser. learning the semantic posterior (-(-) via variational inference, which use for efficient semantic abduction. Image counterfactuals are now generated using Monte Carlo with M particles,
and the guidance scale w > 1 amplifies the counterfactual- conditioned noise estimate. For EØ(-) to amortise condi- tional and unconditional representations, Ø replaces Csem in the denoising objective (Equation (13)) with probability Pæ. Our choice of pa is tuned for counterfactual soundness (Cg),, as opposed to most existing works, who choose Pæ for diverse Le-level sampling (Ho & Salimans, 2022).
z (m) ~ q& (z)e e. (x))
m(m) x hog (xx, ((m))
M
x
1 M
ho(u(m)) (m), )
(15)
The amortised, anti-causally guided semantic mechanism
with Ciem = (z) pa). cirl = (2 pa) and the se- mantic posterior sampled via the reparameterisation trick z (x) + (x) O e with e e~ (~,NO,, I). This depicted in Figure 1b. In Section 4, we generate deterministic image counterfactuals by using the degenerate semantic posterior during abduction such that z Mo (x).
x == fo((,, 2 hig (u, Csem U (Ø))
== (Mije o .... he (u),
(17)
where (.) denotes DDIM using the modified noise estima- tor (Equation (16) and C {Coem: 2,0). 0}. Counterfactual generation follows:
yo (2)e u RJ h1 (x, Csem), x hā (u, Esem (+²)).
Identifiability. The unconditional VAE prior p(z) in our model makes it non-identifiable (Locatello et al., 2019), meaning the true settings of I and o cannot be uniquely determined even with infinite data. This can affect abduc- tion, as multiple values of may yield the same marginal likelihood Po (x pa), resulting in different counterfactuals under the same intervention. Khemakhem al. (2020a) showed that identifiability can be improved by condition- ing the semantic prior on observed variables, e.g. p(z(pa). Incorporating these ideas in our framework may require modifications akin to De Sousa Ribeiro al. (2023), which we leave for future work.
(18)
Amortised, anti-causal guidance can also be used with our spatial mechanism by using Vx log pe ps (pa(x)) to derive the noise estimator (Equation (16)). In this case, counterfactual generation becomes
u R h (x, pa),
X N hg (u, pa U (z)),
(19)
The option to use guidance with both mechanisms is de- picted in Figures la and 1b, via blue edges.
Dynamic Semantic Abduction. Recall that counterfac- tual soundness is assessed through composition (reconstruc- tion) and effectiveness (interventional faithfulness). Amor- tised anti-causal guidance ensures sound effectiveness by boosting cuunterfacuullcconiiioont noise estimates. How- ever, high guidance scales may suppress unconditional esti- mates, compromising composition and, therefore, identity preservation. Analogous tradeoff's are observed in diffusion- based image editing (Yang et al., 2024; Song et al., 2024; Tang et al., 2024) with text-conditional latent diffusion. No- tably, Mokady et al. (2023) notice that DDIM inversion produces poor reconstructions when amortised guidance is used for editing with Stable Diffussion (Rombach et al., 2022). They address this by optimising guidance tokens at each timestep align the inverse and guided trajectories.
3.3. Amortised, Anti-Causally Guided Mechanisms
The main limitation of conditional diffusion models their tendency to ignore conditioning signals during generation. This is because noise injection during training often weakens the association between e and x (Dhariwal & Nichol, 2021; Nichol & Dhariwal, 2021), leading the model to rely on the shortcut pe (x(c) R (x) to maximise likelihood (Chen et al., 2016). To address this, we incorporate amortised guidance into our mechanisms by reframing classifier-free guidance (Ho& & Salimans, 2022) through a causal lens. This procedure is more parameter-efficient, easier train, and encourages sample diversity (Dinh et al., 2023), compared to DiffSCM (Sanchez et al. (2022b)), who use separately trained classifier for guidance (Dhariwal& & Nichol, 2021).
Building on this idea, we propose counterfactual trajectory alignment (CTA) to improve composition by tuning guid- ance tokens, now modelled as exogenous noise terms. We find that a linear update for guidance tokens for each image at each timestep (from T to 1) is more computationally ef- ficient than Mokady et al. (2023); Yang et al. (2024) and sufficient to improve identity preservation:
We introduce amortised guidance into the semantic mech- anism to enhance the counterfactual prediction step. We use modified noise estimator within the denoiser (Equa- tion (4)) derived from the sharpened, anti-causal score func- tion Vx log Pe (Ceeml |x)a (Appendix B.1):
c Ø,t) + ((e((X,, Csem, t) co G, t)), (16)
of + CTA (Ø), X111
where c0(x;, Ø,t) represents the unconditional noise es- timate, Ø is a guidance token introduced during training,
:= ot - Vo; ||x:- - x
(20)
5

---

Diffusion Counterfactual Generation with Semantic Abduction
5 5 8 5
Table I. Soundness of Morpho-MNIST image counterfactuals gen- erated under do(d) from DSCMs modelling the relationship d + x, which the digit class (d) the only parent the image (x), with data generated from the true SCM in Appendix E. Amor- tised, anti-causally guided mechanisms are trained with Pa 0.1. Effectiveness (EFF.) is measured by the accuracy (Ace) of pre- trained classifier. Diffusion counterfactuals are normalised to [0. 1] to measure composition (COMP.) and reversibility (REV.) while ensuring faithful comparison to VAE/HVAE baselines.
6
6 7 7 7
6
6
7
0
0
0
0
(a) Morpho-MNISS ManphO-MIISTIIII DSCM
(b) Counterfactual Soundness
3
4
5
6
7
8
9
CоMP.
EFF.
MECHANISM
L 4 (x10-)
Acc
L (x10-2)
6
3
5
DIFFSCM (Sanchez & Tsaftaris, 2022)
0.410
17.02
1.31
<-1
2-1
del
C-4
2-
2103
0-03
1450
1-18
Eo1
VCI (Wu al. 2024)
2.05
92.48
6.71
3
4
5
6
7
9
6
6
6
6
6
6
SPATIAL (w - L.5]
0.615
99.63
2.56
SPATIAL (w - 3)
1.92
90.95
3.42
X
6
6
6
SEMANTIC [w = 1.5)
0.342
97.46
2.53
SEMANTIC {u = 3}
1.20
99.90)
3.17
9
3
4
S
6
7
8
9
9
9
9 9 9 7
9
9
4. Experiments
(c) Morpho-MNISS MNIST Image Counterfactuals
We present three case studies using our mechanisms for counterfactual image generation 1. We begin with a toy scenario where we control the true causal data-generating process, and progressively scale up our mechanisms for causal face modelling and novel medical artefact removal problem. We compare our mechanisms against VAE, HVAE and diffusion-based alternatives (Pawlowski et al., 2020; De Sousa Ribeiro et al., 2023; Wu et al., 2024; Sanchez & Tsaftaris, 2022) using counterfactual soundness metrics.
Figure 2. Morpho-MNISS (28 x 28) counterfactuals generated using amortised, anti-causally guided semantic mechanism (Pe = 0.1, w = 1.5) based on the DSCM shown in (a). (b) illustrates counterfactual soundness (Obs: Observation, Comp: Composition, CE: Counterfactual, Rev: Reversibility). (c) depicts image counterfactuals: interventions are shown above the top row and the bottom row visualises total causal effects (red: increase, blue: decrease), refer to (Appendix A.2) for details.
Morpho-MNIST We begin by applying our proposed mechanisms within a DSCM a Morpho-MNIST dataset (Castro et al., 2019) generated from a known SCM (Ap- pendix E). The corresponding computational graph, shown in Figure 2a, extends the work of (De Sousa Ribeiro et al., 2023) to introduce a challenging causal relationship between digit class (d) and slant (s). We also use this dataset to im- plement DSCMs modelling a subset of mechanisms from the true SCM, {d - x} and [t + i, + x + i), with re- sults for the latter presented in Appendix E.2. Additionally, we construct an SCM for colourised variant of Morpho- MNIST, where the digit class (d) causes hue (h), presented in Appendix F. We use the true-data generating mechanisms for pa within these DSCMs. Effectiveness is measured us- ing pre-trained classifier for d and measurement functions provided by the Morpho-MNIST library for i, and
where 21. initialised to Ø, 0;_1 f- 23 is set at the end of each timestep, x lije= (x²)) with C' = {Caom, Øi, 0}, X--1- = 1 2)) with C = (Ceem: 0}. 1 is the step size and we use w > 1. We provide the full algorithm in Appendix B.2. We use CTA within dynamic semantic abduction for our guided semantic mechanism. Guidance tokens are modelled as independent exogenous noise terms via p(E) = p(z)p(u) IImr: 8(2, Ø). The exogenous posterior is approximated by
T
= Pes PPs (2)x))))es (u|x, csem) II Pe (e. x4)
(21)
T
= (z)e)((()))) - hg" (xx, csem)
)
II 8(0,, - Ø1), t=1
Table reports the counterfactual soundness results for sim- ple DSCMs modelling only the mechanism d - x, as- sessed under random interventions do(d). Our amortised, anti-causally guided mechanisms outperform VCI in both effectiveness and identity preservation, as measured by com- position and reversibility. Notably, semantic mechanisms achieve better identity preservation than their spatial coun- terparts for the same w, albeit with minor reductions in ef-
and counterfactuals are generated as
X Rd hū (u, Tzem U ( = (hije,
ho (u),
(22)
where e = (Ceem: Ø7, 0}. This is similar to Equation (18) with Ø set to the result of CTA at each timestep.
"trppe / //git hub cpn /uo jus Rasse 1 //b Fuxiion Count we Factuala
6

---

Diffusion Counterfactual Generation with Semantic Abduction
Table Soundness of Morpho-MNiss image counterfactuals under do(s) and do(d) using DSCMs modelling the SCM Appendix E. Effectiveness for digit class (d) is measured using accuracy (Ace) from a pre pre-trained classifier and mean absolute percentage error (MAPE) for slant (s), thickness (t) and intensity (i). Counterfactuals are normalised to [0, 1] to measure composition (COMP.) and reversibility (REV.). Metrics are scaled by x10 except for MAPE (s), which is scaled by x x10 3 and Ace (d), which remains unscaled.
SLANT INTERVENTION (dole)
CLASS INTERVENTION (dol))
NULL
BIFECTIVNESS
EIVECTIVENESS
REV.
COMP.
MECHANISM
MAРE (1)
MAPE (8)
MAРЕ (2)
Ace (d)
I/2 4
MAPE (1)
MAPE (F)
MAPE (8)
Acc (d))
1/1
(Pawlowski et 2020)
4.63
6.98
94.92
2.44
HVAE (De Sousa Ribciro ot al., 2023)
3.39
0.193
3.88
95.02
0.015
1.39
0.508
1.23
95.31
1.62
0.008
VCI (Wa at.,
3.08
0.652
0.907
94.82
0.990
0.655
SPATIAL:
2.78
0.552
2.45
96.62
2.74
3.11
0.592
2.13
96. 29
3.78
0.555
(w = 1.5, Pa
0.1)
2.17
0.501
4.50
2.012
[+ = 3, pa 0.1]
1.85
0.291
0.885
99.84
5.25
1.87
0.292
0.855
99.90
5.70
3.29
(o= 4.5. Pa
0.607
=
0.5)
2.88
0.739
2.43
97.75
2.91
3.41
0,802
1.29
93, 95
4.31
0,660
3. Pe 0.5)
2.01
0.368
0,315
(0 = 4.5. 0.5)
2.02
0.393
1.84
99.71
4.49
2.08
0,380
0.783
98.14
5.30
1.77
SEMANTIC:
0
5:60
0.139
[A - 1.5, 0.1)
2.83
1,20
1.51
98.44
2.98
3.88
1.13
1.43
97. 86
4.33
0.940
pa 0.13
2.14
1.00
5.35
2.17
(2 - 4.5, 0.1)
2.13
0.796
1.28
99.80
5:35
7.67
0.941
0.762
99.93
6.15
2.87
fectiveness. DiffSCM struggles to generate counterfactuals that faithfully reflect the intervention, as such they attain the best identity preservation because of poor effectiveness. We present strategies to improve DiffSCM's effectiveness and their trade-offs with identity preservation in Appendix E.3.
O
X
Figure 2c illustrates that our mechanisms can model com- plex causal relationships: increasing and using larger val- ues of result in increases in i and 8, respectively, while and can be controlled independently of their parents. Counterfactual soundness results, seen in Figure 2b, are summarised in Table 2 for random interventions ds(s) and do(d), with additional results for do(() and do(i) in Ap- pendix E.2. Our mechanisms perform comparably or bet- ter than baselines, with guidance improving effectiveness beyond them. Increasing w improves effectiveness at the cost of identity preservation (composition and reversibil- ity). For chosen w, mechanisms trained with pe 0.5
Ø
I.T
(a) CelebA-HQ DSCM
(b) Counterfactual Soundness
mosevation
2
2o 0.22
32 0-0002
Semeetc
Semark
small drop in effectiveness compared those trained with Pa 0.1, while exhibiting better identity preserva- tion. This improvement is attributed to reduced sample diversity at higher Pæ (Ho & Salimans, 2022), which we deem advantageous for image editing tasks. Notably, for the same w and pa. guided semantic mechanisms achieve com- parable effectiivnness to guided spatial mechanisms across many confounders of while substantially improving iden- tity preservation. While VCI achieves good MAPE on I, and under do(s) interventions, it exhibits significantly lower d accuracy here compared to many ablations of our proposed mechanisms. This suggests that, while localised interventions are faithfully obeyed, global edits to digit class remain challenging. In contrast, our guided mechanisms achieve higher d accuracy, with comparable effectiveness for other covariates, demonstrating more balanced trade- off between intervention faithfulness and preservation of core image characteristics.
Ozsevation
de Py-Ri
ty-tt
I
(c) Semantic Abduction improves Identity Preservation
Figure 3. CelebA-HO HQ (64 x 64) counterfactuals generated using amortised, anti-causally guided semantic mechanisms. (a) DSCM with spatial (u), semantic (z) and dynamic (Øirr) exogenous noise terms for x. (b) shows counterfa counterfactual soundness using semantic abduction with Ps 0.1 and = 2. (c) shows that semantic mechanisms improve identity preservation, and dynamic abduction further improves backgrounds, hairstyle, skin colour and facial structure. Here, choice r) fine-tuned each observation.
7

---

Diffusion Counterfactual Generation with Semantic Abduction
Table 3. Soundness of CelehA-HQ image counterfactuals generated using our proposed diffusion-hased mechanisms under simulated interventions. Effectiveness is measured using the Fl scores from pre- trained classifiers for eyeglasses (g) and smiling (s).
EVEGLASSES INTERVENTION (do(
SMILING INTERVENTION (do(s))
NULL
EPPECTIVENESS
REV.
IDP
EPPECTIVENESS
REV.
IDP
COMP:
MECHANISM
FI(+) 1
Fi(g) 1
L1 4
LPIPS
FI(+) 7
F1(g)
L;
LPIPS
L2
SPATBAL:
95.65 (0.12)
94.08 (0.24)
0.084 (000))
(0.0002)
95.65 (0.04)
95.32 (0.
0.078 (0. (0.00033)
0. (0.0003)
0.034 (100000))
[w-22
98. 08.33 00)
99.07 (0.03)
0.211 (0.9002)
0.171 (0.0002)
99.00 (0.12
99.23 (0.00)
0.183 (0.0003)
0.139 (0.0007)
0.130 (0. (0..004))
98.34 25)
90.27 (0:11)
0.297 (0.0006)
0.197 (0.0009)
99.50 (0 16)
99.19 (0.00)
0.264 (0.0004)
0.161 (0. (0.0005)
0.196 (0. (0.00004
0.5
07:12 (0 35)
90.85 (0.13)
0.141 (0.0004)
0.138 (0.0000)
96.51
96.30 (0.38)
0.127 (0.0002)
0.110 (0.0003)
0.090 (0 (0.00)01)
0.55
07.73 30)
96.33 (0.15)
0.214 (0.0001)
0.161 (0.0007)
36.33 (0)
98.12 (0.37)
0.188 (0.0002)
0.128 (0.0004)
0.159 (0.0001)
SEMANTIC:
00.54
(0:18)
0.046 0001)
0.077
00000
92.25
(0.0000)
0.007
0002)
35)
0.130
90.
(0.11) 13)
0002)
(0.0002)
38
(0.0002)
0.077
0.174 10.0001)
98 20
(0.12)
(0.000 (0.0001)
(0.0002)
38
(0.0002)
0.066 (0.0002)
0.122 (0.0002)
(wm3. P0 0.22
(0.17)
(0.16)
(0.0001)
(0.00044)
(9..000.1))
34)
(0.0002)
0.080 (0.0002)
0.368 (0.0003)
Eyeplasses Inservention
Smiling Intervention
Dyeglasses Intervention
Smiling Intervention
0:30
100.
1310
0.12
91.5
Compositon
0.20
92.5
0:15
920
50.0
0.00
815
0:05
Soturt:
825
Suura
$2.5
820
0.05
0.30
0..35
0.00
0.15
IDP
IDP
Step
Step
Figure 4. CelebA-HQ: Effect of guidance scale (w). depicted by the number on each dot, on composition, model identity preser- vation (IDP) and effectiveness of amortised anti-causally guided mechanisms trained with p= 0.1.
Figure 5. CelebA-HQ: Effect of step size (n) and guidance scale (w) on model identity preservation (IDP) when using dynamic ab- duction with amortised anti- aati-causally guided semantic mechanism (pa = 0.1) on 200 randomly chosen images from the val. set.
We now scale-up our mechanisms for causal modelling of real-world images in CelebA-HQ (Karras, 2017). Here, we model the binary variables Smiling (s), Mouth Open (m), and Eyeglasses (g) as parents of x, and m, shown by the computational graph in Fig- ure 3a. We conduct interventions on g by toggling its ob- served value, do(g"), and simulate interventions on s with do(s')= = do(s = s', m = s'), reflecting associative L1- level statistics in CelebA-HQ and our knowledge of facial expressions, i.c. smiling causes the mouth to open. In preliminary experiments, we observed that spurious corre- lations under do(s') sometimes led unintended effects like hair loss or removal of glasses. To mitigate this, we concatenate confounding attributes Male, Wearing Lip- stick, Bald, and Wearing Hat into a variable o, modelled as an independent parent of x, which we do not subject to interventions. We discuss baseline models in Appendix H.2.
CelebA-HQ
improvement in composition, while reversibility and IDP re- main comparable. Notably, semantic mechanisms improve identity preservation metrics over their spatial counterparts with the same pe and w, by better preserving high-level facial characteristics as shown in Figure 3c, albeit incurring small drop in effecttivnesss Figure 4 further illustrates the effect of w on identity preservation and effectiveness. Specifically, IDP and composition increase linearly with w, indicating a decrease in identity preservation, while effec- tiveness increases across both mechanisms. Importantly, the semantic mechanism consistently outperforms the spatial mechanism in identity preservation across all values of
Dynamic semantic abduction improves the preservation of intricate facial attributes, such as hairstyle, skin tone/illumi- nation, facial structure, and backgrounds, as illustrated in Figure 3c. Figure 5 demonstrates the effect of step size (1) in CTA, revealing that global interventions such as do(g) may require larger 17 to improve IDP compared to localised in- terventions do(s). Moreover, effectiveness improves along- side IDP in CTA. However, these improvements come with additional computational cost: dynamic semantic abduction requires ~ 3 minutes per image, compared to ~ 3 min- utes and ~ 3.5 minutes for the guided spatial and semantic mechanisms, respectively, using a batch size of 128 on an NVIDIA GeForee RTX 4090.
The counterfactual soundness trends observed in Morpho- MNIST, due to the choice of pe and w, are further amplified in CelebA-HQ, with results summarised in Table 3 and vi- sualisations in Figure 3b. In spatial mechanisms, setting Pa = 0.5 improves reversibility more under do(g) than the localised intervention do(s). but at the cost of lower effec- tiveness, as reflected in the reduced Fl-scores. In semantic mechanisms, increasing Pa from 0.1 to 0.2 yields a slight
8

---

Diffusion Counterfactual Generation with Semantic Abduction
Using prior insights, we apply our mechanisms to woal-world artefact removal task on the EMory BrEast imaging Dataset (EMBED) (Jeong et al., 2022). Schueppert et al. (2024) observe that triangular and circular skin mark- ers are spuriously associated with breast cancer in classifiers due to shortcut learning (Geirhos et al., 2020), and man- ually labelled 22,012 affected mammograms. Using this dataset, we train a significantly scaled-up. amortised. anti-
EMBED
Observation
Dynamic
Observation
Semantic
Remove Triang liannlee
Remove Circles
causally guided semantic mechanism (p2- = 0.1, w 1.2) to remove skin markers. We model triangular markers (0) circular markers (c), breast density (d), and cancer (1) as independent parents of the mammogram x, and remove arte- facts by intervening on and с while holding d and y fixed. Figure 6 shows that our mechanisms effectively remove arte- facts and can disentangle representations for triangles and circles. We successfully remove 95.16 + 1.34% of triangles and 91.69 = 1.06% of circles in our test set notewor- thy result given the dataset's 's small size and the scarcity of labelled skin markers (Appendix I).
Figure 6. Skin marker removal on EMBED (192 x 192) using an amortised, anti-causally guided semantic mechanism with and without dynamic abduction; (orange: circle, green: triangle).
which limits mechanisms' ability to learn robust strided trajectories, resulting in slower counterfactual inference. Future work could address this by using larger datasets to enable better strided generation and distillation techniques (Salimans & Ho, 2022; Song et al., 2023) to improve effi- ciency. Our focus on Markovian SCMs assumes that causal effects are identifiable, however, this assumption need not hold in practice. Noisy labels (Lingenfelter et al., 2022: Thyagarajan et al., 2022) and low-tusolution images can challenge the assumptions of our causal graph, potentially affecting the learned representations under untested inter- ventions. Future work could tackle assessing the impact of noisy labels in a controlled setting or using corrected labels where available (Wu et al., 2023). Additionally, we do not guarantee that pa 1L xx but instead set the dimensionality of and value of 8 such that we can control pa whilst im- proving identity preservation. Incorporating techniques for model identifiability (Khemakhem et al., 2020a; Yan et al., 2023; De Sousa Ribeiro et al., 2023) and providing guaran- tees on the contents of (Chen et al., 2025; Von Kügelgen et al., 2021) present useful directions for future research.
Conclusion
Our work highlights trade-off's inherent in using diffusion models, studied extensively on associative (L1) and inter- ventional (L2) applications, for counterfactual inference (Lz). We introduce diffusion-based causal mechanisms with semantic abduction capabilities and demonstrate their enhanced identity preservation in image counterfactuals, as measured by counterfactual soundness metrics. Notably, we show that large w and smaller Pa, popular at L2, compro- mise identity preservation (composition and reversibility) in favour of causal control (effectiveness) at C3. Current diffusion approaches are biased towards L2, limiting their soundness at L3 and widespread adoption at L1. Since L3 subsumes L1 and L2, we argue that a causal lens on diffu- sion modelling at L3 is essential for tackling a broad range of problems, rather than relying on models optimised for ran- dom sampling Lg. This would better generalise diffusion models to real-world tasks that demand creative reasoning and causal understanding- where diffusion already excels.
Our findings present new opportunities for using diffusion models at Lz. For example, fast generation with spatial mechanisms could enable efficient counterfactual data aug- mentation (Roschewitz et al., 2024). Enhanced identity preservation via semantic mechanisms could improve coun- terfactual explanations for downstream models (Augustin et al., 2022). Dynamic abduction could support stress testing by providing higher image fidelity for challenging test cases Béérez-Garcin et al., 2023). Lastly, these mechanisms could inspire sausally-informer generative recognition models, in- tegrating causal representations into the human- allined per- ceptual capabilities of diffusion models (Jaini et al., 2023).
A Acknowledgements
We thank Pavithra Manoj, Mélanie Roschewitz, and Michael Tänzer for their detailed and insightful feedback on early versions of this manuscript. R.R. is supported by the Engi- neering and Physical Sciences Research Council (EPSRC) through Doctoral Training Partnerships PhD Scholarship. A.K. is supported by UKRI (grant no. EP/S023356/1), as part of the UKRI Centre for Doctoral Training in Safe and Trusted AI. B.G. received support from the Royal Academy of Engincering as part of his Kheiron/RAEng Research Chair. B.G. and F.R. acknowledge the support of the UKRI Al programme, and the EPSRC, for CHAI- EPSRC Causal- ity in Healtheare AI Hub (grant no. EP/Y028856/1).
Limitations This work uses high-quality but small datasets,

---

Diffusion Counterfactual Generation with Semantic Abduction
Impact Statement
Bareinboim, E., Correa, J. D., lbeling, D., and Icard, T. On Pearl's Hierarchy and the Foundations of Causal Inference, pp. 507-556. Association for Computing Machinery, New York, NY, USA, 1 edition, 2022. ISBN 9781450395861. http:::::: /tpp:://doii /ttp:://///oo.o///////// (145/3501714.3500 3501744
This paper demonstrates the trade-offs inherent using diffusion models, widely optimised for Cz-level random sampling, to perform C3-level counterfactual inference. By uncovering these trade-offs, our work demonstrates the need to evaluate diffusion models from a perspective that consid- ers fairness and mitigates spurious correlations. Counter- factual inference shares strong similarities with text-driven image-to-image translation and editing, and our findings encourage practitioners to rethink the design of founda- tional diffusion models, which underpin these applications, moving beyond focus high-quee high-quality random sampling. Notably, in latent diffusion models, text control alone often fails to account for causal dependencies, instead defaulting to biases learned from training data, which can result in unin- tended consequences. In rext-guided image editing, prompts alone cannot be used to infer causal structure, leaving the model to infer relationships based solely on correlations observed in training data. As result, even minor edits can produce spurious changes. Adopting a structured causal perspective- where models implicitly discover or explicitly learn SCMs - enables developers to transparently state their model's assumptions and update them as new information becomes available through the causal graph, and restricts misuse by enforeing causal rules. These considerations are critical in real-world applications like face modelling and medical imaging, where dataset biases can lead to flawed decision- decision-making. In such high-stakes scenarios, ensuring that models are interpretable and transparent is essential, as Mlac-box models often fail to provide the clarity needed to justify their outputs.
Batzolis, G., Stanczuk, J., and Schönlieb, C.-B. Vari- ational diffusion auto-encoder: Latent space extrac- tion from pre-trained diffusion models. arXiv preprint arXi::2304. /2141, 2023.
Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. Understanding disen- tangling in Veta-VAE. arXiv preprint arXiv.180..03599, 2018.
Castro, D. C., Tan, J., Kainz, B., Konukoglu, E., and Glocker, B. Morpho-MNIST. Quantitative assessment and diagnostics for representation learning. Journal of Machine Learning Research, 20(178), 2019.
Chao, P., Blöbaum, P., and Kasiviswanathan, S. P. Interven- tional and counterfactual inference with diffusion models. arXiv preprini urXiv:2302.00864 2023.
Chen, B., Zhu, Y., Ao. Y., Caprara, S., Sutter, R., Rätsch, G., Konukoglu, E.,, and Susmelj, A. Generalizable single- source cross- coos--to medical image segmentation via invariant causal mechanisms. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 3592-3602. IEEE, 2025.
Chen, T. Q., Li, X., Grosse, R., and Duvenaud, D. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, 2018.
References
Abstreiter, K., Mittal, S., Bauer, S., Schölkopf, B., and Mehrjou, A. Diffusion- based representation learning. arXiv preprint arXiv:2105.14257. 2021.
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adver- sarial nets. Advances in neural information processing systems, 29, 2016.
Anonymous. Scaling in-the-wild training for diffusion- based illumination harmonization and editing by impos- ing consistent light transport. In Submitted to The Thir- teenth International Conference on Learning Represen- tations, 2024. URL hatps://openrevieee forum? 1 1ddullllle 2orum??d==lco@xxRTT cQYRRI1H. under review.
Child, R. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011.1068 2020.
Atad, M., Schinz, D., Moeller, H., Graf, R., Wiestler, B., Rueckert, D., Navab, N., Kirschke, J. S., and Keicher, M. Counterfactual explanations for medical image classifi- cation and regression using diffusion autoencoder. arXiv preprint wxXiv:240800157.. 2024.
Cho, W., Ravi, H., Harikumar, M., Khuc, V., Singh, K. K., Lu, J., Inouye, D. I., and Kale, A. Towards en- hanced controllability of diffusion models. arXiv preprint 230.:23302.13668. 2023.
Chung, H., Kim, J., Park, G. Y., Nam, H., and Ye, J. C. Cfg++: Manifold-constrainedd classifier free guidance for diffusion models. arXiv preprint arXiv:2406.08077 2024.
Augustin, M., Boreiko, V., Croce, F., and Hein, M. Diffusion visual counterfactual explanations. Advances in Neural Information Processing Systems, 35:364-377 2022.
10

---

Diffusion Counterfactual Generation with Semantic Abduction
Clark, K. and Jaini, P. Text-to-image diffusion models are zero shot classifiers. Advances in Neural Information Processing Systems, 36, 2024.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014.
Couairon, G., Verbeek, J., Schwenk, H., and Cord, M. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint arXiv:2210. 11427, 2022. Dash, S., Balasubramanian, V. N., and Sharma, A. Evalu- ating and mitigating bias image classifiers: A causal perspective using counterfactuals. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 915-924, 2022.
Gu, Y., Yang, J., Usuyama, N., Li, C., Zhang, S., Lungren, M. P., Gao, J., and Poon, H. Biomedjourney: Coun- terfactual biomedical image generation by instruction- learning from multimodal patient journeys. arXiv:2310.10763 2023.
jourenyj arXiv preprint
Haas, R., Huberman- Huberman--piegelllass I., Mulayoff, R., Graßhof, S., Brandt, S. S., and Michaeli, T. Discovering inter- pretable directions in the semantic latent space of diffu- sion models. In 2024 IEEE 18th International Conference Automatic Face and Gesture Recognition (FG), pp. 1-
De Sousa Ribeiro, F., Xia, T., Monteiro, M., Pawlowski, N., and Glocker, B. High fidelity image counterfactuals with probabilistic causal models. In Proceedings of the 40th International Conference on Machine Learning PP- 7390-7425, 2023.
9. IEEE, 2024.
Halpem, J. Y. Axiomatizing causal reasoning. Journal of Artificial Intelligence Research, 12:317-337, 2000.
Dhariwal, and Nichol, A. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8793 8794, 2021.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01622 2022.
Dinh, A.D Liu, D., and Xu, C. Rethinking condi- tional diffusion sampling with progressive guidance. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, vol- ume 36, pp. 42285 42297. Curran Associates, Inc., 2023. URL https::/prooceedingg neurips. filees/paperrr222///1ee ce/paper_fiiiee
Higgins, I., Matthey, L., Pal, A.. Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta-VAE: Learning basic visual concepts with con- strained variational framework. In International Confer- ence Learning Representations, 2017. URL https: S2229g Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint 1rXiv:2207.12599 2022.
/ //openreview. net fereencee
pdf.
Epstein, D., Jabri, A., Poole, B., Efros, A., and Holynski, A. Diffusion self guiddnc for controllable image generation. Advances in Neural Information Processing Systems, 36: 16222-16239, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Fang, Y., Wu, S., Jin, Z., Wang. S., Xu, C.,, Walsh, S., and Yang, G. Diffexplainer: Unveiling black box models via counterfactual generation. In International Conference on Medical Image Computing and Cossuter-Assisted Intervention, pp. 208-218. Springer, 2024.
Hwa, J., Zhao, Q., Lahiri, A., Masood, A., Salimi, B., and Adeli, E. Enforcing conditional independence for fair representation learning and causal image generation. In Proceedings of the IEEE//VV Conference on Computer Vision and Pattern Recognition, pp. 103-112, 2024.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.00 01661888... 2022.
Hyvarinen, A. and Morioka, H. Unsupervised feature ex- traction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Galles, D. and Pearl, J. An axiomatic characterization of causal counterfactuals, Foundations of Science, 3:151- 182, 1998.
Hyvarinen, A., Sasaki, H., and Turner, R. Nonlinear ica us- ing auxiliary variables and generalized contrastive learn- ing. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859-868 868. PMLR, 2019.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. A. Shortcut leam- ing in deep neural networks. Nature Machine Intelligence, 2(1 1 1):665-6777 2020.
Ibrahim, Y., Warr, H., and Kamnitsas, K. Semi-supervised learning for deep causal generative models. arXiv preprint arXiv:2403. 18717, 2024.

---

Diffusion Counterfactual Generation with Semantic Abduction
Jaini, P., Clark, K., and Geirhos, R. Intriguing properties of generative classifiers. arXiv preprint arXiv:2309.16779, 2023.
Komanduri, A., Wu, X., Wu, Y., and Chen, F. From identi- fiable causal representations controllable counterfac- tual generation: A survey on causal generative modeling. arXiv preprint wrXXv:2330.110111 2023.
Jeong, J. J., Vey, B. L.,, Reddy, A., Kim, T., Santos, T., Correa, R., Dutt, R., Mosunjac, M., Oprea-llies, G., Smith, G., ct al. The emory breast imaging dataset (embed); a racially diverse, granular dataset of 3.5 m screening and diagnostic mammograms. arXiv preprint arXiv:2202.04077 2022.
Komanduri, A., Zhao, C., Chen, F., and Wu, X. Causal diffusion autoencoders: Toward counterfactual genera- tion via diffusion probabílistic models. arXiv preprint arXi::2404.1773 2024.
Kumar, A., Sattigeri, P., and Balakrishnan, A. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018
Karras, T. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 10196, 2017.
Karras, T. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04448, 2019.
Li, S., Hooi, B., and Lee, G. H. Identifying through flows for recovering latent representations. arXiv preprint arXiv: /2555, 2019.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 10- 81 8119, 2020.
Lin, S., Liu, B., Li, J., and Yang. X. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 5404-5404444 1114,,11,, 2024.
Lingenfelter, B., Davis, S. R., and Hand, E. M. A quantitative analysis ofnbsp:labeling issues imbbs:theenss::celbba dataset. In Advances in Vi- sual Computing: 17th International Symposium, ISVC 2022, San Diego, CA, USA, October 3-5, 2022, Pro- ceedings, Part I, pp. 129-141, Berlin, Heidelberg, 2022. Springer- ISBN 978-3 978-3-031-2001 12-9. doi: 10. 10.1007//97833003-2071111 10. URL https: /doi. org/10.1007/9788 200-300---1271---11 6_00
Karras, T., Aittala, M., Kynkäänniemi, T., Lehtinen, J., Aila, T., and Laine, S. Guiding a diffusion model with a bad version of itself. arXiv preprint arXiv:2406.000 02507, 2024.
Khemakhem, I., Kingma, D., Monti, R., and Hyvarinen, A. Variational autoencoders and nonlinear ica: A unify- ing framework. In International conference on artificial intelligence and statistics, pp. 2207-2217. PMLR, 2020a.
Khemakhem, I., Monti, R., Kingma, D., and Hyvarinen, A. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Infor- mation Processing Systems, 33:12768-11278 2020b.
Liu, Z., Luo, P., Wang, X., and Tang. X. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning. pp. 4114 4124. PMLR, 2019.
Kim, H. and Mnih, A. Disentangling by factorising. In International Conference Machine Learning, 2018.
Kingma, D. Auto-encoding variational bayes. arXiv preprint 1rXiv: 2013.
Locatello, F., Bauer, S., Lucie, M., Rätsch, G., Gelly, S., Schölkopf, B., and Bachem, O. A sober look at the unsupervised learning of disentangled representations and their evaluation. Journal of Machine Learning Research, 2 21(209): 2020.
Kingma, D. P. Adam: A method for stochastic optimization. 2014.
arXiv preprint 14X2::
Kladny, K.-R., von Kügelgen, J., Schölkopf. B., and Muchlebach, M. Deep backtracking counterfactuals for causally compliant explanations. arXiv preprint arXiv:2310.07665. 2023.
Luo, C. Understanding diffusion models: A unified perspec- tive. arXiv preprint a2Xiv:2208.1 11900, 2022.
Kocaoglu, M., Snyder, C., Dimakis, A. G., and Vish- wanath, S. Causalgan: Learning causal implicit gen- erative models with adversarial training. arXiv preprint 1709v:/709.000 02003, 2017.
Melistas, T., Spyrou, N., Gkouti, N., Sanchez, P., Vlont- zos, A., Papanastasiou, G., and Tsaftaris, S. A. Bench- marking counterfactual image generation. arXiv preprint arXiv:2403.0m 20287, 2024.
12

---

Diffusion Counterfactual Generation with Semantic Abduction
Meng, C., He, Y., Song. Y., Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations, arXiv preprint arXiv:2108.01073, 2021.
Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., and Zhu, J.-Y. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1- 11, 2023.
Mirza, M. and Osindero, S. Conditional generative adver- sarial nets, arXiv preprint arXiv:1411.1784 2014.
Pawlowski, N., Coelho de Castro, D., and Glocker, B. Deep structural causal models for tractable counterfactual in- ference. Advances in Neural Information Processing Systems, 33:857-869, 2020.
Mittal, S., Abstreiter, K., Bauer, S., Schölkopf, B., and Mehrjou, A. Diffusion based representation learn- ing. In Krause, A., Brunskill, E., Cho, K., Engel- hardt, B., Sabato, S., and Scarlett, J. (eds.), Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learn- ing Research, pp. 2023. http:t httptt/::pppppopeeieeee proceedings mlr press// html.
Pearl, J. Direct and indirect effects. Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intel- ligence, pp. 411-420, 2001.
Pearl, J. Causality. Cambridge University Press, edition, 2009. doi: 10.1011/CB09780511 1803161.
PMLR, 23-29 Jul
Pearl, J. The seven tools of causal inference, with reflections on machine learning. Communications of the ACM, 62 (3):54660, 2019.
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen- Or, D. Null-text inversion for editing real images us- ing guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6044 2023.
Peebles, W., Peebles, J., Zhu, J.-Y., Efros, A., and Torralba, A. The hessian penalty: A weak prior for unsupervised disentanglement. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pp. 581-597. Springer, 2020.
Monteiro, M., Ribeiro, F. D. S., Pawlowski, N., Castro, D. C., and Glocker, B. Measuring axiomatic sound- of counterfactual image models. arXiv preprint arXiv:2303.01274 2023.
Pérez-García, F. Bond-Taylor, S., Sanchez, P. P., van Breugel, B., Castro, D. C., Sharma, H., Salvatelli, V., Wetscherek, M. T., Richardson, H., Lungren, M. P., et al. Radedit: stress-testing biomedical vision models via dif- fusion image editing. arXiv preprint wxXiv:2312.12800 2023.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1071,, 2021.
Peters, Janzing, D., and SchIkopf, B. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017. ISBN 0262037319.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffu- sion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Pan, Y. and Bareinboim, E. Counterfactual image editing. arXiv preprint arXiv:2403.0m 006633,,999..... 2024.
Poinsot, A., Leite, A., Chesneau, N., Sébag, M., and Schoe- nauer, M. Learning structural causal models through deep generative models: Methods, guarantees, and challenges. arXiv preprint urXiv:2405.05025 2024.
Pandey, K., Mukherjee, A., Rai, P., and Kumar, A. Dif- fusevae: Efficient, controllable and high-fidelity gen- eration from low-dimensional latents. arXiv preprint arXhv:2201.00308, 2022.
Prabhu, V., Yenamandra, S., Chattopadhyay, P., and Hoff- man, J. Lance: Stress-testing visual models by generating language-guiin counterfactual images. Advances in Neu- ral Information Processing Systems, 36:25165-25188 2023.
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan B. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 12(57):1-64 2021.
Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwa- janakorn, S. Diffussion autoencoders: Toward meaning- ful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Park, Y.H.., Kwon, M., Choi, J., Jo, J., and Uh Y. Un- derstanding the latent space of diffusion models through the lens riemannian geometry. Advances in Neural Information Processing Systems, 36:24129-24111 2023.
13

---

Diffusion Counterfactual Generation with Semantic Abduction
Rahman, M. M., Jordan, M., and Kocaoglu, M. Condi- tional generative models are sufficient to sample from any causal effect estimand. arXiv preprint arXiv:2402.07419, 2024.
Saharia, C., Chan, W., Saxena, S., Li, L.., Whang. J., Denton, E. I.., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35:
Rasal, R., Castro, D. C., Pawlowski, N., and Glocker, B. Deep structural causal shape models. In European Confer- ence on Computer Vision, pp. 400-432. Springer, 2022.
36479-36494, 2022.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
Ravi, D., Alexander, D. C., Oxtoby, N. P., and Initiative, A.
D. N. Degenerative adversarial neuroimage nets: gen- erating images that mimic disease progression. In Inter- national Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 164-172. Springer, 2019.
Sanchez, P. and Tsaftaris, S. A. Diffusion causal models for counterfactual estimation. arXiv preprint arXiv:2202. 10166, 2022.
Sanchez, P., Kascenas, A., Liu, X., O'Neil, A. Q. and Tsaf- taris, S. A. What is healthy? generative counterfactual diffusion for lesion localization. In MICCAI Workshop on Deep Generative Models, pp. 3444.. Springer, 2022a.
Reinhold, J. C., Carass, A., and Prince, J. L. A struc- tural causal model for mr images of multiple sclerosis. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Confer- ence. Strasbourg, France, September 27 27-October 1, 2021, Proceedings. Part V 24, pp. 782-792. Springer, 2021.
Sanchez, P., Voisey, P., Xia, T., Watson, H. I., O' O'Neil, A. Q., and Tsaftaris, S. A. Causal machine learning for healthcare and precision medicine. Royal Society Open Science, 98)2220638, 2022b.
Rezende, J. and Viola, F. Taming vaes. arXiv preprinl arXiv:1810.00597, 2018.
Sauer, A. and Geiger, A. Counterfactual generative net- works. arXiv preprint arXiv:2101. 060066 2021.
Ribeiro, F. D. S. and Glocker, B. Demystifying varia- tional diffusion models. arXiv preprint arXiv2400.002811, 2024.
Schueppert, A., Glocker, B., and Roschewitz, M. Radio- opaque artefacts in digital mammography: automatic de- tection and analysis of downstream effects. arXiv preprint arXiv:2410.03809, 2024.
Roeder, G., Metz, L., and Kingma, D. On linear identifia- bility of learned representations. In International Con- ference on Machine Learning, pp. 9030-9000 90099909...00 PMLR, 2021.
Schut, L., Key, O., Mc Grath, R., Costabello, L., Sacaleanu, B., Gal, Y., et al. Generating interpretable counterfac- tual explanations by implicit minimisation of epistemic and aleatoric uncertainties. In International Conference on Artificial Intelligence and Statistics, pp. 1756-1764. PMLR, 2021.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp. 10684-10695, 2022.
Shen, X., Liu, F., Dong, H., Lian, Q., Chen, Z., and Zhang, T. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23(241):1-55, 2022.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention-MICCA 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 241. Springer, 2015.
Shen, Y., He, G., and Unberath, M. Promptable counterfac- tual diffusion model for unified brain tumor segmentation and generation with mris. In International Workshop on Foundation Models for General Medical AI, pp. 81-90. Springer, 2024.
Roschewitz, M., de Sousa Ribeiro, F., Xia, T., Khara, G., and Glocker, B. Counterfactual contrastive learning: robust representations via causal image synthesis. In MICCAI Workshop on Data Engineering in Medical Imaging, pp- 22-32. Springer, 2024.
Shu, R. and Ermon, S. Bit prioritization in variational au- toencoders via progressive coding. In Chaudhuri, K., Jegelka, S., Song. L.. Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of P Proceedings of Machine Learning Research, pp. 20141-2000 201555. PMLR, 17-23 Jul 2022. URL ht batpps: //t:://prrr http///peeiiiicccc ngs mllr. press/ /v1 62. 62./////.eee shu22a html.
Ruiz, N., Li, Y., Jampani,
V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500-2510, 2023.
14

---

Diffusion Counterfactual Generation with Semantic Abduction
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 07- 09 Jul 2015. PMLR, URL wwwphhtpps/tcc /ttp::::////c//eingg///e/ mlr. press / vv7 sch1 wwwcchh--iiisssee www.diiiicccicieeeeeeeeeeiee htmll. html.
Thyagarajan, A., Snorrason, E., Northcutt, C., and Mueller, J. Identifying incorrect annotations in multi-labe classifi- cation data. arXiv preprint arXiv:2211. 133899 2022.
Tian, J., Aggarwal, L.,, Colaco, A., Kira, Z., and Gonzalez- Franco, M. Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. arXiv preprint arXiv:2308. 12469, 2023.
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 1921-1930, 2023.
Sohn, K., Lee, H.,, and Yan, X. Learning structured output representation using deep conditional generative models. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https:/ oe cc/ httppee_iilleeepppper//////////1 meerii neurips. 8d55a24 425522442222 wwweeeeaaaaao00000222222200 pdf.
Turner, R. E.,, Diaconu, C.-D., Markou, S., Shysheya, A., Foong, A. Y., and Mlodozeniec, B. Denoising diffusion probabilistic models in six simple steps. arXiv preprint arXi::2402.04384 2024.
Paper.
Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint urXiv:2010.02502, 2020a.
Vahdat, A. and Kautz, J. Nvae: A deep hierarchical vari- ational autoencoder. Advances in neural information processing systems, 33:19667-19679, 2020.
Song, X., Cui, J., Zhang, H., Chen, J., Hong, R., and Jiang, Y.-G. Doubly abductive counterfactual inference for text- based image editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9162-9171, 2024.
Van Looveren, A. and Klaise, J. Interpretable counterfactual explanations guided by prototypes. In Joint European Conference on Machine Learning and Knowledge Dis- covery in Databases, pp. 650-665. Springer, 2021.
Song. Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Von Kügelgen, J., Sharma, Y., Gresele, L.., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F. Self- supervised learning with data augmentations provably isolates content from style. Advances in neural informa- tion processing systems, 34:16451-16461 2021.
Song, Y., Sohl-Dickstei,, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-hased generative modeling through stochastic diffcrential equations. arXiv preprint arXiv:2011.13455 2020b.
Wang, B. and Vastola, J. J. The hidden linear structure in score-based models and its application. arXiv preprint 2xXiv:23/1.10083 2023.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. arXiv preprint ar arXi:22303.01469, 2023.
Wang. K., Yang, F., Yang, S., Butt, M. A., and van de Weijer, J. Dynamic prompt learning: Addressing cross- attention leakage for text-based image editing. arXiv preprint arXiv:2309. 15664466666,,,11111 2023a.
Sorrenson, P., Rother, C., and Köthe, U. Disentanglement by nonlinear ica with general incompressible-flow networks (gin). arXiv preprint arXiv:2001.044 0000000..48722222 2020.
Wang, Q. Zhang, B., Birsak, M., and Wonka, P. In- structedit: Improving automatic masks for diffusion- based image editing with user instructions. arXiv preprint arXiv:2305. 18047, 2023b.
Tang, C., Wang, K., Yang, F., and van de Weijer, J. Locinv: Localization-awant inversion for text-guided image edit- ing. arXiv preprint arXiv:2405.01 01966. 2024.
Tang, R., Liu, L., Pandey, A., Jiang, Z., Yang. G., Kumar, K., Stenetorp, P., Lin, J., and Ture, What the daam: Interpreting stable diffusion using cross attention. arXiv preprint arXiv:2210.044880 2022.
Wang, Z., Gui, L., Negrea, J., and Veitch, V. Concept alge- bra for score-basedl text-controlled generative models. Advances in Neural Information Processing Systems, 36, 2024.
W. Sadeghi, Z., and Matwin, S. Causal generative explainers using counterfactual inference: a case study on the morpho-mmin mnist dataset. Pattern Analysis and Applications, 27(3):89, 2024.
Weng, N., Pegios, P., Feragen, A., Petersen, E., and Bigdeli, S. Fast diffusion-baase counterfactuals for shortcut re- moval and generation. arXiv preprint 23v:2312.11422. 2023.
15

---

Diffusion Counterfactual Generation with Semantic Abduction
Willetts, M. and Paige, B. don't need u: Identifiable non-linear ica without side information. arXiv preprint arXiv:2106.05228, 2021.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang.
O. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Winkler, C., Worrall, D. E., Hoogeboom, E., and Welling, M. Learning likelihoods with conditional normalizing flows. 2019.
Zhang. Z., Zhao, Z., and Lin, Unsupervised represen- tation learning from pre-trained diffusion probabilistic models. Advances in neural information processing sys- tems, 35:22117-221133. 2022.
Wu, H., Bezold, G., Günther, M., Boult, T., King, M. C., and Bowyer, K. W. Consistency and accuracy of celeba attribute values. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 3258-3266, 2023.
Wu, Y. McConnell, L., and Iriondo, C. Counterfactual generative modeling with variational causal inference. arXiv preprint 12XV:2440.12230, 2024.
Xia, K. M., Pan, Y., and Bareinboim, E. Neural causal models for counterfactual identification and estimation. In The Eleventh International Conference on Learning Representations, 2023.
Xia, T., Roschewitz, M., Ribeiro, F. D. S., Jones, C., and Glocker, B. Mitigating attribute amplification in counterfactual image generation. arXiv preprint 24X1::2403.09422, 2024.
Yan, H., Kong, L.., Gui, L.., Chi, Y., Xing, E., He, Y., and Zhang, K. Counterfactual generation with identifiability guarantees. Advances in Neural Information Processing Systems, 11556266-562777 2023.
Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., and Wen, F. Paint by example: Exemplar image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18381-18391, 2023a.
Yang, F, Yang, S., Butt, M. A., van de Weijer, J., et al. Dynamic prompt learning: Addressing aots-attontion leakage for text- image editing. Advances in Neural Information Processing Systems, 36, 2024.
Yang, M., Liu, F., Chen, Z., Shen, X., Hao, J., and Wang, Vausalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9593 9593-9602, 2021.
Yang, T., Wang, Y., Lv, Y., and Zheng, N. Disdiff: Unsuper- vised disentangiement of diffusion probabilistic models. arXiv preprint arXiv:2301.13721 2023b.
Zečević, M., Willig, M., Singh Dhami, D., and Kersting, K. Pearl causal hierarchy on image data: Intricacies & challenges. arXiv e-prints, pp. arXiv-2212, 2022.
16

---

Diffusion Counterfactual Generation with Semantic Abduction
A. Background
A.1. Evaluating Counterfactuals
To formalise counterfactual soundness metrics, we use the notion of an image counterfactual function Fe(-), which generates counterfactuals as x Fx(x, pa. pa) = ff (x, pa), pa). Composition measures Le-reconstrrutton error by comparing observed images with counterfactuals under null interventions,
Cxmp pa) == L1 (x, (x. pa, pa)).
(23)
Reversibility assesses cycle-consistency via the L1 distance between the observed image and its cycled-back counterfactual,
Rev(x, pa, pa) == L1 (x, (FFo pa, pa); pa, pa)).
(24)
Effectiveness quantifies faithfulness to interventions. We use a metric (-) to compute the enror between the counterfactual parent pag and the output of an anti-causal parent predictor p(pa /x) sampled using the function Pac(x).
Eff pa, pa) == Lp(ap, Pay (Fo pa, pa))).
(25)
A.2. Causal Mediation Analysis
The general causal effect defined as
CE pa) == --F(x,, pa, pa),
(26)
where X is varied in order to compute the direct, indirect and total causal effects. Following (Pearl, 2001), we define the direct effect using
XDE Fx pa, pape), påDE = (pa) (pa)( U (pac).
(27)
where mediators remain fixed to their observed values, and only pag is modified. The indirect effect is computed
ĀDDE Fo pa, paIDE), paDD (pa))) {pa,}
(28)
where mediators are set according do(pa)) while pa, retains its observed value. The total causal effect combines both direct and indirect effects,
ĂTF Fx pa, pa).
(29)
We verify our computations using the fact that X + XIDE = XTE. For our purposes, this framework provides qualitative insights for debugging and explaining the model's 's causal behaviour.
17

---

Diffusion Counterfactual Generation with Semantic Abduction
B. Methods
B.1. Guided Counterfactual Prediction Step
Vx, logp(pa(x)) = Vxx log p(p/(p)) w(Vx p(x pa) Vx, log)(x;)) x w((ex,,, pa) (,,,,
(30)
B.2. Counterfactual Trajectory Alignment
Algorithm 1 "Contterfactual Trajectory Alignment
1: Input: Images Xx = u.... Xo = x] from h (x, Csem), step size 1), guidance scale w > 1, guidance token Ø.
2: Output: Optimised exogenous guidance tokens Øi.
xi u. 27
4: for T T..,. do
5:
e' (Caem: 27, O}
6:
x;_ t e (x+)
27 CTA Xt-1, x(-1)
8:
0;-1 27
9:
end for
10: Return Ø1.
18

---

Diffusion Counterfactual Generation with Semantic Abduction
C. Extended Related Work
Controllable Representation Learning. Our work contributes to integrating Pearlian causality (Pearl, 2009) into unsu- pervised disentangled representation learning (Higgins et al., 2017; Burgess et al., 2018; Kim & Mnih, 2018; Chen et al., 2018; Kumar et al., 2018: Peebles et al., 2020). These approaches aim to learn semantically meaningful, uncorrelated latent factors using modified VAEs (Kingma, 2013), which can aid with controllable generation. However, Locatello al. (2019; 2020) demonstrate that unsupervised disentanglement is impossible without inductive biases in both models and datasets. To address this, our mechanisms operate within an SCM, with control over the data generation process when possible. Recent work, builds diffusion models with controllable representations (Batzolis et al., 2023; Pandey et al., 2022; Yang et al., 2023b; Mittal et al., 2023; Zhang et al., 2022). Some works introduce semantics into diffusion by interpreting low-rank projections of intermediate images (Park et al., 2023; Haas et al., 2024; Wang et al., 2024), or by employing diffusion models as decoders within VAžS (Preechakul al., 2022: Pandey et al., 2022: Zhang et al., 2022; Batzolis et al., 2023; Abstreiter et al., 2021), where encoders capture controllable semanties. These approaches improve perceptual quality and identity preservation in image editing, inspiring our semantic mechanisms. Additional constraints on diffusion autoencoding frameworks (Hwa et al., 2024; Cho et al., 2023) further enhance interpretability and fairness. Explicit control, essential for structured counterfactual reasoning, can be attained via guidance using discriminative score functions, either amortised (Ho & Salimans, 2022; Karras et al., 2024: Dinh et al., 2023) or trained independently (Dhariwal & Nichol, 2021).
Counterfactuals Image Generation. Generalising Pearl's Causal Hierarchy (Pearl, 2009; Peters et al., 2017; Bareinboim et al., 2022) to high-dimensional data like images poses significant challenges (Zečević et al., 2022; Poinsot et al., 2024). Methods for generating image counterfactuals typically follow the Pearlian method
(Pawlowski et al., 2020; De Sousa Ribeiro al., 2023; Xia et al., 2023; Wu et al., 2024; Pan & Bareinboim, 2024; Dash et al., 2022), relying deep SCMs (DSCMs) where neural networks trained on observational data implement mechanisms. These DSCMs have used models such as normalizing flows (Papamakarios et al., 2021; Winkler et al., 2019), VAEs (Kingma, 2013; Pandey et al., 2022: Sohn et al., 2015). GANS (Goodfellow et al., 2014; 4; Mirza & Osindero, 2014), and HVAEs (Vahdat & Kautz, 2020; Child, 2020) to generate images. DSCMs have also reduced bias counterfactual image edits (Xia et al., 2024), perform data imputations (lbrahim et al., 2024), and several medical imaging scenarios (Ravi et al., 2019; Reinhold Rasal et al., 2022). Some approaches generate counterfactuals without explicitly performing abduction (Sauer & Geiger, 2021). For example, (Shen et al., 2022; Yang al., 2021) incorporate an SCM prior into the latent space of VAE, but these methods can be difficult to train and scale. Some approaches avoid explicit abduction (Sauer & Geiger, 2021), 1), while others embed an SCM prior into a VAE's latent space (Shen et al., 2022; Yang et al., 2021) which are difficult to train and scale. Methods focusing solely on interventional distributions (Kocaoglu et al., 2017: Rahman et al., 2024) cannot generate counterfactuals. Many studies loosely use the term "counterfactual" to describe struetured image perturbations aimed at explaining or interpreting model behaviour (Van Looveren & Klaise, 2021; Fang et al., 2024; Shen et al., 2024; Schut et al., 2021; Kladny et al., 2023; Taylor Melanson et al., 2024; Augustin et al., 2022; Atad al., 2024). Recent diffusion-based counterfactual approaches include (Sanchez & Tsaftaris, 2022), whose method diverges from Pearlian abduction and is not demonstrated on large parent sets with complex causal relationships. Potentially, our closest work is (Komanduri et al., 2024) which extends (Preechakul et al., 2022) who implicitly learn their SCM on latent variables given a causal graph in the style of Yang et al. (2021). They, however, use the aggregate diffusion posterior for spatial abduction, followed by classifier-free guidance; we instead present an independent DSCM mechanism for stability, scalability and identity presentation on complex, high-resolution datasets and explicit control over parent sets. We leave identifiability considerations for future work (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Khemakhem al., 2020a; Li et al., 2019; Sorrenson et al., 2020; Khemakhem et al., 2020b; Roeder al., 2021; Willetts & Paige, 2021).
et al., 2021;
Semantic Image Editing. Latent diffusion models (Rombach al., 2022; Podell et al., 2023; Saharia et al., 2022; Nichol et al., 2021) have become standard for creative text-driven image editing (Wang et al., 2024) and mage-to-image translation tasks (Meng et al., 2021; Yang et al., 2023a), often using structured analysis and manipulation of cross-attention maps (Hertz et al., 2022; Tang et al., 2022; Epstein et al., 2023; Tumanyan et al., 2023; Sanchez et al., 2022a; Tian et al., 2023). Some methods fine-tune entire models for small datasets (Ruiz et al., 2023; Wang et al., 2023b; Gal et al., 2022) or use masking to localise edits and avoid spurious correlations (Couairon et al., 2022; Pérez-García et al., 2023). We focus on test-time optimisations, using objectives that align inverse and generative trajectories to enhance identity preservation by tuning guidance tokens (Mokady et al., 2023; Wang et al., 2023a: Tang et al., 2024), for their relative efficiency and robustness without masking. SCMs have been applied to text-guided diffusion editing (Zečević et al., 2022; Song et al., 2024; Gu et al., 2023; Prabhu et al., 2023), but text control often lacks the precision for counterfactual reasoning in worl-world scenarios.
19

---

Diffusion Counterfactual Generation with Semantic Abduction
D. Architectures
D.I. Diffusion Mechanisms
We modify the architecture in DiffAE to be trained for conditioning and classifier-free guidance, which projections the conditions are added the timestep embedding. We also use EMA on model parameters every training step. Additionally, modify the encoder include mean and log-variance, which reparameterise during training with Equation (13). All images normalised between [-1, 1]. In the of Morpho-MNIST datasets, the digit class (d) is one-hot encoded. For CelebA and CelcbA-HQ, we use torchvision transforms processing. For EMBED, we use the preprocessing used by (Schueppert et al., 2024) to train their classifiers. We outline the architectures and training procedures of our semantic mechanisms in Table 4; spatial mechanisms are implemented without the semantic encoder components.
wotiooolniiiiioo (p=0.5) for pre-
Table Network architecture of our semantic mechanisms.
PARAMETER
NORPHO-MNIST SMORPHO-MNISS
CELEBA
CELEBA-HQ
EMBED
TRAINING SET
50000
50000
162770
24000
13207
VALIDATION SET
10000
10000
19867
3000
3300
TEST SET
10000
10000
19962
3000
5503
RESOLUTION
28 X 1
28 x
64 x
64
192 x 192
BATCH SIZE
128
128
128
128
32
EPOCHS
1000
1000
6000
6000
1000
BASE CHANNELS
64
64
96
ATTENTION RESOLUTION
[16]
[16]
CHANNEL MULTIPLIERS
[1,4,8]
[1.4.8]
[1,2,4.8]
[1,2,4.8]
[1.1,2,2,4.4]
RESNET BLOCKS
2
2
2
RESNET DROPOUT
0.1
0.1
0.1
SEM. ENCODER BASE CH.
64
64
32
32
96
SEM. ENC. ATTN. RESOLUTION
[16]
[16]
SEM. ENC. CH. MULT.
[1,2,4.8]
[1,2,4.8]
[1,2,4.8,8]
[1,2,4,8,8]
[1,1,2,2,4.4.44]
SEM. ENC. RESNET BLOCKS
2
2
2
SEM. ENC. RESNET DROPOUT
0.1
0.1
0.1
0.1
0.1
8
32
32
512
NUM. pa VARIABLES
7
SIZE
13
13
7
7
4
NOISE SCHEDULER
LINEAR
LINEAR
LINEAR
LINEAR
COSINE
LEARNING RATE
lc-4
OPTIMISER
ADAM (NO WEIGHT DECAY)
EMA DECAY FACTOR
0.9999
TRAINING
1000
DIFFUSION LOSS
MSE WITH NOISE PREDICTION
D.2. Effectiveness Classifiers
Morpho-MNIST. We evaluate the effectiveness of the digit class using the following simple CNN trained for 100 epochs with learning rate le and batch size 256, achieving zs 99.5% accuracy:
from torch import
class Classifier (nn, :odule):
super () initttt_()))) self.model nn. Sequential ( nn. nn.Flatten(),
Linear (28 28, 128),
nn.
20

---

Diffusion Counterfactual Generation with Semantic Abduction
m..innea (128, 64), m.ReLU
mn.iinear (64, 10),
def (self, x) return self. mode.oode@ (x)
CelebA-HQ. We evaluate the effectiveness of Eyeglasses and Smiling using a classifier with a ResNet backbone:
from torchvision models looeles import resnet50
from torch import
class Classifier (nn. Module):
def _in_i_itt_s__ (self) super (). in_ini))_)
self.model resnet50 (wei weeeightssseeeeett__eeeeeee whtt. MAG self self.model.fc nn Sequentiii
mn. Lm..inner www.:rssee........mmmmmmmmmmm www.mmooe.....ooooom..mmoomoo.. ff 1024), nn.ReLU(), nn.Dropout (0.25), Linein (1024, 1).
def forward (self, x) return self.model (x)
We train this classifier for 100 epochs with batch size of 128 using the Adam (Kingma, 2014) optimiser, with starting learning rate of le - 3, B1 = 0.9, 32 = 0.999 and weight decay 0.01. To improve regularisation we use transfforotttaa Randomtor zontalFlip 0)) for preprocessing. address the imbalance in Eyeglasses and Smiling, we also use weighted random sampler with replacement. stabilise training, we use EMA on the model parameters with decay rate 0.999, Both classifiers achieve 97% accuracy.
EMBED. We downscale the multi-label classifier used in (Schueppert et al., 2024) for skin and circular marker detection for 192 192 mammograms, achieving ROC-AUC of .91 the test set.
21

---

Diffusion Counterfactual Generation with Semantic Abduction
E. Morpho-MNIST
E.1. Dataset Details
We construct the following SCM using the Morpho-MNIST library (Castro et al., 2019), to extend the casual modelling
scenarios in (Pawlowski al., 2020; De Sousa Ribeiro et al., 2023). Mechanisms are defined follows:
d fa(ea) Cd.
cd MNIST
(31)
= fa,d, Ea) = -27 d 3.3.6.
6, (0,
(32)
t = fe(E)) = -0.5 Et,
6, T(10,5)
(33)
i = Si(t, 6;) = 191 -0.5 6, + 2t -). 5),
6, ~ (0,
(34)
ix 6x) (i, d, Se,,(1, d, Set, (8, d, 6x)));
6x MNIST
(35)
where Set, (-) and (-), (.) functions provided by Morpho-MNIST that change the intensity i, thickness and slant of image in the original MNIST dataset 6x We use the true mechanisms fi(-). fe()), f.(-) and fa(-) to implement
our DSCM (Figure 2a), with the image generating mechanism implemented using our diffusion-based formulations. When
using these mechanisms to generate our dataset, we ensure that all digits have full support of the slant:
(d, c.):
i 0, where (0r2(0.2) and c, NN00 1),
(36)
Est
Otherwise, where ~ MNIST
Our dataset follows the original MNIST dataset splits.
Thickness
Slant
Width
Intensity
2
2
3
2
6
G 6
G
6
7
/
Z
120
200
Figure? 7. Examples from Morpho-MNID MNIST
Figure Features distributions Morpho-MNIST
E.2. Extra Results
Table Soundness of Morpho-MNIST image counterfactuals under do(t) and do(i) using DSCMs modelling the SCM Appendix E. Effectiveness for digit class (d) is measured using accuracy (Acc) from a pre-trained classifier and mean absolute pereentage crror (MAPE) for slant (s), thickness (t) and intensity (t). Counterfactuals normalised to [0, measure composition (COMP.) and reversibility (REV.). All metrics are scaled by 10 except for MAPE (s), which is scaled by x x and Ace (d), which remains unscaled.
THICKNESS INTERVENTION (do(t))
INTENSITY INTERVENTION (do(E)
EERECTIVENESS
REV.
ERFECTIVENESS
REV.
MECHANISM
MAPE (8)
MAPE (1)
MAPE (+)
Acc (c)
L,
MAPE (8)
MAPE (6)
MAPE (+)
Ace (c)
L,
VAE (Pawlowski al. 2020) HVAB Sousa VCI (Wi at, 2024)
4.48
6.78
3.58
97.75
4.26
8.08
7.55
4.83
97.85
4.31
2023)
3.05
0.676
1.82
95.61
0.678
3.92
01471
1.50
94.92
0.580
3.08
0.626
0.913
02.97
2.10
6.99
2.61
3.97
82.52
3.75
SPATIAL:
0.506
1.75
1.34
1.35
[2 1.5,
0.1)
1.96
0.355
1.14
99.61
2.35
3.46
0.496
3.07
97.17
2.28
0.1)
1.95
01.296
1.16
90.80
4.78
3.04
0.625
2.10
95.84
4.5, Pe 0.1]
1.95
0.327
1.26
199.98
5.70
2.43
0.936
1.91
98.63
5.70
1.5, 0.5]
2.29
01.778
2.78
08.93
1.48
2.91
0.637
2.92
07.75
1.07
0.5)
( 4.5,
0.5]
2.03
0.342
2.15
50.500
3.37
2.56
1.02
1.59
30.12
4.78
SAMANTIC:
4.17
0.718
3.18
1.72
5.90
3.00
87.50
2.23
[- 1.5, pa 0.1)
2.38
1.29
2.30
08.54
1.86
3.38
1.40
2.12
96.39
1.70
[ 0.1)
2.16
1.01
1.56
00.51
3.52
3.41
1.45
2.11
07.75
3.27
4.5, pa 0.1)
0.757
99.61
22

---

Diffusion Counterfactual Generation with Semantic Abduction
Table Soundness Morpho-MNNS, image counterfactuals generated under do(d) from DiffSCM modelling the relationship d x, in which the digit class (d) is the only parent of the image (x), with data generated from the true SCM in Appendix E. Effectiveness (BFF.) is measured by the accuracy (Acc) of pre- pre-trained classifier. Counterfactuals are normalised to [o, 1] to measure composition (COMP.) and reversibility (REV.).
COMP:
BFF,
REV.
MECHANISM
L, J (x10
Acc f
L
(x10
DIFFSCM (ORIGINAL)
0.410
17.02
1.31
DIFFSCM (OURS) {w 10)
0.605
73.83
3.92
DIFFSCM (OURS) {w 20)
0.945
96.00
3.94
DIFESCM (OURS) ww 30)
1.22
98.73
4.42
Table Soundness of Morpho-MNIST image counterfactuals with mechanisms conditioned on intensity (i) and thickness (f) from the dataset in Appendix E. Effectiveness for thickness (t) and intensity (i) is measured using mean absolute percentage error (MAPE). Here, we use simulated interventions do(i) and dt do i t). Counterfactuals are normalised to [0, 1] to measure composition (COMP.) and reversibility (REv.). All metrics scaled by x10
THICKNESS INTERVENTION (do(e)
INTENSITY INTERVENTION (do(i)
NULL INT. (do(pa))
EPPECTIVENESS
REV.
EFFECTIVENESS
REV.
COMP.
MECHANISM
MAPE (t)
MAPE J
I
MAPE (f)
MAPE 4
In
In
VCI (Wu et al, 2024)
5.09
0.949
1.17
9.58
5.89
2.74
0.665
SPATIAL: J 1.5, Pa10.1)
3.45
0.631
1.595
5.52
1.07
1.705
2.44
0
2.075
4.42
0.737
2.49
0.39005
( 0.10.1
0.357
0.615
E.3. Improving DiffSCM
The results in Table demonstrate that DiffSCM's original selection of guidance scale, based their proposed counterfactual latent divergence metric, substantially limits effectiveness. By exploring higher guidance scales beyond those originally reported and training the unconditional diffusion model for 500K steps, compared to the 30K used in the original work, we are able to improve effectiveness significantly. However, this gain in effectiveness comes at the expense of identity preservation, as reflected by the increasing composition and reversibility errors. As w grows, the influence of the unconditional diffusion model diminishes relative to the anti-causal classifier, meaning counterfactuals increasingly satisfy the desired class intervention while deviating from the spatial exogenous noise characteristics of the original observation. Comparing ablations 20 and 30, where effectiveness becomes comparable to mechanisms in Table DifISCM's composition is similar our spatial mechanisms but does not outperform reversibility remains substantially worse compared to our methods. We hypothesise that learning amortised anti-causal generative classifier for guidance, as in our method, is more advantageous for reversibility than using a separately trained classifier.
find that semantic mechanisms, while
23

---

Diffusion Counterfactual Generation with Semantic Abduction
E.4. Morpho-MNIST Counterfactuals
d-d
ded
de2
Cot
deb
del
dos
THE
2-0
P+3
and
ded
d+5
3-2
ded
del
10:0
3
4
7
9
0
2
3
4
5
6
7
9
27
<nd
del
dee
des
vod
4*3
des
a=?
des
and
des ved
de
VHE
ax2 del
des
10:00
2-
20
2 3
4
5
7
8
9
2
2
2
2
3
O
a
3
4 5 6
7
8
33 9
3
3
3
3
4
2
2
E<
3
3
X-3
4-1 2-2
4-3
9-5
6-1
T
d-t
1-3
(+0
1-0
X40
P-1
1-2
d-3
2-4
d4
4
9+:00
1-66
6-1
4 0
2
3
4
2
6
7
8
9
4
4
4
5
O
a
3
4
5 6
7
8
9
5
5
5
5 5
4
5
55
6
5
5
5
7
2-1
2-2 d-2
a-5
a-2
2-E
2-3
X-0
d-2
d-5
a-2
6-9
1+:00
1-64
3
4
5
6
7
9
6
O
7
0
2
3
4
S
6
7
8
8
7
7
7
7 7
7
3
30
7
2
7
X-3 #-1
2-2
6-3
9-5
9-7
2-6
X-0
2-1
1-2
4-3
2-4
d-
9-7
4-9
1+:00
1:44
8
O
2
3
4
5
6
7
8
9
8
8
8
8 8
9 6
2
3
4
S
6
7
8
9
9
9
9
9
9
7 9
9
9
9
Figure 9. MorS (28 x 28) counterfactuals generated using an amortised, anti-causally guided semantic mechanism (pe 0.1,w- 1.5) based on the DSCM shown in Figure 2a. Interventions are shown above the top row and the bottom row visualises total causal effects (red: increase, blue: decrease), refer to (Appendix A.2) for details.
V- 4=1
2=2 de3 god
an5
def
4e7
ded Yet
1-1
1-2
d+3
THE
4x7
des
des
10:00 Joft te
O
2
3
4
7
9
O
O
2
3
4
5
7
9
(
(1
3
4/
N
2-3 9-1
d-2
4-3 2-4
8-5
9 7
2-c (-)
2-0 2-3 2-2 4-3 2-A
d-1
9-2
6-9
16:00
1-66
1+1
20
/
2
3
4
Z
2
2
2
2
C
3
4
5
6
7
8
9
3
3
3
3
9
R 3
2
77
3
3
4-7
des
4
O
2
3
4
5
6
7
8
9
4
4
4
4
4
4
50
23
4
5
6
7
8
9
5
5
5
5
5
5
2
5Z
K
5
6
6
2
3
4
5 6 7
8
9
6
6
7 O
2
3
4
5
6
7
8
9
7
7
7
7
7
7
7
D7
7
2
8
OOR
2
3
4
S
6
7
8
9
8
8
P
8
8
9 0
2
3
4
S
6
7
8
9
9
9
9
9
9
9
0
29
40
Figure 10. MorPho-MNIST (28 28) counterfactuals generated using an amortised, anti-causally guided spatial mechanism (pe 0.1, w = 1.5) based on the DSCM shown in Figure 2a. Interventions are shown above the top row and the bottom row visualises total causal effects (red: increase, blue: decrease), refer to (Appendix A.2) for details.
24

---

Diffusion Counterfactual Generation with Semantic Abduction
F. Colourised Morpho-MNIST
F.I Dataset Details
We construct a colourised variant of the dataset in Appendix E. by using the Morpho-MNIST library to implement the
following structural causal model, which extends the causal modelling scenario in (Monteiro et al., 2023):
Sa(Ea):
6 MNIST
(37)
1 = S((Ee) = 0.5 + E2:
6, 5(10,5)
(38)
= co) = 277 + 6
(00,1)
(39)
= fn(d, 0.05 + 0.05 C₁
6, ~ (0,11)
(40)
x = fx 8, Ex) = Sen, d, Sete(t, d, Set, (s, d, 6x)));
6x ~ MNIST,
(41)
where (-) and (-t, are functions in Morpho-MNIST that the thickness and slant of an image, and we implement (-) to set its hue h. We use the true mechanisms (-).), fa()). fa(+) and fa(+) to implement DSCM (Figure 13a), with the image generating mechanism implemented using our diffusion-based formulations. When using these mechanisms to generate our dataset, we ensure that all digits have full support of the slant and hue:
fa Es),
if b 0, where b Bern(0.2) and E, (0,, Otherwise, where C2 ~ MNIST,
(42)
fu ek),
if b where b (0r5(0.5) and 6₁ NN 1); Otherwise, where E ~ Ld(0 1].
h
(43)
Ch:
Our dataset follows the original MNIST dataset splits.
Thickness
Slant
Hue
2
2
2
2
2
3
DUR
J
S
S
S
5
S
6 G
6
G
g
8
R
A1
Q
9
9
-25"
25'
Figure 11. Examples from colour Morpho-MNIS'
Figure 12. Features distributions in colour MorPhO-MNIST
F.2. Qualitative Results
Cbr.
Comp
CF
Rrc
9
3
4
5
7
8
9
9
9
9
9
9
9
9
4
7
3 3 3 3
9
9
9
9
9
9
9
9
9
9
9
9
9
9
f
to
Ø 6 4
/
4
666 6
0
5
3
S
8
9
9 9 9 9
9
9
1221 2
2R
3
(a) CMorpho-MNIST DSCM
(b) Causal Mediation Analysis
(c) Counterfactual Soundness
Figure 13. Colourised MorPhO-MNIST (28 x 28) counterfactuals generated using an amortised, anti-causally guided semantic mechanism (pe 0.1, w 1.5). (a) depicts the DSCM: is hue, d is digit class, s slant, t is thickness and the image. (b) depicts image counterfactuals and causal mediation analysis (Appendix A.2): interventions are shown above the top row and the bottom row visualiscs total causal effects (red: increase, blue: decrease). (c) illustrates counterfactual soundness (Obs: Observation, Comp: Composition, Cf: Counterfactual, Rev: Reversibility).
25

---

Diffusion Counterfactual Generation with Semantic Abduction
G. CelebA
Dostridee
Composten
Countrtacter
Penavel
Orppostenn
Coumatticta
Renasel
Dbsovattrr
Compesttan Cunterrattal
Reverall
Stsevvtter
Compestion
Cunterratal
Rewerul
(a) Guided spatial mechanism (left) with dynamic abduction (right). (b) Guided semantic mech. (left) with dynamic abduction (right).
Figure 14. CelebA (64 x 64) counterfactuals generated using our amortised, anti-causally guided mechanisms the DSCM Figure 3a.
H. CelebA-HQ
H.1. Additional Results
240-
0012 -11
021 0 201211 0013
000
2o0 31
0
(a) Guided spatial mech. with ps 0.1 (left) and pe = 0.5 (right).
(b) Guided sem mech. with ps = 0.1 (left) and pa = 0.5 (right).
Figure CelebA-HQ (64 64) counterfactuals using our amortised, anti-causally guided mechanisms DSCM Figure with
2. Notice that identity preservation improves in each figure from left to right.
Semantic Dynamic Abd. Pg 0
Semantic Dynamic Abd.. Pa 0.
Spatial Pa
Semantic Pa Pa= 0.1
Semantic Pa
Observation
dats O)
im
do1g = 11
Figure CelebA-HQ (64 x 64) counterfactuals generated using amortised, anti-causally guided mechanisms. Notice that identity preservation improves as we use increasingly complex semantic abduction procedures and increase the value of pa-
26

---

Diffusion Counterfactual Generation with Semantic Abduction
H.2. Baselines
We refer readers Figures and 2g (Monteiro al., 2023) for empirical comparison of face image counterfactuals under do(s) and do(g) interventions. Their results exhibit notable background changes and reduced image fidelity, issues that our dynamic abduction method largely corrects. Additionally, their Table 6 reports effectiveness levels comparable to ours, displaying similar trade-off between effectiveness and identity preservation, in this case depending on the number of hierarchical latent variables abducted. This model forms the backbone of the HVAE used by (De Sousa Ribeiro et al., 2023),
and was subsequently adopted by (Melistas et al., 2024) for causal modelling face images on CelebA (Liu et al., 2015). To implement the computational graph in Figure 3a for CelebA-HQ, HQ, we train the model provided by (Melistas et al., 2024).
However, we observe that without extensive hyperparameter tuning, particularly when handling many confounders, and without post-hoe counterfactual fine-tuning, the model exhibits very poor effectiveness. VCI (Wu et al., 2024) introduces counterfactual regularisation term and adversarial loss into the autoencoding objective. They extend the models from (De Sousa Ribeiro et al., 2023; Monteiro et al., 2023) to generate face counterfactuals on the CelebA -HQ subset of images taken directly from CelebA. As such, VCI applies center-cropping to all images following the preprocessing procedure of DEAR (Shen et al., 2022). In contrast, when we train VCI directly on CelebA-HQ and without center-cropping, to ensure compatibility with our pre-trained classifiers, we observe a significant drop in counterfactual fidelity. Table provides metrics for the failure cases of the baselines we have discussed above. We acknowledge that with additional hyperparameter tuning and alternative evaluation setups, i.e. using random-cropping, these models may perform better. However, in comparison, our diffusion-based mechanisms with sufficient model parameters achieve strong performance without the need losses or extensive hyperparameter tuning. We anticipate that future work
for center-cropping, addversaiial/counteffaee leveraging high-resolution images, where diffusion models are known to excel, and adopting an LDM-style architecture will enable scaling to even higher resolutions and further improve identity preservation.
Table 8. Soundness of CelebA-HO image counterfactuals generated using VCI (Wu et al., 2024) without center center- cropping in preprocessing HVAE (Melistas et al., 2024) without counterfactual fine-tuning. Effectiveness measured using the Fl-scores from pre-trained classifiers for eyeglasses (g) and smiling (s).
EYEGLASSES INTERVENTION (do(g))
SMILING INTERVENTION (do(s))
MECHANISM
Fl (s)
FI (g)
FI (+) f
Fl (g)
VCI (Wu et al. 2024)
97.84
3.30
33.81
99.58
HVAE (Melistas 2024)
90.05
65.31
75.33
95.82
27

---

Diffusion Counterfactual Generation with Semantic Abduction
I. EMBED
Obscrvation
Conterfocuaa
Counterfacuall
Remove Cirdes
Remove Circess
Remove
Remove Circess
Remove Cioees
Remoove
Figure 17. EMBED (192 x 192) counterfactuals using an amortised, anti- causall guided semantic mechanism with (ps = 0.1, = 1.2) for circular marker removal. find that larger w improves effectiveness whilst compromising illumination and breast density.
Counteficttal
fecital
Remove Triangles
Triangles
Ramovee Thaangies
Ramove Trangles
Remove Trienples
Remove Thangies
Figure 18. EMBED (192 x 192) counterfactuals using an amortised, anti-causally guided semantic mechanism with (pe = 0.1, w = 1.2) for triangular skin marker removal. We find that larger w improves effectiveness whilst compromising illumination and breast density.
Obsorvation
Semantic
Dynamic
Observation
Semantic
Dynamic
Remove Triingles les
Remove Cirdes
Figure 19. EMBED (192 x 192) counterfactuals using an amortised, anti-causally guided semantic mechanism with (pe = 0.1, = 1.2) with dynamic abduction (1 0.001) for triangular and circular marker removal. Counterfactual inference takes 5 mins per image. In these cases, dynamic abduction improves the sharpness and brightness of our images.
28


Reviews:
[{"review_id": "tsGkPpifbH", "reviewer": "Reviewer_v9YA", "summary": "The paper “Diffusion Counterfactual Generation with Semantic Abduction” explores the use of diffusion models for counterfactual image generation, a task that requires maintaining identity, visual fidelity, and causal consistency. The authors argue that while diffusion models have achieved state-of-the-art synthesis quality, they lack structured semantic control for counterfactual reasoning. To address this, the paper introduces a new framework that integrates diffusion models with structural causal models (SCMs), leveraging semantic abduction to improve identity preservation in generated counterfactuals. The proposed approach introduces mechanisms such as spatial, semantic, and dynamic abduction to refine causal interventions while maintaining perceptual consistency. The study evaluates its methods across synthetic and real-world datasets, including face images and medical imagery, demonstrating improvements over existing generative approaches like VAEs and GANs. However, challenges such as computational efficiency, dataset limitations, and reliance on human-defined causal graphs are acknowledged. The work presents an important step towards using generative models for more controlled, interpretable, and robust counterfactual reasoning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and for recognising the significance of our work. We're encouraged by reviewers acknowledging our framework as \"an important step towards using generative models for more controlled, interpretable, and robust counterfactual reasoning\". We also appreciate the positive remarks regarding our integration of diffusion models with counterfactual inference, the novelty of semantic abduction, and the strength of our experimental design and ablation studies. Additionally, we want to highlight further the novelty, generality, and effectiveness of our dynamic abduction framework, which also offers opportunities to integrate alternative test-time optimisations from LDM-based image editing into structured counterfactual inference beyond CTA.\n\n> **\"It lacks human evaluation to assess the realism and plausibility of generated counterfactuals.\"**\n\nWe agree that incorporating human evaluation to assess the plausibility and realism of generated counterfactuals would add value to our analysis. In particular, expert assessment by radiologists for EMBED and experienced annotators for CelebAHQ could provide additional insights beyond our chosen perceptual metrics. However, such user studies are time-consuming and challenging to set up and not feasible as part of the rebuttal. We acknowledge this limitation and plan to include it as a key direction for future work in our discussion section.\n\n> **\"However, it overstates the ease of defining causal structures, as real-world causal relationships are often ambiguous.\"**\n\nWe recognise that our manuscript may have understated the challenges involved in specifying causal structures, especially in real-world applications. We will expand the limitations section to acknowledge the ambiguity in causal discovery, the assumptions required to define SCMs, and the fact that our results are contingent on the accuracy of our SCM. Note that if assumptions in an SCM are shown to be incorrect, the transparency of our framework means that developers can redesign their SCM, and either retrain our mechanisms under new assumptions or perform more faithful simulated interventions.\n\nFor MorphoMNIST, we leverage the known data-generating process within our DSCM directly for the parents of the image. In the CelebAHQ experiments, we adopt a simplified SCM involving attributes such as smiling and eyeglasses with mild label noise, which can be reliably predicted from observed images using pseudo-oracles, consistent with prior work (see Appendix F, [1]). For the EMBED dataset, our SCM is derived from clinical insights presented in [2] and validated by radiologists.\nWe also note that discovering causal structure is an active and open research area [3,4], and future work could focus on jointly learning causal structure and counterfactual generation, taking inspiration from the theoretical discussion in Appendix C in [5].\n\n> **\"Generalization to more complex, diverse datasets is not fully explored. More testing on real-world counterfactual scenarios would strengthen its conclusions.\"**\n\nWe appreciate the reviewer’s concern regarding generalisability. While we agree that broader testing is important, we would like to highlight that the EMBED artefact removal task addresses a clinically important and underexplored problem in a real-world medical imaging scenario. The EMBED variant proposed by [2] contains many patients with significant variations in tissue density and anatomical structure. We believe this makes EMBED a suitably complex testbed for evaluating the robustness of our counterfactual generation methods, directly addressing a real-world challenge of breast cancer detection in screening mammography. Compared to many existing works, we felt that including this real-world medical imaging application sets our work apart from the many ML papers that primarily test on readily available benchmarks.\n\nFurthermore, we have obtained improved results on EMBED since the initial submission, achieving over 90% accuracy in triangular artefact removal. These updated findings will be included in the paper's final version to support our semantic mechanisms' efficacy further.\n\n[1] https://arxiv.org/abs/2303.01274\n\n[2] https://arxiv.org/abs/2410.03809\n\n[3] https://arxiv.org/abs/2004.08697\n\n[4] https://arxiv.org/abs/2307.05704\n\n[5] https://arxiv.org/abs/2404.17735", "claims_and_evidence": "The paper’s core claims—that diffusion models generate high-quality counterfactuals and semantic abduction improves identity preservation—are well-supported through experiments and ablation studies. However, it overstates the ease of defining causal structures, as real-world causal relationships are often ambiguous. Additionally, while results on faces and medical images are promising, generalization to more complex, diverse datasets is not fully explored. More testing on real-world counterfactual scenarios would strengthen its conclusions.", "methods_and_evaluation_criteria": "The proposed method—integrating diffusion models with causal reasoning—is a logical and promising approach for counterfactual image generation, as diffusion models excel at high-quality synthesis while causal inference ensures meaningful interventions. The use of semantic abduction to refine counterfactual edits also makes sense, as it helps preserve identity and realism.\n\nHowever, the evaluation criteria have some limitations. While the paper uses perceptual similarity metrics, identity preservation scores, and qualitative comparisons, it lacks human evaluation to assess the realism and plausibility of generated counterfactuals. Additionally, the datasets (faces and medical images), though relevant, are relatively small and lack diversity, making it unclear how well the method generalizes to more complex, real-world counterfactual reasoning tasks. More robust benchmarking on larger and varied datasets would improve the evaluation.", "theoretical_claims": "The paper does not contain formal, theorem-based proofs that require rigorous correctness checks. Instead, it presents a theoretical framework for integrating diffusion models with causal inference, specifically through semantic abduction.", "experimental_designs_or_analyses": "Yes, I reviewed the experimental design and analysis validity, the experimental design is well-structured and supports key claims, but the lack of large-scale, diverse datasets and human validation weakens its generalizability. More real-world testing and sensitivity analysis on causal assumptions would improve robustness.", "relation_to_broader_scientific_literature": "This paper builds on diffusion models for high-quality image synthesis (Ho et al., 2020) and extends them to counterfactual generation, an area previously dominated by GANs and VAEs (Goyal et al., 2020). Unlike prior methods, it integrates Structural Causal Models (SCMs) to enforce causal consistency, bridging generative modeling with causal inference. The introduction of semantic abduction refines counterfactual edits, improving identity preservation and realism beyond standard interventions.", "essential_references_not_discussed": "The related works are well discussed", "other_strengths_and_weaknesses": "Please refer to previous section", "other_comments_or_suggestions": "Please refer to previous section", "questions_for_authors": "Please refer to previous section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yB2fMXlEYh", "reviewer": "Reviewer_vbtG", "summary": "This paper studies the image counterfactual generation problem using diffusion models. Specifically, the authors propose a suite of deep causal mechanisms, spatial mechanism, semantic mechanism, and anti-causal mechanism, for tractable counterfactual generation with respect to composition, reversibility, and effectiveness. The authors also propose a dynamic classifier-free training approach to study the trade-off between composition and effectiveness of generated counterfactuals. Experiments on three datasets (one synthetic and two real-world) are performed to evaluate the soundness of generated image counterfactuals.\n\n\n## Update After Rebuttal\nI appreciate the authors taking the time to address my questions and concerns. I believe this work has some limitations, however, after the rebuttal period, I believe the authors did a good job responding to all the reviewers' concerns. I will keep my score at 3 and lean towards **acceptance** of this paper. However, it would be good if the authors made their contribution more clear in the writing and distinguish it clearly from different approaches (DCM, CausalDiffAE, DiffSCM). Since there are only a few diffusion counterfactual baselines and the authors have included some results from DiffSCM (in response to Reviewer kmep), I am satisfied with comparisons. Ideally there would be comparisons with more diffusion counterfactual baselines. I will note that DiffSCM is not completely related to the paradigm proposed here. Although DiffSCM proposes a general formulation for arbitrary causal graphs, it only applies to classifier-based guidance (e.g., image and its label) in practice. Therefore, I do not think this baseline is all that much related to methods such as this paper, DCM, and CausalDiffAE which explicitly focus on modeling the causal structure.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback and are pleased that the paper was found to be “written exceptionally well with clear intuitions” and that the experiments “are extensive and show that the proposed abduction mechanisms are quite effective”. Below we address weaknesses and questions:\n\n> **“Although the comprehensive formulation is important, the main contribution is not clear”...**\n\nConcretely, our work distinguishes itself from existing approaches by:\n- Spatial Abduction: We use causally conditioned DDIM inversion rather than the unconditional forward process [1,2,3], such that spatial noise can help preserve image structure [4]. We go beyond DiffSCM, by incorporating multiple causal conditions (discrete or continuous), and DCM, by demonstrating our mechanisms on complex vision datasets. Also, we adopt a new SCM for MorphoMNIST, including a challenging relationship between slant and digit.\n- Semantic Abduction: Our semantic mechanisms condition diffusion models on semantic encoding and causal conditions from the parent’s SCM, whereas CausalDiffAE [2] incorporates [5] into diffusion models, widely acknowledged to be sensitive to a large number of parents and susceptible to unstable training [6,7], diminishing the benefits of the stable training objective of diffusion models. DiffAE does not provide a method for amortised guidance. Our analysis goes significantly beyond CausalDiffAE and DiffAE, exploring how our amortised guidance procedure trades off effectiveness for identity preservation.\n- Dynamic Abduction: We introduce a novel, general, dynamic semantic abduction framework, which we implement via CTA, that learns image-specific guidance tokens during the abduction step in counterfactual inference. These tokens enhance the retention of attributes that cannot be trivially measured and included as causal conditions via the SCM, such as background, hairstyle, and skin colour (Fig 3c, App H - Fig 16).\n\nOur formulation provides a unifying perspective between image editing using pre-trained diffusion models and counterfactual image generation methods. As such, we set the stage for future work on model identifiability and incorporating techniques from LDM-based methods to improve dynamic abduction, which has been a primarily empirical field. Additionally, by carefully considering which variables should be regarded as exogenous or observed within our formalism, we can evaluate counterfactuals via metrics grounded by intuitive soundness axioms and more formally explain tradeoffs between intervention-faithfulness (effectiveness) and identity preservation (composition) when using diffusion models for counterfactual inference or image editing.\n\n> **Difference between their causal interpretation of classifier-free guidance in Eq (16) and the one proposed by CausalDiffAE**\n\nWe condition on $c_{sem} = (z_{sem}, pa)$, whereas they condition solely on $z_{sem}$ which also encodes the parent SCM in the style of [5].\n\n> **What is the motivation behind learning the ϕ token for dynamic classifier-free guidance?**\n\nAs stated in sec 3.3, we train our guided mechanisms in the standard way [8]. We only use CTA to perform dynamic semantic abduction during counterfactual inference. Given that the CTA optimisation is performed w.r.t composition, the guidance tokens can learn additional semantic information not captured by $z_{sem}$ or $pa$, thereby improving the preservation of hair colour, skin tone and backgrounds in counterfactuals (Fig 3c, App H - Fig 16).\n\n> **How does this framework translate to text-to-image diffusion models?**\n\nOur general dynamic abduction procedure (eq 21-22) is analogous to many test-time optimisations for LDM-based image editing [4, 9, 10, 11], with the fundamental difference that fine-grained causal control can be challenging via text conditioning alone [12]. For an LDM, dynamic abduction using the spatial mechanism with amortised guidance (eq 19) can be formulated trivially within our framework. Natural extensions of our work involve replacing CTA with the test-time optimisation techniques cited above. Further, we demonstrate that shorter DDIM strides with single-step guidance token updates at each timestep improve identity preservation over semantic abduction. In contrast, many LDM-based methods opt for longer DDIM strides with computationally expensive multi-step guidance token optimisations. \n\n[1] https://arxiv.org/abs/2210.11841\n\n[2] https://arxiv.org/abs/2202.10166\n\n[3] https://arxiv.org/abs/2404.17735\n\n[4] https://arxiv.org/abs/2211.09794\n\n[5] https://arxiv.org/abs/2004.08697\n\n[6] https://ieeexplore.ieee.org/document/10021114\n\n[7] https://arxiv.org/abs/2411.19556\n\n[8] https://arxiv.org/abs/2207.12598\n\n[9] https://arxiv.org/abs/2309.15664\n\n[10] https://arxiv.org/abs/2405.01496v1\n\n[11] https://arxiv.org/abs/2403.02981\n\n[12] https://arxiv.org/abs/2212.12570\n\n[13] https://arxiv.org/abs/2410.12730", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, I checked the soundness of the experimental design and analysis. Specifically, the use of the theoretically grounded compositionality, reversibility, and effectiveness metrics have been shown to be good axiomatic metrics to evaluate the soundness of generated counterfactuals. Furthermore, I checked the visual quality of generated counterfactuals for all datasets. Overall, the evaluation is sound, extensive, and holistic.", "relation_to_broader_scientific_literature": "This paper provides a general framework and formalization for counterfactual generation using diffusion models. Specifically, the paper extends the general structure of Pawlowski et al and Ribeiro et al to provide a suite of deep causal mechanisms for tractable counterfactual generation. The causal mechanisms serve as formal generalizations of previous diffusion counterfactual methods, namely DCM (Chao et al), CausalDiffAE (Komanduri et al), and DiffSCM (Sanchez et al). \n\n\nPawlowski et al. Deep Structural Causal Models for Tractable Counterfactual Inference. NeurIPS 2020.\n\nRibeiro et al. High Fidelity Image Counterfactuals with Probabilistic Causal Models. ICML 2023.\n\nChao et al. Modeling Causal Mechanisms with Diffusion Models for Interventional and Counterfactual Queries. TMLR 2024.\n\nKomanduri et al. Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models. ECAI 2024.\n\nSanchez et al. Diffusion Causal Models for Counterfactual Estimation. CLeaR 2022.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n- The paper is written exceptionally well with clear intuitions and formulations contextualizing diffusion-based counterfactual generation with respect to composition, reversibility, and effectiveness, which are fundamental notions in counterfactual inference.\n- The experiments are extensive and show that the proposed abduction mechanisms are quite effective in counterfactual generation on several different counterfactual soundness metrics. Furthermore, the authors experiment on facial image data and a breast cancer image dataset, which further underscores the application of the method in real-world domains.\n\n## Weaknesses\n\n- The proposed mechanisms seem to be generalizations of existing methods. For instance, the spatial mechanism is essentially a generalized version of DCM (Chao et al.), the semantic mechanism is a generalized version of DiffAE (Preechakul et al.) and CausalDiffAE (Komanduri et al.), and the anti-causal mechanism is a generalization of DiffSCM (Sanchez et al.). Although the comprehensive formulation is important, the main contribution is not clear.\n- The authors take a causal interpretation of classifier-free guidance to show that the masking probability of the semantic conditioning acts as a trade-off between composition and effectiveness. However, Komanduri et al also take a very similar causal interpretation of classifier-free guidance. There does not seem to be a substantial difference between the two interpretations.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Could the authors provide clarifications on the differences between the proposed mechanisms and related work as pointed out in the weakness section?\n- Could the authors explain the difference between their causal interpretation of classifier-free guidance in Eq (16) and the one proposed by CausalDiffAE (Komanduri et al)?\n- What is the motivation behind learning the $\\phi$ token for dynamic classifier free guidance? Traditionally, one would just use a zero-mask an jointly train a conditional and unconditional diffusion model.\n- How does this framework translate to text-to-image diffusion models? For existing pretrained models, such as Stable Diffusion, it can be unrealistic to retrain them using this sort of paradigm. How would one perform high-fidelity counterfactual image generation using these pretrained models?\n\nKomanduri et al. Causal Diffusion Autoencoders: Toward Counterfactual Generation via Diffusion Probabilistic Models. ECAI 2024.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dRojHofH4z", "reviewer": "Reviewer_wWq6", "summary": "This paper explores diffusion models for counterfactual image generation by incorporating semantic abduction to enhance high-level semantic identity preservation causal consistency. The authors propose a structural causal model (SCM)-based framework that integrates diffusion models for counterfactual reasoning, leveraging spatial, semantic, and dynamic abduction mechanisms. The paper introduces amortized anti-causal guidance to improve intervention fidelity and evaluates the approach using counterfactual soundness metrics across datasets, including Morpho-MNIST, CelebA-HQ, and EMBED.\n\n## update after rebuttal\nThank the authors for their rebuttal. Some of my concerns have been addressed. The experimental section could be further improved by offering a more comprehensive comparison with strong baseline models. Additionally, it would have been beneficial to see results on a broader range of real-world datasets, which would better highlight the practical value of the approach to the vision community. I maintain my original score of weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments regarding our framework’s “promising applications in medical imaging and causal reasoning tasks” with “well-designed” experiments and our integration of existing concepts in a “novel way”. \n\n> **Scalability and Computational Costs**\n\nLike most diffusion-based approaches, our method incurs high computational costs at generation time. We acknowledge that techniques such as distillation [1], consistency models [2], and higher-order solvers for sampling [3] can improve efficiency; however, these optimisations are beyond the scope of our current work. We focus on exploring the trade-offs involved in using diffusion models for counterfactual generation, thereby setting the stage for future work addressing efficiency concerns.\n\nNotably, our approach to dynamic semantic abduction via CTA leverages causally conditioned diffusion models to improve identity preservation with a single-step optimisation. This contrasts with methods that rely on less robust, image-specific test-time optimisations - such as expensive multi-step optimisations or heuristics about self and cross-attention maps - often requiring extensive fine-tuning for each image [4, 5, 6, 7]. Our single-step method simplifies and enables rapid testing on larger datasets with a simple model selection process (Fig 5). Dynamic semantic abduction via CTA improves identity preservation of backgrounds, hairstyles, and skin colours (see Fig 3c and App H - Fig 16).\n\nFurthermore, using the latent diffusion model paradigm, our approach can be readily scaled to higher-dimensional images [8]. Our work performs comparably to existing variational methods like HVAEs [9] and VDVAEs [10], which can be challenging to scale to deeper architectures for modelling complex images due to the need to compute and match posteriors [11, 12]. In contrast, the diffusion paradigm uses a fixed posterior, offering a more computationally scalable alternative. We can provide further exposition about this in sec 3.1 for clarity.\n\n> **Comparison to Causal Disentanglement Approaches in Generative Modelling**\n\nIn response to reviewer kmep, we provide additional results on VCI [13], whose results improve upon DEAR [14], a popular causal disentanglement generative modelling approach. Our results show that our methods improve effectiveness in the scenario in Fig. 2c, and in a simpler causal modelling problem involving only thickness and intensity, whilst reversibility decreases as guidance scale increases due to trajectory deviation, as described in Sec 3.3. We also focus on methods where an SCM can be assumed instead of methods that infer the assumed causal graph but may struggle with scaling to larger parent sets [16]. We intend to provide further exposition about the challenges associated with defining an SCM in our limitations, and future work could explore incorporating causal discovery frameworks akin to [17].\n\nWhen compared to CausalDiffAE [17], we condition on $c_{sem} = (z_{sem}, pa)$, whereas they condition solely on $z_{sem}$, which also encodes the parent SCM in the style of [16]. Unfortunately, we could not reproduce the results in CausalDiffAE for the main paper or during the rebuttal period; instead, we have included another baseline from the aforementioned work on VCI [13] in response to reviewer kmep. On inspection of Fig 2a in CausalDiffAE, we believe that their framework may perform poor abduction when all parents are unobserved, as in many real-world scenarios, given their poor preservation of digit style.\n\n[1] https://arxiv.org/abs/2303.01469\n\n[2] https://arxiv.org/abs/2202.00512\n\n[3] https://arxiv.org/abs/2206.00927\n\n[4] https://arxiv.org/abs/2211.09794\n\n[5] https://arxiv.org/abs/2309.15664\n\n[6] https://arxiv.org/abs/2405.01496v1\n\n[7] https://arxiv.org/abs/2212.12570\n\n[8] https://arxiv.org/abs/2112.10752\n\n[9] https://arxiv.org/abs/2306.15764\n\n[10] https://arxiv.org/abs/2303.01274\n\n[11] https://arxiv.org/abs/2401.06281\n\n[12] https://arxiv.org/abs/2208.11970\n\n[13] https://arxiv.org/abs/2410.12730\n\n[14] https://www.jmlr.org/papers/volume23/21-0080/21-0080.pdf\n\n[15] https://arxiv.org/abs/2004.08697\n\n[16] https://arxiv.org/abs/2004.08697\n\n[17] https://arxiv.org/abs/2404.17735", "claims_and_evidence": "Most claims are well-supported with experimental results and theoretical justification.\nHowever, claims related to scalability and complexity may meed further validation.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-aligned with the problem, but the benchmark datasets are relatively simple, and additional evaluations on more complex, real-world datasets would strengthen the work.", "theoretical_claims": "The paper presents theoretical justifications for integrating semantic abduction and counterfactual trajectory alignment (CTA) within diffusion models, particularly in the context of structural causal models (SCMs). The high-level formulation appears sound.", "experimental_designs_or_analyses": "The overall experimental setup is well-designed, including multiple real datasets and quantitative metrics. \nFurther improvement could consider including large-scale and more complicated datasets, and include some human perceptual evaluation.", "relation_to_broader_scientific_literature": "The key contributions of this paper build on prior work in counterfactual generation, causal inference, and diffusion models, integrating these areas in a novel way.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The approach has promising applications in medical imaging and causal reasoning tasks.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "- What is the computational cost of the method and have you considered efficiency optimizations to make diffusion-based counterfactual generation more practical?\n\n- How does your method compare to causal disentanglement approaches in generative modeling?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9AV5IIUPGK", "reviewer": "Reviewer_kmep", "summary": "The authors proposed a method that incorporates semantic abduction in diffusion models for the preservation of exogenous noise in counterfactual generations. The inference is conducted with a CFG-style amortised, anti-causally guided DDIM sampling. Results are shown on three datasets: Morpho-MNIST, CelebA-HQ and EMBED.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and are encouraged by their comments that semantic abduction is “interesting and intuitive”. Below, we address the major points raised:\n\n> **“Semantic Abduction” and Novelty**\n\nRefer to our response to reviewer vbtG16 for details regarding the novelty and our contributions.\n\n> **Quality of Abducted Exogenous Noise**\n\nWe acknowledge the reviewer’s observation regarding Fig 3b. The denoising training procedure may influence the apparent challenges in exogenous noise abduction. However, we note that in Fig 3c and App H - Fig 16, dynamic semantic abduction and $p_\\varnothing > 0.1$ improves identity preservation of hairstyle, skin colour, facial structure and backgrounds, which is especially challenging given that we choose not to center crop faces as in existing methods, such as [2,3]. Additionally, we refer to Fig 5 in [2], which shows changes in skin colour, and Fig 2e/2g in [4], which shows changes in background and image fidelity, which the dynamic abduction method largely corrects. Additionally, referring to Table 6 in [4], our effectiveness is on par with theirs.\n\n> **“For MorphoMNIST, there is no reason to use composition...”**\n\nWe agree that the method suggested by the reviewer is valid. Instead, we follow [5,6], which uses ground truth morphological measurement functions provided by the MorphoMNIST library to measure the intensity, thickness, and slant of the counterfactual digits directly. Both methods are valid since they use ground truth mechanisms to evaluate counterfactuals. For MorphoMNIST, we only used pseudo-oracles for attributes which don’t have a ground truth mechanism, i.e. digit class ($d$).\n\n> **“Proposed method does not beat HVAE by a noticeable margin…”**\n\nTable 1 shows that for many settings, i.e. spatial and semantic mechanisms with $\\omega=3, p_\\varnothing=0.1$, our model exceeds HVAE effectiveness. Notably, the spatial and semantic mechanisms improve digit class accuracy by ~4%. When considering effectiveness across all parents holistically, while the target intervention may be successful in the baselines, the image's defining characteristic (digit class) is not preserved.\n\nAs requested, we also train VCI (which outperforms DiffSCM) for the scenario in Fig 2a. Again, all interventions cause large drops in digit class effectiveness:\n\n| Interv. | MAPE(t) | MAPE(i) | MAPE(s) | Acc(d) | Rev. ($L_1$) |\n|-|-|-|-|-|-|\n| do(s) | 3.08e-2 | 6.52e-3 | 9.07e-2 | 90.04 | 3.19e-2 |\n| do(d) | 2.63e-2 | 6.32e-3 | 9.12e-2 | 94.62 | 1.98e-2 |\n| do(t) | 3.08e-2 | 6.26e-3 | 9.13e-2 | 92.97 | 2.10e-2 |\n| do(i) | 6.99e-2 | 2.61e-2 | 3.97e-1 | 82.52 | 3.75e-2 |\n\nwith Comp. ($L_1$) = 1.31e-2, with full table [here](https://imgur.com/a/BYuLIgx).\n\nWe also compare against VCI trained in a simpler setting using only $i$ and $t$ from the dataset generated with Fig 2a, and also notice large margins of improvement:\n\n|  | MAPE(t) | MAPE(i) | Rev. ($L_1$) | MAPE(t) | MAPE(i) | Rev. ($L_1$) | Comp. ($L_1$) |\n|-|-|-|-|-|-|-|-|\n| VCI | 5.09e-2 | 9.49e-3 | 1.17e-2 | 9.58e-2 | 5.89e-2 | 2.74e-2 | 6.65e-3 |\n| Spatial | 3.45e-2 | 6.31e-3 | 3.19e-2 | 5.52e-2 | 1.07e-2 | 3.53e-2 | 9.26e-4 | \n\nwhere the first three cols are for do(t) and the next three are for do(i), with the full table provided [here](https://imgur.com/a/IojOgnj).\n\nNote that given our choice of metrics differs from those in VCI, the margins of improvement are not comparable across works. \n\nAdditionally, we run VCI on CelebAHQ without center cropping, such that effectiveness can be evaluated with our pseudo-oracles, and we notice a large drop in the counterfactual fidelity: \n\n|  | F1(g) | F1(s) | \n|-|-|-|\n| do(g) | 3.39 | 97.84 |\n| do(s) | 95.58 | 33.81 | \n\nwith examples [here](https://imgur.com/a/DRhkuUY).\n\nWe will include DDCM[8] in our related work. DDCM, which uses text-based LDMs, is challenging to compare fairly with our method due to model size, dataset requirements, and prompt engineering. Their sampling method in Alg 1 in [8] complements our work and naturally extends our mechanisms alongside other sampling strategies[7,9]. This is outside the scope of this paper.\n\nWe appreciate the reviewer’s detailed feedback and believe these clarifications and additional experiments demonstrate that our work offers an improved and valuable perspective on diffusion-based counterfactual generation.\n\n[1] https://arxiv.org/abs/2212.12570\n\n[2] https://arxiv.org/abs/2410.12730\n\n[3] https://www.jmlr.org/papers/volume23/21-0080/21-0080.pdf\n\n[4] https://arxiv.org/abs/2303.01274\n\n[5] https://arxiv.org/abs/2306.15764\n\n[6] https://github.com/biomedia-mira/causal-gen/blob/e0e4e22f8ad972b9d3b7dd662fa77d9d7c845078/notebooks/eval_example.ipynb\n\n[7] https://arxiv.org/abs/2206.00927\n\n[8] https://arxiv.org/pdf/2312.04965\n\n[9] https://arxiv.org/abs/2406.08070", "claims_and_evidence": "It's ambiguous. The claim is that this is the first work to consider high-level semantic abduction for diffusion counterfactual. If you strictly consider the meaning of the terminology \"semantic abduction\" that the authors used, this might be the first, however, I think the proposed method is not novel and has minor contributions among the existing diffusion counterfactual models. Using diffusion models for exogenous abduction is already proposed in DiffSCM [1] and using amortised, anti-causally guided inference is already proposed in DDCM [2], which entails high-level exogenous abduction. Both of these works are not compared in either related work or experiments.\n\n[1] Sanchez, Pedro, and Sotirios A. Tsaftaris. \"Diffusion causal models for counterfactual estimation.\" CLeaR (2022).\n\n[2] Xu, Sihan, et al. \"Inversion-Free Image Editing with Language-Guided Diffusion Models.\" CVPR. 2024.", "methods_and_evaluation_criteria": "No. On Morpho-MNIST, there is no benchmark comparison to prior diffusion-based counterfactual models and no benchmark comparison to state-of-the-art counterfactual models. On CelebA-HQ, there is no benchmark comparison, period. Poor benchmark comparisons had unfortunately been a bad norm of this field up until 2023, but it is not the norm anymore. Not everyone just simply use VAE and HVAE anymore. The readers should very reasonably expect *at least* three benchmark models being compared to this work:\n\n- DiffSCM [1] – as the prior effort in applying diffusion model in counterfactual modeling\n\n- DDCM [2] – as the state-of-the-art diffusion model for image editing that already proposed the CFG-style [4] *amortised, anti-causally guided inference*.\n\n- VCI [3] – as the current state-of-the-art in counterfactual modeling.\n\nI know this sounds like a lot, but unfortunately that just means this work hasn’t done enough. In my judgement, it is not ready for publication. Again, any reader should *very reasonably* expect these three works being compared to the proposed method for the reasons I listed. I noticed that [1] and [3] are cited, but [2] is not even cited. (Note that image editing is essentially the same as image counterfactual, especially for a work that isn't theoretically-driven. On an empirical level, the difference is simply whether you use the term \"counterfacutal\" and frame it within a causal formulation or not. On observational datasets, there is no difference.) As for the results, the proposed method does not beat HVAE by a noticeable margin (if it beats HVAE at all) according to Table 1, while VCI [3] beat HVAE by a wide margin. So, it is very reasonable for me to assume that this work performs sub-par compared to state-of-the-art without further benchmark comparison.\n\nSince I do not have benchmark comparisons to evaluate the results, I tried to judge the results by empirical comparison. However, the empirical results the authors showed in Figure 3. b) do not show good ability in exogenous noise abduction either – in most of these pictures, the intervention *greatly* changed the person’s hair style and exhibited some denoising ability (which you do not want for counterfactual modeling) – this seems very sub-par compared to DDCM and VCI, and I’m not sure if it even beats the old non-variational and non-diffusion based method [5].\n\n[1] Sanchez, Pedro, and Sotirios A. Tsaftaris. \"Diffusion causal models for counterfactual estimation.\" CLeaR (2022).\n\n[2] Xu, Sihan, et al. \"Inversion-Free Image Editing with Language-Guided Diffusion Models.\" CVPR. 2024.\n\n[3] Wu, Yulun, Louie McConnell, and Claudia Iriondo. \"Counterfactual Generative Modeling with Variational Causal Inference.\" ICLR (2024).\n\n[4] Ho, Jonathan, and Tim Salimans. \"Classifier-free diffusion guidance.\" (2022).\n\n[5] Zhu, Jun-Yan, et al. \"Unpaired image-to-image translation using cycle-consistent adversarial networks.\" ICCV. 2017.", "theoretical_claims": "N/A. No theoretical claims in this paper.", "experimental_designs_or_analyses": "I checked the soundness of experimental design, and I think the evaluation metrics used for MorphoMNIST are problematic. For a dataset where *the counterfactual truth is available*, there is no reason to use composition and reversibility in favor of the error between counterfactual generation and counterfactual truth. I believe the authors are well-aware that composition, effectiveness, and reversibility are metrics proposed to evaluate results on datasets where *counterfactual truth is not available*. For MorphoMNIST, there is no reason to use composition – the MSE between reconstruction and original image, and reversibility – the MSE between cycle reconstruction and original image, in favor of the counterfactual prediction error – the MSE between counterfactual construction and counterfactual image. A naive model that does nothing other than returning the original image can achieve the perfect composition and reversibility. The counterfactual prediction error, on the other hand, is a true counterfactual evaluation metric on layer 3 of Pearl’s 3 layers of causality and should always be preferred when it’s attainable.\n\n## Update After Rebuttal\n\nI have raised the score to a strong acceptance as the authors have put in an abundant amount of efforts in additional experiments. However, this is the one issue that the authors did not give me an adequate answer for, and since this year's ICML does not seem to allow further discussion with the authors after their **Reply Rebuttal Comment** is posted, I am highlighting my intended response to their **Reply Rebuttal Comment** here for readers' information:\n\n---\n\n*Thanks for the detailed response. Great work in rebuttal. I have raised my score to a 4, assuming that the comparison to DiffSCM will actually be in the final version of the paper. I do not think your argument make sense regarding issue 1, and the concern you [linked](https://openreview.net/forum?id=oeDcgVC7Xh&noteId=is9vzvhbQQ) was literally a clarification that composition and reversibility should be used in CelebA, where counterfactual truth is **not** attainable. So I do not see how this is related to the issue I raised here.*\n\n> *MSE/MAE remain meaningful for composition and reversibility because the target is uniquely defined (ie. the observed image)*\n\n*I think this is such a poor argument for using composition and reversibility over counterfactual error. The counterfactual image **is** uniquely defined, it is just not observed for real-world datasets. It is literally a unique target by the definition of counterfactual and the consistency assumption. You can correct me on this if I'm wrong: I don't believe there is any randomness to the counterfactual image generated by morpho-MNIST when intervention is given. Not only is the target uniquely defined, it **is** the target you care about -- ultimately, you don't care about how good your model can do reconstruction or cycle reconstruction, you care about how good your model can do counterfactual generation. The former is an indirect estimate of model's ability in exogenous noise abduction when the direct estimate, i.e. the latter, cannot be evaluated.*\n\n> *These metrics are also sensitive to sharpness and local artefacts. A poor but blurry counterfactual may yield low MSE [1,2,8].*\n\n*This is the flaw of pixel-wise MSE, it has nothing to do with the comparison of counterfactual error vs. composition & reversibility as all of them are pixel-wise MSE. If this is your argument why counterfactual error is flawed, I can use the exact same argument to tell you composition & reversibility are flawed as well.*\n\n*You are contradicting yourself with these two arguments. You said \"MSE/MAE remain meaningful for composition and reversibility because the target is uniquely defined (ie. the observed image)\" yet you also link works [1,2,8] where MSE is calculated on uniquely defined observed image to argue MSE/MAE is flawed. Just like the reviewer in the concern you linked, you don't have a coherent argument why composition & reversibility should be preferred over counterfactual error.*\n\n*[1] Zhang et al. \"The unreasonable effectiveness of deep features as a perceptual metric.\" CVPR 2018*\n\n*[2] Wang & Bovik, \"Mean squared error: Love it or leave it?\" IEEE 2009 - Fig 2*\n\n*[8] Wang et al, \"Image quality assessment: from error visibility to structural similarity.\" IEEE 2004*\n\n---", "relation_to_broader_scientific_literature": "As discussed in \"Claims And Evidence\" and \"Methods And Evaluation Criteria\", I do not find any evidence of this work exceling prior works in diffusion-based counterfactual models and state-of-the-art counterfactual models.", "essential_references_not_discussed": "Yes, prior works in diffusion-based counterfactual models [1][2] and state-of-the-art counterfactual models [3] are not discussed in related work or compared in the experiment section. See \"Claims And Evidence\" and \"Methods And Evaluation Criteria\" for details.\n\n[1] Sanchez, Pedro, and Sotirios A. Tsaftaris. \"Diffusion causal models for counterfactual estimation.\" CLeaR (2022).\n\n[2] Xu, Sihan, et al. \"Inversion-Free Image Editing with Language-Guided Diffusion Models.\" CVPR. 2024.\n\n[3] Wu, Yulun, Louie McConnell, and Claudia Iriondo. \"Counterfactual Generative Modeling with Variational Causal Inference.\" ICLR (2024)", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper is clear and well written. Figures are clean and illustrative.\n\nWeaknesses:\n\nSee \"Claims And Evidence\", \"Methods And Evaluation Criteria\", and \"Experimental Designs Or Analyses\".", "other_comments_or_suggestions": "I think trying to abduct high-level semantics to aid diffusion models is interesting and intuitive, and I do not want to discourage the authors from keeping exploring this direction. But I think you definitely have to show more evidence of how it works better than / different from existing diffusion models to be more convincing.", "questions_for_authors": "No further questions beyond the weaknesses I listed. If the authors can address those that'd be great.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers generally appreciated the work, felt the paper was well-written, and believed the experiments were convincing.

The main concerns were about comparison to diffusion-based methods but the author rebuttal and discussion was able to resolve these concerns to reasonable satisfaction.

Please make sure to add the new experiments and metrics along with anything else discussed throughout the author-reviewer discussion period. These new results and explanations were critical for the final paper's quality.
Decision: Accept (poster)