Title: Counterfactual Image Editing with Disentangled Causal Latent Space
Venue: NeurIPS 2025
Accepted: yes
Keywords: Causal Inference

Generative model

Causal representation learning

Image editing
Abstract: The process of editing an image can be naturally modeled as evaluating a counterfactual query: “What would an image look like if a particular feature had changed?” 
While recent advances in text-guided image editing leverage powerful pre-trained models to produce visually appealing images, they often lack counterfactual consistency -- ignoring how features are causally related and how changing one may affect others.
In contrast, existing causal-based editing approaches offer solid theoretical foundations and perform well in specific settings, but remain limited in scalability and often rely on labeled data.
In this work, we aim to bridge the gap between causal editing and large-scale text-to-image generation through two main contributions. First, we introduce Backdoor Disentangled Causal Latent Space (BD-CLS), a new class of latent spaces that allows for the encoding of causal inductive biases. One desirable property of this latent space is that, even under weak supervision, it can be shown to exhibit counterfactual consistency.
Second, and building on this result, we develop BD-CLS-Edit, an algorithm capable of learning a BD-CLS from a (non-causal)  pre-trained Stable Diffusion model. This enables counterfactual image editing without retraining. Our method ensures that edits respect the causal relationships among features, even when some features are unlabeled or unprompted and the original latent space is oblivious to the environment’s underlying cause-and-effect relationships.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework for performing causally consistent image editing by leveraging pre-trained text-to-image diffusion models, without requiring further training or fine-tuning. The authors model image editing as the estimation of counterfactual queries. They propose a new class of Structural Causal Models (SCMs), called Backdoor Disentangled Causal Latent Space (BD-CLS), which satisfy key properties of causal image editing: intervention consistency, invariance of non-descendants, and controlled change of descendants. The authors also present an algorithm for obtaining a BD-CLS from a pre-trained diffusion model and sampling causally faithful counterfactual images. Lastly, the proposed method is assessed through experiments on both synthetic and real-world image data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper tackles a significant gap in the literature on how to leverage text-to-image foundation models for performing causally consistent counterfactual image generation.\n- The authors provide a solid theoretical framework regarding how to design a BD-CLS SCM. They clearly justify why the proposed BD-CLS achieves the desired properties for faithful causal editing.\n- The proposed algorithm 1 for extracting a BD-CLS from a pre-trained diffusion model offers a compelling, causally-aware alternative to existing diffusion-based image editing methods, which may not respect assumed causal relationships when edits are performed.\n- The algorithm 1 is efficient, as it does not require updating the diffusion model parameters.\n\n\n**Weaknesses:**\n\n- The proposed method's qualitative evaluation lacks depth.\n  - More qualitative comparisons should have been included in the main paper.\n  - The authors qualitatively compare their method with only two baseline image editing techniques.\n- The paper lacks a thorough quantitative evaluation.\n  - The authors do not include tables with quantitative results or use metrics that are commonly applied in counterfactual image generation [1, 2, 3].\n- For real-world images, the authors do not evaluate their approach on a standard benchmark dataset (e.g., the datasets used by compared baselines such as DDPM inversion [4], or alternatives datasets with text annotations like Multi-Modal-CelebA-HQ [5]).\n- Although the paper is well-structured, in some parts the notation becomes complex and difficult to follow.\n\n**References:**\n\n[1] Monteiro, Miguel, et al. *Measuring axiomatic soundness of counterfactual image models.* arXiv preprint arXiv:2303.01274, 2023.  \n[2] Ribeiro, Fabio De Sousa, et al. *High fidelity image counterfactuals with probabilistic causal models.* arXiv preprint arXiv:2306.15764, 2023.  \n[3] Melistas, Thomas, et al. *Benchmarking counterfactual image generation.* Advances in Neural Information Processing Systems, 37 (2024): 133207–133230.  \n[4] Huberman-Spiegelglas, Inbar, Vladimir Kulikov, and Tomer Michaeli. *An edit-friendly DDPM noise space: Inversion and manipulations.* Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.  \n[5] Xia, Weihao, et al. *TediGAN: Text-guided diverse face image generation and manipulation.* Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.\n\nQUESTIONS:\n- In Algorithm 1, Step 1, how do you identify the largest backdoor set?\n- In Figures 6 and 9, which present qualitative comparisons, could you clarify which are the source and target prompts for performing the causal edits?\n- What diffusion backbone model do you use for the ColorMNIST-Bars dataset? Is it a latent diffusion model or a pixel-based diffusion model?\n- I believe the paper would benefit from some form of quantitative evaluation. If this is not feasible, it would still strengthen the work to include some qualitative results on additional known synthetic datasets (e.g., MorphoMNIST) or other real-world datasets such as CelebHQ, where causal graphs can be assumed as in [1] and [2].\n\n**References:**\n\n[1] Yang, Mengyue, et al. *CausalVAE: Disentangled representation learning via neural structural causal models.* Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.\n\n[2] Melistas, Thomas, et al. *Benchmarking counterfactual image generation.* Advances in Neural Information Processing Systems, 37 (2024): 133207–133230.\n\nLIMITATIONS:\nThe limitations are adequately described in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of counterfactual image editing—\"What would this image look like if feature $X$ were changed from $x$ to $x'$?\"—by marrying causal inference with pre-trained diffusion models. The key steps are: \n\n1. Augmented Structural Causal Models (ASCM) formalize the true generative process, partitioning factors into labeled (V) and unlabeled (L) sets (Def.1, p.3).\n\n2. Backdoor Disentangled Causal Latent Space (BD-CLS) is defined (Def.4, p.5) as a proxy SCM over $\\{X, B, Z\\}$ that (i) matches the true conditional distribution $P(I|X, B)$, (ii) disentangles non-descendant features from interventions, and (iii) respects the causal graph. Theorem 2 (p.6) proves that sampling from any BD-CLS yields intervention consistency, non-descendant invariance, and counterfactual consistency on descendants.\n\n3. BD-CLS-Edit (Alg. 1, p.7) shows how to extract a BD-CLS from a pre-trained Stable Diffusion model without any retraining. By identifying a backdoor set B, inverting the image to noise $n$, and optimizing a prompt-mixing schedule $\\theta_t$ via gradient updates (Eqs. 6–8), one can sample counterfactual images $i'\\sim P_{\\rm{BD-CLS}}(I_{X=x'}|i)$. \n\n4. Experiments:\n\n* Colored MNIST & Bars: BD-CLS yields the correct flip probabilities for descendant features (bar width) within theoretical bounds, while non-causal baselines (CDiffusion, CGN) violate invariance or descendant behaviour (Figs. 6–8).\n\n* Text-to-Image Editing: On real-world prompts (weather → rainy, season → fall, scene → garden), BD-CLS-Edit preserves non-descendants (pose, layout) and induces realistic descendant changes (umbrella, clothing, removal of grocery bag), outperforming DDPM inversion and DDS (Fig. 9).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe ASCM → BD-CLS framework gives formal guarantees on varying and invariant concepts under intervention without costly retraining. Moreover, only target feature $X$ and backdoor set $B$ need labels. All other factors remain label-free. Comprehensive quantitive and qualitative demonstrations via experiments convincingly demonstrate causal consistency where baselines fail. \n\n**Weaknesses**\n1. The approach presented assume a fully specified SCM over image factors $V$ and backdoor set $B$, which can be difficult to obtain in practice. The authors should consider ways of automatically extracting or at least validating the causal graph from data.\n\n2. The setup assumes an exact and invertible mapping between image space and diffusion latents, which is not always available. \n\n3. Each edits requires a gradient search over the per-timestep mixing weights which can be quite expensive. There is a lack of discussion in terms of the overhead for inference time. \n\n4. The prompt-mixing schedule $\\theta_t$ and its learning rates (Eqs. 6–8) require manual tuning. The paper lacks an extensive ablation on how robust the method is to these settings.\n\n5. Validated mostly on synthetic datasets (Colored MNIST, Bars) and a handful of text-to-image edits. It remains unclear how well the approach scales to complex, real-world scenes with many entangled factors.\n\n6.  Although only weak labels for $X$ and $B$ are needed, it still demands annotated examples of each intervention and backdoor factor. For richly structured images, obtaining such labels can be laborious.\n\n7. Disentanglement emerges indirectly via $\\theta_t$ optimization rather than explicit architectural constraints. There’s no guarantee that unrelated factors won’t bleed into each other under challenging edits.\n\nQUESTIONS:\n1. Robustness: How sensitive is BD-CLS-Edit when the assumed backdoor set B omits a confounder or includes an extraneous node?\n\n2. Inference efficiency: What is the average runtime and GPU/memory footprint for a single 512×512 edit?\n\n3. Generalization: Can the method handle continuous interventions (e.g. brightness levels) or multiple simultaneous edits (changing both season and weather)?\n\n4. Automated graph discovery: Is it possible to infer or verify the necessary causal graph and backdoor sets from data, reducing manual effort?\n\nLIMITATIONS:\n1. Manual causal input: The need for a domain-expert causal graph and feature labels may hinder adoption in uncontrolled settings.\n\n2 .Optimization overhead: Per-instance gradient searches for $\\theta_t$ may not suit real-time applications.\n\n3. Evaluation breadth: The paper does not demonstrate performance on large-scale or highly-diverse image datasets beyond a few case studies.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to study the counterfactual image editing problem with pretrained diffusion models. Current image editing works often ignore causal consistency, where some works change non-relevant parts and others fail to edit logically relevant parts. The authors claim there lack of scalability and existing methods rely on full supervision. On the contrary, the authors propose a theoretically grounded (based on Structural Causal Models) image editing method, which (1) considers the causal relationship between different factors; (2) is efficient without training the full model; (3) can work with weak supervision. The proposed method, BD-CLS-Edit, learns to optimize the linear combination parameters of conditional variables $c$ and $c'$ during the denoising process, under the constraints grounded by SCM. The method is demonstrated on Colored MNIST and Bars dataset and stable diffusion, where qualitative examples show better performance than baselines: LS inversion and DDS.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The method is highly theoretically grounded and novel.\n2. The motivation is clear and interesting, counterfactual image editing.\n3. The method is shown to be working on both toy models trained on MNIST and the stable diffusion model.\n4. The actual implementation of BD-CLS-Edit by optimizing linear combination coefficients guided by the gradients is novel.\n\nWeaknesses\n1. Although some qualitative examples are shown, the paper shows no quantitative measurements (LPIPS, FID, human evaluation, etc) with large-scale evaluation datasets. This is a key limitation since the method's effectiveness may not be valid.\n2. Although novel, the method requires a predefined causal graph as input. In practice, such a graph may not be easy to obtain. The paper does not provide a practical solution to get such graphs. Furthermore, in practice, the gained graphs may be incomplete or noisy, and it is worth analyzing whether the method is robust to such causal graphs.\n3. The paper lacks some experimental details, such as how $\\theta_i$ is regularized, whether it is learned per edited image, or can be shared across certain groups of images.\n4. Current illustrations of the method are still constrained within simple causal relationships and do not seem scalable to more complex editing scenarios. For example, whether emotional relationships can be reflected as well?\n5. Some writings are not clear, for example, the term LS inversion are used several times before defining it. The introduction of SCM related parts is also not clear. For example, the notion $\\mathbf{P a}_{V_j}$ (Parents of $V_j$) is never introduced but used as granted.\n\nQUESTIONS:\nSee weakness. Major points are 1 and 2. It would be better if the paper could provide quantitative measurements and compare the proposed method with the baseline. It would be better if the evaluation dataset could practically cover diverse editing scenarios. It would also be better to provide robustness analysis on the scene graphs. Other points are minor and would be better clarified.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper introduces Backdoor Disentangled Causal Latent Space (BD-CLS), a method that enables edits to reflect causal effects without altering unrelated features, even under weak supervision. It proposes BD-CLS-Edit, an algorithm that extracts this structure from pre-trained diffusion models like Stable Diffusion without retraining. The approach ensures edits are causally consistent, scalable, and effective even when only partial labels are available. Results on synthetic and real-world tasks show improved causal fidelity compared to existing baselines.

STRENGTHS AND WEAKNESSES:
The paper presents a solid theoretical framework for counterfactual image editing. It integrates well with pre-trained diffusion models and requires no retraining. The structure is clear however the paper relies heavily on the supplementary material which make reading harder. The work addresses a critical gap in generative modeling. The ability to edit images in a causally grounded way under weak supervision is valuable. The formulation is original and timely.

QUESTIONS:
While results on real-world images are promising, most examples involve relatively small-scale causal graphs (3–5 variables). How does BD-CLS-Edit scale when editing in scenes with richer semantics?

LIMITATIONS:
Yes, but only in the supplementary materials.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02