Facial Counterfactual Generation via Causal Mask-Guided Editing
Authors: Pei Sze Tan
Sailaja Rajanala
Arghya Pal
Raphael CW Phan
Huey Fang Ong
Venue: Accepted by TMLR ()

Abstract: Generating counterfactual facial images is an important tool for interpretable machine learning, fairness analysis, and understanding the causal relationships among facial attributes.  In this work, we propose a novel neuro-symbolic framework for causal editing, which integrates causal graph discovery, mask-guided counterfactual generation, and semantic interpretation to produce facial images that are both realistic and causally consistent. We first employ the Fast Causal Inference (FCI) algorithm to uncover latent causal relationships among facial attributes, enabling the identification of direct and indirect factors for target interventions. Using these causal graphs, we construct spatially informed masks that guide a DDPM-based generative model, ensuring that only regions relevant to the causal factors are modified. Finally, we leverage CLIP-based embeddings to provide logical, human-understandable explanations of the semantic changes in the counterfactuals. Experiments on CelebA and CelebA-HQ demonstrate that our approach produces high-fidelity counterfactuals, achieves superior performance on sparsity and realism metrics, and mitigates bias compared to state-of-the-art methods. This framework offers a principled approach to causally grounded, interpretable facial image editing.

Published in Transactions on Machine Learning Research (03/2026)
Facial Counterfactual Generation via Causal Mask-Guided Editing
Pei-Sze Tan
peiszetan266@gmadlcco com
CyPhi (vr) AT Lab, School of Information Technology Monash University, Malaysia Compus
Sailaja Rajanala
CyPhi (V4) AI Lab, School of Information Technology Monash, University, Malaysia Campus
Arghya Pal
orghya poleyyaspl@@mmmshh... edu
CyPhi (24) Al Lab, School of Information Technology Monash University, Malaysia Campus
Raphaël C.-W. Phan
rephael.phan@moonce edu
CyPhi (VO) AI Lab, School of Information Technology
Monash Universily, Malaysia Campus
&
Dept of Software Systems & Cybersecurity, Faculty of IT Monash University, Clayton compus
Huey-Fang Ong
wng.hueyfeng@monas.ccom edu
CyPhi (VD) AI Lab, School of Information Technology Monash Universily, Malaysia Campus
Reviewed on OpenReview: https: // openreview. net/ forum? id= ssamE00j00
Abstract
Generating counterfactual facial images is an important tool for interpretable machine learn- ing, fairness analysis, and understanding the causal relationships among facial attributes. In this work, we propose a novel neuro-symbolic framework for causal editing, which integrates causal graph discovery, mask-guided counterfactual generation, and semantic interpretation to produce facial images that are both realistic and causally consistent. We first employ the Fast. Causal Inference (FCT) algorithm to uncover latent causal relat among facial attributes, enabling the identification of direct and indirect factors for target interventions. Using these causal graphs, construct spatially informed masks that guide DDPM-based generative model, ensuring that only regions relevant to the causal factors are modified. Fi- nally, we leverage CLIP-based embeddings to provide logical, human-understanddbble expla- nations of the semantic changes in the counterfactuals. Experiments on CelebA and CelebA- HQ demonstrate that our approach produces high-fidelity counterfactuals, achieves superior performance on sparsity and realism metrics, and mitigates bias compared to state-of-the-ars methods. This framework offers principled approach to causally grounded, interpretable facial image editing. The code is available at: ictps://githut hwtpsh/....oo/oobbaaaaaaaaaaaaaa
1 Introduction
Counterfactual reasoning has bocome an essential tool in interpretable and responsible machine learning (Velazquez al., (2023). By considering how changes in inputs would alter outputs, counterfactuals al- low researchers and practitioners explore "what-if" scenarios in controlled and understandable man-

---

Published in Transactions on Machine Learning Research (03/2026)
ner (Goyal et al. 2019 In the context of facial images, the goal extends beyond explaining model's decision: it involves generating realistic, causally consistent counterfactual instances that reflect. plausi- ble alternative appearances or attributes. Unlike traditional counterfactual explanation methods, which aim to minimally perturb inputs to reveal a model's decision boundary, counterfactual generation focuses on creating images that maintain fidelity to the underlying causal relationships among facial features.
Realism and causal consistency are criti- cal in applications such identity ver- ification, biometric authentication, and human-computer interaction, as shown in Figure [] an example in the visa ap- plication scenario and consequences of non-causal inference counterfactual gen- eration. Existing image-based counter- factual methods often treat all features as independently manipulable, ignoring dependencies between attributes (Jean- neret et al. 2023 2022) As a result, ed- its may produce implausible or entangled changes. For example, altering hair colour might unintentionally modify perceived age or gender, and chang- ing gender could introduce unrelated expressions. Such inconsistencies not only reduce trust in the generated images but can also amplify dataset biascs, undermining fairnoss in downstream tasks (Xu et al. 2020).
Acceptance Criteria:
FoCER
of Emy Interpret
Smiling, No Heavy Makeup
of Clearly Explained
Rejection Scenario
? Confusiin
VISA
Unelow Explanation
Baseline
Figure Motivation scenario: a rejected visa photo. Our method highlights the actual causal reasons, enhancing user understand- ing.
Several methods have been proposed for generating visual counterfactual explanations. The work in (Goyal et al. 2019) developed method for generating visual explanations with a counterfactual perspective. An- other work that emphasises causal relation in counterfactual estimation is (Sanchez & Tsaftaris) 2022). a
deep structural causal model that. leverages recent advancements in diffusion models that involve iteratively sampling gradients from the marginal and conditional distributions that are part of the causal model. Peng et al. (Peng et al. 2022) simplified the explanation of model-detected face swap images by introducing au- tomatic methods to create more authentic and manipulated counterfactual versions. Similarly, (Hendricks)
et al. 2018 introduced phrase-critic model to enhance candidate explanations generated with inverted phrases.
However, methods that do not account for causal dependencies can yield visually unrealistic or non-actionable counterfactuals (Slack et al.] (021). De Sousa Ribeiro et al. (De Sousa Ribeiro et al. 2023) show that incorporating a causal model helps distinguish true causes from spurious associations, enabling accurate, high-fidelity counterfactual image generation. Similarly, Melistas et al. (Melistas et al. 2024 emphasise
that realistic image edits must respect causal relationships inherent to the data generation process. These works demonstrate that causal models help isolate genuine causal factors and avoid spurious correlations. Existing methods face limitations that hinder the generation of causally consistent and interpretable facial comtteracctulls. These gaps highlight the need for a framework that not only enforces cansal consistency during counterfactual generation but also provides
explanations.
To address these challenges, we propose framework that explicitly incorporates causal knowledge into the generation process. We introduce Facial Counterfactual Generation via Cansal Quided Editing, method that first learns causal graph over facial attributes and perceptible features. This graph identifies direct causes and confounders, which then guide mask-based interventions during reverse diffusion. By modifying only causally relevant regions while holding confounders fixed, our approach ensures that generated counterfactuals are both realistic and semantically coherent, To further enhance interpretability, we leverage CLiP-based embeddings to provide textual descriptions of the visual edits, allowing users to understand which features were causally modified without implying an explanation of any specific model.
This work makes four primary contributions: (i) we shift the focus from counterfactual explanation to the causally coherent generation of facial images in a neuro-symbolic generation framework, emphasizing realistic and plausible outputs; (ii) we introduce a framework that learns causal structure over attributes
2

---

Published in Transactions on Machine Learning Research (03/2026)
and applies mask-guided reverse diffusion to intervene selectively on direct causes; (iii) we provide visual and textual outputs that narrate the applied edits, improving interpretabilityy and user understanding: and (iv) we demonstrate both quantitative and qualitative improvements in realism, sparsity, and causal consistency compared to statc-of-the-art basclines on CelebA and CelebA-HQ.
2 Literature Review
There are two primary research gaps and challenges motivating this work: first, the difficulty of accurately discovering cansal relationships among facial attributes in images; and second, the prevalence of stercotypical biases in current generative models.
Causality in Facial Images Analysis The literature on causality in facial attributes has explored meth- ods to improve model interpretability and achieve counterfactual fairness, particularly with respect to sen- sitive attributes such as race and gender. While prior work has addressed fairness by explicitly modeling the role of protected attributes (Mazumder & Singh, 2022), methods in facial attribute editing still face important limitations. For example, Kang et al. (Kang et al., 2022] propose generating synthetic facial replicas under factual and counterfactual assumptions using causal graph-based attribute translation, pro- ducing realistic counterfactual images. However, their work focuses on generating facial replicas for fairness evaluation rather than counterfactual explanations with emphasis on sparsity and causal consistency, which is fundamentally different objective from ours. Other approaches incorporate wavelet scattering trans- forms [Liu et al. 2024) and channel attention for efficient face attribute classification, while NCINet (Wang & Boddeti 2022 aims to identify causal relations from higl-dimensional image representations.
Recent advances in diffusion- based counterfactual generation have made significant progress. Rasal et al. (Rasal et al.] 2025) propose semantic abduction mochanisms for diffusion models, introducing spatial, semantic, and dynamic abduction to balance identity preservation with causal faithfulness during image editing. Wu et al. (Wu et al., 2025) present. Variational Causal Inference (VCT), framework that enables end-to-end counterfactual supervision during training with disentangled exogenous noise abduction. How- ever, both approaches focus primarily on image editing preciss control over generated images and identity preservation and assume known causal graphs as imput. In contrast, our work addresses inter- pretable counterfactual explanations for fairness auditing and learns the causal structure from data via FCI, which is critical for real-world settings where ground-truth causal graphs are unavailable.
Despite these advances, existing methods face several limitations that hinder their practical application for causally consistent facial counterfactual generation. First, while some approaches build causal graphs to guide attribute changes, they rarely integrate these graphs directly into the image generation process, resulting in edits that may violate causal dependencies. Second, previous methods focus solely on visual output, without providing interpretable or human-underssanddable explanations of the applied interventions. Third, many causal discovery techniques rely on approximations that are sensitive to noise or missing variables, limiting robustness and scalability to large, higl-dimenssooal facial features.
Stereotyping in Generative Models Recent studies have highlighted significant concerns regarding stereotyping in image generation models, particularly in text-to-image systems. For instance, Bianchi et al. (Bianchi et al. (023) demonstrated that widely accessible text-to-image models can amplify demographic stercotypes at large scale, producing biased images even from neutral prompts. Similarly, Barve et al. Barve let al. (2025) found that while prompt refinement can mitigate stereotypes, it often limits contextual align- ment, indicating trade-off between bias reduction and semantic accuracy. These issues underscore the need for more nuanced approaches to address bias in generative models.
However, existing literature predominantly focuses on Lexl-lo-image models, leaving a gap in understanding and mitigating bias in facial image generation. This oversight is particularly concerning given the widespread use of facial images sensitive applications such as identity verification and biometric asscssments. The lack of causal reasoning in current models often leads to unrealistic and biased facial edits, as they fail to account. for the interdependencces among facial attributes.
3

---

Published in Transactions on Machine Learning Research (03/2026)
CelehA
pccputt
Patevel
Label
Label
Mask the commppand region of de
Graph
D e
Concol
Preserved Inpainted
prelm
M
Data Preprocessing
Learning Maskguided Refinement
Figure 2: Our proposed causal editing pipeline. Data preprocessing constructs facial attribute matrix D, input to FCL. We assume d 5, N 10 for illustration. Using PAG G, mask direct causes of target Qi- The mask is obtained as binary difference between Jzef and Tprelim, then filtered by causal structure to produce counterfactual I Color coding: red bold node = target attribute (Qt); red nodes with white text = non-inteneeee interineeeeded attributes; grey nodes = direct causes DC(Qi) (inpainted regions). Mask: white = impainted//regeneraat regions; black preserved regions.
n
3 Methodology
In Figure 2 it illustrates our complete pipeline. Starting from facial attribute matrix D, FCI discovers causal graph G. The reference and preliminary images are compared to construct mask M, which guides diffusion produce the final counterfactual.
Table 1: Edge lypes in a Partial Ancestral Graph (PAG) and their interpretations.
Edge Type
Interpretation
(a) Vm + Vn
Directod causal link from Vm to Vn (i.c., V, is a child of Vm..
(b) V Vn
Rules out V,, ancestor of Vm; may reflect the pres- ence of unmeasured confounders C. Undetermined orientation; no d-separation (an un- blocked path).
(c) m o-o Vn
(d) Vm + Vn
Suggests an unobserved confounder between Vn and Vn.
3.1 Causal Graph Learning
To generate the causal graph, let D E {0, Nxd be the data matrix of binary facial attributes, where each row corresponds to an instance and each column to an attribute. Given D, we use the Fast Causal Inference (FCT) algorithm Spprtess (2001) to estimate Partial Ancestral Graph (PAG) over the d attributes. Unlike conventional causal graphs, PAGs encode both dirccted causal odges and odges with uncertain orientation, as summarised in Table
Causal Graph Discovery. To ensure causally consistent counterfactual generation, first uncover the underlying causal structure among facial attributes. We compared established causal discovery algorithms, focusing on the FCI (Spirtes) 2001). While Peter-Clark (PC) (Spirtes el al. 2000) may suggest simplistic direct links, FCI uncovers richer, more nuanced structures that capture indirect effects and potential con- founding relationships, allowing for more precise and valid interventions during counterfactual generation. Ultimately, we adopted FCI for its superior handling of hidden confounders comfundess-umbseereed unobserved variables that affect multiple facial traits.

---

Published in Transactions on Machine Learning Research (03/2026)
Mathematically, FCI employs chi-square tests to evaluate conditional dependencies between categorical vari- ables:
F0 ~= E
=
(1)
where O is the observed frequency of a category and E is the expected frequency under independence. Higher x² values indicate stronger dependence, with significance determined via comparison to a critical value or a p-value threshold 0.05.
Our goal is infer the causal structure among these attributes. Formally, we seek a graph = (V,E), where Xa) is the set of nodes and & is the set of edges encoding causal relations.
V = (X1, xx,
Practical Advantages. FCI combines conceptual robustness with computational efficiency. Its use of conditional independence tests avoids the heavy resource demands of deep learning-basee cansal discovery methods, making scalable for large facial attribute datasets. For a complete mathematical treatment of FCI, see [Spirtes 2001).
We provide extensive validation of the learned causal graph in Appendix A including synthetic ground- truth recovery experiments, bootstrap stability analysis, and ablation studies demonstrating that causal knowledge-not graph topology alonc-drived our performance improvements.
3.2 Reference Image Generation via Guided Diffusion
Notation. We use the following notation throughout this section: In (original image), r=s (reference counterfactual with full guidance), 7prelim (preliminary counterfactual with weak guidance), In (final coun- terfactual), Qi (target attribute), DC(Qi) (direct causes of Qi, identified as type (a) edges in the PAG), M (binary mask), and À (mask threshold).
The first stage of our counterfactual generation produces a reference image that achieves the desired target attribute change without any causal constraints. Let In denote the original image and Qi the target attribute. We use guided DDPM to generate reference counterfactual Tree.
Ft-1 ~ (Po( + vv, 1o),(Qi(aa,.of)))
where (a(z,, t) is the predicted mean from the diffusion model, of is the variance schedule, and y controls the guidance strength toward the target attribute Q
This reference image serves as a minimaaly perturbed counterfactual that successfully flips the target label. At this stage, edits are guided purely by the attribute change, and no causal reasoning applied. The image may include unrelated or unrealistic changes in other regions, which are addressed in the subsequent mask-guided refinement.
3.3 Mask-Guided Counterfactual Generation with Causal Graphs
After obtaining fref, we identify regions relevant for causal intervention through two-step process illustrated in Figure 2
Step 1: Generating Reference and Preliminary Images.
Reference image pref, Generated via guided diffusion toward the target attribute Q with full guidance strength 7.
Preliminary image T /prelim, Generated with weaker guidance to isolate attribute-specific changes, serving as a bascline for comparison.
Constructing the Causally-Filterss Mask. The difference between these images highlights Step 2:
regions that changed due to the target attribute. We construct binary mask M that incorporates both the
5

---

Published in Transactions on Machine Learning Research (03/2026)
pixel-level difference and the learned causal structure:
Mij - [ 1, (i.j) - 1prelim (i,j)l > X and (i,j) E DC(Q1)
(3)
0, otherwise
where is a threshold controlling sensitivity to changes (sec Appendix B for scnsitivity analysis), and DC(Q1) denoles the spatial regions corresponding to direct couses the target abtribute Qt. Direct causes are identified from the learned PAG as attributes connected to Qi via type (a) edges (+), representing direct causal relations (Table
Step 3: Causal Filtering. Using the PAG, filter the mask to ensure only causally relevant regions are modified:
Direct cause regions (type (a) edges: +): Included in mask (whitte/inaiinte))
Confounder or indirect cause regions (bidirectional tt, circle endpoints -, o-o): Excluded
from mask folack/preserver
Step 4: Counterfactual Generation. The mask is applied within the DDPM generative process:
r, - G(do(In, M));
(4)
where G is the DDPM and do(In, .M) denotes intervening on In along the causally relevant regions defined by M. This ensures that the final counterfactual I modifies only the direct causes of Qt while preserv- ing confounded or unrelated attributes, resulting in both realistic and causally consistent counterfactual explanations.
Scope of causal intervention. We emphasise that framework performs causally-constrained editing rather than formal pixel-level counterfactual inference. We conduct attribute-level do- intervention interveniooniideniiyiine identifying direct causal parents from the PAG- and use this to constrain spatial editing via the mask M. The ablation in Appendix A.4 confirms that specific causal relationships, not graph topology alone, drive our improvements.
3.4 Logical Interpretation
To provide a logical and human- human-underean understandable in- terpretation of the generated counterfactuals, we leverage CLIP (Radford et al. 2021 to map visual changes into semantic descriptions,
Norm
illustrated in Figure 3 Let In denote the original image and its counterfactual counterpart, and let T denote the space of textual descriptions. CLIP provides en- coders E₁ and Er that embed images and texts into a shared d-dimensional space, enabling cross-modal reasoning.
E
"Top Changes: elderly person
Ex
Norm
E
Ae
We first compute the normalised image embeddings:
Figure 3: Our proposed counterfactual-to-teet inter- pretation model with CLIP CLIP-inspired image and text encoder.
(T,), d' ET,,),
(5)
and define the semantic transformation vector
Ac c
(6)
This vector captures the overall change induced by the counterfactual transformation in the semantic em- bedding space.

---

Published in Transactions on Machine Learning Research (03/2026)
To interpret these changes logically, compare Ae with curated set of text embeddings (Er(d)) where each d; E 'D C T represents an interpretable facial attribute or semantic concept. The relevance of each description is quantified using cosine similarity:
Ae (r,),)
S(d))
(7)
| || (d+(dl)
Descriptions with high alignment are selected the logical explanations for the visual changes:
D* = {d. ED S(d: > >]
(8)
where T is a similarity threshold.
This logical interpretation framework allows us to translate causal manipulations the image into human- readable semantic statements, providing transparent insights into which facial attributes were modified and how these changes relate causally to the target attributes. By connecting visual counterfactuals to linguistic concepts, we bridge the gap between low-level image transformations and high-level reasoning about attribute causality.
4 Experiments
We conducted all our experiments on two large facial datasets to evaluate the performance of our model. Comparison is conducted with the current state-of-tho-art counterfactual explanations model. Our exper- imental pipeline was implemented using PyTorch 2.0.1, with computations accelerated by NVIDIA A-100 GPUs on SLURM-managed computing cluster using CUDA 11.7. The diffuusion process is performed with 500 steps, with timestep respacing to 50, stochastic sampling fraction of 0.1, and an inpainting ratio of 0.15. The causal discovery component of our work leveraged the causal-learn library (Zheng et al. 2023 comprehensive Python package that implements both classical and cutting-edge causal discovery algorithms.
4.1 Datasets
Our work draws strength from the diversity and scale of the CelebA dataset. (Liu et al. 2015). which is a collection featuring over 200,000 facial images with 40 precisely annotated attributes. To ensure our findings generalise across different image quality conditions, [Lee et al.) 2020). which is high-resolution counterpart featuring 30,000 meticulously detailed images of 1,000 celebrities. These two datascts together provide a sufficiently diverse and large sample, covering a wide range of facial variations, attribute combinations, and image resolutions, making them well-suited for robust evaluation our counterfactual image generation method. Additionally, CelebA and CelebA-HQ are standard benchmarks used by all baseline methods (ACE, DIME, STEEX, DiVE), enabling direct and fair comparison. We also conducted the attribute correlation and analysod the class imbalance of the CclebA dataset in Appendix C
also conducted experiments CelebA-HQ
4.2 Evaluation Metrics
Our evaluation methodology employs four complementary metrics, each capturing different aspects of coun- terfactual quality.
Realism of Generated Counterfactuals: Fréchet Inception Distance (FID) serves as our primary gauge of generated image quality, with lower scores indicating better quality and closer resemblance to real images. Let pr. E, be the mean and covariance of features from real images, and pg Mg be the corresponding statistics for generated images. FID is computed as:
FID = I|le - 1 Tr Er + Eg 2(E2
(9)
7

---

Published in Transactions on Machine Learning Research (03/2026)
Goaa Cansundor
Correiation with Smiting
Poutive Commaation
Mogulae Sompatton
ecretter
Elery
avary Deverour
Pay
Cheery
twary Anhlgg
Chevetedds
Hedt Siskll Open
Mostaote
Pein Uniee
dang
Cuucal Colfounder
towe Erve
add
avpr
Be rour
mick ser
thery
even Har
sune Saveowee
Chadey
duste
Hale Chemetrren
Hosterte
Eiae
Aodt Rvic
Pewe Cloets
saary star
Pertow
Surr
reeving
gewe
Figure 4: Correlations vs confounders in causal relationships of attribute "Smile" heatmap. The high purple bar in correlation ("High_ "cceccbcccccs"" and "Mouth SSigh_Sllllhh_SS_ll _0pen") shows the conflict between highly correlated attributes and the confounders the causal algorithm identifies. The conflict is that while causal algorithms correctly identify confounders as indirect causes, traditional correlation- models mistakenly learn these confounders direct predictors simply because they are correlated with the target attribute.
We supplement this with Scaled Fréchet Inception Distance (sFID), which adjusts the feature distri- butions to better capture differences across attribute scales:
2 + (at 02/
Σ
sFID 2
(10)
(o+)2 +E
where d is the feature dimension, n² and a² are the mean and standard deviation of the k-th feature, and is small constant for numerical stability.
We introduce Mcan Number of Attributes Changed (MNAC) to quantify how many facial altributes are modified in the counterfactual generation process. Let x denote the original image and a the generated counterfactual, with corresponding binary attribute vectors A(x) and A()). MNAC is defined as:
Sparsity of Counterfactual Changes:
N d
1 MNAC = N ==
Σ Σ
I(A())))
(1))
where is the total number of test images, d is the number of auributes, and 1() is the indicator function. Counterfactual explanations should exhibit minimal visual variations from the original image while achieving the desired change in target attributes. Lower MNAC indicates sparser, more targeted modifications a desirable property for interpretable counterfactual explanations.
Identity Preservation: To ensure that counterfactual edits preserve the identity of the subject, we employ Face Verification Accuracy (FVA) Cao et al. 2018); ; Jeanneret et al. (2023). FVA measures whether the original image and its counterfactual are recognised as the same individual by a face verification network. Specifically, we extract deep embeddings using ArcFace (Deng et al. 2019) and compute the cosine similarity between the original and counterfactual image embeddings. FVA is defined as:
N FVA
Σ 1=1
(( cos ((())).(()))>)))
(12)
N
where f(-) denotes the ArcFace embedding function, cos(..)) is the cosine similarity, and T is the verification threshold. Higher FVA indicates better identity preservation. This metric is crucial for counterfactual explanations, as meaningful explanations should modify only the target attribute while keeping the subject's identity intact.
8

---

Published in Transactions on Machine Learning Research (03/2026)
Notably, we deliberately avoid using correlation-based metrics such as Correlation Difference (CD) because our method is specifically designed to account for causal relationships rather than mere correlations wetween facial attributes, as illustrated in Figure
4.3 Baseline Methods
We compared our model with several state-of-the-art baselines for counterfactual explanation generation. Adversarial Counterfactual visual Explanations (ACE) (Jeanneret et al. 2023 leverages adversar- ial attacks on a diffusion model to refine counterfactual images, focusing on generating subtle yet effective modifications that change the target attribute while maintaining realism. Steering Counterfactual Ex- planations with Semantics (STEEX) Jacob ct al. (2022) generates counterfactuals by steering the latent representations of images in a pretrained generative model, guided by semantic attribule changes, al- lowing controlled modifications of specific attributes while preserving other visual details. Diffusion Model Counterfactual Explanations (DiME) Jeanneret et al. (2022) uses diffusion-based generative models to produce realistic counterfactual images through iterative denoising of latent reprosentations conditioned on larget attributes, enabling smooth transitions between original and modified attribute states. Counterfac- tual Explanations with Diverse Valuable Explanations (DiVE) Rodriguez et al. (2021) focuses on gencrating a diverse set of counterfactual explanations that are both visually plausible and informative, em- ploying a diversity-promoolng objective to ensure multiple counterfactuals for given input caplure different plausible modifications.
These basclines cover a wide spectrum of counterfactual generation paradigms, from adversarial refinement and latent-space steering to diffusion biffe synthesis and diversity-driven explanations, making them suitable benchmarks for ovaluating the performance of our proposed method.
We restrict comparison to methods that (i) generate image-level facial counterfactuals, (ii) operate on attribute-conditionee diffusion or generative models, and (iii) preserve identity in facial editing tasks. While other causal generative frameworks exist (c.g., diffusion-based causal models evaluated non-facial datasets such as BDD or ImageNet), they either assuine known causal graphs, do not focus on facial attribule editing, or require retraining under different supervision regimes. Our selected baselines therefore represent the most directly comparable methods for facial counterfactual generation under the same evaluation setting.
4.4 Results and Discussions
We presented and discussed both the causal graph results as illustrated in Figure 5 counterfactual explanations images quantitative and qualitative evaluation are concluded in Table and
6 and the generated
Weering
cheek
Attractive
Double
Heavy
Seedding bg
Eye- glasses
Mouth
Sealle
Ond Pace
Blury
Age
Cat
Smile
Car
[+]
[b]
Pale
Toing
Bask
Chis
Deal
Wear,
fr.
Gray
Gray
R_e_dlos
M
Figure 5: Partial Ancestral Graph (PAG) for target attributes "Young" and "Smiling". Blue: larget attribute QF grey: confounders C: white: related attributes I, Bidirectional odges (++) indicate unobserved confounders (e.g. Smiling ++ Mouth Open implies Smiling C. + Mouth Open). See Table for edge interpretations.
Figure 6: CelebA dataset causal relation of the larget variable (in blue) with the PC algorithm. Note that the label in this graph indicates the attribute class type without specifying its actual binary class value in CelebA.
9

---

Published in Transactions on Machine Learning Research (03/2026)
(a) Hervy Makeup
(b) Narrow Eyes
(g)
Smile
Original Ours ACE Original
Ours
ACE
Original
ACE Original Ours ACE
Figure 7: Qualitative counterfactuals results for facial attributes demonstrate the effectiveness of our ap- proach. For "Age", the bascline often generates features correlated with aging, such as eye bags and glasses. However, our causal graph analysis from Figure 5 ence "Age". For "Smile", our method successfully disentangles correlated attributes like "High Cheekbones" and "Opening Mouth", providing more accurate and realistic counterfactual explanations. For additional examples and higher-resolution images, please refer the supplementary material. Zoom for best view.
indicates that "Bags Under Eyes" does not directly influ-
Table 2: Quantitative results for CelebA and CelebAHQ datasets.
CelebA
CelebAHQ
Smile
Age
Smule
Age
Method
FIDL
sFIDI
MNAC|
PVAT
FIDI
=FIDI
MNACL
FVAt
FIDI
sFID MNACL
FVAT
FIDL
sFID]
MNACJ
FVAT
STEEX JJnob el ml.
2022
10.2
4.11
96.9
11.8
3.41
97.5
21.9
5.27
97.6
26.8
5.63
96.0
DIVE
Rodriguis
2021
33.8
4.58
98.2
107.0
7.11
35.7
32.3
DIME
Jeanerret dt al
2122
3.17
4.89
3.72
98.3
4.15
5.89
3.13
95.3
18.1
27.7
2.63
96.7
18.7
27.8
2.10
95.0
ACE
Gronnhor of nl.
2024
1.27
3.97
2.94
90.9
1.45
4.12
3.20
98.6
3.21
20.2
1.56
100.0
5.31
21.7
1.53
90.6
ACE 8g
Jemneerol et
2023
1.90
4.56
2.77
90.9
2.08
4.62
2.44
98.6
6.33
22.0
1.87
100.0
16.4
28.2
1.92
9846
Ours
1.08
3.79
1.63
99.9
1.51
4.10
1.92
99.8
1:27
5.12
1.55
99.9
12.28
18.83
1.06
99.9
4.4.1 Causal Graph Analysis
The learned causal structures are illustrated in Figure 5 (PAG) and Figure 6 (DAG obtained via the PC algorithm). For the query labels "Young" and "Smile" as child nodes: For "Young", the graph includes direct edge from "Gray Hair" to "Young", while "Eyeglasses", "Black Hair" and "Double Chin" appear indirect (upstream) factors. Because "Black Hair" and "Gray Hair" mutually exclusive and may be confounded with age-related appearance, avoid direct interventions these attributes when orientation is uncertain. Accordingly, our masking targets regions associated with "Eyeglasses" and "Double Chin". For "Smile", the graph indicates upstream influences from "Double Chin", "High Cheekbones", and "Gray Hair". We therefore apply masks to the corresponding regions when guiding the generation process, while avoiding edits along edges marked as uncertain or confounded in the PAG.
4.4.2 Quantitative Results
As shown in Table 2 our model consistently outperforms prior methods across both CelebA and CelebAHQ datasots. It achieves the lowest FID and sFID scores in most cases, especially for the "smile" attribute, indicating superior image quality and more realistic feature preservation. Moreover, ours obtains significantly better MNAC scores, highlighting enhanced diversity and causal disentanglement in the generated images. Our analysis reveals that the classifier often relies on spurious correlations learned during training rather than causal relationships, resulting in lower MNAC values for these attributes, limitation also acknowledged by DiME (Jeanneret et al. 2022).
10

---

Published in Transactions on Machine Learning Research (03/2026)
Table 3: Quantitative Assessment for CelebA. H. Makeup and N. Eyes are heavy makeup and narrow eyes attributes.
Table 4: Quantitative bias metrics. Higher DI and lower SPD indicate reduced bias.
CelebA
DI (f)
SPD (1)
Attr. Gender
H. Makeup
N. Eyes
ACE
2.37
0.019
MNAC (4)
Age-HiigChhee
Ours
2.52
0.011
ACE
3.95
5.18
3.44
ACE
6.23
0.159
Ours
3.91
5.18
2.84
Age- Makeup
Ours
6.24
0.152
sFID (t)
ACE
0.019
0.016
ACE
7.56
11.29
4.04
Gender-H.
Ours
0.020
0.015
Ours
7.42
10.98
4.44
4.4.3 Qualitative Results
Figure 7 (more in Appendix D illustrates this, showing how the baseline's counterfactuals for "young" often include ageing cues like eye bags and glasses. Similarly, confounding influences are evident for "smile", with high checkbones and open mouths being incorrectly emphasised. In contrast, our ageing effects appear more realistic. Also, ours generates images with more noticeable heavy makeup than the baseline. In contrast, our approach successfully corrected the classification by generating impactful counterfactuals. Our method integrates causal knowledge to produce realistic, less biased counterfactual images, revealing the classifier's decision process. This enhances interpretability and utility in real- applications where human understanding is crucial. For example, the lines generated by baseline ACE in Figure 7 for age may be confusing to the general public, they represent ageing effects.
We present qualitative results for the "Heavy Makeup" and "Narrow Eyes" attributes in Figure 7 (a) and (b), with additional bias evaluations in the supplementary materials. These attributes, often linked to sensitive factors like gender and race, are not explicitly labeled in the datasct. Although not considered in prior work, we find them important for uncovering potential biases and enhancing the fairness of counterfactual explanations.
4.4.4 Fairness Evaluation
Table 4 presents bias metrics for different attribute pairs evaluated on the CelebA dataset. We used a ResNeXt-50 classifier trained for 35 epochs to validate attribute intensity in generated counterfactual images. Each row examines a specific attribute pair where the first attribute represents the protected group (e.g., Age, Gender) and the second denotes the unprivileged group (e.g., HighCheck, H.Makeup).
We evaluate fairness using two widely-adopted metries: Disparate Impact (DI) and Statistical Parity Difference (SPD).
Disparate Impact (DI): DI measures the ratio of positive outcomes between the unprivileged and protected groups. It is defined as:
DI = Pr(f = 1| A unprivileged)
(13)
Pr(ý = 1 protected)
where denotes the predicted attribute, and A represents the group membership. A D1 value closer to 1 indicates equitable treatment, with higher values suggesting reduced bias against the unprivileged group.
Statistical Parity Difference (SPD): SPD quantifies the difference in positive prediction rates between the protected and unprivileged groups:
SPD = Pr(Y = A= protected) Pr(Y = A unprivileged).
(14)
Lower absolute values of SPD indicate smaller disparities, with SPD = representing perfect statistical parity.
We compare against ACE as it achieves the strongest performance on realism metrics among all basclines (Table 2 Other baselines such DiME and STEEX exhibit. substantially weaker image quality (FTD
11

---

Published in Transactions on Machine Learning Research (03/2026)
> 3 for DIME, 10 for STEEX), and evaluating fairness lower-quality counterfactuals would conflate generation quality issues with fairness metrics, making interpretation difficult.
Connection to Stereotype Bias Reduction: Stereotypes arise when spurious correlations are treated as causal relationships. Our method reduces stereotype bias through three mechanisms:
1. Causal disentanglement: The learned PAG identifies direct causes versus spurious correlations. For example, in Figure 7 ("age" column), baselines add "Double Chin" when aging (a correlated but not causal attribute), while our PAG shows Double_Chhie t Age indicates confounding so wo exclude this region from modification.
2. Mask-guided intervention: We only modify regions corresponding to direct causal parents. If "Smiling" and "Female" are correlated but not causally related, our mask excludes gender-associate regions, preventing inadvertent. feminization (see Appendix E
3. Quantitative evidence: Table 4 shows improved DI (closer to 1.0) and SPD (closer to 0.0) com- pared to baselines.
By filtering correlation from causation only direct causal edges (+) are included while all other edge types are exchuded-ond our PAG prevents stereotyped attributes from propagating into counterfactuals.
Metrics for Measuring Stereotype Bias: DI and SPD directly quantify group fairness disparities. Ad- ditionally, MNAC (Mean Number of Attribute Changes) indirectly measures stereotype bias: lower MNAC indicates the model does not automatically add stereotypically correlated attributes when modifying the target attribute. Our method achieves MNAC of 1.63 compared to 2.94 for baselines without causal con- straints.
Our method consistently achieves lower SPD values and comparable higher DI values across all evaluated attribute pairs compared to the ACE baseline. This demonstrates that approach effectively mitigates bias by reducing prediction disparitics between different demographic groups while maintaining equitable performance across altributes in counterfactual explanations. The improvement stems from our causal disentanglementt by learning the PAG and restricting edits to direct causal parents only, we prevent spurious correlations (which often encode societal biases) from propagating into the generated counterfactuals. In other words, generated counterfactuals using our method exhibit fairer distribution of attribute changes, ensuring that the protected and unprivileged groups treated more equitably.
4.4.5 Logical Interpretation
To quantitatively assess the semantic alignment between images and their cor- responding text descriptions, we report two evaluation metrics (Radford et al.
Table CLIP-based semantic alignment evaluation comparing FaCER with SOTA baseline.
Category Method
Mean Sim.
R@1 (%)
2021 Faghri et al., 2018): : 1ecall@1 (R@1) and mean similarity. R&1 mea- sures whether the intended attribute change is the top-detected semantic change, reflecting how precisely the coun- terfactual targets the desired modifica-
ACE
6.22
49.09
Smile
FaCER (Ours)
5.91
72.95
ACE
4.45
62.98
Age
FaCER (Ours)
5.03
61.34
tion. Mean similarity denotes the average CLIP similarity score between each image and its descriptive caption, measuring overall semantic alignment (see example in Appendix
We use CLIP [Radford el al. 2021] as our vision-language model due to its open-source availability, compu- tational efficiency, and status as the standard embedding space for Aision-language aligument tasks. Unlike prior work that applics CLiP-based interpretation arbitrary attribute changes, our approach grounds these explanations in the learned causal structure-oni direct causes identified by FCI are modified, ensuring that semantic interpretations reflect causally meaningful edits rather than spurious correlations.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5 compares FaCER against ACE, the strongest baseline realism metrics. For Smile, FaCER achieves substant ially higher R@1 (72.95% vs. 49.09%), indicating that our counterfactuals more precisely target the intended attribute as the primary semantic change. For Age, results are comparable (61.34% vs. 62.98%). Notably, for Smile, FaCER achieves higher R@1 despite a lower mean similarity score (5.91 vs. 6.22). This combination suggests that our method produces more focused, allribule-specific modifications: the counter- factual changes are concentrated on the target attribute rather than introducing broader modifications that may inflate similarity scores while affecting unintended attributes. This behaviour aligns with our goal of generating minimal, interpretable counterfactual explanations- edits should change only what is causally necessary.
5 Conclusion
We introduced a framework for cansal editing of facial images that unifies causal discovery, mask-guided counterfactual generation, and semantic interpretation. Motivated by the need for interpretable and causally grounded visual reasoning, our approach employs the Fast Causal Inference (FCI) algorithm to uncover re- lationships among facial altributes and uses spatially informed masks to guide a diffusion-based generator, ensuring that only causally relevant regions are modified. Experiments on CelebA and CelebA-HQ show that method produces realistic, sFID and MNAC metrics. The inlegration of CLIP-based semantic explanations further enhances inter- pretability and supports fairness analysis. While our framework assumes reasonably accurate causal graphs, which may not always hold in noisy real-world settings, it provides a principled foundation for interpretable generative modelling. Future work will address uncertain causal structures and feature-level alternatives, and extend our framework broader domains such as medical imaging and scene understanding.
counterfactuals and surpasses statc-of-thc-art basclines
Broader Impact Statement
Our work introduces a causal mask-guided framework for generating counterfactual facial images to improve interpretability and fairness evaluation in machine learning. While such methods can help reveal causal relationships and reduce reliance on spurious correlations, techniques for editing facial images may also pose risks of misuse, such as generating misleading manipulated visual content. Additionally, biases in the underlying datasets may still influence generated counterfactuals. Therefore, this work is intended primarily for rescarch on interpretable and fair machine learning, and its use should consider appropriate safeguards and responsible ment.yment.
Acknowledgement
This work was supported in part by the Uxpress Project, funded by the Frader-Willi Syndrome Association Victoria & Monash Faculty of Information Technology, Australia, and by the Æinstein Project funded by Petronas Research.
References
Saharsh Barve, Andy Mao, Jiayue Melissa. Shi, Prerna Juneja, and Koustuv Saha. Can we debias social stereotypes in ai-generated images? examining text-to-image outputs and user perceptions. orXiv preprint arXiv:2505.20692, 2025.
Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily accessible text-to-image generation amplifies demographic stereotypes large scale. In Proceedings of the 2023 ACM conference on fairness, accountability, and transporency, pp. 1493-1504, 2023.
Qiong Cao, Li Shen, Weidi Xic, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A datasct for recognising faces across pose and age. In IEEE International Conference on Automatic Face & Gesture Recognition, 2018.
13

---

Published in Transactions on Machine Learning Research (03/2026)
Fabio De Sousa Ribeiro, Benoît Ding, Grzegorz Kruszewski, and Julia Kantorovitch. Structural causal interventions on generative models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 7745 7763. PMLR, 2023. URL
https:/ haapss///arxv...o http//aavvvv....raa.. hwwg/abb/2306..... 2206 08276
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. +se++: Improving visual-semantic embeddings with hard negatives. 2018. URL
Yash Goyal, Ziyan Wu, Jan Erust, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pp. 2376 2376-2384. PMLR, 2019.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akala. Grounding visual explanations. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Paul Jacob, Éloi Zablocki, Hedi Ben-Younes, Mickaël Chen, Patrick Pérez, and Matthieu Cord. Steex: steering counterfactual explanations with semantics. In European Conference on Computer Vision, pp. 387-403. Springer, 2022.
Guillaume Jeanneret, Loic Simon, and Frédéric Jurie. Diffusion models for counterfactual explanations. In Proceedings of the Asian Conference on Computer Vision, pp. 858-876, 2022.
Guillaume Jeanneret, Loic Simon, and Frédéric Jurie. Adversarial counterfactual visual explanations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognation, pp. 16425-16435, 2023.
Yixue Kang, Yuchen Guo, Zhouxing Ren, Feng Zhang, Simon Dib, Vincent Y. F. Lee, and Nicu Sebe, Causal inference-based explanations for fairness in facial image classifiers. Sensors, 22(2):654, 2022. URL htttps://ww. mdpi.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pallern Recognition (CVPR), 2020.
Na Liu, Fan Zhang, Liang Chang, and Fuqing Duan. Seattcring-based hybrid network for facial attributc classification. Frontiers of Computer Science, 18(3):183313, 2024.
Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Compuler Vision (ICCV), December 2015.
Pratik Mazumder and Pravendra Singh. Protected attribute guided representation learning for bias mitiga- tion in limited data. Knowledge-Baad Systems, 244:108449, 2022.
Alexander Melistas, Nicolas Mayer, Toshihiko Takeuchi, and Severin Suter. Causal inverse graphics: Context- aware modular causal counterfactual generation. In Advances in Newral Information Processing Systems, 2024. URL
Goran Paulin and Marina. Ivasic-Kss. Review and analysis of synthotic dataset generation methods and techniques for application in computer vision. Artificial Intelligence Review, 56(9):9221 9265, 2023.
Bo Peng, Siwei Lyu, Wei Wang, and Jing Dong. Counterfactual image enhancement for explanation of face swap deepfakes. In Shiqi Yu, Zhaoxiang Zhang, Pong C. Yuen, Junwei Han, Tieniu Tan, Yike Guo, Jianhuang Lai, and Jianguo Zhang (eds.), Pattern Recognition and Computer Vision, pp. 492-508, Cham, 2022. Springer Nature Switzerland.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamcla Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Rajat Rasal, Avinash Kori, Fabio De Sousa Ribciro, Tian Xia, and Ben Glocker. Diffusion counterfactual generation with semantic abduction. In ICML, 2025.
Pau Rodriguez, Massimo Caccia, Alexandre Lacoste, Lee Zamparo, Issam Laradji, Laurent Charlin, and David Vazquez. Beyond trivial counterfactual explanations with diverse valuable explanations. In Pro-
ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1056-1065, 2021.
Pedro Sanchez and Sotirios A. Tsaftaris. Diffusion causal models for counterfactual estimation. First Conference on Causal Learning and Reasoning, 2022. URL httpss//ppareerree.n////oouuue
LAAZLZIMN-O
Dylan Slack, Himabindu Lakkaraju, Julie Shah, and Sorelle A. Friedler. Counterfactual explanations can be manipulated. In Advances in Neural Information Processing Systems, volume 34, pp. 19511-29523, 2021.
Peter Spirtes. An anytime algorithm for causal inference. In Thomas S. Richardson and Tommi S. Jaakkola (eds.), Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, volume R3 of Proceedings of Machine Learning Research, pp. 278 285. PMLR, 04 07 Jan 2001. URL https://
proceedings mlr. pttss//33/pprrtestt... html Reissued by PMLR on 31 March 2021.
Peter Spirtes, Clark N Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2000.
Diego Velazquez, Pau Rodriguez, Alexandre Lacoste, Issam Laradji, Xavier Roca, and Jordi Gonzàlez. Explaining visual counterfactual explainers. Transoctions on Machine Learning Reseorch, 2023.
Lan Wang and Vishmu Naresh Boddeti. Do learned representations respect causal relationships? In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 264-26 274, June 2022.
Yulun Wu, Louie McConnell, and Claudia Triondo. Counterfactual generative modeling with variational causal inference. In ICLR, 2025.
Tian Xu, Jennifer White, Sinan Kalkan, and Hatice Gunes. Investigating bias and fairness in facial expression recognition. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part VT 16, pp. 506-523, Springer, 2020.
Yujia Zheng, Biwei Huang, Wei Chen, Joseph Ramsey, Mingming Gong, Ruichu Cai, Shohei Shimizu, Peter Spirtes, and Kun Zhang. Causal-learn: Causal discovery in python. arXiv preprint or arXiv:2307.100 164 2023.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Appendix
A Causal Graph Validation
We provide additional validation experiments for the learned causal graph as requested by reviewers.
A.1 Synthetic Ground-Truth Recovery
To verify that our causal discovery pipcline correctly identifics known causal structures, we constructed a synthetic Structural Causal Model (SCM) with ground-truth edges. Both PC and FCI algorithms correctly recover the adjacency structure (precision 1.0, recall = L.0), confirming that our implementation faithfully identifies causal relationships when the true structure is known.
A.2 Bootstrap Stability Analysis
To assess the stability of learned causal structures, performed 500 bootstrap runs CelebA. Table G reports the frequency with which each candidate parent appears in the learned graph for selected target attributes.
Table 6: Bootstrap stability analysis 500 runs on CelebA. Frequencies indicate how often each candidate parent appears in the learned causal graph.
Target
Candidate Parents
Bootstrap Freq.
Gray Hair
28%
Young
Bald
22%
Allractive
14%
Smiling
High
Cheekbones
30%
The moderate frequencies reflect genuine structural uncertainty due to latent confounding precisely what motivates our use of Partial Ancestral Graphs (PAGs) learned from FCI rather than Directed Acyelic Graphs (DAGs) Icarned from algorithms such as PC. As shown in Figure of main paper, relations such as Smile + U2 + Mouth Open indicate latent confounders that make parent sets only partially identifiable. This is not limitation but an accurate representation of causal ambiguity inherent in observational data.
Importantly, implausible parents (c.g., Wearing Hat Smiling) never appear in any bootstrap run, con- firming that our approach appropriately filters spurious relationships.
A.3 Algorithm Sensitivity
We evaluated sensitivity to the choice of causal discovery algorithm and hyperparameters:
Algorithm comparison: FCI and PC show high structural agreement (Jaccard similarity 0.95).
CI-test scnsitivity: Results are stable across different conditional independence tests (x2; G²).
Significance level: Structure remains stable across a E {0.01, 0.13,
This consistency confirms that our findings are not artifacts of specific algorithmic choices.
A.4 Ablation: Impact of Causal Graph
To isolate the contribution of causal knowledge, we compare our full method against. a version without the causal graph (equivalent to ACE with & regularization).
Without the causal graph, our method reduces to baseline performance. The substantial increase in MNAC
(1.63 + 2.94) demonstrates that causal knowledge not masking alone drives our improvements in attribute
16

---

Published in Transactions on Machine Learning Research (03/2026)
Table 7: Ablation study on the contribution of the causal graph.
Configuration
FID
MNAC
With causal graph (Ours)
1.08
1.63
Without causal graph (ACE (1)
1.27
2.94
specificity. This ablation directly confirms that the learned causal structure is essential for generating minimal, largeted counterfactual explanations.
B Hyperparameter Sensitivity Analysis
B.1 Mask Threshold A
The mask threshold À controls which pixels are included in the editable region based on the difference between the reference and preliminary images (Equation main paper). We set 0.1 (normalized pixel intensity threshold) based on the sensitivity analysis presented Table
Table 8: Sensitivity analysis for mask threshold À on CelebA (Smile attribute). Lower À produces more focused masks, while higher À lcads excessive modifications.
A
FID 4 MNAC 4
0.10
1.08
1.63
0.50
1.21
2.32
1.00
1.68
3.63
As shown in Table g increasing À leads to progressively worse performance both metrics. With À == 1.0, MNAC increases from 1.63 to 3.63, indicating that the model modifies more than twice as many attributes as necessary. This occurs because a higher threshold allows more pixels to be included in the mask, leading to unnecessary modifications in regions unrelated to the target attribute. The degradation in FID (1.08 + 1.68) suggests that these excessive edits also harm image realism.
We select 0.1 as achieves the best balance between targeted attribute modification (lowest MNAC) and image quality (lowest FID). This value ensures that only pixels with substantial differences between the reference and preliminary images are included in the mask, effectively isolating the regions most relevant to the target attribute change.
B.2 Complementary Relationship between FID and sFID
We use both FID and sFTD complementary metrics for evaluating counterfactual realism, as they capture different aspects of image quality.
FID (Fréchet Inception Distance) compares the full feature distributions between real and generated images, including feature covariances. This makes FID sensitive to incoherent attribute combinations generated image contains an implausible combination of fcatures (c.g., young face with gray hair), FID will detect this a deviation from the real data distribution. FID thus provides holistic measure of overall image realism and distributional faithfulness.
sFID (Scaled Fréchct Inception Distance) complements FID by measuring per-feature realism without considering cross-feature covariances. This is useful for assessing whether individual attributes are rendered realistically, independent their combinations. sFID can reveal issues specific feature dimensions that might be masked by the aggregate FID score.
The wwo metrics serve complementary purposes:
FID captures global
the overall image and attribute combinations are realistic
and in-distribution.
17

---

Published in Transactions on Machine Learning Research (03/2026)
sFID captures locol fidelity heetheey-whether
whether individual features are rendered with high quality.
By reporting both metrics, we provide more complete picture of counterfactual quality. A method with good FID but poor sFID might produce globally coherent but locally degraded images, while method with good sFID but poor FID might render individual features well but combine them in implausible ways. Our method achieves strong performance on both metrics, indicating that the generated counterfactuals are both globally coherent and locally realistic.
C Understanding Attribute Correlations
30
Dan
Hkgn, (hhviheeee
Revv pair
Bowy Cheeies
Yearns Carringe
Vewy, lat
Hesky_Mkkcxp
Eyoplorses
Chubbty
Weuring_
3, 3.e_clox shadow
de oa mim Semmon
Figure 8: Dendrogram and correlation heatmap for the CelebA auribute classes. The intensity of color (blue or red) indicates the strength of correlation between attribute pairs. Hierarchical clustering shows which attributes frequently co-occur in the dataset.
As shown in Fig. we visualize the correlation patterns between different facial altributes in CelebA using hierarchical clustering dendrogram and heatmap. The dendrogram reveals how attributes cluster based correlation patterns, with shorter branches indicating higher similarity. However, it is crucial to emphasize that these correlations do not imply causation.
For example, consider two variables X and Y with correlation coefficient r = cov(X,Y) Despite a strong correlation value, this does not necessarily mean that changes in X cause changes in 8 or vice versa. Other factors, including confounding variables, may be responsible for the observed relationships.
In our analysis, we found that existing methods consistently associate "smiling" with high chockboncs and open mouth due to dataset correlations, limiting the diversity of generated images. Our causal relationship- aware model effectively mitigates such biases, producing more realistic facial attribute manipulations.
18

---

Published in Transactions on Machine Learning Research (03/2026)
C.1 Addressing Class Imbalance in CelebA
Cierk shadow
Arched Evairces
Under Eves
Hooe
Blond liar
Blany
Busty Bythican
Cuulity
Cysglaseo
Costos
Honsy Makoou
High Choolbaree
Mouth signtty
Mustach
Beand
Oval Joce
Porny Rooe
Recoding Hullfs
smiling
Shaight
Wosth gasiing
Wear
Weattng beddine
-200k
-10k
-000
-50
23k
100k
A5Ck
Count
Figure 9: Class distribution for facial attributes the CelebA dataset, highlighting significant imbalances
across many attribule categories.
The CelebA dataset exhibits substantial class imbalance across different attributes, shown in Fig. This visualization highlights how altributes such 'Smiling" have disproportionate sample distributions bet ween positive and negative classes. Such imbalances can significantly impact model performance, particularly when generating images with underrepresented attributes. Our method addresses this challenge by incorporating causal relationships to generate diverse images with balanced attribute intensities, improving both quality and diversity.
Addressing Perceptual Interpretability While counterfactual explanations are crucial for understanding model behavior, their effectiveness hinges their interpretability humans. Existing methods often struggle in this regard, with high flip rates resulting in imperceptible nonsensical changes to human observers.
Our approach focuses on enhancing the perceptual interpretability of counterfactual explanations by priori- tizing changes that are more noticeable and coherent to humans. FaCER identifies and emphasizes features salient human perception, such altering the hairstyle or adding/removing glasses. By doing so, we ensure that the counterfactual images technically valid, visually compelling, and aligned with human expectations.
D Additional Evaluation Results
We compare our method with the state-of-tho-art approach Jeanneret et al. (2023) on both ColebA and CelebAHQ datasets. The results in Figures 10 and 13 demonstrate that our method generates more realistic images with diverse attributes such as smiling and young. The difference maps highlight how our method makes more targeted changes relevant facial features while preserving unrelated attributes. Our approach effectively captures the attribute intensity and diversity in the generated images, enhancing both realism and quality compared to existing methods.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Qualitative comparison on CelebAHO for the smiling attribute, From left to right: (1) Original image, (2) Baseline method result, (3) Difference map between basclinc and original, (4) Our method result, (5) Difference map between our method and original. Note how our method generates more realistic attribute manipulations with fewer unintended changes other facial features.
Figure 10:
Figure 11: Qualitative comparison on CelebAHQ for the young attribute. From left right: (1) Original image, (2) Baseline method result, (3) Difference map between baseline and original, (4) Our method result, (5) Difference map between method and original. Our method better preserves identity features while realistically altering age appearance.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Qualitative comparison on CelebA for the smiling attribute. From left to right: (1) Original image, (2) Baseline method result, (3) Difference map between baseline and original, (4) Our method result, (5) Difference map between our method and original. Our approach generates more natural smiles with fewer artifacts.
Figure 12:
21

---

Published in Transactions on Machine Learning Research (03/2026)
Figure 13: Qualitative comparison on CelobA for the young attribute. From left to right: (1) Original image, (2) Baseline method result, (3) Difference map between baseline and original, (4) Our method result, (5)
Difference map between our method and original. Our method produces more consistent age transformations while maintaining facial identity.
E
Stereotype Bias Reduction: Qualitative Analysis
This section provides qualitative evidence demonstrating how our method reduces stereolype bias in coun- terfactual generation through causal disentanglement and mask-guided intervention.
E.1
Mechanisms for Stereotype Reduction
Stereotypes in generative models arise when spurious correlations in training data are treated causal relationships. For example, if "Smiling" and "Female" correlated in the dataset (but not causally related), naive counterfactual method might inadvertently feminize face when adding a smile. Our method addresses this through three mechanisms:
1. Causal Disentanglement via PAG. The learned Partial Ancestral Graph (PAG) distinguishes be- tween direct causal relationships and spurious correlations:
Direct causal edges (+): Indicate attributes that should be modified together (c.g., Gray Hair Age).
22

---

Published in Transactions on Machine Learning Research (03/2026)
Bidirectional edges (+)) Signal confounding due to latent variables these regions excluded from modification to prevent stereotype propagation.
2. Mask- Guided Intervention. We modify only regions corresponding to direct causal parents DC(Qi) of the target attribute. Regions associated with confounded or correlated (but non-causal) attributes are preserved, preventing unintended changes that reflect dataset biases rather than true causal effects.
Quantitative Validation. Table 4 in the main paper shows that our method achieves improved fairness metries (higher DI, lower SPD) compared basclines, confirming that causal constraints reduce bias in practice.
3.
E.2 Case Study: Gender Attribute
Figure E Iillustrate how our method handles the sensitive "Gender" attribute compared to the ACE baseline.
Original
Ours
ACE
W IIIIIa
michan WT mimma
Figure 14: Qualitative comparison for the Gender attribute. Red boxes highlight alterations in facial expressions affecting classifier decisions. Row 3: ACE fails to change the classification, while our method succeeds by focusing on causally relevant regions. Our approach avoids modifying spuriously correlated attributes (e.g., "mouth opening"), producing more meaningful and less biased explanations for sensitive attributes.
Observations:
Bascline (ACE): The classificr's judgment is influcnced by irrelevant facial attributes such as "opening mouth." the third row, ACE fails to alter the classification result because modifies regions that are correlated with but not causally related to gender.
Ours: By using the learned PAG to identify direct causes of the "Gender" attribute, our met hod modifies only causally relevant regions. This produces counterfactuals that successfully change the class while avoiding sterootype-rcinforcing modifications.
Why This Matters for Fairness: When generating counterfactual explanations for sensitive attributes like gender, it crucial to avoid reinforcing societal stereotypes. method associates "smiling" with "female" (a spurious correlation in many datasets), it may inadvertently:
23

---

Published in Transactions on Machine Learning Research (03/2026)
Orghnflmece
Contanfactaa
Orginal Irige
Conterlettelingg
To Chart Miticd
Figure 15: Examples of CLIP-based similarity scores between facial images and descriptive lext expressions before and after counterfactual modifications ("Smile": above, "Age": below). Higher similarity scores for relevant facial expressions the modified images demonstrate the targeted nature our transformations.
Add smiles when changing gender to female, or Remove smiles when changing gender to male.
Such behavior reflects dataset bias, not causal relationships. Our PAG-based approach identifies that Smiling + Gender (bidirectional edge indicating confounding), and therefore excludes smile-related regions from the gender intervention mask. This ensures that gender counterfactuals modily only genuinely gender-related features, producing fairer and more interpretable explanations.
E.3
Case Study: Age Attribute
Similar principles apply to the "Age" attribute (see Figure in the main paper):
Stercotype example: Baselines often add "Double Chin" when aging face, reflecting correlation in the training data.
Our approach: The PAG shows Double Chin <> Age (bidirectional edge), indicating confounding rather than direct cansation. We therefore preserve the chin region and modify only direct causes such as hair color (Gray Hair + Age).
This causal filtering prevents the propagation of age-related stereotypes while still producing effective coun- terfactual explanations.
E.4 Summary
Our method reduces stereotype bias by:
24

---

Published in Transactions on Machine Learning Research (03/2026)
1. Learning the causal structure among facial attributes via FCI
Identifying and excluding confounded relationships (bidirectional edges)
3. Restricting modifications to direct causal parents only
Validating improvements through fairness metrics (DI, SPD) and qualitative inspection
This principled approach ensures that counterfactual explanations reflect true causal relationships rather than spurious correlations that encode socictal biases.
F Instances of CE in Language via CLIP
To exlend the usability of our counterfactual explanations, we employ CLIP to quantify semantic alignment between images and descriptive text. This approach allows us to assess whether the changes introduced by our method are perceptible and scmantically meaningful by comparing embedding similaritics. 1 presents examples of CLIP-based uextual alignment. scores before and after counterfactual modifications. These visual comparisons illustrate how approach effectively alters specific facial attributes while maintaining overall image coherence.
G Contribution to Downstream Task
In the synthesis training data perspective. (1) The increased realism of these counterfactuals generated by FaCER helps train more robust. and accurate classification models. (2) Realistic counterfactuals provide meaningful variations of the data, helpful in causal inference scenarios that align better with the true un- derlying distribution, thereby enhancing the generalization capability of the classifier Paulin & Ivasic-Kos (002). The debiased results in the classification are also discussed. Diverse yet realistic training sets can improve classification by providing more exposure and knowledge of underrepresented groups in the real world.
25


Reviews:
[{"review_id": "AhhXCtIHTG", "reviewer": "Reviewer_aA6Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a way to generate facial \"counterfactual\" images --- i.e., images that answer \"what would this same face look like if attribute X were different\" --- while trying to keep the edits realistic and causally consistent. The main idea is to learn a causal graph over facial attributes, then use it to build spatial masks that constrain a diffusion-based edit so only regions tied to the direct causal factors are modified, and finally produce a CLIP-based text explanation of what changed.\n\n## Strengths\n### 1. Clear, coherent \"causailty + generation\" story\n* The paper presents a systematic study, which leans a causal graph over facial attributes, and then use it to constrain diffusion-based editing via a mask, and finally adds CLIP-based text explanations. This end-to-end framing is easy to follow and aligns the components toward \"causally grounded\" counterfactuals.\n\n### 2. Clear motivation over \"true causes\" vs \"correlates/confounders\"\n* This work highlights an important practical issue: counterfactual generators may exploit **spurious corrections** (e.g., adding correlated \"aging cues\" when editing \"young/age\"), which stresses the necessity of causal graph modeling.\n\n### 3. Quantitative improvements on standard benchmarks\n* This work reports best FID/sFID and lower MNAC than existing baselines on CelebA / CelebA-HQ dataset.\n\n### 4. Attempts at interpretability beyond images\n* Though simple, I really favor the design of CLIP-based explanation module which answers \"what changed\". This interepretability is especially helpful in image editing.\n\n## Weakness\n### 1. The central dependency - the learned causal graph - is not sufficiently validated\n* This work does not provide strong evidence that the discovered PAG is correct (beyond qualitative plausibility and a comparison to PC). Since the *claimed benefit* relied on \"direct causes vs confounder\", the lack of graph validation is a majot risk.\n\n### 2. Figures are low-res not clearly perceivable\n* Many figures (especially faces) are in low resoluation, making it hard to observe the editting differences. The color scheme in figure 4 is also not read-friendly.\n\n### 3. The \"causal intervention\" is somewhat heuristic / loosely connected to causal semantics\n* The mapping from \"direct causal parents\" (attribute space) to \"corresponding facial region\" (pixel space) is not formally interpreted. It relied on the observed diff map and heuristic region association. This makes it hard to argue that the method truly performs a causal do-intervention rather than a constrained edit guided by a graph-shaped prior.\n\n### 4. Evaluation scope is narrow, which limits claims of generality\n* The primary quantitative tables focuses on **smile** and **age**. There is some extra qualitative evaluation regarding gender, makeup and narrow eyes, which however remains a small subset of CelebA dataset.\n\n### 5. Metric choices leave important properties under-measured\n* It's unconvincing to to me that metrics such as identity/face-similarity metrics are not used for evaluation. How could the authors ensure that identity are not changed?", "claims_and_evidence_yn": "No", "claims_explanation": "Partially supported. \nThe submission provides clear quantitative evidence for improved **image-quality / sparsity metrics** on the tested settings, but the evidence is **weaker or incomplete for the stronger claims around causal correctness, identity preservation, fairness, and explanation quality/generalization**.\n\nThe paper defines the metrics clearly, but does not clearly specify key evaluation protocol details such as how many images were used for FID/MNAC, confidence intervals.\n\nThere is no direct validation that the learned PAG is correct (and the method heavily depends on it).\n\nThe conclusion explicitly claims identity preservation. But in the evaluation section they state FS/FVA are not the primary focus and only claim “qualitatively strong performance” on them, without reporting numbers.", "audience_yn": "Yes", "audience_explanation": "Yes—researchers in causal ML, counterfactual explanations, and diffusion-based editing would be interested.", "requested_changes": "1. Causal graph reliability: Can the authors validate key edges (e.g., via synthetic data with known ground-truth graph, or perturbation-based checks)? How robust is performance to graph perturbations or alternative discovery methods?\n2. Ablations: What happens if you remove the graph refinement and use only the diff-based mask? Or keep the graph but randomize/permute parent sets? This would isolate how much gain comes from “causal knowledge” vs masking/inpainting heuristics.\n3. Identity preservation: Either report FS/FVA (even if not primary) or provide a stronger quantitative identity metric, since the claim appears in the conclusion but is not rigorously measured.\n4. Please also refine the figure for better readability.", "broader_impact_concerns": "The topic is relevant and could benefit auditing/interpretability, but the face-editing capability introduces non-trivial misuse and fairness/over-trust risks.", "additional_comments": "I'm not an expert in image editing. I'm holding low confidence of my review.", "extra_scores": {}, "extra_text": {}}, {"review_id": "yS86SbOfoX", "reviewer": "Reviewer_4Y3G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose an approach to counterfactual generation for images, especially considering facial images. They leverage results from graph theory and previous works on counterfactual generation to build a more fair and better performing model. The conceptual idea is based on previous works using DAGs in counterfactual generation generated from an In addition, the authors provide the idea of CLIP supported labeling to enhance interpretability and understanding of the modified facial features. \n\nThese propositions are then evaluated empirically by looking at the FID, sFID and MNAC scores to evaluate the quality of the counterfactual generation. Further, fairness is evaluated for the proposed method and one baseline.", "claims_and_evidence_yn": "No", "claims_explanation": "While the theoretical parts of the work are presented sound, there is no proof given on why this method should work better than previous methods. Since the addition of DAGs to generate counterfactual images is somewhat novel it would be interesting to have an indicator of why this method works better than previous methods. However, the approach is simply stated. In addition, the labeling with CLIP has been seen in other works or even pure LLM prompting has been applied to provide explainability. \n\nFurthermore, the experimental results are lacking detailed comparisons to baselines and even to current baselines. Only for the score evaluation the baselines stated in the paper have been evaluated. The comparison of fairness under the bias DI and SPD metrics is only given for the ACE baseline and the proposed model. Further, no detailed description of the experiments and no hyperparameters are given, making the experiments hard to reproduce. \n\nSee also the requested changes for a list of current weaknesses in the paper.", "audience_yn": "Yes", "audience_explanation": "The findings of this paper include counterfactual generation using DAGs, which could be of interest to both vision and graph learning communities. Especially since facial image generation was subject to significant improvements in the last few years, counterfactual generation plays an important role for future research. Nonetheless, interest has been rather low in recent years with only few works in this area.\n\n In addition, the highlighted results indicate an improved performance over previous baselines. However, these results remain questionable as outlined above. Therefore, with the current lack of both theoretical and empirical guidance given by these results the interest of the TMLR community would be rather low.", "requested_changes": "Considering the reasons lifted above I would request the following changes:\n\nCritical:\n1. Include current SOTA baselines for comparison. The selected baselines are not very recent and further models have been developed since 2023 such as [1] and [2]. Therefore, it is crucial to compare the proposed model to recent baselines and outline why the inclusion of the DAG should give an advantage. \n\nMajor:\n\n1. Explain or provide proof for the selection of FCI over more recent methods as for example listed in [3]. The explanation given in the paper is not very extensive and remains somewhat unclear given the existence of many different methods for the generation of the DAG. \n\n2. Evaluation of other baselines for fairness metrics. Since the topic of fairness and bias is mentioned significantly in the paper the current comparison to a single baseline is lacking for a comprehensive study. In addition all baselines are only evaluated for the model scores and not for the remaining metrics showcased in section 4. \n\nMinor:\n\n1. There is a lot of related work still missing in section 2. Since there exist more recent contributions to the topic of counterfactual vision models it would be good to discuss and/or add these models to the related works section. \n\n2. It would be nice to see the results obtained on additional datasets in order to evaluate the fairness and the scores of models better since both datasets consist of only celebrity faces. \n\nQuestions: \n\n1. Why is DDPM used to generate the counterfactual. Would methods such as flow matching also work for this type of task? \n\n2. How does the approach differ from previous guided diffusion approaches, except for the usage of a DAG to generate the counterfactual?\n\n3. Were other models than CLIP evaluated for the generation of text for interpretability?\n\n4. A minor question: Is there a reason for torch version 2.0.1 in the experiments?\n\n[1] Diffusion Counterfactual Generation with Semantic Abduction, Rasal et al., ICML 2025\n\n[2] Counterfactual Generative Modeling with Variational causal Inference, Wu et al., ICLR 2025\n\n[3] https://arxiv.org/pdf/2305.10032", "broader_impact_concerns": "Since there is no broader impact statement section in either the main paper or the appendix I feel like it is important to note that for topics such as image editing with people and fairness evaluation there should be at least a short broader impacts statement highlighting the risk of abuse of such methods. However, this statement could be rather general since the authors simply propose a novel version of an existing type of algorithm.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IIwwTy3ePW", "reviewer": "Reviewer_5b6j", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a framework for causal editing of facial images that unifies causal discovery, mask-guided counterfactual generation, and semantic interpretation. Its main ideas and contributions can be summarised as follows:\n-  The method learns a causal graph of facial attributes using the Fast Causal Inference (FCI) algorithm. This helps identify which attributes are true causal factors, which are indirect effects, and which are confounders.\n-  Based on this graph, the method builds spatial masks that guide a diffusion model so that only causally relevant regions of the face are edited. This avoids unrealistic or biased changes caused by correlated but non-causal features.\n-  The framework also uses CLIP to translate visual edits into short semantic descriptions, helping users understand which attributes changed and why.\n-  Experiments on CelebA and CelebA-HQ show that the method generates high-quality, identity-preserving counterfactuals and outperforms prior work on realism, sparsity, and causal consistency.\n-  The approach also reduces bias compared to standard counterfactual generation methods that rely only on correlations.\n-  The authors acknowledge that their method depends on reasonably accurate causal graphs and suggest that future work should handle uncertain causal structures and extend the framework to other visual domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The assertions contained in the paper are very well substantiated. The critical analysis of existing work provides a sound justification for the proposed methodology, which is quite convincing. However, certain sections need to be more precise and clear in order to improve the paper's rating on this criterion.", "audience_yn": "Yes", "audience_explanation": "The work and solutions proposed in this paper are timely and provide an interesting solution for generating facial counterfactual images.", "requested_changes": "The paper is generally well structured and well organized: the section titles are precise, each part is clearly summarized, the context is well explained, and previous work is thoroughly reviewed, which helps position the paper’s contribution within the literature.\nBelow is a list of questions and suggestions:\n\n* Why is the approach of Kang et al. (Kang et al., 2022), mentioned in the literature review, not included as a competitor?\n* The section *“Stereotyping in Generative Models”* clearly explains the importance of avoiding bias in generated images. However, how does the proposed method address or reduce this stereotype bias in practice?\n* Introduction of FCI: the reference needs to be added (or the term “existing” should be used).\n* Figure 2 is not explained in the main text.\n* Equation (1): the Chi² distance can only be interpreted if the degrees of freedom are known. Likewise, different Chi² values are only comparable when the degrees of freedom are the same. The p-value also depends on the degrees of freedom. Why not use the Chi² statistic normalized by the degrees of freedom (such as Cramer’s V)?\n* Subsections 3.2 and 3.3 use notations that are not defined beforehand, which makes the presentation harder to follow. For example, in Section 3.2 we have (x_t), (t), (I_n^{\\text{prelim}}), etc.\n* Subsection 3.3 is a key step but is not very clear: it is not obvious whether the (I_n^{\\text{prelim}}) are generated multiple times to identify the causal features involved in the target attribute change. How does (Q_t) appear in Equation (3)?\n* The mask definition seems overly simplistic. For instance, if a change affects a region unrelated to the target attribute, the indicator will still be 1. Since the goal is to isolate modifications that influence the target attribute, why not use feature variations rather than directly comparing images?\n* Equation (3): how is (\\lambda) chosen? Is there a sensitivity or impact analysis?\n* Clarify the connection between Subsection 3.3 and Figure 2, including the color coding (red, grey).\n* Subsection 4.2 is unclear because it states “three complementary metrics” in the *Realism of Generated Counterfactuals* paragraph, but only two are listed. Does this include MNAC, which appears in a separate paragraph?\n* Why does the sFID measure ignore covariance between features? This metric may capture realism at the feature level but not at the level of feature co-occurrence—e.g., a person smiling while frowning.\n* Which metric is used to measure stereotype bias in the generated images? Since this is an objective of the work, it should be clearly identified.\n* Table 2 lacks uncertainty estimates. A k-fold evaluation should be used to assess robustness relative to other methods.\n* Figure 5: what do the “Cu” elements represent?\n* Table 4: it is unclear why the other competitors are not included. The table is also poorly placed—directly below the subsection header—which affects readability. In addition, the caption appears above the table.\n* Table 5 and Subsection 4.4.5: should these statistics not also be reported on a baseline (real images or other generated images) to better compare the approach?", "broader_impact_concerns": "NA", "additional_comments": "The paper and methodology are interesting and well justified, but I think the methodology section needs to be clearer and more educational.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)