Counterfactual Visual Explanation via Causally-Guided Adversarial Steering

## Abstract

Abstract Recent work on counterfactual visual explanations has contributed to making artificial intelligence models more explainable by providing visual perturbations to flip the prediction. However, these approaches neglect the causal relationships and the spurious correlations behind the image generation process, which often leads to unintended alterations in the counterfactual images and renders the explanations with limited quality. To address this challenge, we introduce a novel framework CECAS, which leverages a causally-guided adversarial method to generate counterfactual explanations. It innovatively integrates a causal perspective to avoid unwanted perturbations on spurious factors in the counterfactuals. Extensive experiments demonstrate that our method outperforms existing state-of-the-art approaches across multiple benchmark datasets and ultimately achieves a balanced trade-off among various aspects of validity, sparsity, proximity, and realism.

## 1 Introduction

Despite the impressive performance of deep neural networks (DNNs) across domains such as language, vision, and control (Devlin et al., 2019; Bojarski et al., 2016; Silver et al., 2017), explaining their predictions remains challenging due to their non-linear, overparameterized nature (Li et al., 2022). This challenge is particularly critical in safety-sensitive applications like healthcare, finance, and autonomous driving (Mersha et al., 2024). As a result, explainable AI (XAI) has gained momentum (Dwivedi et al., 2023), with counterfactual explanations (CEs) Wachter et al. (2017) emerging as a powerful approach. CEs aim to answer: What minimal change to an input would alter the model’s prediction? They are valued for producing intuitive and preferably actionable insights that can support decision auditing and planning. As a powerful XAI tool (Wachter et al., 2017), CE has been particularly flourishing in computer vision (Jacob et al., 2022; Jeanneret et al., 2022; Rodriguez et al., 2021; Jeanneret et al., 2023). Unlike attribution-based methods that highlight pixels associated with the output, counterfactuals in images simulate realistic alternative scenarios, allowing us to ask: What would the image need to look like for the model to decide differently? For example, modifying a few facial features to change a classification from “neutral” to “happy” provides insight into the model’s decision boundary. Successful visual counterfactuals not only satisfy conventional CE criteria such as being valid (flip the prediction), proximal (close to the original image), and sparse (modify as few features as possible) (Mothilal et al., 2020; Russell, 2019), but also are semantically meaningful and perceptually natural, avoiding unrealistic artifacts. These methods offer intuitive and human-understandable explanations, making them especially valuable for analyzing high-dimensional visual models.

While counterfactual visual explanation has seen rapid progress in recent years Jacob et al. (2022); Jeanneret et al. (2022); Rodriguez et al. (2021); Jeanneret et al. (2023), several critical aspects of the problem remain overlooked. A key limitation lies in the lack of a principled understanding of the underlying causal mechanisms in the data, as well as the true target of explanation. Most existing methods (Petsiuk et al., 2018; 2021; Vasu & Long, 2020) optimize directly over the entire input feature space in search of perturbations, which often leads to changes in spurious factors that are correlated with the label but not causally related. While such perturbations may achieve a prediction flip, they rarely reflect meaningful or actionable changes in the real world and can even produce misleading or harmful outcomes. For example, in autonomous driving scenarios, changing the shape of a traffic sign (spurious factor) to alter a “stop” vs. “go” prediction is far less informative than changing the content of the sign, which is causally relevant to the model’s decision.
More importantly, conventional CE strategies often result in low-quality and unrealistic images. The pixel space of images is vast, and unconstrained perturbations can introduce noise or artifacts that are visually implausible or semantically incoherent. Recent advances Jeanneret et al. (2022; 2023) have begun to address this by leveraging generative models, particularly diffusion-based methods, which operate in semantically meaningful latent spaces and are better equipped to produce realistic counterfactuals. These approaches benefit from the ability of diffusion models to suppress high-frequency noise and structure perturbations more coherently.
However, most current methods still rely heavily on predefined loss functions, such as $l_{1}$ loss constraining the difference between original and counterfactual images, to enforce sparsity and proximity. While effective at a global level, these constraints often lead to unwanted changes in local artifacts such as color shifts, shape distortions, or the emergence of spurious attributes (Andonian et al., 2021). This results in a persistent trade-off between validity, proximity, sparsity, and realism, making it difficult to generate counterfactuals that are both high-quality and causally meaningful. To the best of our knowledge, no existing method well addresses these challenges, underscoring the need for a new approach that incorporates causal reasoning into the generation of counterfactual visual explanations.

To address the aforementioned challenges, we propose a novel framework, Counterfactual Explanation via Causally Adversarial Steering (CECAS), which leverages a causality-guided adversarial method to generate counterfactual explanations that are more semantically faithful and causally grounded. Adversarial attacks have been an effective tool to generate CEs due to their common aim to minimally alter an input to change a model’s prediction Pawelczyk et al. (2022). Uniquely, our approach integrates causality into the adversarial objective: While the adversarial component ensures prediction flips (validity), the causal guidance directs perturbations toward features that are causally relevant to the label. This promotes sparsity, proximity, and realism, by avoiding changes to spurious or irrelevant features. As a result, our method produces high-quality, semantically meaningful counterfactual images that align with real-world interventions. We conclude our contributions as follows:

- •
We provide a new perspective on the problem of counterfactual visual explanation by highlighting the critical role of causality. We show how causality can improve the quality of counterfactuals by avoiding spurious perturbations.
- •
We propose a novel framework with a causally guided adversarial approach to generate counterfactual visual explanations. Our method steers perturbations toward causally relevant features, enabling the generation of valid, sparse, proximal, and realistic counterfactual images.
- •
We conduct extensive experiments to evaluate our framework on multiple benchmark datasets. Our approach outperforms current state-of-the-art methods across various metrics both quantitatively and qualitatively.

## 2 Related Work

Explainable AI (XAI) XAI refers to a set of techniques aimed at enhancing the transparency and explainability of AI models—making their decisions and internal mechanisms more understandable to humans (Mersha et al., 2024). The field is commonly divided into two major categories: ad-hoc and post-hoc methods. The key difference between them lies in the stage at which they are implemented (Guidotti et al., 2018). The former is often deployed during the training phase and advocates the design of inherently interpretable architectures (Alaniz et al., 2021; Alvarez Melis & Jaakkola, 2018; Bohle et al., 2021; Huang & Li, 2020; Nauta et al., 2021). In contrast, the latter focuses on understanding and explaining the trained models without modifying their internal structure (Selvaraju et al., 2017; Shrikumar et al., 2017; Simonyan et al., 2013). The post-hoc category includes global and local explanations, distinguished by the scope of the explanations they provide (Mersha et al., 2024). Global explanation provides a comprehensive description of the model (Lipton, 2018), while the latter focuses on explaining the model’s prediction for a specific instance (Arrieta et al., 2020). The studied problem of our paper belongs to local explanations. This line of work mainly focuses on the following three directions: saliency map (Chattopadhay et al., 2018; Jalwana et al., 2021; Kim et al., 2022), concept attribution (Ghorbani et al., 2019; Kim et al., 2018; Kolek et al., 2022), and model distillation (Ge et al., 2021; Tan et al., 2018).

Counterfactual Explanations As a branch of post-hoc explanations, CE has emerged as a highly influential approach in recent years (Pawelczyk et al., 2022), especially in computer vision (Jeanneret et al., 2023).
Some methods generate counterfactual samples by locating key regions that distinguish two contrasting images and substituting the relevant region from one image into the other (Goyal et al., 2019; Wang et al., 2021).
Other methods utilize the gradient of the input image with respect to the target label while also leveraging robust models to generate human-recognizable modifications (Schut et al., 2021; Zhu et al., 2021). Additional approaches, including prototype-based algorithms (Van Looveren & Klaise, 1907) and invertible convolutional networks (Hvilshøj et al., 2021), have also shown promising abilities in generating counterfactual examples. Generative techniques are inherently well-suited for CE. With their rapid development, an increasing number of related works have emerged, which can be broadly categorized into conditional (Singla et al., 2019; Van Looveren et al., 2021) and unconditional approaches (Khorram & Fuxin, 2022; Rodriguez et al., 2021). Diffusion models (DDPM) are cutting-edge generative models. Jeanneret et al. (2022) and (Jeanneret et al., 2023) both utilize DDPM to generate counterfactual images. The difference lies in that the former modifies the sampling process of the DDPM, while the latter employs DDPM as a regularizer.

Adversarial Attacks Adversarial attacks and CEs share a similar objective: flipping the model’s prediction, which has led some studies to focus on exploring the relationship between them (Akhtar et al., 2021; Ignatiev et al., 2019; Pawelczyk et al., 2022). However, the key difference lies in the fact that adversarial attacks on undefended models often result in imperceptible noise rather than semantically meaningful changes. Adversarial attacks include white-box and black-box settings. In white-box attacks, the attacker has access to the model’s architecture and data (Costa et al., 2024). It typically generates adversarial examples by utilizing the gradient of the loss function with respect to the input (Goodfellow et al., 2014; Madry et al., 2017; Moosavi-Dezfooli et al., 2016). In contrast, black-box attacks (Andriushchenko et al., 2020; Wicker et al., 2018) restrict the attacker to accessing only samples from an oracle or pairs of inputs and corresponding outputs.

Adversarial Defense The development of adversarial attacks has, in turn, stimulated significant progress in adversarial defense. Recent works can be primarily divided into two categories: certified defense and empirical defense. Certified defense (Raghunathan et al., 2018; Cohen et al., 2019; Singla & Feizi, 2020) offer provable robustness by ensuring that the model’s predictions remain stable under perturbations constrained within a specified norm ball. However, empirical defense (Goodfellow et al., 2014; Madry et al., 2017; Zhang et al., 2019), particularly adversarial training (Madry et al., 2017), is still highly effective. It typically generates adversarial samples by using an adversarial attack first and then incorporates such samples into the training process. It is worth noting that recent work has begun to integrate causality into adversarial defense strategies (Zhang et al., 2020; Carlini & Wagner, 2017; Mitrovic et al., 2020; Zhang et al., 2021), sharing a common objective with our work.

## 3 Preliminaries

Figure: Figure 1: Causal graph of the natural and counterfactual images generation process. Each node means a variable, where blue nodes are involved in the generation of natural images, green nodes contribute to the generation of counterfactual images, and two-colored nodes participate in both.
Refer to caption: x1.png

Problem Definition We first consider a classification task with $x$ representing an input image and $y$ standing for the corresponding label, where $x\in\mathcal{X},y\in\mathcal{Y}:=\{1,...,K\}$, $\mathcal{X}$ and $\mathcal{Y}$ represent the image space and the label space, respectively. A DNN classifier trained
for this task can be denoted by $f:\mathcal{X}\rightarrow\mathcal{Y}$. Then the counterfactual visual explanation can be defined as the task of designing a semantically meaningful, perceptible yet minimal perturbation $\delta$ that, when added to an input image $x$, generates a counterfactual image $x^{\prime}:=x+\delta$ and flips the classifier’s prediction $\hat{Y}$ from the original prediction $y$ to a target label $y^{\prime}$.

Causal View on Image Generation Figure [1](https://arxiv.org/html/2507.09881v2#S3.F1) is a well-recognized causal graph $\mathcal{G}$ (Zhang et al., 2021) that can represent the generation processes of both the natural and counterfactual images.
As shown, $C$ denotes the causal factors that directly determine the label $Y$ and predicted label $\hat{Y}$. $S$ denotes the spurious factors, which are independent of $C$, as reflected in the $v$-structure $C\rightarrow{X}\leftarrow{S}$. $X$ is the natural image, and $\Delta$ represents the semantically meaningful perturbation; together, they form the counterfactual image $X^{\prime}$. $\theta$ is the set of parameters of the DNN. It is evident from the graph that the perturbation $\Delta$ is determined by $\hat{Y}$, $X$ and $\theta$. The inclusion of $\theta$ is due to the use of white-box attacks, such as projected gradient descent (PGD). Note that $S$ can also have an influence on $Y$ because of the spurious paths in $\mathcal{G}$. When $X$ is given, there is a spurious path ${Y}\leftarrow{C}\rightarrow{X}\leftarrow{S}$ associate $Y$ and $S$.

Denoising Diffusion Probabilistic Models DDPM Ho et al. (2020) can filter out the noise introduced by traditional adversarial attacks without changing the classifier’s weights. It relies on two Markov chains, corresponding to the forward and reverse processes, respectively. The forward process gradually adds Gaussian noise to the data at the state of $t$ to generate data at the next state $t+1$. In practice, the data at time step $t$, denoted as $x_{t}$ can be directly obtained from the clean data $x_{0}$ through

$$ $x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\,\epsilon,\quad\epsilon\sim\mathcal{N}(0,I),$ (1) $$

where $\bar{\alpha}_{t}:=\prod_{s=1}^{t}\alpha_{s}$, ${\alpha}_{t}:=1-{\beta}_{t}$ and $\{\beta_{t}\}_{t=1}^{T}$ is the variance schedule. The reverse process removes the noise from $x_{t}$ to get a less noisy $x_{t-1}$ through the following equation:

$$ ${x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left({x}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\,\bm{\epsilon}_{\rho}({x}_{t},t)\right)+\sigma_{t}\epsilon,$ (2) $$

where ${\sigma_{t}}^{2}={\beta}_{t}$, $\bm{\epsilon}_{\rho}$ is a function approximator intended to predict $\epsilon$ from $x_{t}$.

Projected Gradient Descent Attack PGD is a white-box attack that leverages the gradient of the loss function $L$ with respect to the input image $x$ to iteratively modify the image in a direction that maximally confuses the classifier. PGD operates through multiple iterations, and after each iteration $\tau$, the perturbed image $x_{\tau}$ is projected back onto an $l_{\infty}$-norm ball centered at the original input, with a predefined radius. This process can be summarized as follows:

$$ $x_{\tau+1}=\Pi_{x+\mathcal{S}}\left(x_{\tau}+\alpha\,\text{sgn}\left(\nabla_{x}L(\theta,f(x_{\tau}),y)\right)\right),$ (3) $$

where $\Pi$ denotes the projection operator, $\mathcal{S}$ defines the radius of the $l_{\infty}$-norm ball, $\alpha$ is the step size, sgn is the sign function, and $\theta$ represents the parameters of the target model.

## 4 Methodology

Figure: Figure 2: CECAS Framework: The upper part of the figure illustrates the first stage of the framework, which involves generating the initial CE images. Here, $x$ denotes the original image, $x^{\prime}$ represents the causally-guided CE. The lower part of the figure depicts the second stage of the framework, namely post-refinement. Here, we refine $x^{\prime}$ with a precise pixel-level mask $m$ (computed based on the difference between $x$ and a $l_{1}$-based CE $x^{\prime\prime}$), subsequently used in conjunction with image inpainting techniques to produce the high-quality final CE image.
Refer to caption: x2.png

We propose the CECAS framework, which consists of two key components, as shown in Figure [2](https://arxiv.org/html/2507.09881v2#S4.F2). The first component is the initial generation of CE images, achieved through a novel causally guided adversarial method. The second component is the causally guided refinement of the generated counterfactuals, where we adopt an image inpainting approach to better preserve the details of the unmodified regions of the original image. We detail them in the following sections.

### 4.1 Initial Causally-Guided Counterfactual Visual Explanation Generation

The first and central component of the proposed framework is a causally guided adversarial method to generate counterfactual visual explanations.
As previously mentioned, adversarial attacks and CE share a common goal: altering a model’s prediction through minimal input perturbations. Inspired by prior work bridging the two domains (Pawelczyk et al., 2022; Jeanneret et al., 2022; Augustin et al., 2022), we adopt adversarial strategies—such as PGD attacks—to generate image-based counterfactuals.
The procedure is outlined as follows:

$$ $x_{\tau+1}=\Pi_{x+\mathcal{S}}\left(x_{\tau}-\alpha\,\text{sgn}\left(\nabla_{x}L(f_{\theta}(x_{\tau}),y^{\prime})\right)\right),$ (4) $$

where the subscript $\tau$ denotes the iteration in PGD, $L$ is the classification loss, typically the cross-entropy loss, and $y^{\prime}$ is the target label. Upon completing $T$ iterations, the resulting $x_{T}$ is taken as the counterfactual image we aim to generate. It is worth noting that, in form, Equation (4) differs from traditional PGD attacks. Here, the gradient is subtracted from the perturbed image rather than added. This is because the objective is not to mislead the classifier into making an arbitrary incorrect prediction, but rather to guide it toward a specific target label $y^{\prime}$. This allows CEs to be generated with the gradient-based perturbation $\delta$ added on the original image.

However, relying solely on the above adversarial attack-based method is often overly idealized, as these methods typically impose no constraints on spurious factors. As discussed in the Introduction, although the explained model may rely on such factors during prediction, perturbing them often does not yield meaningful or actionable explanations. Furthermore, altering these spurious factors can introduce artifacts or noise, which severely compromise the core goals of CE of providing realistic and constructive explanations.

To address this issue, it is essential to constrain changes to spurious factors. Given the central role of gradients in the above CE generation process, we aim to identify a new gradient direction—one that enables effective modification of the image while preserving spurious factors unchanged. Since our goal is to explain the classifier $f$, we are not allowed to make any modifications to it. Considering the architecture of the explained model $f$, a DNN commonly viewed—following (Bengio et al., 2013)— as a combination of a feature extraction module and a linear classifier module, we learn to capture the spurious factors out of the representations $R$ learned by the feature extraction module (details of how to capture $S$ from $R$ are given in the final part of this subsection).
With the captured $S$, we use the original data to train an auxiliary classifier $g$ which predicts the label $Y$ based on the spurious factor. More specifically, after training, $g$ can make predictions as follows: $\hat{z}=g(s),\hat{z}^{\prime}=g(s^{\prime})$, where $\hat{z}$ denotes the logit predicted by $g$ from the spurious factor $s$ in the original image, while $\hat{z}^{\prime}$ is the logit predicted by $g$ from the spurious factor $s^{\prime}$ in the modified image. We can consider $z$ to capture the spurious factors that are most relevant to the label $Y$—that is, the spurious features most likely to be unintentionally altered during counterfactual generation.
On top of this, we introduce a penalty loss $L_{S}$ that encourages these spurious factors to remain invariant during the counterfactual generation process, and thereby we compute the gradient of the penalty loss with respect to the image and incorporate its direction into the guidance for image modification:

$$ $x_{\tau+1}=\Pi_{x+\mathcal{S}}\left(x_{\tau}-\alpha\,\text{sgn}\left(\nabla_{x}L(f_{\theta}(x_{\tau}),y^{\prime})+\nabla_{x}L_{S}(z_{\tau},z_{\tau}^{\prime})\right)\right),$ (5) $$

where $L_{S}$ can typically be implemented as the soft cross-entropy loss, which takes two logits as input. The purpose of this loss is to align the spurious distributions between the natural and counterfactual representations by penalizing the deviation of spurious factors in the counterfactuals from those in the original inputs. In summary, the generation of counterfactual images involves modifying the images via adversarial attack, while preserving the spurious factors through a causal perspective.

Furthermore, inspired by the effectiveness of diffusion models in enhancing CE image quality (Jeanneret et al., 2022; Augustin et al., 2022), we integrate diffusion models to filter out high-frequency, semantically meaningless noise introduced by adversarial attacks. Specifically, we apply the forward process of DDPM to add Gaussian noise to the image first, and then use the reverse process to denoise it before feeding it into the classifier for loss computation. In this work, we define the joint classifier $F$ as the combination of DDPM and the base classifier $f$. The generation of counterfactual images can be simplified and reformulated in an optimization form:

$$ $\arg\min_{x^{\prime}}L(F(x^{\prime});y^{\prime})+\alpha L_{S}(z,z^{\prime}),$ (6) $$

where $\alpha$ is the hyperparameter that controls the strength of $L_{S}$.

The only remaining problem is how to obtain the spurious factor $S$. Fortunately, numerous recent works have explored disentangling causal factors from spurious ones (Lv et al., 2022; Creager et al., 2021). Building on the causal model illustrated in Figure [1](https://arxiv.org/html/2507.09881v2#S3.F1), we adopt the widely used assumption that the causal factor $C$ is independent of the spurious factor $S$. In this work, similar to (O’Shaughnessy et al., 2020), we pre-train a VAE $h$ to disentangle $C$ and $S$ from the latent space. $h$ takes the representation
$R$ as input and is optimized with two objectives: (1) maximizing the causal influence of $C$ on $Y$; and (2) ensuring that $C$ and $S$, when jointly passed through the decoder, can faithfully reconstruct the distribution of $R$, thereby decomposing the latent space into two independent components.

### 4.2 Causally Post Refinement

After generating the initial counterfactual images, a key challenge lies in ensuring sparsity and proximity. This is particularly difficult because the $L_{S}$ constraint operates at the concept level and often becomes “soft” when translated to pixel space, making it insufficient to enforce strict pixel-level consistency. While we have discussed the limitations of the $\ell_{1}$ loss, its advantages should not be overlooked—particularly its ability to identify precise pixel-level changes. Therefore, we leverage this property of $\ell_{1}$ loss to roughly localize pixels in the image that require modification, and then combine it with our causal constraint $L_{S}$ to avoid unnecessary changes to spurious factors. This supports our goal of causally guided post-refinement, where we preserve the counterfactual changes in $x^{\prime}$ introduced by causal adversarial steering, while restoring the unmodified regions with the corresponding pixels from the original image $x$. Specifically, we first generate a perturbed image using an adversarial attack combined with $l_{1}$ loss as shown in the following equation:

$$ $\arg\min_{x^{\prime\prime}}L(F(x^{\prime\prime});y^{\prime})+\lambda||x^{\prime\prime}-x||_{1},$ (7) $$

where $x^{\prime\prime}$ is the perturbed image, $\lambda$ is a hyperparameter. Then a binary mask is constructed as:

$$ $m_{i,j}=\mathds{1}\left[\max_{(u,v)\in\mathcal{N}_{i,j}}\left(\frac{1}{M}\sum_{ch=1}^{CH}\left|x^{\prime\prime}(ch,u,v)-x(ch,u,v)\right|\right)>\gamma\right],$ (8) $$

where $m_{i,j}$ is the binary mask value at position $(i,j)$, $x$ is the original image, $ch$ is the channel index, $(u,v)$ is the position index in an image, $CH$ is the number of channels, $M$ is the maximum value after the summation, $\mathcal{N}_{i,j}$ is a $d\times{d}$ neighborhood window centered at position $(i,j)$, $\mathds{1}$ is the indicator function and $\gamma$ is a predefined threshold. We apply the dilation operation to adjust the size of the modification region to a reasonable extent, allowing for as many plausible changes as possible.

The mask indicates which pixels need to be modified through $l_{1}$ loss and which do not. Intuitively, we need to combine $x^{\prime}$ generated in the last subsection and the original image $x$ with the mask to achieve causal perturbation in CEs with well-preserved pixels in the rest parts. However, directly combining $x^{\prime}$ and $x$ using the mask is not feasible, as it would result in clearly visible region boundaries in the CE. In our work, we leverage the image inpainting technique (Lugmayr et al., 2022) to tackle this problem. The inpainting process is shown as follows:

$$ ${x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left([(1-m){x}_{t}+mx^{0}_{t}]-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\,\bm{\epsilon}_{\rho}([(1-m){x}_{t}+mx^{0}_{t}],t)\right)+\sigma_{t}\epsilon,$ (9) $$

where $m$ is the binary mask, $x_{t}$ denotes the image being refined at the noise level $t$, and $x^{0}_{t}$ is the original image at the noise level $t$. After a pre-defined $T$ denoising steps, we can obtain the final counterfactual visual explanations $x^{\prime}$ from the previous initial $x^{\prime}$ we gained from Section [4.1](https://arxiv.org/html/2507.09881v2#S4.SS1).

## 5 Experiments

In this section, we present extensive experiments to evaluate our proposed framework. We first compare our method against state-of-the-art baselines through both qualitative and quantitative analyses, including widely used metrics and case studies. Beyond traditional metrics, we further assess the quality of the generated counterfactuals using vision-language models (VLMs) as human-like evaluators. Additionally, we perform parameter studies to better understand the impact of different hyperparameter values in our method.

### 5.1 Experimental Setup

Datasets
We conducted extensive experiments on 4 datasets (10 subsets) in total. We first evaluate our framework on the CelebA dataset (Liu et al., 2015), using images of size 128$\times$128 and focusing on two attributes which constitute three sub-datasets: “Smile”, “Age” and ”Gender”. The CelebA-HQ (Lee et al., 2020) dataset is used with the same attributes and is also constructed into three sub-datasets, but with images of size 256$\times$256. The other datasets include the traffic scene dataset BDD-OIA (Xu et al., 2020), with images of size 512$\times$256, and three sub-datasets extracted from ImageNet (Russakovsky et al., 2015), each containing two classes. These three sub-datasets are: Cougar vs. Cheetah, Sorrel vs. Zebra, and Egyptian Cat vs. Persian Cat. All experiments were conducted on NVIDIA RTX 4090 and RTX A6000 GPUs.

Baselines We selected four recent and state-of-the-art methods as our baselines: STEEX (Jacob et al., 2022), DiVE (Rodriguez et al., 2021), DiME (Jeanneret et al., 2022), and ACE (Jeanneret et al., 2023). Among them, DiVE includes an extended variant called DiVE^100, in which the number of optimization steps is reduced to 100.

Models Weights and Architectures To ensure a fair comparison, we use the same pre-trained DDPM weights as those used in the baselines for each dataset. For ImageNet, we use the ResNet-50 model pre-trained by PyTorch (Paszke, 2019) as the classifier. For the other datasets, we use DenseNet-121 as the classifier and adopt the pre-trained weights provided by the baselines. Additional training details, including hyperparameter settings, are provided in Appendix [A](https://arxiv.org/html/2507.09881v2#A1).

Evaluation Metrics
We evaluate our method and the baselines across multiple dimensions, each assessed using several metrics. The first evaluation dimension is validity, which we measure using Flip Rate (FR) and Counterfactual Transition (COUT). FR denotes the proportion of samples for which the counterfactual successfully causes the classifier to change its prediction. COUT measures the transition scores between the original and counterfactual images, with values ranging from $-1$ to $1$. The second evaluation dimension focuses on sparsity and proximity, which we assess using Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018), Correlation Difference (CD) (Jeanneret et al., 2022), and SimSiam Similarity (S^3) (Chen & He, 2021). LPIPS measures the perceptual similarity between two images based on human visual perception. CD computes the average number of non-target attributes that are unintentionally altered across all samples. S^3 evaluates the cosine similarity between the original and counterfactual images using SimSiam as the encoding network. The third evaluation dimension is realism, which we measure using Fréchet Inception Distance (FID) (Heusel et al., 2017) and split FID (sFID) (Jeanneret et al., 2023). Since counterfactual images typically share most pixels with the original images, FID tends to be underestimated and may fail to reflect their true differences. To address this issue, sFID is proposed. It splits both the original and counterfactual image sets into two subsets, computes FID in a cross-wise manner, and repeats this process ten times to obtain an average score. All results reported in this paper are averaged over multiple experimental runs. We omit standard deviations as they are negligible compared to the corresponding mean values.

**Table 1: Evaluation Results on CelebA-HQ: Smile.**
| Methods | FR $\uparrow$ | COUT $\uparrow$ | LIPIPS $\downarrow$ | S^3 $\uparrow$ | CD $\downarrow$ | FID $\downarrow$ | sFID $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DiVE | - | - | - | - | - | 107.0 | - |
| STEEX | - | - | - | - | - | 21.9 | - |
| DiME | 90.00% | -0.2797 | 0.1689 | 0.9245 | 2.52 | 15.8 | 26.79 |
| ACE | 77.50% | 0.2369 | 0.0353 | 0.9868 | 2.76 | 5.9 | 38.20 |
| Ours | 97.85% | 0.6751 | 0.0331 | 0.9889 | 2.42 | 3.9 | 26.58 |

**Table 2: Evaluation Results on CelebA-HQ: Age**
| Methods | FR $\uparrow$ | COUT $\uparrow$ | LIPIPS $\downarrow$ | S^3 $\uparrow$ | CD $\downarrow$ | FID $\downarrow$ | sFID $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DiVE | - | - | - | - | - | 107.5 | - |
| STEEX | - | - | - | - | - | 26.8 | - |
| DiME | 94.55% | -0.3339 | 0.1689 | 0.9202 | 4.00 | 17.8 | 26.54 |
| ACE | 63.45% | 0.0567 | 0.0448 | 0.9876 | 5.73 | 9.7 | 33.95 |
| Ours | 96.15% | 0.5489 | 0.0466 | 0.9893 | 5.33 | 5.6 | 27.84 |

**Table 3: Evaluation results on ImageNet.**
| Datasets | Methods | FR $\uparrow$ | COUT $\uparrow$ | LPIPS $\downarrow$ | S^3 $\uparrow$ | FID $\downarrow$ | sFID $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Sorrel-<br>Zebra | ACE | 76.67% | 0.0967 | 0.1670 | 0.9285 | 77.76 | 124.66 |
| Ours | 93.33% | 0.2219 | 0.0468 | 0.9338 | 72.26 | 116.28 |  |
| Cougar-<br>Cheetah | ACE | 76.67% | 0.2059 | 0.1809 | 0.9625 | 108.67 | 147.79 |
| Ours | 100.00% | 0.4186 | 0.0470 | 0.9616 | 102.56 | 137.91 |  |
| Persian Cat-<br>Egyptian Cat | ACE | 80.00% | 0.2326 | 0.1531 | 0.8939 | 139.65 | 227.07 |
| Ours | 83.33% | 0.2660 | 0.0466 | 0.8744 | 134.89 | 215.07 |  |

### 5.2 Quantitative Analysis

In this section, we conduct a quantitative analysis of all datasets with results summarized in Table [1](https://arxiv.org/html/2507.09881v2#S5.T1) through Table [6](https://arxiv.org/html/2507.09881v2#A2.T6). We highlight the best performance in bold and the second-best in underline. The results for the baselines DiVE, DiVE^100, and STEEX, we extract the results from the original papers. For the data not reported in the original papers, we leave the corresponding entries blank in the tables.

Tables [1](https://arxiv.org/html/2507.09881v2#S5.T1)and  [2](https://arxiv.org/html/2507.09881v2#S5.T2) show the results on the CelebA-HQ Smile and Age dataset, Table [3](https://arxiv.org/html/2507.09881v2#S5.T3) shows the results on the ImageNet dataset. As observed, our method achieves the best performance on the vast majority of metrics. More specifically, our method achieves the highest FR and COUT scores, while also attaining the lower LPIPS and higher S^3 values, along with relatively low FID and sFID scores. This indicates that our method can achieve the highest flip rate with minimal modifications. In other words, it attains the highest validity while preserving both sparsity and proximity. Although in Table [2](https://arxiv.org/html/2507.09881v2#S5.T2) our method performs slightly worse than DiME on certain metrics, the COUT of DiME is negative. This indicates that in DiME’s method, the classifier lacks sufficient confidence in label flipping. Even though the other metrics may show marginal improvements, they do not necessarily indicate that the counterfactual images are of higher quality than those produced by our method. Due to page limitations, we present the results on the other datasets in Appendix [B](https://arxiv.org/html/2507.09881v2#A2).

Figure: Figure 3: Qualitative results on CelebA (Smile & Age). Red circles highlight unintended changes produced by ACE, while green circles indicate the improvements made by our method.
Refer to caption: x3.png

### 5.3 Qualitative Analysis

In this section, we perform case studies to straightforwardly evaluate the quality of the generated counterfactual images from a human perceptual perspective.
For CelebA datasets, we compare the counterfactual images generated by our approach with those produced by the best-performing method, ACE, to demonstrate the visual superiority of our method.
As shown in Figure [3](https://arxiv.org/html/2507.09881v2#S5.F3), ACE introduces inexplicable color and shape distortions, as well as unintended attributes in the generated images. For example, in the transformation from “young” to “old,” some female samples are incorrectly rendered with dark patches on the chin—an artifact caused by the spurious correlation between beard and age in male samples. This results in unmeaningful CEs. In contrast, our method effectively mitigates such artifacts by avoiding reliance on spurious cues. Figure [4](https://arxiv.org/html/2507.09881v2#S5.F4) shows the results on ImageNet. As our method and ACE yield visually indistinguishable outputs on this dataset, we restrict the comparison to the original images and their counterfactual counterparts.
Qualitative results for other datasets are provided in Appendix [B](https://arxiv.org/html/2507.09881v2#A2).

### 5.4 VLM Evaluations

**Table 4: Performance comparison across datasets.**
| Datasets | DiME | ACE | Ours |
| --- | --- | --- | --- |
| CelebA Smile | 7.49 | 8.14 | 8.24 |
| CelebA Age | 7.06 | 7.13 | 7.57 |
| CelebA-HQ Smile | 6.18 | 8.19 | 8.55 |
| CelebA-HQ Age | 3.95 | 5.76 | 6.19 |
| BDD-OIA | 4.07 | 7.86 | 7.77 |
| Sorrel-Zebra | - | 5.73 | 6.37 |
| Cougar-Cheetah | - | 7.14 | 7.61 |
| Persian Cat-<br>Egyptian Cat | - | 6.93 | 7.63 |

Vision-Language Models (VLMs), through large-scale pretraining on image–text pairs, demonstrate strong reasoning capabilities in tasks such as visual question answering and image captioning (Bordes et al., 2024). Overcoming the limitation of traditional CE metrics that only provide shallow numerical evaluations and fail to capture the semantic meaning of images, VLMs provide more human-like evaluations by leveraging their rich prior knowledge in pertaining (Huang et al., 2025). To better suit the specific nature of counterfactual image generation, we carefully design unique prompts with evaluation criteria covering validity, sparsity, proximity, and realism. The prompt design for each dataset can be found in Appendix [C](https://arxiv.org/html/2507.09881v2#A3). Notably, sparsity in our prompt is partially assessed from a causal perspective, examining whether the modified regions in the image are necessary or causally relevant, while ensuring that unrelated parts remain unchanged. In the context of our task, we utilize GPT-4o to assign scores ranging from 1 to 10 for each of the four evaluation dimensions. Given that these dimensions hold equal importance in counterfactual generation, the final score is computed as the average across all four. As shown in Table [4](https://arxiv.org/html/2507.09881v2#S5.T4), our method outperforms the baselines in 7 out of the 8 sub-datasets. Combined with the powerful visual and textual reasoning capabilities of VLMs, we have strong reason to believe that the counterfactual images generated by our method offer the highest level of interpretability.

### 5.5 Parameters Study

Figure: Figure 4: Line plots of FR and COUT with respect to varying values of $\alpha$.
Refer to caption: x4.png

In this section, we analyze the impact of the hyperparameter $\alpha$ on counterfactual image generation. We present results on the CelebA: Smile dataset, while results on the other datasets are provided in Appendix [D](https://arxiv.org/html/2507.09881v2#A4). As shown in Figure [4](https://arxiv.org/html/2507.09881v2#S5.F4), when $\alpha$ increases, the model is progressively constrained from modifying spurious factors. This trend leads to a monotonic decrease in both the flip rate and the COUT score. The reduction in flip rate indicates that many successful counterfactual transitions at lower $\alpha$ values are largely driven by spurious shortcuts; once these features are restricted, generating effective counterfactuals becomes more challenging. The concurrent decline in COUT further suggests that the transition trajectories become less smooth, with the classifier maintaining high confidence in the original class and showing limited confidence shifts toward the target class throughout the interpolation path. Taken together, these results demonstrate that higher $\alpha$ enforces a stricter focus on causal features, thereby producing counterfactuals that are harder to realize but more faithful to causal semantics. Therefore, it is crucial to strike a balance between flip effectiveness and visual realism. Fortunately, with a properly chosen $\alpha$, our method effectively achieves this balance, producing counterfactuals with both high flip rates and high image quality.

## 6 Conclusion

In this paper, we propose a novel framework, CECAS, which innovatively designs a causally-guided adversarial method to generate counterfactual visual explanations that constrain less informative changes on spurious factors. The framework can be applied to any DNN models. Experimental results demonstrate that our method consistently outperforms existing approaches in quantitative evaluation, qualitative analysis, and VLM-based assessment. CECAS achieves a better balance among the four key properties of counterfactual explanations: validity, sparsity, proximity, and realism. Therefore, we argue that the counterfactual explanations generated by our method offer a more causally grounded understanding of the classifier’s decision-making and prediction mechanisms.

## References

- Akhtar et al. (2021)
Naveed Akhtar, Mohammad AAK Jalwana, Mohammed Bennamoun, and Ajmal Mian.
Attack to fool and explain deep networks.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 44(10):5980–5995, 2021.
- Alaniz et al. (2021)
Stephan Alaniz, Diego Marcos, Bernt Schiele, and Zeynep Akata.
Learning decision trees recurrently through communication.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 13518–13527, 2021.
- Alvarez Melis & Jaakkola (2018)
David Alvarez Melis and Tommi Jaakkola.
Towards robust interpretability with self-explaining neural networks.
*Advances in neural information processing systems*, 31, 2018.
- Andonian et al. (2021)
Alex Andonian, Taesung Park, Bryan Russell, Phillip Isola, Jun-Yan Zhu, and Richard Zhang.
Contrastive feature loss for image prediction.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 1934–1943, 2021.
- Andriushchenko et al. (2020)
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein.
Square attack: a query-efficient black-box adversarial attack via random search.
In *European conference on computer vision*, pp. 484–501. Springer, 2020.
- Arrieta et al. (2020)
Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al.
Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.
*Information fusion*, 58:82–115, 2020.
- Augustin et al. (2022)
Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein.
Diffusion visual counterfactual explanations.
*Advances in Neural Information Processing Systems*, 35:364–377, 2022.
- Bengio et al. (2013)
Yoshua Bengio, Aaron Courville, and Pascal Vincent.
Representation learning: A review and new perspectives.
*IEEE transactions on pattern analysis and machine intelligence*, 35(8):1798–1828, 2013.
- Bohle et al. (2021)
Moritz Bohle, Mario Fritz, and Bernt Schiele.
Convolutional dynamic alignment networks for interpretable classifications.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 10029–10038, 2021.
- Bojarski et al. (2016)
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al.
End to end learning for self-driving cars.
*arXiv preprint arXiv:1604.07316*, 2016.
- Bordes et al. (2024)
Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, et al.
An introduction to vision-language modeling.
*arXiv preprint arXiv:2405.17247*, 2024.
- Carlini & Wagner (2017)
Nicholas Carlini and David Wagner.
Towards evaluating the robustness of neural networks.
In *2017 ieee symposium on security and privacy (sp)*, pp. 39–57. Ieee, 2017.
- Chattopadhay et al. (2018)
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.
Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.
In *2018 IEEE winter conference on applications of computer vision (WACV)*, pp. 839–847. IEEE, 2018.
- Chen & He (2021)
Xinlei Chen and Kaiming He.
Exploring simple siamese representation learning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 15750–15758, 2021.
- Cohen et al. (2019)
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter.
Certified adversarial robustness via randomized smoothing.
In *international conference on machine learning*, pp. 1310–1320. PMLR, 2019.
- Costa et al. (2024)
Joana C Costa, Tiago Roxo, Hugo Proença, and Pedro Ricardo Morais Inacio.
How deep learning sees the world: A survey on adversarial attacks & defenses.
*IEEE Access*, 12:61113–61136, 2024.
- Creager et al. (2021)
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel.
Environment inference for invariant learning.
In *International Conference on Machine Learning*, pp. 2189–2200. PMLR, 2021.
- Devlin et al. (2019)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
In *Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)*, pp. 4171–4186, 2019.
- Dwivedi et al. (2023)
Rudresh Dwivedi, Devam Dave, Het Naik, Smiti Singhal, Rana Omer, Pankesh Patel, Bin Qian, Zhenyu Wen, Tejal Shah, Graham Morgan, et al.
Explainable ai (xai): Core ideas, techniques, and solutions.
*ACM Computing Surveys*, 55(9):1–33, 2023.
- Ge et al. (2021)
Yunhao Ge, Yao Xiao, Zhi Xu, Meng Zheng, Srikrishna Karanam, Terrence Chen, Laurent Itti, and Ziyan Wu.
A peek into the reasoning of neural networks: interpreting with structural visual concepts.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 2195–2204, 2021.
- Ghorbani et al. (2019)
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim.
Towards automatic concept-based explanations.
*Advances in neural information processing systems*, 32, 2019.
- Goodfellow et al. (2014)
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples.
*arXiv preprint arXiv:1412.6572*, 2014.
- Goyal et al. (2019)
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee.
Counterfactual visual explanations.
In *International Conference on Machine Learning*, pp. 2376–2384. PMLR, 2019.
- Guidotti et al. (2018)
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.
A survey of methods for explaining black box models.
*ACM computing surveys (CSUR)*, 51(5):1–42, 2018.
- Heusel et al. (2017)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
*Advances in neural information processing systems*, 30, 2017.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
*Advances in neural information processing systems*, 33:6840–6851, 2020.
- Huang et al. (2025)
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Liangliang Cao, and Shifeng Chen.
Diffusion model-based image editing: A survey.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 2025.
- Huang & Li (2020)
Zixuan Huang and Yin Li.
Interpretable and accurate fine-grained recognition via region grouping.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 8662–8672, 2020.
- Hvilshøj et al. (2021)
Frederik Hvilshøj, Alexandros Iosifidis, and Ira Assent.
Ecinn: efficient counterfactuals from invertible neural networks.
*arXiv preprint arXiv:2103.13701*, 2021.
- Ignatiev et al. (2019)
Alexey Ignatiev, Nina Narodytska, and Joao Marques-Silva.
On relating explanations and adversarial examples.
*Advances in neural information processing systems*, 32, 2019.
- Jacob et al. (2022)
Paul Jacob, Éloi Zablocki, Hedi Ben-Younes, Mickaël Chen, Patrick Pérez, and Matthieu Cord.
Steex: steering counterfactual explanations with semantics.
In *European Conference on Computer Vision*, pp. 387–403. Springer, 2022.
- Jalwana et al. (2021)
Mohammad AAK Jalwana, Naveed Akhtar, Mohammed Bennamoun, and Ajmal Mian.
Cameras: Enhanced resolution and sanity preserving class activation mapping for image saliency.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 16327–16336, 2021.
- Jeanneret et al. (2022)
Guillaume Jeanneret, Loïc Simon, and Frédéric Jurie.
Diffusion models for counterfactual explanations.
In *Proceedings of the Asian conference on computer vision*, pp. 858–876, 2022.
- Jeanneret et al. (2023)
Guillaume Jeanneret, Loïc Simon, and Frédéric Jurie.
Adversarial counterfactual visual explanations.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 16425–16435, 2023.
- Khorram & Fuxin (2022)
Saeed Khorram and Li Fuxin.
Cycle-consistent counterfactuals by latent transformations.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 10203–10212, 2022.
- Kim et al. (2018)
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.
Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).
In *International conference on machine learning*, pp. 2668–2677. PMLR, 2018.
- Kim et al. (2022)
Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky.
Hive: Evaluating the human interpretability of visual explanations.
In *European Conference on Computer Vision*, pp. 280–298. Springer, 2022.
- Kolek et al. (2022)
Stefan Kolek, Duc Anh Nguyen, Ron Levie, Joan Bruna, and Gitta Kutyniok.
Cartoon explanations of image classifiers.
In *European Conference on Computer Vision*, pp. 443–458. Springer, 2022.
- Lee et al. (2020)
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.
Maskgan: Towards diverse and interactive facial image manipulation.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 5549–5558, 2020.
- Li et al. (2022)
Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Xiao Zhang, Ji Liu, Jiang Bian, and Dejing Dou.
Interpretable deep learning: Interpretation, interpretability, trustworthiness, and beyond.
*Knowledge and Information Systems*, 64(12):3197–3234, 2022.
- Lipton (2018)
Zachary C Lipton.
The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.
*Queue*, 16(3):31–57, 2018.
- Liu et al. (2015)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.
Deep learning face attributes in the wild.
In *Proceedings of the IEEE international conference on computer vision*, pp. 3730–3738, 2015.
- Lugmayr et al. (2022)
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11461–11471, 2022.
- Lv et al. (2022)
Fangrui Lv, Jian Liang, Shuang Li, Bin Zang, Chi Harold Liu, Ziteng Wang, and Di Liu.
Causality inspired representation learning for domain generalization.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 8046–8056, 2022.
- Madry et al. (2017)
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.
Towards deep learning models resistant to adversarial attacks.
*arXiv preprint arXiv:1706.06083*, 2017.
- Mersha et al. (2024)
Melkamu Mersha, Khang Lam, Joseph Wood, Ali AlShami, and Jugal Kalita.
Explainable artificial intelligence: A survey of needs, techniques, applications, and future direction.
*Neurocomputing*, pp. 128111, 2024.
- Mitrovic et al. (2020)
Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell.
Representation learning via invariant causal mechanisms.
*arXiv preprint arXiv:2010.07922*, 2020.
- Moosavi-Dezfooli et al. (2016)
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.
Deepfool: a simple and accurate method to fool deep neural networks.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 2574–2582, 2016.
- Mothilal et al. (2020)
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan.
Explaining machine learning classifiers through diverse counterfactual explanations.
In *Proceedings of the 2020 conference on fairness, accountability, and transparency*, pp. 607–617, 2020.
- Nauta et al. (2021)
Meike Nauta, Ron Van Bree, and Christin Seifert.
Neural prototype trees for interpretable fine-grained image recognition.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 14933–14943, 2021.
- O’Shaughnessy et al. (2020)
Matthew O’Shaughnessy, Gregory Canal, Marissa Connor, Christopher Rozell, and Mark Davenport.
Generative causal explanations of black-box classifiers.
*Advances in neural information processing systems*, 33:5453–5467, 2020.
- Paszke (2019)
A Paszke.
Pytorch: An imperative style, high-performance deep learning library.
*arXiv preprint arXiv:1912.01703*, 2019.
- Pawelczyk et al. (2022)
Martin Pawelczyk, Chirag Agarwal, Shalmali Joshi, Sohini Upadhyay, and Himabindu Lakkaraju.
Exploring counterfactual explanations through the lens of adversarial examples: A theoretical and empirical analysis.
In *International Conference on Artificial Intelligence and Statistics*, pp. 4574–4594. PMLR, 2022.
- Petsiuk et al. (2018)
Vitali Petsiuk, Abir Das, and Kate Saenko.
Rise: Randomized input sampling for explanation of black-box models.
*arXiv preprint arXiv:1806.07421*, 2018.
- Petsiuk et al. (2021)
Vitali Petsiuk, Rajiv Jain, Varun Manjunatha, Vlad I Morariu, Ashutosh Mehra, Vicente Ordonez, and Kate Saenko.
Black-box explanation of object detectors via saliency maps.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 11443–11452, 2021.
- Raghunathan et al. (2018)
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang.
Certified defenses against adversarial examples.
*arXiv preprint arXiv:1801.09344*, 2018.
- Rodriguez et al. (2021)
Pau Rodriguez, Massimo Caccia, Alexandre Lacoste, Lee Zamparo, Issam Laradji, Laurent Charlin, and David Vazquez.
Beyond trivial counterfactual explanations with diverse valuable explanations.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 1056–1065, 2021.
- Russakovsky et al. (2015)
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.
Imagenet large scale visual recognition challenge.
*International journal of computer vision*, 115:211–252, 2015.
- Russell (2019)
Chris Russell.
Efficient search for diverse coherent explanations.
In *Proceedings of the conference on fairness, accountability, and transparency*, pp. 20–28, 2019.
- Schut et al. (2021)
Lisa Schut, Oscar Key, Rory Mc Grath, Luca Costabello, Bogdan Sacaleanu, Yarin Gal, et al.
Generating interpretable counterfactual explanations by implicit minimisation of epistemic and aleatoric uncertainties.
In *International Conference on Artificial Intelligence and Statistics*, pp. 1756–1764. PMLR, 2021.
- Selvaraju et al. (2017)
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.
Grad-cam: Visual explanations from deep networks via gradient-based localization.
In *Proceedings of the IEEE international conference on computer vision*, pp. 618–626, 2017.
- Shrikumar et al. (2017)
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.
Learning important features through propagating activation differences.
In *International conference on machine learning*, pp. 3145–3153. PMlR, 2017.
- Silver et al. (2017)
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.
Mastering the game of go without human knowledge.
*nature*, 550(7676):354–359, 2017.
- Simonyan et al. (2013)
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.
Deep inside convolutional networks: Visualising image classification models and saliency maps.
*arXiv preprint arXiv:1312.6034*, 2013.
- Singla & Feizi (2020)
Sahil Singla and Soheil Feizi.
Second-order provable defenses against adversarial attacks.
In *International conference on machine learning*, pp. 8981–8991. PMLR, 2020.
- Singla et al. (2019)
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich.
Explanation by progressive exaggeration.
*arXiv preprint arXiv:1911.00483*, 2019.
- Tan et al. (2018)
Sarah Tan, Rich Caruana, Giles Hooker, Paul Koch, and Albert Gordo.
Learning global additive explanations for neural nets using model distillation.
*stat*, 1050:3, 2018.
- Van Looveren & Klaise (1907)
Arnaud Van Looveren and Janis Klaise.
Interpretable counterfactual explanations guided by prototypes. arxiv 2019.
*arXiv preprint arXiv:1907.02584*, 1907.
- Van Looveren et al. (2021)
Arnaud Van Looveren, Janis Klaise, Giovanni Vacanti, and Oliver Cobb.
Conditional generative models for counterfactual explanations.
*arXiv preprint arXiv:2101.10123*, 2021.
- Vasu & Long (2020)
Bhavan Vasu and Chengjiang Long.
Iterative and adaptive sampling with spatial attention for black-box model explanations.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pp. 2960–2969, 2020.
- Wachter et al. (2017)
Sandra Wachter, Brent Mittelstadt, and Chris Russell.
Counterfactual explanations without opening the black box: Automated decisions and the gdpr.
*Harv. JL & Tech.*, 31:841, 2017.
- Wang et al. (2021)
Pei Wang, Yijun Li, Krishna Kumar Singh, Jingwan Lu, and Nuno Vasconcelos.
Imagine: Image synthesis by image-guided model inversion.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 3681–3690, 2021.
- Wicker et al. (2018)
Matthew Wicker, Xiaowei Huang, and Marta Kwiatkowska.
Feature-guided black-box safety testing of deep neural networks.
In *Tools and Algorithms for the Construction and Analysis of Systems: 24th International Conference, TACAS 2018, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2018, Thessaloniki, Greece, April 14-20, 2018, Proceedings, Part I 24*, pp. 408–426. Springer, 2018.
- Xu et al. (2020)
Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos.
Explainable object-induced action decision for autonomous vehicles.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 9523–9532, 2020.
- Zhang et al. (2020)
Cheng Zhang, Kun Zhang, and Yingzhen Li.
A causal view on robustness of neural networks.
*Advances in Neural Information Processing Systems*, 33:289–301, 2020.
- Zhang et al. (2019)
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.
Theoretically principled trade-off between robustness and accuracy.
In *International conference on machine learning*, pp. 7472–7482. PMLR, 2019.
- Zhang et al. (2018)
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual metric.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 586–595, 2018.
- Zhang et al. (2021)
Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang.
Causaladv: Adversarial robustness through the lens of causality.
*arXiv preprint arXiv:2106.06196*, 2021.
- Zhu et al. (2021)
Yao Zhu, Jiacheng Ma, Jiacheng Sun, Zewei Chen, Rongxin Jiang, Yaowu Chen, and Zhenguo Li.
Towards understanding the generative capability of adversarially robust classifiers.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 7728–7737, 2021.

## Appendix A Training Details

### A.1 Data Preparation

The ten datasets discussed in the main text are categorized into two types: binary datasets and multi-label datasets. CelebA Smile, CelebA Age, CelebA Gender, CelebA-HQ Smile, CelebA-HQ Age, CelebA-HQ Gender, and BDD-OIA belong to the first category, while Cougar vs. Cheetah, Sorrel vs. Zebra, and Egyptian Cat vs. Persian Cat fall into the latter. For binary-label datasets, if the original predicted label is $y$ ($y$ can either be 0 or 1), then the target label is set to $1-y$. For multi-label datasets, since ImageNet contains 1,000 classes, the prediction and target labels differ for each dataset subset. Specifically, Cougar vs. Cheetah: predicted label is 286, corresponding to “cougar”; target label is 293, corresponding to ”cheetah”. We adopt this setting to perform a transition from the “cougar” class to the “cheetah” class. Sorrel vs. Zebra: The predicted label is 339, which corresponds to ”sorrel”; the target label is 340, which corresponds to ”zebra”. We adopt this setting to perform a transition from the “sorrel” class to the “zebra” class. Egyptian Cat vs. Persian Cat: predicted label is 283, corresponding to ”Persian cat”; target label is 285, corresponding to ”Egyptian cat”. We adopt this setting to perform a transition from the “Persian cat” class to the “Egyptian cat” class.

### A.2 Hyperparameters Settings

We provide the detailed hyperparameter settings for each dataset in Table [5](https://arxiv.org/html/2507.09881v2#A1.T5). $\alpha$ is the weight of the adversarial defense loss, $\lambda$ is the weight of the $l_{1}$ loss used during mask generation, and $\gamma$ is the threshold for generating the binary mask.

**Table 5: Hyperparameter Settings for All Datasets.**
| Datasets | $\alpha$ | $\lambda$ | $\gamma$ | PGD Iterations | Diffusion Steps |
| --- | --- | --- | --- | --- | --- |
| CelebA Smile | 0.2 | 0.001 | 0.1 | 50 | 500 |
| CelebA Age | 0.5 | 0.001 | 0.1 | 50 | 500 |
| CelebA Gender | 0.4 | 0.001 | 0.1 | 50 | 500 |
| CelebA-HQ Smile | 0.01 | 0.001 | 0.1 | 50 | 500 |
| CelebA-HQ Age | 0.03 | 0.001 | 0.1 | 50 | 500 |
| CelebA-HQ Gender | 0.005 | 0.001 | 0.1 | 50 | 500 |
| BDD-OIA | 0.005 | 0.001 | 0.05 | 50 | 1000 |
| Sorrel-Zebra | 0.03 | 0.001 | 0.15 | 100 | 1000 |
| Cougar-Cheetah | 0.02 | 0.001 | 0.15 | 100 | 1000 |
| Persian Cat-Egyptian Cat | 0.04 | 0.001 | 0.15 | 100 | 1000 |

## Appendix B Additional Experimental Results

Due to the page limit, we omit the results of CelebA, CelebA-HQ Gender, and BDD-OIA from the main text and provide them in this section as supplementary material. We first present the quantitative analysis. Table [6](https://arxiv.org/html/2507.09881v2#A2.T6), Table [7](https://arxiv.org/html/2507.09881v2#A2.T7), and Table [8](https://arxiv.org/html/2507.09881v2#A2.T8) summarize the results on the CelebA dataset, Table [9](https://arxiv.org/html/2507.09881v2#A2.T9) concludes the results on CelebA-HQ Gender, which exhibit a similar overall trend to that observed on CelebA-HQ Smile and Age. Although ACE achieves slightly better performance than our method on LPIPS and S^3, this comes at the cost of reduced flip rate and FID. Importantly, FID is a key metric, as it measures whether the overall distribution of counterfactual images is close to that of real images. Similarly, although DiME attains slightly better results on sFID, its COUT is extremely low and even negative, indicating that the classifier lacks confidence in the counterfactual transitions produced by DiME. Table [10](https://arxiv.org/html/2507.09881v2#A2.T10) presents the results on the BDD-OIA dataset. Although ACE performs well on several metrics, our method outperforms it in terms of COUT, demonstrating that the classifier retains sufficient confidence in the counterfactual transitions.

**Table 6: Evaluation Results on CelebA: Smile.**
| Methods | FR $\uparrow$ | COUT $\uparrow$ | LPIPS $\downarrow$ | S^3 $\uparrow$ | CD $\downarrow$ | FID $\downarrow$ | sFID $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DiVE | - | - | - | - | - | 29.4 | - |
| DiVE^100 | - | - | - | - | 2.34 | 36.8 | - |
| STEEX | - | - | - | - | - | 10.2 | - |
| DiME | 97.60% | -0.0451 | 0.0896 | 0.9158 | 1.99 | 12.3 | 25.48 |
| ACE | 99.05% | 0.6547 | 0.0273 | 0.9822 | 2.05 | 4.2 | 26.38 |
| Ours | 99.65% | 0.7943 | 0.0299 | 0.9819 | 2.29 | 3.9 | 25.70 |

**Table 7: Evaluation Results on CelebA: Age**
| Methods | FR $\uparrow$ | COUT $\uparrow$ | LPIPS $\downarrow$ | S^3 $\uparrow$ | CD $\downarrow$ | FID $\downarrow$ | sFID $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DiVE | - | - | - | - | - | 33.8 | - |
| DiVE^100 | - | - | - | - | - | 39.9 | - |
| STEEX | - | - | - | - | - | 11.8 | - |
| DiME | 73.10% | 0.0381 | 0.0938 | 0.9158 | 3.81 | 20.2 | 34.37 |
| ACE | 99.00% | 0.5441 | 0.0326 | 0.9843 | 3.37 | 4.3 | 26.47 |
| Ours | 99.75% | 0.6760 | 0.0333 | 0.9829 | 3.17 | 4.3 | 25.76 |

**Table 8: Evaluation results on CelebA: Gender.**
| Methods | FR $\uparrow$ | COUT $\uparrow$ | LPIPS $\downarrow$ | S^3 $\uparrow$ | CD $\downarrow$ | FID $\downarrow$ | sFID $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DiME | 97.05% | 0.0636 | 0.0914 | 0.9158 | 4.91 | 14.3 | 27.9 |
| ACE | 91.75% | 0.4947 | 0.0360 | 0.9826 | 5.44 | 7.3 | 29.6 |
| Ours | 97.15% | 0.6304 | 0.0387 | 0.9807 | 4.53 | 6.3 | 27.9 |

**Table 9: Evaluation results on CelebA-HQ: Gender.**
| Methods | FR $\uparrow$ | COUT $\uparrow$ | LPIPS $\downarrow$ | S^3 $\uparrow$ | CD $\downarrow$ | FID $\downarrow$ | sFID $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DiME | 98.45% | -0.3630 | 0.1671 | 0.9146 | 7.55 | 17.3 | 28.0 |
| ACE | 26.75% | -0.3705 | 0.0376 | 0.9865 | 9.66 | 26.7 | 55.4 |
| Ours | 99.95% | 0.8555 | 0.0627 | 0.9831 | 8.30 | 7.6 | 29.0 |

**Table 10: Evaluation results on BDD-OIA.**
| Methods | FR $\uparrow$ | COUT $\uparrow$ | LPIPS $\downarrow$ | S^3 $\uparrow$ | FID $\downarrow$ | sFID $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| DiME | 60.50% | -0.4865 | 0.1119 | 0.9638 | 62.3 | 225.17 |
| ACE | 99.90% | 0.7278 | 0.0318 | 0.9956 | 9.5 | 91.70 |
| Ours | 99.50% | 0.7745 | 0.0449 | 0.9949 | 15.3 | 92.35 |

We then present the qualitative analysis. Since ACE achieves the strongest performance among all baselines, we focus our comparison on the counterfactual images generated by our method and those produced by ACE. As shown in Figure [5](https://arxiv.org/html/2507.09881v2#A2.F5), similar to the results on the CelebA dataset, the counterfactual images generated by ACE contain many unintended modifications. These include changes in teeth and chin color, the appearance of exaggerated facial creases, and the emergence of extraneous patches on the cheeks. In the transition from “old” to “young”, the counterfactual images generated by our method exhibit comparable visual quality to those produced by ACE, with no obvious undesired changes in either case. As a result, we do not highlight any specific regions for this comparison. However, our method aligns more closely with the intended transformation—for example, by effectively removing age spots and smoothing wrinkles—demonstrating greater semantic consistency with the target concept. In the transition from ”male” to ”female”, we omit the highlight for the same reason.

The extended qualitative results on CelebA, CelebA-HQ, BDD-OIA, and ImageNet are presented in Figures [5](https://arxiv.org/html/2507.09881v2#A2.F5) to Figure [10](https://arxiv.org/html/2507.09881v2#A2.F10). On CelebA and CelebA-HQ, we highlight the unintended alterations present in the counterfactual images generated by ACE, while also marking the corresponding regions where our method yields noticeable quality improvements. For BDD-OIA and ImageNet, since the visual differences between our method and the baseline are minor, we only present the counterfactual images generated by our method, with the regions of change relative to the original images highlighted.

Figure: Figure 5: Qualitative results on CelebA-HQ. Red circles highlight unintended changes produced by ACE, while green circles indicate the corresponding improvements made by our method.
Refer to caption: x5.png

Figure: Figure 6: Extended qualitative results on CelebA. Red circles highlight unintended changes produced by ACE, while green circles indicate the corresponding improvements made by our method.
Refer to caption: x6.png

Figure: Figure 7: Qualitative results on BDD-OIA. Green circles highlight the counterfactual changes. The right column for each transition direction shows a zoomed-in version.
Refer to caption: x7.png

Figure: Figure 8: Extended qualitative results on BDD-OIA. Green circles highlight the counterfactual changes. The right column for each transition direction shows a zoomed-in version.
Refer to caption: x8.png

Figure: Figure 9: Qualitative results on ImageNet. Green circles highlight the counterfactual changes.
Refer to caption: x9.png

Figure: Figure 10: Extended qualitative results on ImageNet. Green circles highlight the counterfactual changes.
Refer to caption: x10.png

## Appendix C VLM Prompt Design

In this section, we provide the prompt designs for each dataset to facilitate evaluation using Vision-Language Models (VLMs). While the evaluation dimensions are consistent with traditional metrics, the key distinction lies in our use of a VLM as a querying agent. Rather than relying solely on conventional metrics—which may be limited in capturing perceptual or semantic nuances—we engage the VLM through human-centric, dialog-style questions. This enables more holistic and perceptually grounded evaluations, aligning better with human judgment and interpretability.

CelebA Smile
You are given two sets of facial images: one set contains the original images, and the other contains the corresponding counterfactual images.
The target attribute to be modified is smile. If the person in the original image is smiling, the target is ”not smiling”; if the person is not smiling, the target is ”smiling”.
Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., smiling or not smiling) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

CelebA Age
You are given two sets of facial images: one set contains the original images, and the other contains the corresponding counterfactual images.
The target attribute to be modified is age. If the person in the original image is old, the target is ”young”; if the person is young, the target is ”old”.
Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., young or old) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

CelebA Gender
You are given two sets of facial images: one set contains the original images, and the other contains the corresponding counterfactual images.
The target attribute to be modified is age. If the person in the original image is male, the target is ”female”; if the person is female, the target is ”male”.
Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., male or female) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

CelebA-HQ Smile
You are given two sets of facial images: one set contains the original images, and the other contains the corresponding counterfactual images.
The target attribute to be modified is smile. If the person in the original image is smiling, the target is ”not smiling”; if the person is not smiling, the target is ”smiling”.
Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., smiling or not smiling) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

CelebA-HQ Age
You are given two sets of facial images: one set contains the original images, and the other contains the corresponding counterfactual images.
The target attribute to be modified is age. If the person in the original image is old, the target is ”young”; if the person is young, the target is ”old”.
Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., young or old) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

CelebA-HQ Gender
You are given two sets of facial images: one set contains the original images, and the other contains the corresponding counterfactual images.
The target attribute to be modified is age. If the person in the original image is male, the target is ”female”; if the person is female, the target is ”male”.
Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., male or female) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

BDD-OIA
You are given two sets of animal images: one set contains the original images, and the other contains the corresponding counterfactual images.
The target attribute is traffic status. If the traffic participants and infrastructure in the image indicate that the current traffic condition is stop or slow down, then the target is moving forward. Conversely, if the image reflects a moving forward traffic state, the target is to change it to stop or slow down. Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., stop or moving forward) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

Sorrel - Zebra
You are given two sets of animal images: one set contains the original images, and the other contains the corresponding counterfactual images. The target attribute to be modified is animal type. The query attribute is ”sorrel”, and the target attribute is ”zebra”. Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., zebra) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

Cougar - Cheetah
You are given two sets of animal images: one set contains the original images, and the other contains the corresponding counterfactual images. The target attribute to be modified is animal type. The query attribute is ”cougar”, and the target attribute is ”cheetah”. Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., cheetah) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

Persian Cat - Egyptian Cat
You are given two sets of animal images: one set contains the original images, and the other contains the corresponding counterfactual images. The target attribute to be modified is animal type. The query attribute is ”Persian cat”, and the target attribute is ”Egyptian cat”. Please evaluate the counterfactual images based on the following four criteria, each rated on a scale from 1 to 10:

- •
Validity: Does the counterfactual image successfully transition to the target attribute (i.e., Egyptian cat) in a visually convincing way?
- •
Sparsity: Are irrelevant attributes and regions left unchanged, with only the necessary parts being modified?
- •
Proximity: From a human perceptual perspective, how close is the counterfactual image to the original one?
- •
Realism: Does the counterfactual image remain close to the data manifold and distribution, i.e., does it appear as a realistic image that could exist in the original dataset?

## Appendix D Additional Parameters Study

In this section, we provide the parameter study plots for all datasets other than those in Section [5.5](https://arxiv.org/html/2507.09881v2#S5.SS5), where the observed patterns are consistent with those discussed in the main text. This highlights the critical role of $\alpha$ in our method, as it controls the degree of causal constraint on the change of spurious factors.

Figure: Figure 11: Line plots of FR and COUT with respect to varying values of $\alpha$ on CelebA Age.
Refer to caption: x11.png

Figure: Figure 13: Line plots of FR and COUT with respect to varying values of $\alpha$ on CelebA-HQ Smile.
Refer to caption: x13.png

Figure: Figure 15: Line plots of FR and COUT with respect to varying values of $\alpha$ on CelebA-HQ Gender.
Refer to caption: x15.png

Figure: Figure 17: Line plots of FR and COUT with respect to varying values of $\alpha$ on ImageNet (Egyptian Cat - Persian Cat).
Refer to caption: x17.png

Figure: Figure 19: Line plots of FR and COUT with respect to varying values of $\alpha$ on ImageNet (Cougar - Cheetah).
Refer to caption: x19.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - How well does the proposed causal–spurious disentanglement generalize across datasets, and how sensitive are the results to the choice of the VAE architecture or latent dimensionality?
- Have you quantitatively verified that the causal loss truly preserves spurious features while altering causal ones?
- Since the method depends on the classifier’s gradients, how robust are the counterfactuals if the classifier itself relies on spurious correlations?
- Can the authors provide evidence that the generated counterfactuals remain perceptually and semantically distinct when evaluated by an independent or robust classifier?
- How consistent are the GPT-4o evaluation results across different prompt formulations or repeated runs?
- Could the causal guidance be implemented more elegantly, for example, directly in latent space or via a unified generative causal model, rather than through multiple post-hoc components?
- Does the causal guidance term risk over-constraining the perturbation, leading to weaker or insufficient edits as seen in the Male↔Female examples?
soundness: 2
strengths: - The paper tackles an important problem in counterfactual visual explanation by aiming to make generated examples causally faithful and semantically meaningful.

- The experimental evaluation is extensive and covers multiple datasets, metrics, and strong baselines, which supports the empirical effectiveness of the results.

- The paper is clearly written and easy to follow, with well-structured sections and illustrative figures that make the proposed method understandable.
summary: This paper proposes CECAS (Counterfactual Explanation via Causally Adversarial Steering), a framework for generating counterfactual visual explanations. The method combines an adversarial perturbation stage with a causal regularization term​, which aims to preserve spurious (non-causal) features while modifying causal ones, using a VAE-based disentanglement of causal and spurious representations. A second stage refines the resulting images using ℓ₁-based masks and diffusion-based inpainting to improve realism and sparsity. Experiments on CelebA, CelebA-HQ, BDD-OIA, and ImageNet subsets show that CECAS achieves higher flip rates and better perceptual quality metrics (FID, LPIPS) than existing counterfactual explanation methods such as DiVE, STEEX, DiME, and ACE.
weaknesses: My main concern with this paper is that the proposed method is overly complex and lacks conceptual elegance. The pipeline combines several loosely connected components: a PGD-based adversarial attack, a VAE for causal–spurious disentanglement, a mask extraction step, and diffusion-based inpainting. Each of them introduces its own set of hyperparameters and loss terms. While these parts collectively improve visual quality, the overall design feels more like a layered engineering solution than a coherent causal formulation. The causal guidance is implemented through an additional regularization term rather than a principled causal mechanism, and the diffusion refinement serves mainly as a post-hoc fix to clean artifacts instead of addressing their root cause within the generation process. As a result, the framework appears heavy, ad-hoc, and difficult to interpret or reproduce, which weakens the overall conceptual clarity and novelty of the work.
- The causal guidance mechanism is largely heuristic and depends on a VAE-based disentanglement that is not guaranteed to separate true causal and spurious factors.
- The method still operates in pixel space, so adversarial perturbations may flip predictions through imperceptible or non-semantic changes rather than meaningful causal edits.
- There is limited evidence that the generated counterfactuals truly preserve spurious features or modify causal ones, as no causal validation or ablation analysis is provided.
- The diffusion refinement improves realism but can also re-generate image regions, which weakens the interpretability of the counterfactual as a minimal intervention on the same instance.
- The GPT-4o–based vision-language evaluation is interesting but may introduce bias and lacks reproducibility or robustness analysis.
- Some qualitative figures are cluttered with red and green circles that distract from the visual comparison and are not essential to interpret the results.
- In Figure 5, the counterfactual edits for gender conversion appear too subtle to convincingly demonstrate a change in the intended attribute. The “Male to Female” counterfactuals still largely resemble male faces, and the “Female to Male” results also retain feminine features. This suggests that the method may struggle to produce sufficiently strong or semantically distinct attribute changes for certain categories.
- The quality and reliability of the generated counterfactuals heavily depend on the classifier used to guide them. If the classifier itself relies on spurious correlations or non-causal features, the resulting counterfactuals will likely reflect and reinforce those same biases rather than reveal true causal changes. This raises concerns about whether the method can produce meaningful explanations when the underlying model is imperfect or biased.
- The overall pipeline involves multiple ad-hoc steps, including PGD-based adversarial attack, VAE-based disentanglement, mask computation, and diffusion-based inpainting, which makes the approach complex and less principled.
- The diffusion-based refinement serves mainly as a post-hoc fix to remove artifacts, rather than addressing the underlying cause of imperfect or unrealistic counterfactual generation within the main optimization process.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 1
questions: Please refer to the weaknesses above.
soundness: 1
strengths: S1. The proposed approach is evaluated on a representative set of datasets with metrics covering the entire evaluation spectrum present in the current literature on CEs.
summary: This paper considers the problem of counterfactual explanation (CE) generation for classifiers in computer vision. The proposed approach adapts the ACE method to additionally limit the editing of spuriously correlated factors. This is achieved by adding a variational autoencoder (VAE) trained on the representations of the explained model, which are later fed into an auxiliary classifier trained to mimic the behavior of the original one. The resulting misalignment is used to regularize the optimization procedure. Empirically, the method is evaluated on a broad range of datasets and problems.
weaknesses: W1. The proposed methodology is flawed at its core. The authors propose generating CEs with an additional constraint on the spurious correlations learned by the explained model that limits their appearance in the explanations. However, the primary role of CEs is to actually reveal these correlations, since they highlight the model's unreasonable failure cases, which is desirable.

W2. The paper claims to be outperforming *recent state-of-the-art* methods for CE generation, while ignoring the actual recent papers on this topic that have appeared in top conferences in the last two years. For example, the paper is missing references to OCTET [1], DiG-IN [2] and RCSB [3]. These baselines are crucial, as they consider similar techniques for constraining the changes in the CEs. Moreover, the paper properly cites DVCE [4], which is a work concurrent to DiME and ACE, but fails to compare with it in any way.

W3. The novelty of the method is limited and the authors do not properly credit the ACE paper [5]. A quick glance at the paper's implementation shows that it is entirely based on ACE's codebase. The method is presented in a way that frames both phases (adversarial-attack-based and refinement) as contributions of the paper, while both are entirely taken from ACE. The only remaining novelty is the addition of the factor-disentangling VAE and the auxiliary classifier, a contribution that is logically flawed (W1).

W4. The paper does not provide any further details on the training procedure for the VAE and the auxiliary classifier, which seem to be the primary novelty claimed by the authors. Does the VAE require causal and spurious concept annotations? If yes, is the method limited exclusively to datasets that provide them? If not, how does its accuracy influence the effectiveness of CECAS? What is the error of the auxiliary classifier? These are just a few sample questions that are never addressed in this work.

W5. As the authors correctly point out, adversarial attacks (AA) result in imperceptible changes that influence the model's decision. Incorporating AA techniques into CE generation risks obtaining a modified image where the semantic edits are not actually responsible for the decision change, but rather the small AA-based pixel changes. The same can be said about the ACE method.

W6. Many less significant errors, some of which are pointed out below, show that the work is currently unpolished and requires crucial refinement:

1.  line 058: The actual problem with standard diffusion-based models is that they do not operate in semantically meaningful latent spaces, but rather in high-dimensional pixel spaces.
2.  "Two-colored nodes" are nowhere to be seen in Figure 1.
3.  Table 2 incorrectly bolds and emphasizes the results. It only considers the authors' method and not others.
4.  The Figure 4 caption incorrectly references the line plot of FR vs COUT. Why do the results of ACE and the proposed CECAS not differ perceptually? Is it because these methods are essentially the same on ImageNet (W3)?
5.  The $\alpha$ variable is introduced in an ambiguous way. It first appears as the step size for the PGD attack (Eq. 5), then is used as the regularization strength for the second term in Eq. 6. Moreover, figures like Figure 4 once again highlight its flawed influence - it restricts the range of edits to deviate from the spuriously correlated ones.
6.  lines 051-053: This example is inherently wrong, since the shape of a traffic sign is not a spurious factor, but rather a property used to characterize types of signs. Stating that either content or shape is *causally relevant to the model’s decision* is rather far-fetched, as such a statement requires access to the ground-truth of the model's explanation, which is almost always unavailable.

[1] Zemni et al., OCTET: Object-aware Counterfactual Explanations, CVPR, 2023

[2] Augustin et al., DiG-IN: Diffusion Guidance for Investigating Networks - Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations, CVPR, 2024

[3] Sobieski et al., Rethinking Visual Counterfactual Explanations Through Region Constraint, ICLR, 2025

[4] Augustin et al., Diffusion Visual Counterfactual Explanations, NeurIPS, 2022

[5] Jeanneret et al., Adversarial Counterfactual Visual Explanations, CVPR, 2023

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: Can you please explain how this makes a significant contribution beyond the myriad of CF image generation methods? You seem to claim you are the first to consider the separation of causal v. non-causal features, and that is the core contribution. This seems a bit of a stretch to me given work such as (but hardly limited to) these [1, 2].

Do you have any indication your method would be useful for tasks such as model improvement, HCI collaboration, teaching humans concepts learned by the model etc...

***

[1] Kocaoglu, M., Snyder, C., Dimakis, A.G. and Vishwanath, S., 2018, February. CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training. In International Conference on Learning Representations.

[2] Pawlowski, N., Coelho de Castro, D. and Glocker, B., 2020. Deep structural causal models for tractable counterfactual inference. Advances in neural information processing systems, 33, pp.857-869.
soundness: 3
strengths: The paper addresses an interesting problem of causality in images, and appears to avoid some unnecessary changes in the images based on the figures.

The experimental evaluation is heavily in their method's favor, suggesting it is a worthwhile contribution.

An extra qualitative evaluation adds weight to the actual image quality.
summary: The paper introduces a method for counterfactual image generation called CECAS. The core idea is to take a causal perspective on the image generation process to avoid spurious features, and instead focus only on the causally relevant ones. The paper's experiments show their method achieves better scores on their metrics of counterfactual flip rate and other plausibility metrics.
weaknesses: In general, my main critique of the paper is that it appears to be quite incremental in its contributions. There are quite a few papers addressing the issue of counterfactual image generation, and the contribution of this one is not overly clear to me.

For example, in the paper's claimed contributions they state "We provide a new perspective on the problem of counterfactual visual explanation by highlighting the critical role of causality". But please see e.g. [1], this just isn't true at all, plenty of work has highlighted this.

Most of the evaluation only compares against a single method ACE, which is quite weak.

I also wonder of the validity of these UNet-based CF methods (e.g. [2]) in light of current image generation tools such as Nano Banana, which can generate far superior image edits to these approaches. I understand of course that adapting such an approach to this would be non-trivial given the whole setup, but far more interesting in my opinion (and likely far more useful).

I am quite wary of metrics such as FID, I would much rather the authors evaluate the usefulness of the method compared to baselines in downstream tasks. That would be better and more convincing.

***

[1] Melistas, T., Spyrou, N., Gkouti, N., Sanchez, P., Vlontzos, A., Panagakis, Y., Papanastasiou, G. and Tsaftaris, S.A., 2024. Benchmarking counterfactual image generation. Advances in Neural Information Processing Systems, 37, pp.133207-133230.

[2] He, Z., Zuo, W., Kan, M., Shan, S. and Chen, X., 2019. Attgan: Facial attribute editing by only changing what you want. IEEE transactions on image processing, 28(11), pp.5464-5478.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: Restating or repeating some of the weaknesses:
1.  In Table 2, the metrics contradict the claims of the paper. Correlation Difference (CD) score for CECAS is 5.33, worse than the baseline DiME (4.00). The authors don't explain satisfactorily why CD should be discounted. Could you elaborate on why your causally guided method performed worse on this specific metric?
2. Can you expand on the inpainting logic? it's possible that the $ x_t $ and $  x_t^0 $ are flipped in the equation.
3. The paper's central claim rests on the VAE's ability to disentangle C and S. Are there any quantitative or qualitative evidence that this disentanglement was successful and how it was validated?
soundness: 2
strengths: - Strong motivation: The paper clearly articulates a significant weakness in existing visual counterfactual methods: they often latch onto spurious correlations (e.g., beards and age) rather than causal features, leading to unrealistic and unhelpful explainations.
- Introducing a causal perspective to guide the adversarial generation process is a novel and meaningful contribution. It directly tackles the reasoning behind poor counterfactuals.
- The experimental design tests the method across four distinct datasets with varying attributes (CelebA, CelebA-HQ, BDD-OIA, and ImageNet) and uses a robust set of quantitative metrics. So it's thorough.
- The authors incorporate a novel, human-like evaluation using Vision-Language Models. This adds a valuable layer of semantic validation to their claims.
- Convincing qualitative results: The visual examples demonstrate that CECAS avoids the unrealistic artifacts produced by other methods, such as the spurious beard growth on female faces during an age transition.
- The parameter study on the causal constraint weight (`α`) effectively demonstrates its impact on the trade-off between flip rate and causal faithfulness, reinforcing the method's core claims.
summary: This paper proposes a new framework for producing images for counterfactual explanation. This framework, CECAS, integrates causal reasoning with adversarial perturbation methods. The primary contribution lies in addressing spurious correlations during counterfactual generation by constraining modifications to causally relevant features while preserving spurious factors unchanged. The approach combines adversarial attacks (PGD), diffusion models (DDPM), and a causal disentanglement mechanism to produce counterfactuals that achieve high validity, sparsity, proximity, and realism across multiple benchmark datasets.
weaknesses: - Causal assumption is invalid (or at the least, unvalidated). The paper assumes the independence of causal (`C`) and spurious (`S`) factors (`C⊥S`)  but never validates it. This v-structure in Figure 1 is a strong assumption.
- The VAE-based disentanglement is underspecified, which is a major weakness . The paper provides no details on the VAE architecture, its training, or any quantitative or qualitative evidence that the disentanglement was successful. Without this, there is no proof that the `Ls` loss is actually constraining spurious factors; it's just constraining an arbitrary, unvalidated latent subspace.
- The baselines are narrowly focused on other recent generative methods (STEEX, DiVE, DiME, ACE). A comparison against a wider range of non-diffusion CE methods would have strengthened the paper.
- The VLM evaluation is presented without any scientific rigor. There is no discussion of prompt bias, nor reliability (e.g., running the same prompt multiple times), and no validation against human-in-the-loop judgments.
- In the mask generation equation, the paper divides the pixel difference by M, which it defines as the maximum value after the summation. Is M calculated once per image, or is it a constant for all images, or calculated based on a local window? M is ambiguous, and hence not reasonably reproducible.
- In equation 9, if `m=0`, $ [(1-m)x_t + mx_t^0] $ becomes $ x_t $. If `m=1`, it becomes $  x_t^0 $. This tells the model to keep the counterfactual in the unchanged areas and the original image in the changed areas? This seems backwards (or I could be misunderstanding it).
-  In Table 2, the metrics contradict the claims of the paper. Correlation Difference (CD) score for CECAS is 5.33, worse than the baseline DiME (4.00). The authors don't explain satisfactorily why CD should be discounted.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. Why should we disregard spurious features in counterfactual explanations, given that exposing reliance on them is a key purpose of XAI? Provide a concrete use case where your approach yields more valuable insight.  
2. Precisely how is the VAE trained to obtain \(C\) and \(S\)? What losses, architectures, and supervision are used? How do you verify that the split corresponds to causal versus spurious factors?  
3. What is the exact role of the classifier $g$?
4. Provide ablations isolating each component: causal loss only, post-refinement only, full method. How much improvement stems from the simple inpainting step?  
5. Discuss or cite causal counterfactual generation literature and clarify how your method differs.
soundness: 1
strengths: - Ambitious goal of integrating causal reasoning with counterfactual visual explanation.
- Combines diffusion models with adversarial perturbations, which is technically non-trivial.
- Proposes a post-hoc refinement step that masks only important regions for modification.
- Includes experiments on multiple datasets with qualitative and quantitative results
summary: The paper proposes CECAS, a framework for generating causally guided counterfactual visual explanations. The method combines (1) a PGD-based adversarial perturbation guided by a causal penalty that aims to modify only causal (not spurious) features, and (2) a post-refinement stage that uses thresholding and diffusion-based inpainting to improve image quality. The authors claim this approach achieves better balance between validity, sparsity, proximity, and realism. Experiments on CelebA, CelebA-HQ, and several ImageNet subsets show improved quantitative metrics compared to prior diffusion-based counterfactual methods (ACE, DiME). They also evaluate realism with a vision-language model (GPT-4o) acting as a human-like judge.
weaknesses: 1. **Weak motivation.** The central premise—that counterfactuals should change only causal, not spurious, features—is not convincingly justified. In explainability, highlighting spurious dependencies is *often desirable* to audit models. The paper gives no scenario where suppressing spurious cues yields better insight.  
2. **Unclear and unsound causal mechanism.** The role of the auxiliary classifier \(g\) and the VAE-based decomposition \(R\!\to\!(C,S)\) is vague. The method trains \(g\) on “spurious” features to predict \(Y\), then penalizes the change in its logits, but this contradicts the notion of “spurious.” The causal factorization is neither theoretically grounded nor empirically validated.  
3. **Missing ablations.** The architecture is complex but not decomposed experimentally. There is no evidence separating contributions from (a) the causal projection, (b) diffusion denoising, and (c) post-refinement. Qualitative examples suggest most gains come from the simple mask + inpainting step.  
4. **Weak evaluation.**  
   - Only a few datasets and two baselines are compared on all metrics.  
   - Standard deviations are omitted (“negligible”), preventing assessment of significance.  
   - No metrics measure *causal faithfulness* despite the causal framing.  
   - Reliance on GPT-4o scoring lacks validation against human judgments.  
5. **Poor figure and writing quality.** Figure 2 is overloaded and incomprehensible; several notations (orthogonalization, R′, g) are undefined in text.  
6. **Omission of relevant literature.** Prior work on causal generative models and causal counterfactual generation (e.g., causal diffusion, SCM-based image editing) is ignored, overstating novelty.


Decision: Reject