V-CECE: Visual Counterfactual Explanations via Conceptual Edits
Authors: Nikolaos Spanos
Maria Lymperaiou
Giorgos Filandrianos
Konstantinos Thomas
Athanasios Voulodimos
Giorgos Stamou
Venue: NeurIPS 2025 poster ()
Keywords: counterfactual explanations
diffusion models
LVLMs
XAI
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Recent black-box counterfactual generation frameworks fail to take into account the semantic content of the proposed edits, while relying heavily on training to guide the generation process. We propose a novel, plug-and-play black-box counterfactual generation framework, which suggests step-by-step edits based on theoretical guarantees of optimal edits to produce human-level counterfactual explanations with zero training. Our framework utilizes a pre-trained image editing diffusion model, and operates without access to the internals of the classifier, leading to an explainable counterfactual generation process. Throughout our experimentation, we showcase the explanatory gap between human reasoning and neural model behavior by utilizing both Convolutional Neural Network (CNN), Vision Transformer (ViT) and Large Vision Language Model (LVLM) classifiers, substantiated through a comprehensive human evaluation.

V-CECE: Visual Counterfactual Explanations via Conceptual Edits
Nikolaos Spanos
Maria Lymperaiou Giorgos Filandrianos
Konstantinos Thomas Athanasios Voulodimos
Giorgos Stamou
National Technical University of Athens
( mar,ialmpp, geof ila, kthomas Baahomas@@alll H@ails. ece .tta..rrr gr thanosv@mail ntua gr, gstamêes .tua.gr
Abstract
Recent black-box counterfactual generation frameworks fail take into account the semantic content of the proposed edits, while relying heavily on training guide the generation process. We propose novel, plug-and-play black-box counterfactual generation framework, which suggests step-by-step edits based on theoretical guarantees of optimal edits to produce human-level counterfactual explanations with zero training. Our framework utilizes pre-trained image editing diffusion model, and operates without access the internals of the classifier, leading to an explainable counterfactual generation process. Throughout our experimentation, we showcase the explanatory gap between human reasoning and neural model behavior by utilizing both Convolutional Neural Network (CNN), Vision Transformer (ViT) and Large Vision Language Model (LVL.M) classifiers, substantiated through comprehensive human evaluation. Project page and code are available at htttts::/iickppaanots httpk/appaan55..i.i.... github. io/vcece
Introduction
Ensuring fairness and trust in artificial intelligence (AI) applications has been of paramount impor- tance, especially since the vast adoption of large and uninterpretable models, such as Large Language Models (LLMs) (33] and Diffusion Models [17]. For this reason, explainable AI (XAI) has become an essential research field, fostering understanding and accountability of black-box models, while rendering them ethically deployable in practical high-stakes circumstances [35]. Among the various XAI techniques, counterfactual explanations (CEs) stand out as powerful tool for providing action- able and human-centric insights 14]. The what-if nature of CEs underpins cause-effed relationships in human reasoning [18], aligning with the concept of performing minimal input perturbations to simulate those what-if scenarios. Observable outcome changes indicate that the corresponding input perturbation, though subtle, sufficiently influential, uncovering reasoning path inside the model. By aggregating these reasoning paths within a well-structured CE framework, general patterns of the model's 's decision- makiing are revealed, highlighting potentially flawed reasoning directions.
The parallel venture of counterfactual generation considers generated image x² as an imaginary counterpart of existing image which succeeds in altering the prediction of classifier C. To this end, an appropriate CE framework should be able to answer why x" was classified in class L" rather than L * L in interprrateeeee terms. The real value of CEs in comparison to any possible perturbation that achieves flipping from L to L"" lies in the semantics [5] considered for classification, well for the generation of =*: high-level concepts succeed in explaining why L* instead of L, while superficial changes -such as altering a pixel in an image- may flip the classification label but fail to provide meaningful tractable explanation for the reason behind the change.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The counterfactual image generation literature frequently reports dispersed edits B 25 32 23
42 22 all of which do not support the semantic desiderata of CEs in the first place. Even though high-quality generations are often achieved, they are not reproducible or interpretable by humans [21), undermining the primary
non-actionable outcomes [36] or uninterpretable changes overall [99 T0
goal of enhancing human understanding of model decision-making [ 39]. Additionally, methods
that apply edits to generate images without user explanations, such as those relying on diffusion models trained with classifier labels [23 21). can yield misleading results due to their heavy reliance on the diffusion process. For example, Stable Diffusion may add a blob resembling a bus, leading to the mistaken conclusion that the bus influences the classifier's output, while it is unclear if the classifier indeed identifies the bus or reacts to pixel distribution changes.
At the same time, semantic- CE algorithmss underline biases of black-box models 8 9 ensuring actionability and interpretability of edits under striet explanation frameworks. However, they assume a a-priori that the classifier under explanation holds the same understanding of semantics as humans. To this end, we observe a significant gap in CE literature: there is no investigation on what characteristics are important for CEs from the perspective of neural classifiers and humans, and where these two perspectives disagree. In this paper, we argue that this issue is even more problematic than the case of uninterpretable adversarial edits, it introduces ambiguity regarding whether the edits are comprehensible and reproducible for humans. This ambiguity can result in misleading CEs, ultimately compromising the effectiveness of related explainability algorithms. By acknowledging these limitations, we break down the problem of defining CEs into two discrete steps. The first one addresses the question: "Is there a discrepancy between human and neural perspectives?" In other words, "Can the classifier's 's decision-making process be explained using human-level semantics?" If the answer to the first question is affirmative, the second step is to determine: "What are the minimum edits that actually change the classifier's 's label?"
Existing counterfactual frameworks often address the second question only partially, either bypassing the first question entirely or assuming an ungrounded answer. Works such as [23] 21 disregard semantics entirely, making it difficult to accurately interpret the second question, as the context provided by the first is essential for understanding its implications. On the other hand, approaches that assume the classifier operates at semantic level [9] g fail to provide evidence or indications to substantiate this assumption, effectively sidestepping the first question altogether.
Our proposed V-CECE is the first to systematically address both of these questions, introducing a CE generation pipeline focused on two main directions: first, we exclusively support conceptual edits based on well-defined visual semantics, guaranteeing the optimality of edits. Secondly, we apply the extracted optimal edits using state-of-the-ar (SoTA) diffusion models for counterfactual image generation. Ultimately, the effectiveness of these images is used as a proxy for identifying the discrepancy between the classifier's and human perspectives. Overall, our generation pipeline operates in a fully black-bou setting in which we do not train any of the participating modules, nor do we optimize over the final CEs, offering a highly efficient plug-and- and-play solution.
2 Related work
Counterfactual explanations comprise a human- interprrinttr way of explaining deep learning models, applicable in several scenarios [14], thus they have been favored in recent explainability literature. By probing a pre-trained model and observing its output changes, we can approximate its behavior without accessing its internals [40]. requirement that is on par with the development of proprietary models. A line of work focuses on concept- based counterfactuals 13 2
[ driven by the claim that there can be explanation without semantics [5]. In that case, level concepts are used to explain why a classifier made a decision and what could have been different in order to alter the classification outcome. In this paper, we exclusively work with conceptual CEs.
Diffusion models for counterfactuals The advent of diffiusion models 17 has rapidly elevated the field of image synthesis, also allowing high-quality counterfactual image generation. Initial attempts manage to modify observable regions of an image to change its classification [3), even though they do not deal with well-defined semantics. Causal white-box frameworks enhance counterfactual generation with theoretical constrains (36). even though their actual generations alter class identity
in somehow non-realistic directions. Also in the white-box spectrum 7 21 22 37 25 (2), access
to the classifier's gradients is required to generate counterfactuals, thus being unable to explain

---

CMRA
Local Balls IUYLM Ortwtng
Beire Orapks
Meaaica
Melric
Senartk Prepoel
Gloaal Esta Impertance
+ Stable Ohaato Haparriig
(3t inpurtinie
Groandsgooo
Didaning
Ingilatinn Mabet
Exp Masil Sep
Explaration
Camposset
Figure I: Outline of V-CECE to address the explanatory gap between humans and models. The semantic edit framework reviews the image and proposes edits and their iterative sequence. The edits are then implemented through a combined object recognition and diffusion model. The edited images are reviewed from the respective models to ascertain whether or not the edit had the desired effect. The edited images are evaluated through visual metrics, counterfactual metries and human survey.
proprictary classifiers. [10] deviate towards model-agnostic approaches, though they do not focus on high-level concepts, while misgenerations are also present. In the black-box setting also lies the work of [23], producing dispersed edits within the images. Other endeavors target to manipulate the presence or absence of features rather than flipping the classification label [42]. Our work belongs in the model-agnostic setting, treating both the classifier as well the diffusion model as black boxes, while only human- iitmannntterrrttable conceptual edits are considered over other perturbations.
Method
Our V-CECE pipeline mainly consists of an explanation component and generative component (Figure D. In the explanation stage, we query for the closest image pair in terms of semantics that belong to distinct visual classes L, L", given any pre-trained classifier C. To optimally transit from to L", a set of conceptual edits E' is calculated, incorporating feasible insertions I, deletions D and substitutions S of concepts. The guarantee of optimality of the proposed method stems from the underlying mechanism used to compute the closest image pair, as well as the correspondingly optimal calculation of the edit set F. Once computed, F is passed to the generative stage which ultimately executes them to produce the counterfactual image. A grounding module masks the area to be edited, driving the generation process of diffusion model. In each generation step, C' decides whether the generated image actually changed its class or not, terminating the generation process in the first case.
3.1 Guarantees of optimality
The explanation component is responsible for guaranteeing that the proposed semantic explanations to achieve T, - I.* are optimal, meaningful and actionable. Specifically, it suggests insertions deletions D) and replacements R of concepts driven by their distances on pre-defined knowledge graph. our work, we utilize WordNet [31], due to its inclusion of multiple semantic meanings and its presence in previous work, such in 8 9 I]] Given a semantic concept s; from sample r, L and another semantic concept sj from a sample x} E L (8; + $ÿ), the cost c(Sa,-);) of substituting optimally s, with sy is equivalent to finding the shortest path min (dist sz)) between these concepts on the knowledge graph using pathfinding algorithms, such as Dijkstra. Similarly, and D operations require traversing the knowledge graph up to its root, invoking a related edit cost (c(Is;) for I and c(D,.) for D) equal to the distance of this path. Actionability of edits is also guaranteed, as non-actionable edits can be excluded by assigning them an infinite cost, thereby removing them from consideration. Overall, the edits to be performed in each step of the I. - L transition are defined by the following optimization function:
min( Σ c(Is;)
C(D...). R.EL
Σ c(S..13)))
(1)
ELL
S,EL,JELL"
The solution of equation I can be deterministically provided using bipartite matching, where concepts from L and L items are placed onto a bipartite graph G. with each edge (s.,s;) having corresponding weight We(ne)g) = c(Sa..os;) ), while also dummy nodes da, ia; to simulate and D operations are
3

---

added, with associated costs of Wecr)da)) cDDs.), w m.g = c(Ts)). The minimization of
this matching is performed via the Hungarian algorithm [26]. resulting in a set E = D, of minimum cost edits that satisfy the L + Le transformation. Bipartite matching is analogous to an m x n assignment problem, where m and n correspond to the number of source and target concepts respectively; this problem is solvable in (mm log n.) time via the Hungarian algorithm.
3.2 Selection of edits
The explanation component returns the optimal set of semantic edits E to transform an input image belonging to class L into existing image classified L" Importantly, E' constitutes the provably minimal set of edits such that, if all are applied, the predicted class is guaranteed to change. However, there is guarantee that this set is uniquely minimal, nor that proper subset E' C E could not also suffice to induce the same class change. To approximate such a minimal effective subset E', we iteratively apply edits from E until the classifier's 's prediction transitions from L to L To determine which edits (s1.6)) E E will be actually performed, we employ three distinct methods.
Local Edits After extracting the edits ((e,a)) E E as driven from the explanation component, we must decide their order. Lacking human context cues, we delegate this ordering procedure to Large Vision-Language Model (LVLM). At every step, the LVLM receives the current image plus the remaining edits from E and selects the next action-insert, delete, or subsitute. We then update the image and repeat. Supplying the updated image each round prevents the logical inconsistencies that arise when the whole sequence is produced at once. Prompts provided in App. B.
Global Edits Local reasoning overlooks systematic biases present in the classifier. Inspired by [ we ask: Which semantic edits most often flip images from class L. to L Running the Section 3.1 algorithm over all images in L,, we tally every edit ((5.18)) and score it according to the following:
Lag1
D2. + 18,, -23 -8.3-a.!
(mportance((ss.s3))
(2)
le(xs,0)) E E51
We then apply edits in descending importance: e.g. if delete bed ranks highest and a bed is present, we remove test if the label altered, or proceed the next edit until the class changes.
Local-Global Edits Global ordering exploits classifier shortcuts based on global biases but ignores scene details, whereas local ordering does the opposite. A balanced approach applies only the local edits proposed for the specific image, yet orders them by the global importance scores, thus retaining scene-aware changes while still taking advantage of biases imbued in the classifier.
3.3 Performing the Edits
We apply edits to the image using a pre-trained, frozen diffusion model. While training the diffusion model could improve the alignment of generated images with the dataset, it would also transfer statistical biases present in the data to the generation process [12 15), leading to artificially favorable counterfactual images; this runs counter to our goal of using semantic edits to test the classifiers' semantic comprehension. By relying on a model not directly trained on our data, we ensure that any inherent bias remains consistent across all experiments, creating a fair foundation for comparison.
We leverage the Stable Diffusion v1.5 Inpainting model D to edit images while closely following the prompts and fully repainting the masked regions. Each image is processed for 40 steps with the DPM++ 2M SDE sampler (34] and an automatically selected scheduler. We opt for the default random seed, which can be fixed for reproducibility, while we abstain from applying a variation seed to keep outputs consistent. A high-resolution fix is enabled, adding an extra upscaling pass that improves final image quality.
To minimize commonsense artifacts introduced during editing, additional information should be provided. This information includes the optimal placement within the image for any object that has to be added. For instance, if a pillow is to be inserted, the most suitable location for it must be
Model card: ruwnamm//sabll wiw.-eleeddifuui--iinniiineiiiii

---

determined, such as on a couch. Moreover, when removing an object, it is pertinent to consider what most likely be behind it and replace it accordingly in order to maintain image continuity. Both of these steps are performed using the reasoning and common-sense understanding capabilities of a LVLM 4 (), and specifically Claude 3.5 Sonnet, which processes the image along with a task-specific prompt (to add or remove an object) as input. More details can be found in App. B.
4 Experiments
Datasets We experiment with distinct datasets for which the semantics play definitive role. First, we utilize BDD100K [43] that focuses on real-world autonomous driving situations, where semantics are important for whether car has to stop move. This dataset has been favored in previous counterfactual generation works [21] [3 thanks to the well-defined semantics representing each class, Moreover, following the state-of-the-art work on semantic counterfactuals [9], we replicate the Visual Genome experiments on the VG-Random subse 2 upon which we generate the final images. The object annotations of each image and dataset comprise the concept sets considered the bipartite graph construction of the explanation component.
Classifiers To ensure a fair comparison of our results with other methods for the BDD100K dataset, we employ the same DenseNet-122 121 classifier as in [23]. We extend to more convolutional classifiers, and specifically ConvNext 30) and EfficientNet [38], as well as to transformer-basee architectures, such as Swin [29]. Similarly, for Visual Genome, we use the ResNet18 classifier from 88 (.. The value of our method is further demonstrated when explaining closed-source, proprietary LVLMS on both datasets. In particular, we deploy Claude 3. Claude 3.5 Sonne and Claude 3 haiku S
as classifiers, prompting them to classify given images accordingly. In the case of Claude 3.7, experimentation is conducted with and without thinking. Self-consistency [41] is also employed to guarantee robustness of finally predicted labels, since hallucinations may be present in the LVLMM as-classifiers case. To this end, we repeat the classification process 7 times per image, considering the labels of each run as an indicator of each model's intrinsic classification ambiguity. Finally, we obtain the final label via majority voting.
Generative Module We leverage Stable Diffusion v1.5 Inpainting in the core of the generation process. The proposed edits are first passed through a pipeline of GroundingDINO (28) and SAM (Segment Anything Model) 24 to generate the concept masks. For inpainting, the positive prompt is decided by each edit from E, while we also make use of negative prompt to facilitate image manipulation and enhance realism. Only the masked area and a small expansion around it are affected, so that cohesion of the editing process is encouraged and common generation pitfalls are reduced. This process is repeated until label flip is achieved or the edit set E is exhausted. All experiments are conducted L40S GPU (48GB) with average memory usage of 70% (33.6GB). Technical details are provided in App.
Evaluation Following Jeanneret et al. 23 evaluate our counterfactual examples (CEs) along
three complementary dimensions. Realism is quantified with Fréchet Inception Distance (FID) [16 and the more robust CLIP Maximum Mean Discrepancy (CMMD) (0) We also evaluate through
SimSiam Similarity between the two generated domains (21. Effectiveness is captured by the Success Rate (the fraction of CEs that flip the classifier) and by the mean number of semantic edits [E] required to reach the target class, a cost-related metric overlooked in earlier CE work (9) Stability estimated from the proportion of identical labels obtained across seven independent model runs. Finally, because counterfactuals are meant for human interpretation, we conduct user study in which volunteers inspect the original image followed by the edited sequence, mark the step where they believe the label should change, and judge whether the final image appears free of noticeable artifacts. More details are provided in App. D
As we do not consider inter-concept edges, VG-Dense subset proposed in the same paper does not provide any new insights. hatprppppcccuudd------mm--------
"anthropic.claude ---aaadd------onee-t-e 2=xommet-22244102222422
authropic.claudde 3-haiku 201kk-20240307-v1100
5

---

Comparisons Our primary goal is mostly to explore the explanatory gap between humans and models, hameessing interpretable semantics in black-box manner rather than proposing new editor. To verify the framework's s potential, we also compare with both SoTA approaches for counterfactual image generation (STEEX [99], DIME 22), ACE
21 TIME [33), as well as semantic-based counterfactual algorithms that do not generate images 8 [9. In both cases, we compare with the same datasets utilized by these works, i.e BDDIO0K and Visual Genome (Random split) respectively. Of the prior work, STEEX conditions on semantic masks and ACE uses binary mask of the difference between the explanation and input image for refinement. We do not utilize masks for training, but only for object recognition and editing.
4.1 Results on BDD100K
Table 1 summarizes the outcomes of multiple counterfactual generation approaches evaluated on BDDIOOK. These methods are categorized by the extent of their access to the classifier, whether they require training, and whether they rely on an optimization approach produce counterfactual images. This distinction is crucial, as white-box methods are trained on the dataset and utilize an optimization strategy which, as evidenced by the results, allows them achieve nearly flawless performance, producing high-quality counterfactual images of exceptionally high SR. Nevertheless, since they depend heavily the classifier and dataset, the modifications they make often subtle and users may struggle to comprehend why the class changes, undercutting the core objective of explaining the classifier. In scenarios where the edits are more overt (e.g., not merely introducing imperceptible noise that shifts the class), these methods still provide no guarantees about the classifier's 's semantic reasoning, leaving the interpretation of the results highly variable. TIME likewise embeds latent inter-class differences to craft counterfactuals. yet its classifier's s vague semantic level still clouds interpretation. V-CECE instead fixes the semantic level and does not require datasct-specific training. letting us ask: "Does the classifier reason like a human?" For CNNs (DenseNet, ConvNeXt) the answer is No, as SR falls to 84.8-88.9%9 and the image quality as denoted by FID and CMMD) degrades heavily as the number of steps increase. However, this should not be seen as flaw in V-CECE itself, as its distinet behavior when using LVLMMs as classifiers achieves near- sear-perfect SR and even outperforms TIME-despit the latter being trained on the dataset-in generating higher-quality images. Consequently, any attempt to explain CNN classifiers at a human-like semantic level may result in poor image quality at best, and potentially misleading outputs at worst.
To validate the above hypothesis, we compare the CNN-based DenseNet with the LVLMs as classifiers not only in terms of visual metrics, but also by juxtaposing their average number of semantic edits [E] needed for label flip (Avgl E| column). We observe a significant disparity in both the avg |E and the visual metrics between DenseNet and LVLM classifiers. Without relying on the previous analysis and metrics, and by considering only the number of edits, this discrepancy could arise from either differences in the level of explanation required by DenseNet or the need for more semantic edits to alter its classification. Among the CNNs, EfficientNet flips labels fastest, requiring markedly fewer optimization steps. We attribute this to its leaner architecture: a smaller computational footprint and compound scaling constrain activations to semantically pertinent regions, reducing drift into irrelevant areas during training 4 This suggests that architecture can play role in bridging the semantic gap without sacrificing fidelity, but neural classifiers still lag behind LVLMs.
A finer-graines comparison of Claude 3.7 Sonnet with versus without the thinking module underscores the role of prompting. Activating the module demands more edits to change the label and yields lower FID scores, even though the underlying weights are identical. This aligns with reports that can hamper visual recognition tasks, likely because verbal reasoning struggles to capture visual cues [27].
The human survey sheds some light to the underlying cause of the human- model explanation level discrepancy. The related findings are detailed in Table Table (more results in App. E), where the average number of edits by both the respective models and humans on the same annotated BDD subset is presented. These results suggest that, from a human perspective, the label change in DenseNet should ideally occur three edits earlier on average than currently necessary, This points to a potential misalignment between human judgments and classifier perspectives on this task. Although this discrepancy does not necessarily indicate that DenseNet operates at different
Human evaluation
'Aeraagigg numbers for all ordering methods per classifier.

---

Table Comparison of CE image generation methods metrics and model design choices. denotes that related results are not reported by authors. Bold indicates best black-box results.
Method (1)
FID
CMMD
S3
SR
Avg. [bl
Access
Training
Optimiz.
(1)
(T)
(1)
(1)
STEEX
58.8
99.5
white-box
days
DiME
7.94
0.9463
90.5
white-box
days
ACE &
1.02
0.9970
99.9
white-hox
days
ACE b
1.56
0.9946
99.9
white-box
days
TIME
51.5
0.7651
81.8
black-box
hours
x
V-CECE DenseNet classifier
V-CECEL.cal
90.42
1.101
0.6254
88.9
4.77
V-CECEGbbu Global
99.37
1.232
0.5489
85.8
5.37
black-box
N/A
V-CECF Gloal-Globl
81.90
1.092
0.6169
84.8
5.23
V-CECE ConvNext classilier
V-CECEL.c 84.08
1.111
0.6130
95.36
4.91
V-CECE
99.21
1.243
0.5560
81.4
5.86
black-box
N/A
X
V-CECE, acai-Gilatal
92.94
1.312
0,5510
82.09
5.81
V-CECE EfficientNet classifier
V-CECELocal 67.27
0.767
0.6950
98.07
3.76
V-CECE Eloball
69.31
0.733
0.6940
87.82
4.1
black-box
N/A
x
V-CECELoc Local Global
69.31
0.758
0.6930
93.03
4.01
V-CECE
Swin classifier
V-CECE ocal
82,93
1.027
0.6160
94.06
4.71
V-CECE
92.76
1.025
0,5860
83.76
5.3
black-box
N/A
V-CECEr.ede alall-Cluull
87.93
1.034
0.6040
82.58
5.35
V-CECE
Claude 3 Haiku classifier
V-CECELoca
56.93
0.566
0.7667
95.14 14
3.19
V-CECE
59.94
0,516
0.7646
95.64
3.13
black-box
N/A
X
V-CECEL.ch
55.05
0.527
0.7528
95.17
3.21
V-CECE
Claude Sonnet classifier
V-CECE
62.64
0.524
0.7593
96.60
3.10
V-CECE
45.22
0.427
0.7635
97.80
2.65
black-box
N/A
X
V-CECEL.calee Locall-Goaall
42.76
0.364
0.7970
98.10
2.44
V-CECE Claude 3.7
Sonnet classifier (No thinking)
V-CECEL.cal
67.78
0.565
0.7679
99.5
3.03
V-CECEGloa
70.65
0.620
0.7394
98.51
3.47
black-box
N/A
x
V-CECELocal Global
68.17
0.529
0.7591
99.76
3.45
V-CECE Claude 3.7 Sonnet classifier (Thinking)
V-CECFL.CC Loocal
73.36
0.762
0.6165
98.2
3.78
V-CECEGbaa C.Gbbll
79.28
0.829
0.6490
96.41
4.37
black-box
N/A
X
V-CECE, aealCChlcalllllhhll
73.51
0.876
0.6750
97.73
4.07
semantic level from humans. this hypothesis gains support from the fact that the 59.7% of the counterfactual images, are visually incorrect (contain generation artifacts defy commonsense), as indicated by DenseNet's ss visually correct images rate average. As result, DenseNet does not flip its label concurrently with human judgments whilst the images appear more visually accurate.
Table Average human-survey results regarding perception of quality.
Avg. E Model (L)
Avg. [b] Human (L)
Visually correct images (%)
DenseNet
5.22
2.21
59.71
ConvNext
7.35
2.27
34.24
EfficientNet
5.96
2.66
30.17
Swin
6.31
2.25
56.66
Claude-3-Haike
2,91
1.88
69.58
Claude-3.5- 3.5-Sonnet
2.19
1.33
81.20
3.7Sonet
2.50
1.37
79.98
Claude-3.7-Somee Thinking
4.33
2.69
70.01

---

Bource imag
clssaffer
Clauda 3.5 Sonnet classifier
Stap1
Huraan ldbu Tkp
Step 6
Liriouhaa lavit Thp
Slup
Cloude Scnnet lobel tie
Figure 2:
An example of counterfactual generation using the global-local method for two different classifiers on the same input image, including the source image, the intermediate image generated from the edit in step I (removal of the car), and the final counterfactual image that prompted label change. In the of the LVLM, the counterfactual image is the same the image from step
does, however, change classes after an average of three additional edits, by which time the 59.7% of generated images start showing artifacts, denoting that their semantic integrity is compromised. Notably, EfficientNet, has the worst performance, indicating that despite the small amount of steps and relatively high fidelity, the images are not interpretable by human standards.
Qualitative results Figure presents an example of CE generation using the local-global edit method, displaying the source image from the class "Stop", alongside the outcomes the steps needed until label flip for DenseNet and Claude 3.5 Sonnet classifiers. DenseNet requires 6 steps of step-by-step generation to alter its label to "Move", leading to a highly edited image: note that the pavement the left and the building the right have been unnecessarily removed, while the halo added decreases the correctness of the generation. Inserting such semantically unrelated features raises significant doubts about DenseNet's semantic understanding, as its label flipping relies on such features. The car blocking the way is correctly deleted; however, this edit has been already performed in Step even though DenseNet has not effectively handled this semantic change to alter its label. On the other hand, humans agree that "Stop" - "Move" label flip should happen at Step 1. This perspective correlates with the outcome of Claude 3.5 Sonnet, which requires exactly one step to alter its label. This finding strengthens the assumption that Claude 3.5 Sonnet effectively grasps the semantics of each class, resulting in fewer passes from the generative module.
Discovering biases Another interesting observation pertains to the global edits required for classifier to change its label. By analyzing the edits, we find that for Claude 3.5 Sonnet the most common modifications required to transit from "SStop" to "Move" involve removing the concepts "car", "pole", "streetlight". and "person". ". Similar are the most common edits for the Claude 3.7 Sonnet with the thinking and not thinking module. For Claude 3 Haiku, the most common concepts are "car", "vegetation" " "pole", ", and "person" ". contrast, the edits for the CNN based networks sporadic, indicating that consistent steps could reliably alter the class, resorting label- to randomness. To further analyze this finding, we calculate the importance of cach concept.
Table presents the importance value of the most prominent concept, along with its standard deviation and the number of concepts with an absolute importance value greater than O, for each classifier. From this table, is evident that there is a significant dis- crepancy between the maximum im- portance values and the number of im- portant concepts in the CNN- classifiers compared to the LVLMs. This suggests that CNN-based classi- fiers tend to modify more objects over- all, with less consistency, indicating that there is a randomness in their decision making process.
Table 3: Importance and standard deviation of the most prominent concept for each classifier.
Classifier
Importance
#(Importance >>
DenseNet
0.23 1 0.04
35
Swin
0.17 + 0.03
55
ConyNeXt
0.16 + 0.03
40
EfficientNet
0.20 0.03
49
Claude Haiku
0.23 + 0.04
39
Claude 3.5 Sonnet
0.37 + 0.05
31
Claude 3.7 (no thinking)
0.40 + 0.06
28
Claude 3.7 (thinking)
0.32 + 0.05
27
These results reinforce that DenseNet's semantie space differs from V-CECE's. Claude Haiku inaccurately links roadside vegetation to the "Stop" label, requiring about more semantic edit than
8

---

Claude 3.5 Sonnet. For Claude 3.7 Sonnet, the critical edits remain the same with and without the thinking module, but their weights fall sharply when thinking enabled. confirming that the module adds randomness (see Table D. This global edits analysis underscores the significance of V-CECE, which due to its model-agnostic design, is able to operate even on proprietary models and provide insightful results regarding classification biases. This becomes prominent in the -lLassa--l case, where the semantic levels between the classifier and human explanations align. Despite its popularity in prior CE work, DenseNet yields explanations that are only partially aligned with our human-level semantic annotations. This suggests that, at least for our dataset and metrics, CNN features driven by statistical dependencies may be difficult to translate into clear, concept-level explanations.
Ambiguity in deciding the final label can shed more light regarding the classifiers' behavior. For this reason, we extract the label probabilities of the final layer in CNN and transformer-hased classifiers, while for LVMLs we keep the prediction for each of the 7 runs; the classification is performed on the source images, as well as on the generated ones for as many steps as required until label flipping. The label probabilities regarding the final class are illustrated in Figure 3 revealing interesting behaviors of the models under scrutiny. Regarding the non-LVLM classifiers, EfficientNet arises as the most consistent model, with label probabilities lying well above the rest. Overall, downtrend is observed for most classifiers and edit selection strategies, denoting that the more edits are performed, the less confident the classifier is. This is an expected outcome, since artifacts tend to occur when more edit steps are performed, strengthening the requirement for performing as few edit steps as possible. As for LVL.ss, The downtrend is less visible in the LVLMSs as classifiers case, showcasing advanced classification robustness despite artifacts in comparison to non-LVI.M classifiers, even though some decrease in classification confidence is unavoidable after numerous edit steps.
6.0
Crade on
2:00
Swi
6.0
Caade
Connon
Metrod
Clitall
0.00
Mettod
Catal
5.75
Local
6901
dee
mor
saagee
Figure 3: Ambiguity in different LVLMs (left) and CNNs (right) across different stages of the counterfictual generation process.
4.2 Results Visual Genome
Table shows the average number of semantic edits avg required to change class Visual Genome, compared with previous works on semantic counterfactuals [8] 9. This comparison reveals that V-CECE not only generates counterfactual images but also produces CEs with significantly fewer semantic edits than the rest. We also report the SR for each classifier employed by V-CECE (SR is not applicable for the non -eneererre methods we compare with). To evaluate the impact of each edit ordering technique on the generated image quality, we present a related breakdown for all our classifiers in Table 5
Table 4: Mean count of required edits and SR on VG. Bold entries represent the best values.
ResNet18
Claude-3-Haiku
Claude-3.5-Sonen Avg. |3] (4)
Method
Generate Image
Avg. [a] (L)
SR (t)
Avg. [E] (4)
SR (1)
SR (1)
Dervakos et al.
12.15
NIA
12.83
N/A
12.81
N/A
X
Dimitriou al. 9
12.18
N/A
12.88
N/A
12.84
N/A
V-CECE
2.53
96.41
2.06
96.59
2.68
93.36
/
V-CECEaaaall
2.54
97.77
2.74
99.49
2.68
98.13
2.56
98.43
2.62
99.77
2.96
98.87

---

Table S: Comparison of methods Visual Genome using ResNet18 and Claude classifiers.
R8
Claude-3-Haiku
Claude-3.5-oonet
Method
FID (1)
CMMD (J)
S3 (1)
FID (C)
CMMD ())
S3 (1)
FID (L)
CMMD (L)
S3 (1)
Local
90.42
0.233
0.6459
78.94
0.144
0.7154
78.94
0.212
0.6640
Global
82.05
0.295
0.6388
82.63
0.302
0.6286
90.98
0.292
0.6195
Local-Global
81.90
0.291
0.6169
75.66
0.302
0.6589
79.73
0.268
0.6259
Additionally, there is substantial discrepancy between the global explanations provided by B 9]. and the ones of V-CECE. Specifically, [8] report total of 121 edits with non-zero importance for changing the label of images initially classified as "Bedroom" using Claude 3.5 Sonnet. In contrast, V-CECE returns only 12 edits non-zero importance for the same dataset and classifier, denoting that its global edits proposed after generation significantly less noisy.
5 Limitations and Future Work
We recognize certain limitations within our framework and evaluation. Regarding our human evaluation experiments, the current cohort modest in size and scope, which limits statistical power, precision, and generalizability; accordingly. results should be regarded as sarly-insights into significant problem AI explainability. To address this, we will broaden the cohort include participants from varying disciplines and quantify inter-individual variability, for example, how single sample interpreted by different human raters, to assess inter-rater reliability and identify sources of disagreement. Apart from enhancing the human survey as a part of future work, we would like to extend the evaluation the framework across additional disciplines, including the medical domain, to surface challenges and explanation needs in settings with greater field dependence. We also plan to assess robustness to realistic noise in real-world data and measure its impact on both performance and interpretability. In parallel, we will evaluate white-box generative models and examine whether additive bias is detrimental for the editing modules, as well as the effects of masking and segmentation choices. Collectively, these steps are intended strengthen external validity, reduce uncertainty around effect estimates, and guide design refinements.
Conclusion
In this work, present the V-CECE framework which aims explore the explanatory gap between classifiers and humans driven by semantics. We prove that when employing LVLMs as classifiers we achieve a compatible semantic comprehension with humans. whereas that does hold for favorable CNN or ViT classifiers utilized in prior literature. Our black-box framework is able to incorporate any classifier for explanation without any training, providing human-level and discrete CEs, the classifier's degree of semantic understanding, and general classification biases. We hope this work proposes a new frontier for explainability analysis, where semantic coherence for artificial intelligence models is at the forefront.
7 Acknowledgment
This work was supported by AWS resources, which were provided by the National Infrastructures for Research and Technology GRNET and funded by the EU Recovery and Resiliency Facility.
10

---

References
[1] Abubakar Abid, Mert Yuksekgonul, and James Y. Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations. In International Conference Machine Learning, 2021.
[2] Arjun Akula, Shuai Wang, and Song-Chun Zhu. Cocox: Generating conceptual and counterfac- tual explanations via fault-lines. Proceedings of the AAAI Conference on Artificial Intelligence, 34(03):2594-2601, Apr. 2020.
[3] Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein. Diffusion visual counterfactual explanations. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc.
[4] David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations, 2017.
[5] Kieran Browne and Ben Swift. Semantics and explanation: why counterfactual explanations produce adversarial examples in deep neural networks. ArXiv, abs/2012.100766 2020.
[6] Boyuan Chen, Zhuo Xu, Sean Kinmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 14455-14465, June 2024.
[7] Kamil Deja, Tomasz Trzciński, and Jakub M. Tomezak. Learning data representations with joint diffusion models. In Danai Koutra, Claudia Plant, Manuel Gomez Rodriguez, Flena Baralis, and Francesco Bonchi, editors, Machine Learning and Knowledge Discovery in Databases: Research Track, pages 543-559, Cham, 2023. Springer Nature Switzerland.
[8] Edmund Dervakos, Konstantinos Thomas, Giorgos Filandrianos, and Giorgos Stamou. Choose your data wisely: A framework for semantic counterfactuals. In Edith Elkind, editor, Proceed- ings of the Thirty-Second International Joint Conference on Artificial Intelligence, 1JCAI-23, pages 382-390. International Joint Conferences on Artificial Intelligence Organization, 8 2023. Main Track.
[9] Angeliki Dimitriou, Maria Lymperaiou, Georgios Filandrianos, Konstantinos Thomas, and Giorgos Stamou. Structure your data: Towards semantic graph counterfactuals. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 10897-10926. PMLR, 21-27 Jul 2024.
[10] Karim Farid, Simon Schrodi, Max Argus, and Thomas Brox. Latent diffusion counterfactual explanations, 2023.
[11] Giorgos Filandrianos, Konstantinos Thomas, Edmund Dervakos, and Giorgos Stamou. Concep- tual edits as counterfactual explanations. In Proceedings of the AAAI 2022 Spring Symposium on Machine Learning and Knowledge Engineering for Hybrid Intelligence (AAAI-MAKE 2022), CEUR Workshop Proceedings, Stanford University, Palo Alto, California, USA, mar 2022. CEUR-WS.oo CC BY 4.0.
[12] Eric Frankel and Edward Vendrow. Fair generation through prior modification. 2020.
[13] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pages 2376-2384 PMLR, 2019.
[14] Riccardo Guidotti. Counterfactual explanations and how to find them: literature review and benchmarking. Data Min. Knowl. Discor, 38(5):277--2224, April 2022.
[15] Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock. systematic study of bias amplification, 2022.

---

[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 1629-6640 Red Hook, NY, USA, 2017. Curran Associates Inc.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[18] Niki Van Hoeck, Patrick D. Watson, and Aron K. Barbey, Cognitive neuroscience of human counterfactual reasoning. Frontiers in Human Neuroscience, 9:420, 2015.
[19] Paul Jacob, Éloi Zablocki, Hedi Ben- Younes, Mickaël Chen, Patrick Pérez, and Matthieu Cord. STEEX: stecring counterfactual explanations with semantics. In ECCV, 2022.
[20] Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. In 2024 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 9307-9315, 2024.
[21] Guillaume Jeanneret, Loic Simon, and Frédéric Jurie. Adversarial counterfactual visual explana- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023.
[22] Guillaume Jeanneret, Loic Simon, and Frédéric Jurie. Diffusion models for counterfactual explanations. Computer Vision and Image Understanding, 249:104207, 2024.
[23] Guillaume Jeanneret. Loic Simon, and Frédéric Jurie. Text-to-image models for counterfactual explanations: a black-box approach. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2024.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Wan-Yen! Lo, Piotr Dollár, and Ross Girshick. Segment anything, 2023.
[25] Aneesh Komanduri, Chen Zhao, Feng Chen, and Xintao Wu. Causal diffusion autoencoders: Toward counterfactual generation via diffusion probabilistic models, 2024.
[26] Harold W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics (NRL), 52, 1955.
[27] Ryan Liu, Jiayi Geng, Addison J. Wu, Ilaa Sucholutsky, Tania Lombrozo, and Thomas Griffiths. Mind your step (by step): Chain-of-thought can reduce performance on tasks where thinking makes humans worse, 2024.
[28) Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024.
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
[30] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s, 2022.
[31] George A. Miller. Wordnet: An electronic lexical database. 1994.
[32] Franz Motzkus, Christian Hellert, and Ute Schmid. Cola-dce concept-guided latent diffusion counterfactual explanations, 2024.
[33] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. comprehensive overview of large language models, 2023,
12

---

[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2022.
[35] Bukhoree Sahoh and Anant Choksuriwong. The role of explainable artificial intelligence in high-stakes decision-making systems: a systematic review. Journal of Ambient Intelligence and Humanized Computing, (4(6):7277-7843 2023.
[36] Pedro Sanchez and Sotirios A. Tsaftaris. Diffusion causal models for counterfactual estimation. In Bernhard Schölkopf, Caroline Uhler, and Kun Zhang, editors, Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pages 647-668. PMLR, 11-13 Apr 2022.
[37] Bartlomiej Sobieski, Jakub Grzywaczewski, Bartlomiej Sadlej, Matthew Tivnan, and Przemys- law Biecek. Rethinking visual counterfactual explanations through region constraint, 2024.
[38] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
[39] Simon Vandenhende, Dhruv Mahajan, Filip Radenovic, and Deepti Ghadiyaram. Making heads or tails: Towards semantically consistent visual counterfactuals. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision -ECCV 2022, pages 261-279, Cham, 2022. Springer Nature Switzerland.
[40] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr, 2018.
[41] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023.
[42] Nina Weng, Paraskevas Pegios, Eike Petersen, Aasa Feragen, and Siavash Bigdeli. Fast diffusion- based counterfactuals for shortcut removal and generation. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024, pages 338 358, Cham, 2025. Springer Nature Switzerland.
[43] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEESCVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[44] Gengyuan Zhang, Yurui Zhang, Kerui Zhang, and Volker Tresp. Can vision-language models be a good guesser? exploring vlms for times and location reasoning. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 625 634,, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: All findings mentioned in the abstract and introduction are presented and later on discussed full scope within the manuscript.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in the Appendix section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We provide proofs for the algorithms from prior work in the manuscript.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide code and guidelines for the framework reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We provide code for setting up and evaluating the framework.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.SSubbisioooooiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify dataset and testing details. We do not train anything in our framework.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: On certain metrics, results did not deviate so much as to require an analysis. Other metrics, such as important and ambiguity, have been analyzed as such.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on the memory footprint and hardware utilized for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:::/neurpps.cc/////c/ httpp////.eaaaro..o//o//aall//ta
Answer: [Yes]
Justification:
Paper conforms with all aspects the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Impact discussed the Appendix section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
All modules utilized are pretrained, no new misuse risks are introduced.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All modules utilized cited and already public for open use.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We utilize existing assets for our experiments and analysis.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We discuss the evaluation in the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
IRB is not required for human surveys.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM usage only utilized for writing, editing and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https://neure) /ttpa/rrpp...ccoooeeeeeee/////// for what should or should not be described.
20

---

A LVLMs-as-cassifeers prompts
BDD100K Dataset The prompt for LVL.M-bsee classification is provided Table 6 considering 'start' and 'stop' as the (str_ jat_caegooiis). The LVLM is forced to focus on the semantics that define each driving situation, since they are definitive for classification based on concepts.
Table 6: Classification prompt for BDD100K
Classify each image in their appropriate class according to the driving situation they depict. Valid class stop based its
labels are (str__ateerrr categories) and only these, depending whether the has surroundings. You need to classify the images in one of these classes. Pay attention semantics that define each class. Return me only the label of the scene depicted and nothing else.
Visual Genome Dataset In Table we show the classification prompt used classify an image from Visual Genome in one of its appropriate categories belonging in the {str_ [at__eereoiis]] list. The LVLM is forced to focus on the semantics that define each class, since they are definitive for classification based on concepts.
Table 7: Classification prompt for Visual Genome
Classify each image in their appropriate class according (str categories] and only these, so you need to classify the images in one of these classes. Pay attention to the semantics that define each class. Return me only the label of the seene depicted and nothing else.
scene they depict. Valid classes are
B Prompts for performing the edits
As mentioned in the t.2 sectionn there are three ways to define which edits are going to be performed and in which order.
In the local editing approach, the LVLM serves as the only decision-making module to order the edits produced from the explanation component. In each step, only edit is selected and passed to the generative component. This assists in performing a small number steps until label flip, since label flip may occur before the edits proposed in E is exhausted (an assumption that is verified, based on the results of Table 4 in which our generative V-CECE consistently performs fewer edits that its non-generative counterparts). Other than that, performing one step at time allows for more high-quality generations from the point of the generative component.
The prompt that arranges the local edits at each step is illustrated in Table determining the selection of a 1, D, edit based on its assumed commonsense understanding, which triggered using a suitable example.
The prompts used by the LVLM perform the insert and delete edits are provided in Tables 10 This procedure is needed to ensure commonsense of performed edits. At the same time, it assists the mask generator of the generative component to define the object that should appear after deleting another object, effectively handling occlusion, while also masking suitable area that an existing object spans in case new object has to be added in relation it.
C Generative Component
In configuration, object detection operates with confidence threshold of 0.3, guiding the inclusion or exclusion of specific object classes via textual prompts. The bounding boxes around detected objects expanded by 35 pixels, with a soft boundary applied using a mask blur of 10 pixels. The expansion required in order for fewer artifacts to emerge from the text prompts, as further contextual information is added and the areas be modified restricted.
For inpainting, the process adheres strictly to the provided guidance, with a classifier-free guidance scale of 10, instructing the model to strongly follow the given prompts. A denoising strength of I
21

---

Table 8: Local edits prompt: defined the operations (1, D, that best be performed in each step, based on the remaining edits and the image.
want to remove some objects and add others. I would like you to find the best possible edit for the image, but want only single edit.
You can choose from the following options: Add object from the "Add" list. In this case please give the answer the format: ("add", "added_objeat" "target where the added object will appear in front of"]. Avoid positional description such as "over", "next to". "above" etc. Remove an object from the "Remove" list. In this case please give the answer in the format: ["remove", "removed. object", "the object that is behind the object when it removed e-g. wall, floor, background"].
Replace an object from the "Remove" list with one from the "Add" list. this case please give the answer in the format: l"replace", "removed _ojjectt "added_ "added_object"]].
So, you need to decide whether to add, remove, or replace an object.
For example:
Object list: [couch, lamp, window]
Add list: [bed, curtain, blanket]
Remove list:
[lamp, couch]
Step: Replace couch with bed.
Another valid step might be: Step: I"add", "curtain", "window"]. However, the step ["add", "blanket", "couch"] is not logical step because the couch is on the remove list. If put the blanket on the couch, would still have remove the couch and thus the blanket well. Please respond with only a single step and make the most logical edit you can based on the image have provided.
Object list: objects Add list: added_objs. Remove list: removed_objs
Step:
Table 9: Prompt defining the addition of objects in the image.
want an object in the image. Please specify what object that is target where added object will appear in front of. Avoid positional description such as "over", "next to", "above" etc. Please respond with a single item, without any additional text. I want parse this answer automatically, so is crucial return only single object without any explanation, or additional text! For example:
Add: "painting"
Answer: "wall"
Add: "pillow"
Answer: "bed"
Add: obj
Answer:
used, ensuring the inpainted areas undergo full transformation based the prompt. The Stable Diffusion v1.5 Inpainting model processes the image for 40 steps, using the a DPM++ 2M SDE sampler, with automatically chosen scheduler. The pipeline uses default random seed, ensuring reproducibility with the specification fixed seed, while no variation seed applied, preserving consistency in the output. Additionally, high-resolution fix is enabled, improving the final image quality through a secondary upscaling pass.
22

---

Table 10: Prompt defining the deletion of objects in the image.
want to remove an object from the image. Please specify what is the object that is behind the object when is removed e.g. wall, floor, background. Please respond with single item, without any additional text. I want to parse this answer automatically, it is crucial return only single object without any explanation, or additional text!
For example:
Remove: "painting"
Answer: "wall"
Remove: "pillow"
Answer: "bed"
Remove: obj
Answer:
D Qualitative Results
In the following Figures, we present some additional qualitative results as occurring from V-CECE pipeline. Specifically, in Figures 4 DenseNet-suggested edits. DenseNet tends to perform more steps on average in comparison to the the generative
present some successful generations stemming from
LVLM classifiers (as analyzed in Table 2), which often leads misgenerations, module is unable to handle the complex editing procedure arising as a result of requesting multiple edits in row. However, several cases, DenseNet-driven edits lead successful counterfactual generations, as illustrated below.
Source
Step
Stso
Figure 4: Successful generations after steps of edits for DenseNet classifier. The red arrow denotes the step which humans perceive label- -lbppiiiiiing. In the presented case. DenseNet flips label concurrently with humans and generation terminates.
Step
Stop
Siep 3
Figure 5: Successful generations after steps of edits for DenseNet classifier. The red arrow denotes the step at which humans perceive label-fiippigg In the presented case, DenseNet flips label concurrently with humans and generation terminates.
23

---

Interestingly, the success of the performed edit is non-trivial, since removing large objects easily leads to artifacts. Nevertheless, BDD100K images often depict large cars (being close to our point of view from the driver's s seat), rendering successful edits challenging. This is reason why prioritizing the edits in an influential way with respect to the classifier under explanation is erucial.
There are some cases where the classifiers need more steps to identify label-flipping, contrary to humans. Such scenarios are illustrated in Figure classifiers identify label flipping steps (Ist row) and 2 steps (2nd row) instead of one step that a human perceives as necessary. Therefore, the classifier instruct the generation module to proceed, leading to irrelevant edits the class transition semantics. For example, in the first case, the black car in the same lane as our point of view is removed in Step 1, allowing the transition from "Stop" to "Move" according to humans. The classifier however, cannot perceive this change as influential, concluding to counterfactual image in which the buildings in the front have been removed, and black object has been added on the upper right of the frame. However, these changes are totally irrelevant to the queried driving situation. The classifier is probably biased towards certain semantics, or even pixel distributions, therefore being fooled under such transformation, instead of flipping during the removal of the black car in Step 1. In the second case, he white car in the front is removed at Step I, correctly marked by humans as a "Move" situation. The classifier instructs further generation, resulting in the replacement of the tree with street light in the front. Nevertheless, this semantic edit is not associated with whether one has to brake or move, deeming this operation comprehension of the classifier. In the last case, human and classifier perception of tebppigi agree, since the removal of the car in the front suggests transiting to the "Move" class. However, observe a visual artifact in place of the big car. This example denotes the limitations of the generation module employed in our experimentation, suggesting that even if single step is performed towards counterfactual generation, is not guaranteed that the resulting image will be of good quality. Once again, removing large objects is tough endeavor itself for visual editors, and it is rather unpredictable whether this operation will be performed without any detectable artifact.
extraneous edit, wrongly imposed by limited semantic
Source
Slop
Step 2
Step
Figure 6: Interesting cases of sub-optimal counterfactual generations. The red arrow denotes the step at which humans perceive lahel-fiipping In the first two cases, classifiers flip label later than humans; therefore, generation terminates later than necessary. In the last case, humans and classifier perception align, but generation is not devoid of artifacts.
24

---

E
Human survey
Our human survey on BDDI0OK generated counterfactual images was filled by 31 participants. We gathered personal information about these evaluators. We used the Label Studio platform for evaluation, allowing us to demonstrate image sequences, along with the required descriptions and questions. Specifically, the participants numbered generation steps, as occurring from our experiments (we incorporated all classifiers and all ordering techniques). They were then asked to respond to the following:
provided with source image and sequence of
The step at which they believe label flip is happening, given that the source class is always "Stop". If label flip did not happen at all in this specific image sequence, they can reply with "None of the above".
The visual correctness of the image, given the options "Yes" (if the image is visually correct, meaning that is absent of severe visual artifacts) and "No".
An example of the questionnaire they were asked to fill is presented Figure
You und ugaai frrm henig vos otaae cana ahcu
Mind Stop naads imagear tux you sas decdiig staditer to ove skgp can sorpou rochoo
1) in the teet task, salect the elep shhve mesediiee mor tlop' to 1ilve the crange acture
2) spaci aos snmebaa image seagaenee be senenally correct moseing tisknadd hame savase vesad lakitdd extoda the image guanasded Ognore visually
characleriiiis
respiation
Ylos
akp_ Aa
At witch step de you think the aaatrratto lobel changen' for example, if slege to belo og the diase 'Hoo' and itpp belorge to clss Nove, serect Stap even li
dlass changes age Subeeepent stego. mages bekong Stop sedeet
Stop
step
Chep
Step
tee
Shep TOP
Hore of caneet
light conditonu? l' errreinage mauslly correct anowal You, ohhewwie, answer NNo."
Figure 7: Panel of human annotation instance in Label Studio.
Consequently, we delve into the human evaluation results, since they are crucial in unveiling the explanatory gap between humans and classifiers via V-CECE explanations. Therefore, we analyze
human responses regarding visual correctness (Figures 8 9 10 and steps required (Figures 11 12
13 until label flip for each ordering method, as well as average values for methods.
Commencing with DenseNet classifier in Figure g its average correctness lies around 60% based on human perception of visual quality. Regarding the ordering techniques for edits, local edits, instructed by Claude 3.5 sonnet on the proposed edit set [E], as occurring from the explanation component, arises as the most successful strategy with 64.58% successful counterfactual generations. The most 'greedy" strategies (with respect to label flipping) that consult global edits score lower, with 57.89% for global and 56.67% for local-globe -ooobal edits.
The local edits are proven as the most successful also in the case of Claude 3 Haiku human results in Figure achieving 73.47% visual correctness. On average, Claude 3 Haiku achieves 69.62% correctness indicating a medium agreement with human perception in semantic comprehension for classification.
The patterns changes when Claude 3.5 Sonnet is leveraged as the classifier, where local edits results in only 73.97% correctness, scoring lower than the average of 78.3% on all orderings. Local-global edits lead to 87.88% correctness, the highest percentage overall, suggesting that leveraging model biases in conjunction to LVLM-driven ordering is the best practice for this classifier. Global edits achieve 77.42 77.42% correctness, indicating that a 'greedy' edit selection choice is effective, though sub-optimal without proper ordering.
The average number of steps needed for label flip an informative indicator for the classifier's semantic level as demonstrated on the human survey findings (Table 2). Single-step edits are the most
25

---

Visual Correctness DenseNet
Visual correctries DenseNet- all edits
Visual correciness DenseNet local
50%
60
Percentage
40.00%
Percentage
20-
20
Visual correctness
Visual correctness
Visual DenseNet- global
Visual
local-global
50
42.11%
Percentage
88
20-
20
Visual correctness
Visual comeetnness
Figure 8: Human evaluation results regarding visual correctness with edits driven from DenseNet classifier.
Visual Correctness Claude 3 Haiku
Visual correctness Claude Halku all edits
Visual correctness Claude halku local
20
10
Visual correctness
Visual correctness
Visuel Claude
correctnees
globll
71.45%
70
Percentage
1
28.57%
10
Visual correctness
Visual comeetness
Figure Human evaluation results regarding visual correctness with edits driven from Claude Haiku classifier.
prevalent on average for DenseNet. Interestingly, when local edits are employed, the label-flipping procedure needs two steps the most frequent step frequency. At the same time, local edits associated with the best-quality generations for DenseNet, suggesting that despite often needing two steps, the finally generated images are good as possible, in comparison with other ordering strategies. Furthermore, local and local-global strategies for DenseNet never require more than steps for label flipping, contrary global edits, which presents few cases of 6 and 7 edits. This finding verifies the effectiveness of Claude 3.5 Sonnet as an edit ordering module, which assists in driving counterfactual generations in fewer steps, thanks to its contextual and spatial understanding.
Regarding Claude Haiku (Figure 12 single-step generations are the most frequent scenario. The behavior of this classifier is more predictable, demonstrating often 2 or 3-step generations, but with
26

---

Visual Correctness Claude 3.5 Sonnet
Visual comreciness - Claude Sonnet all edits
Visual correcnness Claude sonnet local
Percentage
50
23.70%
20
10
10
Visual correctness
Visual correctness
Visual
Claude
local-global
00-
77.4%%
77.89%
Percentage
30
22.50%
Visual correctness
Visual coreetnees
Figure 10: Human evaluation results regarding visual correctness with edits driven from Claude Sonnet classifier
striking difference in comparison the single-step ones. In very few cases, 7 8 steps needed, associated with local and gooalgglobal orderings, while for global edits, the steps are at most 5 in few instances. Global edits impose more aggressive editing strategy towards label flipping, as indicated Figure 12 but this does not mean these edits are reasonable with respect to the source image semantics, finding that is veris-ee by the lower image correctness reported previously in Figure
Finally, Claude 3.5 Sonnet presents outstanding dominance of single-step generations as the most frequent case, as exhibited in Figure [3 Very few cases require more than one step to change classification label and are primarily associated with the local edits strategy (and secondly with the local-gr gooaal ordering). This verifies that the edits suggested by Claude 3.5 Sonnet in each generation step suboptimal, agreeing with the visual correctness findings of Figure I0 On the contrary, all generations driven by global edits need only step until label flipping, highlighting this ordering strategy the most successful one for Claude 3.5 Sonnet classifier, both in terms of editing steps and visual correctness.
27

---

Number of steps for label flip on Claude 3 Haiku Number of steps for label flip on DenseNet
Count until
200
frequnccy
Prequency
40
Wep
Mee
Label Hipping step
Label fllpping step
flpp Denselet global
Stops
local-global
200
200
15.0
Irequmcc
Prequency
2.5
Wap
Label flpping step
Label fllpping step
Figure 11: Number steps until label flip distribution for DenseNet-driven edits.
Steps
21.0
Reeguncc
Prequency
Ster
Stap
Label pppiing stop
Label flipiin step
Steps flip Claude halku global
Steps label flip Claude global
23
20
frequency
Prequency
10
1.
Label Ripping step
Label fllpping step
Figure 12: Number of steps until label flip distribution Claude Haiku-driven edits.
28

---

Number of steps for label flip on Claude 3.5 Sonnet
Count
27.0
fraquncc
Frequencc
Stee
Label pipiing stop
Label flipping step
Steps flpp global
Steps flip global
17.5
36.0
frequeccy
Prequency
20
Label fllpping step
Label fllpping step
Figure 13: Number steps until label flip distribution for Claude 3.5 Sonnet Sonnct-driven edits.
29


Reviews:
[{"review_id": "kL0PTwNsKb", "reviewer": "Reviewer_zsNj", "summary": "the paper introduces a blackbox visual counterfactual explanation generation framework aimed towards generating meaningful and human interpretable edits to the given picture so that its label flips. the key idea is to first identify a set of possible semantic edits to the image, apply them in sequence in terms of their \"cost\" of edit, and stop when the minimal cost edit which flips the label is encountered. for identify set of edits, the paper utilizes WordNet which uniquely has \"part-of\" and \"is-a\" kind of relations required in this setting, and use the shortest path metric as a proxy for semantic cost.", "questions": "Besides the list of weaknesses above, I have the following questions: \n\n1.  How sensitive is the method to the quality of the semantic annotations? If the concets are inferred noisily or incompletely, does it still work? \n\n2. can the method handle unseen concepts outside the annotation vocabulary? if so, how? please provide details. \n\n3. it may also be possible that the edits that actually flip the class label are not annotated in the image, for example a general bright background may be able to classify the image as a \"daytime picture\". what kind of counterfactual editing can this model generate in such cases?", "limitations": "Authors have given discussion of limitations and potential negative societal impact.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "__Strengths:__\n\n1. Tackles an important and critical question in explainable AI research\n\n2. Proposed technique is fully blackbox, generalizable (within the WordNet concept space), and clearly interpretable by humans \n\n3. Evaluations are over different datasets and human evaluations clearly demonstrate the utility of the method. \n\n__Weaknesses:__\n\n1. Authors argue that edits produced by diffusion based counterfactual models are \"merely responding to changes in pixel distribution\". While it may be true, the proposed method is not entirely safe either. V-CECE proposed edits still rely on annotations and concepts inferred from pixel distributions (e.g., by object identifiers). Thus, despite the valid goal of aligning edits with human semantics, it can not fully eliminate the influence of low-level pixel patterns on classifier decisions. Please discuss this aspect more explicitly.\n\n2. The uniqueness of optimal edit set is not guaranteed. Although it is not a major limitation of the work, it may result in more edits than actually required, thus leading to a potentially weak explanation. This issue could be acknowledged and its impact can be discussed further. \n\n3. Experiments with semantically diverse set of datasets is critical for strengthening the claim of generalizability of the proposed model. \n\n4. No details of human participants involved is provided. \n\n5. There is a strong dependence on a specific kind of knowledge graph -- it should have (only?) visually groundable relationships organized in a hierarchical manner (e.g., \"part-of\" or \"is-a\") for the proposed model to make sense. Instead of just saying \"knowledge graph\", potentially leading to ambiguities (since Yago or WikiData are also knowledge graphs but quite useless in this task), authors should explicitly list the desiderata from the conceptual graph model that could be used. \n\n6. the proposed model may not be able to handle cases where classifier actually relies on unannotated, global or non-object features (such as brightness, texture, or some other scene-level cue).", "was_revised": true, "final_justification": "While I was quite positive to begin with, at the end of the discussion phase, it seemed like there some critical parts still missing in the paper. Especially the experiments that I had requested for with different knowledge graphs is critical to justify the generic claims made in the paper. Therefore I would like to reduce my rating slightly.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. We truly appreciate the time and effort dedicated to deeply understanding the paper and for providing such detailed comments and suggestions. We are also especially grateful for the kind recognition of the strengths of our work, as well as for highlighting its originality and significance as excellent. \n\nBelow, we provide brief responses addressing the points raised under the “Weaknesses” section. \n\n1. Influence of pixel-level distributions in the proposed method is a very interesting point. In our work, we have used the annotations provided by humans using datasets that include this type of information. However, the same method can also operate using an automated system for extracting annotations, such as an object detector. In this case, indeed, there is an influence from pixel distributions. However, this influence is not directly related to the model under inspection, but rather to the auxiliary model that extracts the objects. For the inspected model, only human-level semantics are used, and the biases or pixel-level dependencies are not explicitly passed on. That said, it is correct that the influence cannot be entirely eliminated unless object detectors improve significantly. We will include a relevant discussion in the final version of the paper. Thank you once again for this interesting observation. \n\n2. Uniqueness of the optimal edit sets is another compelling point. Even though uniqueness is not guaranteed (e.g., multiple edits of equal cost may exist), this does not result in more edits, as that would imply a higher overall cost. The determinism of the underlying counterfactual algorithm ensures that ambiguous or redundant edits are avoided. In cases where alternative edit sets exist, the pipeline proceeds as usual, and we have no theoretical or empirical indication that this affects the quality of the explanation. Having two different optimal paths does not lead to degradation, as both adhere to the minimal cost criterion. \n\n3. Diversity of datasets: VG is a highly diverse dataset, covering a wide range of  topics and semantics. Other datasets such as Flickr30k and COCO often depict very similar scenes, without object annotations, however, making additional evaluation on them redundant. BDD, on the other hand, covers a distinct real-world scenario not directly represented in VG, which adds valuable generalization insight to our study. Both datasets also give us the ability to compare with prior work. \n\n4. Knowledge graph requirements: We use WordNet (line 109), which indeed follows the required hierarchical structure with “part-of” and “is-a” relations. We have not experimented with other graphs, as WordNet already provides satisfactory semantic distances. Nevertheless, V-CECE is not tied to a specific knowledge graph, any conceptual graph that provides semantic distances could be used. We can make this point more explicit in the final version. \n\n5. Global/non-object features: We acknowledge that the current framework does not capture cases where classifiers rely on unannotated, global, or non-object features such as brightness or texture. However, these are inherently difficult to define in a structured semantic form. For example, brightness is subjective and hard to quantify consistently; even if changed, it does not usually alter the core semantics of the image (e.g., a classroom remains a classroom regardless of lighting conditions). The goal of this work is to explore whether a classifier relies on clearly defined, human-level concepts. Global features fall outside the current scope but present an interesting direction for future work. \n\nResponses to Questions \n\n1. Sensitivity to annotation quality: The proposed V-CECE pipeline does not explicitly rely on annotation quality but, like most ML systems, benefits from high-quality data. Incomplete annotations may lead to seemingly lower-cost edits if essential concepts are missing. Noisy annotations, on the other hand, could result in more costly or misleading counterfactuals. While the pipeline can operate in such scenarios, data quality remains a critical pre-processing factor. \n\n2. Out-of-vocabulary concepts: The datasets used in our study include well-known, everyday concepts already represented in WordNet. If V-CECE is applied to a different domain (e.g., medical imaging), the knowledge graph should be adapted accordingly to ensure appropriate coverage. Since V-CECE is agnostic to the specific source of semantic distances, it can work with any suitable knowledge graph, assuming concept coverage is sufficient and existent. \n\n3. Handling cases like “daytime picture”: Our focus is on well-specified, semantically grounded scenes such as playgrounds, classrooms, and theaters-settings that are easily identifiable by both humans and models. Generic features like “daytime” or “brightness” are not within the current scope. If a classifier is influenced primarily by lighting, that is precisely the type of superficial dependency V-CECE aims to reveal. Style-transfer-like transformations that alter superficial cues without changing semantics could be a future extension to further probe classifier robustness. \n\nOnce again, we sincerely thank the reviewer for their time, effort, and thoughtful engagement with our work. Your feedback has been invaluable in helping us improve the quality and clarity of our submission."}, {"review_id": "PqL5JiOiK6", "reviewer": "Reviewer_JVys", "summary": "The authors propose a post hoc, model agnostic, counterfactual explainer for computer vision models. They claim that prior counterfactual frameworks do not adequately address the problem of explaining model decision making processes in terms of how humans understand the model's input features. Their V-CECE method uses a large vision language model to identify how to generate a CF and then uses a diffusion model to actually generate the CF. It may be said that the LVLM is used as a proxy for humans. *How* to generate CFs is decided as follows. Every image has a set of visual semantic concept labels. A counterfactual instance $(x*,y*)$ for an input $(x,y)$ can be found by identifying a set of changes to the set of semantic concepts held by $x$ that lead to $x*$ having semantic concepts characteristic of the class $y*$. Each change incurs a cost. Using these costs as weights, the CF design process can be modeled as bipartite matching with weighted edges. An LVLM is used to identify the *set* of changes to $x$ with a *sequence* of changes to $x$. Each change corresponds to a concept edit. Each concept corresponds to a concept mask. Thus the changes to $x$ can be achieved by using a diffusion model to perform inpainting inside of a concept mask. \n\nThe evaluation is carried out quantitatively by measuring realism with FID/CMMD, effectiveness with success rate of CF generation, and stability by the proportion of identical labels obtained across independent runs. V-CECE is compared with several baseline methods from prior literature.", "questions": "- I think you did not mean to write $s_i$ twice in lines 109-110?\n- This is in response to line 154. How can the authors be sure that the diffusion model they use was not actually trained directly on their data?\n- I wonder if the claim on lines 305-308 that explaining DenseNet is pointless because CNNs learn statistical dependencies and as a result do not recover human-level semantics is either too strong in need of more justification. After all, researchers in biology have found correlations between CNN visual processing and human visual processing (at lower levels).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- V-CECE compares favorably to the baseline methods on the chosen metrics\n- The **Discovering biases** subsection in section 4 shows evidence of differences in the semantic space of DenseNet and V-CECE. This is a neat result. \n- V-CECE requires the execution of fewer edits than competing methods \n- The paper has a human evaluation study. It showed evidence that DenseNet is doing too many edits from a human perspective. \n\n## Weaknesses\n\n- I would have liked more detail in section 3.3 about how the diffusion model executes the edits, or at least a sentence saying that more detail can be found in the **Generative Module** subsection in section 4.", "was_revised": true, "final_justification": "The authors gave a satisfactory response to my small critique. After reviewing their exchange with 9F83 about their user study I have nevertheless decided to retain my scores. I feel a change in XAI where XAI papers must include user studies, and must design user studies according to best practices. However, it was not that long ago that XAI papers could be accepted at top CS conferences without user studies and so I am compelled to be charitable during this time, which I identify as phase where the incorporation of user studies is *becoming* the norm and researchers with CS training are adapting to this new landscape.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. We truly appreciate the time and effort dedicated to deeply understanding the paper, which is evident not only in the valuable comments provided but also in the remarkably concise and accurate summary that effectively captures its essence and demonstrates a clear grasp of its core ideas and contributions. \n\nBelow, we provide brief responses addressing the points raised under the “Weaknesses” section. \n\n1. We thank the reviewer for this helpful observation. We agree that additional clarification is needed regarding how the diffusion model executes the edits. Due to space constraints, we had originally included the relevant information in Appendix Section B. However, we fully acknowledge its importance for the reader’s understanding. In the camera-ready version, given the extra space provided, we will incorporate a clear and self-contained explanation of the generative process around lines 192–193, previously available only in the appendix. This will ensure that the underlying mechanism is properly conveyed within the main text. \n\nResponses to Questions: \n\n1. The second s_i should be s_j, we will fix it for the camera-ready version. \n\n2. The Stable Diffusion checkpoint utilized for the generative task, as per the description of the model card, has been finetuned for inpainting on laion‑aesthetics v2 5+ , as noted in the model card of the checkpoint on HuggingFace, which does not include images from BDD100K or Visual Genome. The original checkpoint also has been trained on the LAION dataset [1]. Both datasets are from the Common Crawl space, which consists of web page documents and images linked with URLs. As such, the dataset does not include curated datasets, which are invisible to the crawl procedure.  \n\n3. The statement in lines 305–308 refers to the DenseNet classifier utilized in prior work and was made with a specific motivation that we now recognize could be explained more clearly. Prior work using the same classifier (DenseNet using BDD100K), such as [2], reports an inability to generate meaningful counterfactual images (Section 4.3, Limitations), attributing this issue to limitations of their algorithm and to the complexity of the dataset. While we acknowledge that these factors may contribute, we argue that the classifier itself also plays a crucial role, which is typically overlooked in previous studies. Through our analysis, we showed that the model’s perception is not aligned with human-level concepts, which significantly limits the potential for meaningful interpretation. In other words, even with a perfect explainability method, the classifier may still fail to produce meaningful outputs, often resulting in outcomes ranging from visually incorrect images to entirely misleading explanations. We will revise the relevant section to articulate this point more clearly and reinforce the reasoning behind our claim.  \n\nOnce again, we would like to thank the reviewer for their review, and specifically for the time and effort spent to deeply understand our work. Your thoughtful engagement truly means a lot to us and has helped us improve the quality of our work. \n\n[1] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. \n\n[2]  Guillaume Jeanneret, Loïc Simon, and Frédéric Jurie. Text-to-image models for counterfactual explanations: a black-box approach. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2024."}, {"review_id": "lVrYhzniWM", "reviewer": "Reviewer_JSSs", "summary": "This paper proposes V-CECE, a black-box, training-free framework for generating visual counterfactual explanations (CEs) through semantically meaningful conceptual edits. The method leverages nearest-neighbor semantic matching and a pre-trained diffusion model to apply optimal edits. It is applied to assess the discrepancy between human reasoning and classifier decisions, including CNNs, ViTs, and LVLMs. The paper emphasizes evaluating models from a human-centered, semantic perspective and offers a novel tool for model auditing.", "questions": "**Questions (and Suggestions to Improve Score)**:\n1. Why are no modified versions of existing CE generation methods (e.g., TIME, ConceptualCounterfactuals) included as baselines?\nEven if exact comparisons are hard, simplified adaptations could offer valuable insights. Including such baselines would significantly strengthen the empirical validation.\n2. What prevents integration of components from related works (e.g., Dimitriou et al.) into your pipeline or comparison setup?\nClarifying this would improve the credibility of the claim that no existing methods are comparable.\n3. Why is [1] not cited? Their method, like yours, proposes black-box counterfactuals using conceptual edits with minimal changes. How does your framework differ?\n4. What exactly is meant by “visually incorrect” images (ln. 256), and how is this judged?\n5. Could Figure 1 be simplified or separated into multiple stages for clarity?\n6. How does the method handle domains or tasks with weakly defined or abstract concepts (e.g., textures, scenes, medical images)?\n7. Why is there no ablation study isolating the effect of the masking step? Since the masking mechanism guides the generative process, it would be helpful to understand its contribution independently.\n\n[1] Conceptual Edits as Counterfactual Explanations (Filandrianos et al., AAAI-MAKE 2022)", "limitations": "Limitations are not sufficiently touched upon.\n\nAdditionally:\n- Limited to object-level concepts: The reliance on masking and localized edits constrains the method to object-based or spatially separable concepts. More abstract or distributed features (e.g. texture, scene layout) are harder to capture or edit.\n- Semantic edit optimality depends on image retrieval: The guarantee of optimal edits depends heavily on the quality and diversity of the dataset used to retrieve the closest image pair. In low-data or biased domains, the optimality guarantee may break down.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- Tackles an underexplored yet important task: semantic-level evaluation of vision classifiers.\n- Training-free and black-box framework with plug-and-play generative capabilities, that supports multiple types of classifiers (CNNs, LVLMs).\n- Methodology is explained clearly.\n- Incorporates human evaluation to measure classifier-human alignment.\n\n**Weaknesses**:\n- No comparisons to existing or adapted CE generation methods like TIME, even in ablation or simplified form. Also, related work that could be used for comparison is omitted [1]\n- The setting in prior comparisons (e.g., Dervakos et al., Dimitriou et al.) is underexplained.\n- Some parts of the writing and figures lack clarity (e.g., Figure 1, Section 4.1).\n- Limited discussion of failure cases and when the method might not work (e.g., ambiguous concept boundaries).\n\n[1] Conceptual Edits as Counterfactual Explanations (Filandrianos et al., AAAI-MAKE 2022)", "was_revised": true, "final_justification": "I am increasing my score to 5 after reading the rebuttal, as several of my initial concerns were due to misunderstandings that the authors clarified with a detailed and thoughtful response—particularly regarding comparisons to prior work. While an ablation on the masking component and exploration of adapting non-generative baselines for human evaluation would further strengthen the work, these omissions are not critical. The authors' commitment to improving clarity and adding missing details in the camera-ready version further supports the recommendation.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. We truly appreciate the time and effort dedicated to deeply understanding the paper and for providing such detailed comments and suggestions.  \n\nWeaknesses: \n1. Comparison with TIME and prior white-box methodologies is provided in Table 1. Regarding the omitted citation [1] (Filandrianos et al., AAAI-MAKE 2022), we would like to clarify that it presents the same core algorithm as in [2], written by the same authors, which we already cite and use for direct comparison. We chose to refer to [2] instead, as it includes more extensive experimentation and a broader evaluation across multiple datasets, making it more appropriate for empirical comparison.   \nThat said, we appreciate the reviewer’s suggestion and agree that including [1] as a reference could provide helpful additional context. We will add it to the related work section of the camera-ready version and briefly explain its relation to the more comprehensive follow-up work of [2]. \n\n2. We did not elaborate on the evaluation setup in detail, primarily due to space limitations. As noted in lines 208–210, we follow the exact evaluation methodology described in the related works we compare against ([2] and Dimitriou et al. [3]), to ensure a fair and meaningful comparison. A more detailed explanation of the evaluation procedure will be included in the camera-ready version. Both [2] and [3] rely on semantic counterfactuals driven by knowledge graphs. Their methods retrieve counterfactual images and extract semantic edits that indicate what needs to change in order to shift from the source class to the counterfactual class. However, they do not generate the proposed edits, and thus no minimally altered image is returned. For this reason, we compare the methods based on label flip and number of edits performance on the Visual Genome dataset (Tables 4 and 5).  \n\n3. We will revise the parts referenced to improve clarity. If the reviewer could point out specific aspects that were unclear, it would help us focus on our improvements more effectively. \n\n4. We have placed discussion for limitations in the Appendix Line 504. There are certain cases in which artifacts may exist, despite the mitigation procedure,a known problem with diffusion inpainting and generation in general [4]. There are also certain cases in which the classifier might flip the label sooner than human perception, but this is highly variable. We will update the camera ready with a clear limitations section in the Appendix.  \n\nQuestions:  \n1. We would like to kindly ask for further clarification regarding what type of adaptation is being suggested. It is not entirely clear to us whether the reviewer is referring to adapting existing methods to operate on new datasets or to modifying their outputs for compatibility with our evaluation pipeline. Below, we provide responses covering both possible interpretations:  \nAdaptation of existing methods to other datasets:  \nWe aim for fair comparisons by evaluating each method in the exact setting proposed by its original authors. We run each baseline using the same dataset, classifier, and data splits as the original work. While adapting each method to a different dataset might seem straightforward in theory, such adaptations often require careful tuning and design choices that may deviate from the original intent. We deliberately avoided introducing such changes to prevent the risk of unfair comparisons.  \nAdaptation of methods for human evaluation:  \nTIME is included in Table 1 for reference. However, an empirical comparison with human studies is difficult because TIME does not apply semantic edits step-by-step, but instead performs large-scale image manipulations. As a result, it is difficult for human evaluators to identify which change led to the label flip. Additionally, we did not include Conceptual Counterfactuals in the human survey, as it does not generate counterfactual images. Since human evaluation requires visual inspection and judgment, such methods cannot be directly incorporated. If we have misunderstood the reviewer’s concern, we would be grateful for further clarification. We would be happy to provide a more targeted response to ensure that all aspects of the question are fully addressed.  \n\n2. We integrate components from [2] and [3]: we base our edits on semantic distances driven by a knowledge graph. Their main limitation is that they do not generate the counterfactual image that corresponds to the semantic edits. This way, not only do they not provide an actual counterfactual image, but by generation it is possible that we need fewer edits to change the class, indicating that both [2] and [3] propose more expensive edit set. Additionally, both [2] and [3] take it for granted that the classifiers under inspection operate based on semantic concepts, without providing any form of evaluation or verification. In contrast, by generating actual counterfactual images and observing whether fewer semantic edits suffice to flip the prediction, we are able to examine and test this assumption explicitly.  \n\n3. We will cite the work. Similarly to the previous answer, the method does not generate counterfactual images. \n\n4. A visually incorrect image has generation artifacts or defies commonsense (ie shadows existing without an object to cast it). This characterization is supported from the human survey, in which participants were asked to identify the visually incorrect images during the generation process.  \n\n5. We will simplify the figure. \n\n6. The framework aims to identify semantic relationships through well-defined object-oriented semantics. For abstract concepts to be supported, the framework would require a different semantic graph that can provide this information. Since V-CECE is agnostic to the specific source of semantic distances, it can work with any suitable knowledge graph, assuming concept coverage is sufficient. We leave this as future work.  \n\n7. In V-CECE, the Editing Module is treated as a black-box within the generative process. While we conduct ablation studies on all other parts of the algorithm, we did not isolate the effect of the masking step specifically. This is because we do not introduce any novelty regarding the parts of the editing module; the novelty lies on the usage of the module to generate counterfactual images, driven by the novel conceptual counterfactual algorithm. Also, this component has already been indirectly evaluated in the original works (GroundingDINO, Segment Anything Model, Stable Diffusion Inpainting). In those studies, the effect of the masking mechanism on the quality and accuracy of the generated outputs has been analyzed as part of the editing system. We rely on these evaluations for the low-level behavior of the generator, focusing instead on the design and evaluation of the full counterfactual explanation framework. If the reviewer considers this ablation important for evaluating V-CECE, we would be happy to include an analysis. \n\nLimitations:  \n\n1. We recognize this limitation; however, V-CECE can function on a robust enough semantic graph that contains objects. As long as that graph exists for abstract concepts, the framework can extract information. At this stage, this is left as future work. \n\n2. We would like to clarify that the guarantee of optimality in V-CECE is defined relative to the dataset. V-CECE always returns the shortest semantic edit path within that space, regardless of whether the dataset is biased or sparse. In that sense, the guarantee is not invalidated by dataset limitations. In sparse datasets, the absence of intermediate samples can significantly hinder the construction of interpretable counterfactual transitions.  In such cases, concept transitions become coarse, making it harder to capture smooth, interpretable paths. Prior works [2], [3] rely heavily on the dataset structure, and the absence of intermediate examples can lead to unstable or misleading explanations. However, V-CECE directly addresses this limitation by iteratively generating intermediate counterfactuals, one semantic edit at a time, allowing us to densify the semantic space. This gradual construction helps bridge distant concepts through concrete edits, as shown in Table 4, where V-CECE requires fewer steps to reach a decision boundary.  As for bias, we believe this is where V-CECE provides a key advantage. In [2], [3], biased co-occurrences in the dataset are reflected directly in the explanations, regardless of whether they are causal for the model’s predictions. V-CECE breaks this dependency by applying and validating each semantic edit in isolation. If a biased concept, e.g a tree, is introduced and does not change the classifier’s output, V-CECE will not mark it as influential. This ensures that dataset bias is not mistaken for model bias. A similar limitation exists in methods like TIME, which learn edit patterns directly from training data. \n \nPaper Formatting:  \n\n1. We will update the figure and the caption for clarity .  \n\n2. We will correct the spelling.  \n\n3. A visually incorrect image has generation artifacts or defies commonsense (ie shadows existing without an object to cast it). We will update the definition in the main paper.  \n\n4. Table 3 is cited in line 283-284  \n\n[1] Filandrianos et al. Conceptual Edits as Counterfactual Explanations, AAAI-MAKE 2022 \n\n[2] Edmund Dervakos et al. Choose your data wisely: A framework for semantic counterfactuals, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence \n\n[3] Angeliki Dimitriou et al. Structure your data: Towards semantic graph counterfactuals. Proceedings of the 41st International Conference on Machine Learning,  \n\n[4] Cao, Bin, et al. \"Synartifact: Classifying and alleviating artifacts in synthetic images via vision-language model.\" arXiv preprint arXiv:2402.18068 (2024)."}, {"review_id": "MVGPdnvA9B", "reviewer": "Reviewer_9F83", "summary": "The paper introduces V-CECE, a model-agnostic, training-free framework for generating semantically meaningful visual counterfactual explanations by representing image concepts (e.g., objects like “stop sign” or “car”) as nodes in a WordNet-style graph and computing counterfactuals with off-the-shelf diffusion-based models. They then showed that standard vision models were worse compared to LVLMs in terms of visual counterfactual edits that correspond to human-meaningful interpretations of \"distance\".", "questions": "* Can you please describe the user study in detail? So I can assess it properly?\n* Why did you only use your method in the user study?\n* Can you think of any practical applications for your method? Something with a clear purpose-driven goal?", "limitations": "It's ok", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n* I like the model agnostic nature of the framework, although many similar \"model agnostic\" counterfactual image generation papers exist [1,2], it is a strength that should be acknowledged and I think this approach is quite practical.\n* The authors make an attempt to validate this with humans, something often overlooked.\n* The focus on human interpretable edits is also quite a nice touch.\n* Comparing classic CNNs, ViTs, and LVLMs under the same framework reveals a clear semantic “gap” in how these models represent scenes.\n\n### Weaknesses\n* The human study has enormous flaws, it is not described in any detail in the paper, the number of participants, a power analysis, null hypothesis, materials shown, metrics, there is literally almost nothing, not even in the appendix. This is completely unacceptable.\n* For me, the clear flaw with this paper is the same as many similar works in the XAI literature, the authors have proposed a new technique, ran it on a few proxy metrics such as num edits required for crossing the decision boundary, and a basic human task of counterfactual simulation, but they have not demonstrated the technique is useful to intended practitioners of the system (whomever that may be).\n\n### Minor points\n* The images are all raster, they really should be SVG/PDF\n* You cite [5] a lot to justify your arguments, but this work is quite old and never even published (on Google Scholar at least)\n* The claim on line 37 that counterfactual images are not reproducible seems bizarre to me, most methods have reproducible code bases.\n* You should really start using bookends for your tables to look more professional.\n\n***\n\n[1] Kenny, E.M. and Keane, M.T., 2021, May. On generating plausible counterfactual and semi-factual explanations for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 13, pp. 11575-11585).\n\n[2] Zemni, M., Chen, M., Zablocki, É., Ben-Younes, H., Pérez, P. and Cord, M., 2023. Octet: Object-aware counterfactual explanations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 15062-15071).", "was_revised": true, "final_justification": "The paper relies on a user study for the backbone of its evaluation (as it's about human aligned edits), but it was conducted in a somewhat lacking manner in my opinion.\n\nThe authors conducted no power analysis, almost no description of the user base, and did not include the full study in their appendix or supplement. Instead results are reported as a basic average, which is (in my opinion) far below the bar of what is scientifically acceptable.\n\n*'Our human survey on BDD100K generated counterfactual images was filled by 31 participants.'* -- This sentence is the sum total of everything we know about the subjects. We don't even know if they were paid, which is a requirement for NeurIPS Checklist item 14.\n\nAs such, it is impossible for me to say the study coo-berates their claims and findings, so I can't confidently accept it.\n\nAside from this I have no issues with the paper.", "rebuttal": "Thank you for taking the time to review our submission. Below, we respond to the main concerns raised under the “Weaknesses” and “Questions” sections. \n\n1. We would like to respectfully disagree with the claim that the human study “has enormous flaws,” or that important information regarding the human survey is missing, even from the appendix (“completely unacceptable”). We believe these characterizations do not accurately reflect the content of the submission. The points raised, including the number of participants, the annotation process, the materials shown, and the methodology used, are in fact clearly presented in the paper in Appendix E. The section provides detailed information about the setup: it involved 31 annotators; the platform used for the survey (we develop a custom platform using Label Studio); it includes full instructions and example screenshots of the annotation interface; and it describes the exact questions asked. The stimuli and evaluation protocol are also explained. In our view, this information sufficiently documents the human study and is in line with the level of detail expected for an evaluation of this kind. Regarding the absence of a null hypothesis or power analysis; our study is exploratory in nature, aiming to assess semantic interpretability and not designed as a confirmatory statistical test. This is consistent with the way similar human studies have been conducted in [1, 2] that we are directly comparing our work with, where such statistical elements, including null hypotheses and power analyses, were likewise not included. The structure, scope, and reporting of our human evaluation are directly aligned with the format and methodology used in [1, 2], which have been peer-reviewed and accepted at top venues (IJCAI, ICML). We did not design a new or ad hoc procedure; instead, we deliberately followed established, community-accepted protocols. Thus, we do not believe the study contains methodological flaws, and we consider the information reported to be appropriate and complete for the goals it serves. \n\n2. V-CECE introduces a general method to explain any pre-trained classifier in semantic terms, thereby revealing the semantic gap between human and model interpretation; it is not limited to a specific recipient or domain. However, the classifiers we study, CNNs, ViTs, and LVLMs, are among the most widely used architectures in both research and practical applications. In particular, CNNs and ViTs have served as the backbone of image classification for years, making black-box semantic explanation methods for them both timely and important. Similarly, LVLMs are increasingly deployed across domains due to their strong performance and generality, yet they remain opaque by design, warranting targeted explanation endeavors. To this end, V-CECE is designed to be model-agnostic: practitioners can plug in any model to obtain semantic explanations and generate counterfactual images. Beyond previous work, V-CECE provides a novel insight: we uncover and analyze the explanatory gap in generated counterfactuals, highlighting a key mismatch between model behavior and human-understandable semantics. We also include a real-world use case, autonomous driving, where semantic transparency is critical. For example, in a system tasked with deciding whether a vehicle should stop or go, semantic failure could lead to unsafe behavior. Our results show that some classifiers (e.g., DenseNet) may fail in this setting due to limited semantic perception. This finding underscores the importance of deploying semantically-aware models in safety-critical applications and may inform future practices in the design of explainable autonomous systems. \n\nMinor Points  \n\n1. We will change this in the camera ready, should the paper be accepted. \n\n2. We cite [5] as a theoretical claim of the importance of semantics in explainability. However, this claim has been proven in practice: the works of [1, 2] with which we compare with, are based on semantic counterfactuals, supporting the claim that semantics are crucial for explainability. These works have been published in top-tier conferences (IJCAI, ICML) verifying the validity of this claim. It is also a highly influential paper in the domain of explainability in other works, published in CVPR and ECCV as well [3], [4]. The human study (machine teaching) presented in [2] reveals that semantics are not only important but often adequate to explain differences between classes according to humans. Therefore, there is some extra evidence regarding the importance of semantics in explainability apart from [5], which is once again verified in our work, since we reveal that given a semantic edits algorithm, some classifiers disagree with humans regarding the label-flipping step.  \n\n3. This is not what this sentence attempts to point-out; we do not claim that prior works do not have sufficient code bases. Instead, we mean that they are not reproducible by humans in the sense that a human cannot easily detect what has changed in comparison to the source image when generated edits are dispersed or targeted to non-semantic areas. At the same time, we mention that counterfactual generations are not interpretable; a human cannot accurately reproduce what cannot be interpreted from their point of view. \n\n4. We will use bookends in the camera-ready.  \n\nQuestions \n\n1. All requested details are already included in Appendix E. However, if additional clarification is needed, we are happy to provide it. \n\n2. Our human study targets to reveal the explanatory gap between classifiers and humans, given different pre-trained classifiers. It does not target to compare generated images between different methods. To the best of our knowledge, no prior work in counterfactual generation makes such a claim. Therefore, there would be no purpose in including generated samples from prior works, since their goal is different. Other than that, this would be unfair for some prior works, since they mostly target label-flipping, but their edits may not be localized, therefore it is harder for humans to assess when label-flipping occurs and why. In this case, the human study would be questionable, as it targets human perception regarding label-flipping; if humans are unable to accurately point the step where the label changes, the results are unreliable, therefore the findings would obscure the semantic level of human understanding. Even if we wanted to directly compare our method to others using a human survey, this would not be possible. The algorithms proposed by [1, 2] do not generate images, but instead provide only theoretical edit suggestions. Similarly, TIME does not support iterative editing; instead, it modifies all relevant areas at once. As a result, it is not possible to isolate and evaluate each individual edit step in a way that would be meaningful for human annotators. Therefore, a direct human-based comparison would be infeasible and potentially misleading. \n\n3. We presented a practical use-case regarding autonomous driving. Revealing that some popular models, such as DenseNet, diverge from human semantic understanding in this application is crucial and calls for the deployment of semantic-aware models. For example, when engineers would like to deploy a new classifier for a self-driving system, they could check its semantic awareness using VCECE as part of a reliability check pipeline, and compare it to well-known classifiers used in our study (for example, is the new classifier closer to Claude 3.5-the best performer- or closer to CNNs-worst performers- regarding the number of edits?). The reported degree of semantic awareness would suggest improvements in model architecture, if it is found to be non-satisfactory. \n\nWe hope these clarifications have addressed the reviewer’s comments and helped resolve any concerns regarding the paper. Thank you once again for your time and effort in reviewing our submission. \n\n[1] Dervakos, Edmund et al. Choose your data wisely: A framework for semantic counterfactuals. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI-23), pages 382–390.. \n\n[2] Dimitriou, Angeliki et al. Structure your data: Towards semantic graph counterfactuals. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235. \n\n[3] Zemni, Mehdi, et al. \"Octet: Object-aware counterfactual explanations.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023. \n\n[4] Jacob, Paul, et al. \"STEEX: steering counterfactual explanations with semantics.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022."}]

Author rebuttal:

Meta-review: This paper proposes a blackbox visual counterfactual explanation generation framework aimed towards generating meaningful and human interpretable edits to the given picture so that its label flips. the key idea is to first identify a set of possible semantic edits to the image, apply them in sequence in terms of their "cost" of edit, and stop when the minimal cost edit which flips the label is encountered. for identify set of edits, the paper utilizes WordNet which uniquely has "part-of" and "is-a" kind of relations required in this setting, and use the shortest path metric as a proxy for semantic cost.

4 reviewers reviewed this paper. After rebuttal, there were 2 accepts, 1 borderline accept and 1 borderline reject ratings. Reviewers agreed that this paper presents interesting results and addresses an important task. The main negative concerns are around the evaluation, where one reviewer believed that the power of the user study is not enough to support the conclusion this paper makes, and another reviewer concurs with it but suggest leniency because of the general status of user studies in machine learning work. Another reviewer was concerned about whether pixel-level influences would be removed.

Despite the positive scores, AC thinks this is borderline. First of all, this paper did not propose any novel method, what they proposed is a novel approach that utilizes entirely existing modules to perform the task of evaluating the explanation gap between humans and deep networks. This in itself is not a problem, but AC does believe that it puts this paper in the category of "evaluation papers" where the bar on whether the evaluation is conducted fairly and comprehensively would be a bit higher. In this particular case, the user study is conducted on a very biased task of predicting stop/go on BDD100K, which do not cover a broad set of object categories. Besides, as correctly pointed out by the reviewers, a study size 30 is quite small. The authors cited some references, but it should be pointed out that novel algorithms were proposed in all the references the authors cited, whereas this paper only used frozen diffusion models and prompts to generate counterfactual examples. Hence, its main contribution to the field would be the conclusion of running such experiment (e.g. a main claim is that they showed that LVLMs match human preferences better than other deep networks). For that kind of conclusion, we would expect the experiment to be a bit more comprehensive, than only covering 2 classes with limited amount of participants. In fact, Table 5 already shows that the conclusion might be quite different if different categories were covered.

After deliberation, AC decides to accept this paper, for the purpose that this can be considered analogous to a phase-1 clinical trial. but ask the authors to clearly list the limitations of the user study in the final version.
Decision: Accept (poster)