Title: V-CECE: Visual Counterfactual Explanations via Conceptual Edits
Venue: NeurIPS 2025
Accepted: yes
Keywords: counterfactual explanations

diffusion models

LVLMs

XAI
Abstract: Recent black-box counterfactual generation frameworks fail to take into account the semantic content of the proposed edits, while relying heavily on training to guide the generation process. We propose a novel, plug-and-play black-box counterfactual generation framework, which suggests step-by-step edits based on theoretical guarantees of optimal edits to produce human-level counterfactual explanations with zero training. Our framework utilizes a pre-trained image editing diffusion model, and operates without access to the internals of the classifier, leading to an explainable counterfactual generation process. Throughout our experimentation, we showcase the explanatory gap between human reasoning and neural model behavior by utilizing both Convolutional Neural Network (CNN), Vision Transformer (ViT) and Large Vision Language Model (LVLM) classifiers, substantiated through a comprehensive human evaluation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces V-CECE, a model-agnostic, training-free framework for generating semantically meaningful visual counterfactual explanations by representing image concepts (e.g., objects like “stop sign” or “car”) as nodes in a WordNet-style graph and computing counterfactuals with off-the-shelf diffusion-based models. They then showed that standard vision models were worse compared to LVLMs in terms of visual counterfactual edits that correspond to human-meaningful interpretations of \"distance\".\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* I like the model agnostic nature of the framework, although many similar \"model agnostic\" counterfactual image generation papers exist [1,2], it is a strength that should be acknowledged and I think this approach is quite practical.\n* The authors make an attempt to validate this with humans, something often overlooked.\n* The focus on human interpretable edits is also quite a nice touch.\n* Comparing classic CNNs, ViTs, and LVLMs under the same framework reveals a clear semantic “gap” in how these models represent scenes.\n\n### Weaknesses\n* The human study has enormous flaws, it is not described in any detail in the paper, the number of participants, a power analysis, null hypothesis, materials shown, metrics, there is literally almost nothing, not even in the appendix. This is completely unacceptable.\n* For me, the clear flaw with this paper is the same as many similar works in the XAI literature, the authors have proposed a new technique, ran it on a few proxy metrics such as num edits required for crossing the decision boundary, and a basic human task of counterfactual simulation, but they have not demonstrated the technique is useful to intended practitioners of the system (whomever that may be).\n\n### Minor points\n* The images are all raster, they really should be SVG/PDF\n* You cite [5] a lot to justify your arguments, but this work is quite old and never even published (on Google Scholar at least)\n* The claim on line 37 that counterfactual images are not reproducible seems bizarre to me, most methods have reproducible code bases.\n* You should really start using bookends for your tables to look more professional.\n\n***\n\n[1] Kenny, E.M. and Keane, M.T., 2021, May. On generating plausible counterfactual and semi-factual explanations for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 13, pp. 11575-11585).\n\n[2] Zemni, M., Chen, M., Zablocki, É., Ben-Younes, H., Pérez, P. and Cord, M., 2023. Octet: Object-aware counterfactual explanations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 15062-15071).\n\nQUESTIONS:\n* Can you please describe the user study in detail? So I can assess it properly?\n* Why did you only use your method in the user study?\n* Can you think of any practical applications for your method? Something with a clear purpose-driven goal?\n\nLIMITATIONS:\nIt's ok", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes V-CECE, a black-box, training-free framework for generating visual counterfactual explanations (CEs) through semantically meaningful conceptual edits. The method leverages nearest-neighbor semantic matching and a pre-trained diffusion model to apply optimal edits. It is applied to assess the discrepancy between human reasoning and classifier decisions, including CNNs, ViTs, and LVLMs. The paper emphasizes evaluating models from a human-centered, semantic perspective and offers a novel tool for model auditing.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- Tackles an underexplored yet important task: semantic-level evaluation of vision classifiers.\n- Training-free and black-box framework with plug-and-play generative capabilities, that supports multiple types of classifiers (CNNs, LVLMs).\n- Methodology is explained clearly.\n- Incorporates human evaluation to measure classifier-human alignment.\n\n**Weaknesses**:\n- No comparisons to existing or adapted CE generation methods like TIME, even in ablation or simplified form. Also, related work that could be used for comparison is omitted [1]\n- The setting in prior comparisons (e.g., Dervakos et al., Dimitriou et al.) is underexplained.\n- Some parts of the writing and figures lack clarity (e.g., Figure 1, Section 4.1).\n- Limited discussion of failure cases and when the method might not work (e.g., ambiguous concept boundaries).\n\n[1] Conceptual Edits as Counterfactual Explanations (Filandrianos et al., AAAI-MAKE 2022)\n\nQUESTIONS:\n**Questions (and Suggestions to Improve Score)**:\n1. Why are no modified versions of existing CE generation methods (e.g., TIME, ConceptualCounterfactuals) included as baselines?\nEven if exact comparisons are hard, simplified adaptations could offer valuable insights. Including such baselines would significantly strengthen the empirical validation.\n2. What prevents integration of components from related works (e.g., Dimitriou et al.) into your pipeline or comparison setup?\nClarifying this would improve the credibility of the claim that no existing methods are comparable.\n3. Why is [1] not cited? Their method, like yours, proposes black-box counterfactuals using conceptual edits with minimal changes. How does your framework differ?\n4. What exactly is meant by “visually incorrect” images (ln. 256), and how is this judged?\n5. Could Figure 1 be simplified or separated into multiple stages for clarity?\n6. How does the method handle domains or tasks with weakly defined or abstract concepts (e.g., textures, scenes, medical images)?\n7. Why is there no ablation study isolating the effect of the masking step? Since the masking mechanism guides the generative process, it would be helpful to understand its contribution independently.\n\n[1] Conceptual Edits as Counterfactual Explanations (Filandrianos et al., AAAI-MAKE 2022)\n\nLIMITATIONS:\nLimitations are not sufficiently touched upon.\n\nAdditionally:\n- Limited to object-level concepts: The reliance on masking and localized edits constrains the method to object-based or spatially separable concepts. More abstract or distributed features (e.g. texture, scene layout) are harder to capture or edit.\n- Semantic edit optimality depends on image retrieval: The guarantee of optimal edits depends heavily on the quality and diversity of the dataset used to retrieve the closest image pair. In low-data or biased domains, the optimality guarantee may break down.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a post hoc, model agnostic, counterfactual explainer for computer vision models. They claim that prior counterfactual frameworks do not adequately address the problem of explaining model decision making processes in terms of how humans understand the model's input features. Their V-CECE method uses a large vision language model to identify how to generate a CF and then uses a diffusion model to actually generate the CF. It may be said that the LVLM is used as a proxy for humans. *How* to generate CFs is decided as follows. Every image has a set of visual semantic concept labels. A counterfactual instance $(x*,y*)$ for an input $(x,y)$ can be found by identifying a set of changes to the set of semantic concepts held by $x$ that lead to $x*$ having semantic concepts characteristic of the class $y*$. Each change incurs a cost. Using these costs as weights, the CF design process can be modeled as bipartite matching with weighted edges. An LVLM is used to identify the *set* of changes to $x$ with a *sequence* of changes to $x$. Each change corresponds to a concept edit. Each concept corresponds to a concept mask. Thus the changes to $x$ can be achieved by using a diffusion model to perform inpainting inside of a concept mask. \n\nThe evaluation is carried out quantitatively by measuring realism with FID/CMMD, effectiveness with success rate of CF generation, and stability by the proportion of identical labels obtained across independent runs. V-CECE is compared with several baseline methods from prior literature.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- V-CECE compares favorably to the baseline methods on the chosen metrics\n- The **Discovering biases** subsection in section 4 shows evidence of differences in the semantic space of DenseNet and V-CECE. This is a neat result. \n- V-CECE requires the execution of fewer edits than competing methods \n- The paper has a human evaluation study. It showed evidence that DenseNet is doing too many edits from a human perspective. \n\n## Weaknesses\n\n- I would have liked more detail in section 3.3 about how the diffusion model executes the edits, or at least a sentence saying that more detail can be found in the **Generative Module** subsection in section 4.\n\nQUESTIONS:\n- I think you did not mean to write $s_i$ twice in lines 109-110?\n- This is in response to line 154. How can the authors be sure that the diffusion model they use was not actually trained directly on their data?\n- I wonder if the claim on lines 305-308 that explaining DenseNet is pointless because CNNs learn statistical dependencies and as a result do not recover human-level semantics is either too strong in need of more justification. After all, researchers in biology have found correlations between CNN visual processing and human visual processing (at lower levels).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthe paper introduces a blackbox visual counterfactual explanation generation framework aimed towards generating meaningful and human interpretable edits to the given picture so that its label flips. the key idea is to first identify a set of possible semantic edits to the image, apply them in sequence in terms of their \"cost\" of edit, and stop when the minimal cost edit which flips the label is encountered. for identify set of edits, the paper utilizes WordNet which uniquely has \"part-of\" and \"is-a\" kind of relations required in this setting, and use the shortest path metric as a proxy for semantic cost.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n\n1. Tackles an important and critical question in explainable AI research\n\n2. Proposed technique is fully blackbox, generalizable (within the WordNet concept space), and clearly interpretable by humans \n\n3. Evaluations are over different datasets and human evaluations clearly demonstrate the utility of the method. \n\n__Weaknesses:__\n\n1. Authors argue that edits produced by diffusion based counterfactual models are \"merely responding to changes in pixel distribution\". While it may be true, the proposed method is not entirely safe either. V-CECE proposed edits still rely on annotations and concepts inferred from pixel distributions (e.g., by object identifiers). Thus, despite the valid goal of aligning edits with human semantics, it can not fully eliminate the influence of low-level pixel patterns on classifier decisions. Please discuss this aspect more explicitly.\n\n2. The uniqueness of optimal edit set is not guaranteed. Although it is not a major limitation of the work, it may result in more edits than actually required, thus leading to a potentially weak explanation. This issue could be acknowledged and its impact can be discussed further. \n\n3. Experiments with semantically diverse set of datasets is critical for strengthening the claim of generalizability of the proposed model. \n\n4. No details of human participants involved is provided. \n\n5. There is a strong dependence on a specific kind of knowledge graph -- it should have (only?) visually groundable relationships organized in a hierarchical manner (e.g., \"part-of\" or \"is-a\") for the proposed model to make sense. Instead of just saying \"knowledge graph\", potentially leading to ambiguities (since Yago or WikiData are also knowledge graphs but quite useless in this task), authors should explicitly list the desiderata from the conceptual graph model that could be used. \n\n6. the proposed model may not be able to handle cases where classifier actually relies on unannotated, global or non-object features (such as brightness, texture, or some other scene-level cue).\n\nQUESTIONS:\nBesides the list of weaknesses above, I have the following questions: \n\n1.  How sensitive is the method to the quality of the semantic annotations? If the concets are inferred noisily or incompletely, does it still work? \n\n2. can the method handle unseen concepts outside the annotation vocabulary? if so, how? please provide details. \n\n3. it may also be possible that the edits that actually flip the class label are not annotated in the image, for example a general bright background may be able to classify the image as a \"daytime picture\". what kind of counterfactual editing can this model generate in such cases?\n\nLIMITATIONS:\nAuthors have given discussion of limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-05"}