GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers
Authors: Eloi Zablocki
Valentin Gerard
Amaia Cardiel
Eric Gaussier
Matthieu Cord
Eduardo Valle
Venue: Accepted by TMLR ()

Abstract: Understanding the decision processes of deep vision models is essential for their safe and trustworthy deployment in real-world settings. Existing explainability approaches, such as saliency maps or concept-based analyses, often suffer from limited faithfulness, local scope, or ambiguous semantics. We introduce GIFT, a post-hoc framework that aims to derive Global, Interpretable, Faithful, and Textual explanations for vision classifiers. GIFT begins by generating a large set of faithful, local visual counterfactuals, then employs vision–language models to translate these counterfactuals into natural-language descriptions of visual changes. These local explanations are aggregated by a large language model into concise, human-readable hypotheses about the model’s global decision rules. Crucially, GIFT includes a verification stage that quantitatively assesses the causal effect of each proposed explanation by performing image-based interventions, ensuring that the final textual explanations remain faithful to the model’s true reasoning process. Across diverse datasets, including the synthetic CLEVR benchmark, the real-world CelebA faces, and the complex BDD driving scenes, GIFT reveals not only meaningful classification rules but also unexpected biases and latent concepts driving model behavior. Altogether, GIFT bridges the gap between local counterfactual reasoning and global interpretability, offering a principled and extensible approach to causally grounded textual explanations for vision models.

Published in Transactions on Machine Learning Research (02/2026)
GIFT: A Framework Towards Global Interpretable Faithful Textual Explanations of Vision Classifiers
Éloi Zablocki
cloi. zollockiivalloo..... valentin, Sesennii.gerardfii gerard@epft.rc ch amaia. c com eric.gausier Gumioo Gumii--renobble alpes. motthieu. cord valeo. com eduardo. valle inteeroom.tee
Valentin Gerard*1.f
Amaia Cardiel
Eric Gaussier
Matthieu Cord
Eduardo Valle
equal contribution
Valeo.ai, Parts, France
Universilé Grenoble Alpes, France
Sorbonne Universilé, Paris, Fronce
now at EPFL; now at Intercom
Reviewed on OpenReview: https: // openreview. net/ /orum?
? id= OwhW5/MPFD
Abstract
Understanding the decision processes of deep vision models is essential for their safe and trustworthy deployment in real-world settings. Existing explainability approaches, such as saliency maps or concept-based analyses, often suffer from limited faithfulness, local scope, or ambiguous semantics. We introduce GIFT, a post-hoc framework that aims to de- rive Global, Interpretable, Failhful, and Texlual explanations for vision classifiers. GIFT begins by generating large set of faithful, local visual counterfactuals, then employs vi- sion language models translate these counterfactuals into natural-languag, descriptions of visual changes. These local explanations are aggregated by a large language model into concise, human-readable hypotheses about the model's global decision rules. Crucially, GIFT includes a verification stage that quantitatively assesses the causal effect of each pro- posed explanation by performing image- based interventions, ensuring that the final textual explanations remain faithful the model's true reasoning process. Across diverse datasets, including the synthetic CLEVR benchmark, the rcal-world CclebA faces, and the complex BDD driving scenes, GIFT reveals not only meaningful classification rules but also unex- pected biases and latent concepts driving model behavior. Altogether, GIFT bridges the gap between local counterfactual reasoning and global interpretability, offering a principled approach to causally grounded textual explanations for vision models.
1 Introduction
Explainability is crucial for deploying deep vision models high-stakes applications such as autonomous
driving (Omeiza al.] 2022 Zablocki al., 2022) and medical imaging (Tjoa & Guan 2019 where un-
derstanding model decisions ensures trust and safety. Solutions for explainability include feature attribution
methods (Selvaraju et al. 2017 Sundararajan et al. 2017).. concept-based approaches (Kim et al. 2018
Fel et al. 2023b). and model-surrogate methods (Ribciro et al. 2016 Lundberg & Loc 2017). However, those approaches often lack laithfulness, with explanations ions confounded by spurious correlations in the data without causal link to the model's decision. As a result, they fail to accurately reflect the model's true reasoning process, leading potentially mislcading interpretations (Rudin) 2019 Dasgupta et al. 2022), Counterfactual explanations (Wachter et al.) (2077) address that limitation by identifying minimal input changes that alter model outputs, capturing causal relationships. However, counterfactual explanations have

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Post-hoc methods for explaining vision classifiers.
Explainability
Level of output
Faithful to
Scope
family
interpretability
the model
Input attribution
Local
Low (saliency maps)
No
Model-surrogate
Local
Average (surrogate model)
No
Concept-basod
Global
Low (CAV) or High (text)
No
Counterfactual
Local
Average (images)
Yes
GIFT (ours)
Global
High (text)
Yes
their own limitations: they are inherently local, focusing on specific instances; often are hard to interpret, as they depend visual analysis of the generated changes; and can be ambiguous, with single counterfactual modification potentially stemming from multiple plausible causes.
In this work, we address those limitations by introducing GIFT: framework for obtaining more Global, Interpretable, Faithful, and Textual explanations for deep vision models. GIFT builds upon counterfactual explanations while mitigating their weaknesses through three key innovations: (1) Global Reasoning. To move beyond the locality of individual counterfactuals, GIFT aggregates multiple counterfactual explana- tions across the model's input domain. This enables discovering broader, global insights into the model's behavior. (2) Natural Language Interpretability. GIFT leverages natural language transform terfactual explanations into clear, hunan-readable descriptions. By utilizing the reasoning capabilities of Large Language Models (LLMs) (Radford et al. 2019), GIFT resolves ambiguities in counterfactuals and organizes them into coherent global explanations. (3) Verification of Explanations. GIFT includes a novel verification step that measures the causal effect candidate explanations. By intervening on images using image-editing models, GIFT ensures that the derived explanations are faithful to the model being explained.
coun-
As a result, the explanations produced by GIFT are global, textual, disambiguated, and supported by causal verification to the underlying model. Unlike standalone model, GIFT is framework that can be reasoning, and text-guided
instantiated with recent models for counterfactual generation, (vision-)languaal image editing. That flcxibility is essential for handling the diverse data domains and decision models across different tasks, as general-purpose models often struggle with the specificity required in certain domains.
We validate GIFT across diverse binary-classification scenarios: on the CLEVR dataset (Johnson et al. 2017), GIFT uncovers classification rules in a controlled sctting with complex compositionality; on the ColebA dataset (Liu et al., 2015); GIFT discovers fine-grained relationships between data features and classification outcomes; and on the BDD-OLA dataset (Yu et al.) 2020), GIFT identifies biases in challenging domain with driving scenes. Our contributions are:
We introduce the first framework for global, textual, and counterfactual explanations for vision classifiers. The faithfulness of the explanations is supported by causality quantification.
We derive global explanations by combining two ideas: (1) gathering counterfactual signals across the model's input domain, which are inherently causal although very local; and (2) reasoning with an LLM to uncover global insights from those local signals. Both ideas and their combined synergy are novel, to our knowledge.
We analyze two complementary causal metrics, demonstrating their relationship, and providing GIFT with verification tools to measure causal association.
We validate GIFT's ability to generate meaningful global explanations across several natural image domains and use cases in binary classification.
2 Related Work
Explanation methods are typically categorizod as intrinsic (built into models during training for inherent interpretability (Hendricks et al.) 2016 Zhang et. al. 2018)) post-hoe (applied post-training to explain
2

---

Published in Transactions on Machine Learning Research (02/2026)
decisions (Zhang et al, 2018 Chen et al. 2019) ) As shown in Tab. 1 post-hoc methods further divide into local (per-instance) and global (model-level).
Feature Attribution Methods highlight regions of the input image that most influence the model's
predictions, often visualized as a saliency map. Gradient-basee bastd approaches (Zeiler & Fergus 2014 Selvaraju
at. 2017 Sundararajan et al. 2017 Fel al. 2021 Wang et al.] 2024bb back-propagate gradients to identify
influential features, and cwtuurbaiiinnaaaa changes in model output (Fong & Vedaldi 2017 Wagner et al.. 2019 Bacha & George 2025) Though
methods remove or alter parts of the input to observe resulting
widely used, attribution maps offer only local, instance-level explanations. They require user interpretation, introducing subjective biases (Borowski et al. 2021), and can be unreliable, somctimes rosembling edge-
detectors on CNNs.
Model-Suurogate Methods explain black- models through interpretable surrogate models (Eugel et al. 2024 Páez 2024). Approaches such as LIME (Ribeiro ct al., 2016) and SHAP (Lundberg Lee 2017] are widely used to provide local explanations, with some ablempts at global surrogales for more holistic
interpretability (Frosst & Hinton 2017: Zilke et al., 2016 Harradon et al., 2018). These methods simplify inputs, e.g., using superpixels (Ribeiro et al. 2016) or high-level input features (Lundberg & Lee| 2017 which can result in reduced fidelity and limit the reliability of explanations for complex, high-dimensional
data (Zhang et al. 2023).
Counterfactual Explanations, in the context of image classification, are images minimally altered to yield a different classification outcome, thus revealing semantic changes needed to cross the classifier's decision
boundary (Wachter et al. 2017). Retrieval-based [Hendricks et al. 2018 Goyal et al. 2019b Vandenhende
ct al.
2022) and generator- based methods (with GAN (Rodríguez ct al.
2021 Jacob ct al. 2022 Zemni
et al. 2023) or diffusion models (Jeanneret et al.) 2022 Augustin et al. 2022
Jeanneret et al. 2023 Sobieski
& Biecek 2024)) are used to create realistic counterfactuals. While counterfactuals can provide intuitive
insights, they are inherently local explanations. Moreover, they demand careful user interpretation, which
may introduce human biases (Borowski et al. 2021 Zemni et al. 2023
Concept-based Methods explain model decisions through luman-interpretablle 'concepts' (Kim et al.
2018 Lee ct al. 2023 from neuron-level analyses (Bau et al.
2017] to layer-level approaches such as CAVs
[Kim et al. 2018 Most require annotated data for concept
presence/absence (Kim et al. 2018 though (Zhang et al.
2021: Kumar et al. 2021 Fel ltt al. 2023b). These methods typically require model weights, limiting applicability, and are architecture- specific, with most methods only supporting CNN-based models (Kumar et al.) 2021 Kamakshi et al.
some extract concepts unsupervised via matrix factorization
2021 Fel el al. 2023b). Manually defined concepts may overlook biases, while unsupervised ones still require human interpretation (Fel et al. 2023a). In contrast, GIFT uses counterfactual signals instead of predefined concepts, avoiding prior biased assumptions. Its textual explanations are directly interpretable and verifiable for faithfulness, providing more reliable insights than correlation- approaches. Similarly, MATA (Shaham et al., 2024) employs a multimodal agent to automate interpretability experiments by composing tools for neuron description and bias discovery, though it relics heavily on itcrative hypothesis lesting rather than aggregating counterfactin evidence.
Error Explanations and Shortcut Mitigation Methods uncover model failure modes using natural language. Some rely on user- or LLM-generated inputs (Abid et al., 2022 Csurka et al. 2024 Prabhu et al., 2023 Wang al., 2024a),, which can bias the scope of discovered errors. Others identily systematic failures through data partitioning and labeling (Eyuboglu et al. 2022 Wiles et al. 2022), but require ground-truth labels. In contrast, GIFT uses the LLM solely to summarize counterfactual evidence- as a source of domain knowledge- correct and incorrect predictions. Kuhn et al. (2025) propose an unsupervised shortcut detection and mitiga- tion pipeline for transformers that clusters patch key-space, extracts prototypical patches, generates textual prototype descriptions with a VLM, and mitigates shortcuts via patch ablation and last-layer retraining. While complementary to our aims, their method operates on model internals and focuses on ablation-based mitigation tailored to VIT architectures; by contrast, GIFT is counterfactual-cnnerr and model-agnostic, grounding explanations in generated edits and verifying them with causal measurements.
reducing LLM-induced bias. It also goes beyond failure analysis to explain both
3

---

Published in Transactions on Machine Learning Research (02/2026)
Stage Generate countert.
Local
Stage 2:
X Local
Stage Extract candidate
Global Interoretable
Stage 4:
Global
Interpretable Faihful
Translate changes natural language
Interpretable
Verify explanations with interventions
Interpretable Faithfu
explonations
Faithfu
global explanations
Failhful
1
M P
The tiny red metal
Class
presence of a red metal object
PNS:
CEX
Change
tiny brown metal sphere became rubber
block behind
71%
Captioning
EDIT
M
M 1
M 0
--Class = presence of red object
PÑS:
CEX
Change Captioning
The red thing became purple
44%
EDIT
M
M
M
The tiny red metal ball behind the bny brown thing nevly placed
Class presence sphere
PNS: 11%
Change Captioning
CEX
EDIT
M
Figure 1: Overview of GIFT. Given classifier M (here discriminating images with 'red metal object') GIFT extracts explanations in four stages: Stage 1 generates local visual counterfactual explanations for several images. The counterfactuals are by nature faithful to the classifier as they reveal semantic and minimal changes to the query images that flip the classifier's output. Stage 2 translates in natural language the visual differences between original and counterfactual images with an image change captioning model this enhances interpretability but risks introducing potential noise. Stage 3 applies an LLM to aggregate local explanations into candidate global explanations this disambiguates local evidences. Lastly, Stage 4 filters out validates these global explanations with intervention studies, to ensure faithfulness with respect to the classifier.
3 GIFT Framework
GIFT automatically identifies and validates explanations for a differentiable classifier M. An explanation. here, describes features, attributes, or concepts that are significant to the classifier's decision. Explanations should be meaningful to humans and faithful to the classifier's behavior.
Overview. GIFT follows four stages Fig. 1.. In Stage Sec. 3.1p, creates local explanations by generating several counterfactual input- pairs, highlighting faithful visual features relevant to M's decision at the individual input level. In Stage 2 (Sec. 3.2). it uses Vision Language Model (VLM) to translate the differences between each image in a counterfactual pair into interpretable text descriptions. In Stage 3 Seec. 3.3). GIFT employs an LLM to identify recurrent patterns on those local explanations to obtain global candidate explanations. Finally, in Stage 4 (Sec. 3.4), it evaluates the candidate explanations on causal metrics, mcasuring their faithfulness M's decision process.
3.1 Stage 1: Faithful visual and local explanations
GIFT begins by producing local, faithful visual explonations for the target model M through counterfactual image gencration. Counterfactual explanation methods (Zemni et al. 2023 Jeanneret et al. 2023 scarch for minimal, semantically meaningful modifications of an input image x that flip the model's prediction. Unlike adversarial attacks (Szegedy et al. 2014), which often rely on imperceptible perturbations counterfactuals aim expose the model's semontic decision boundaries by altering visual features that the model truly relies |Freiesleben, 2022). By explicitly requiring the model's output to change, these methods are faithful by definition they do not rely surrogate approximations but instead directly probe the model's decision boundary. This ensures that the resulting explanations capture a causal link to M's decision.
Concretely, given a sample set I of N input images, we use a counterfactual generator CEX that directly accesses the model M and its decision M(x) to synthesize a counterfactual x' such that M(x') # M(x). This yields a set of image pairs:
P= ((,,2') II x' CEX(a, M(),
(1)

---

Published in Transactions on Machine Learning Research (02/2026)
class change
nn class change
Class change
Model M
Target
Target Model Mf -
<c...>>>
Target
Target Nodel M
Target Model
Image edition
Image edition
Image edition
XXX
Add red object
Add red object
Remove red objects
Figure 2: Causal interventions, Stage 4. For a candidate explanation (e-g., 'class 1 presence of red object'), we use an image-editing model to add or remove the underlying concept ce (e.g., 'red object') and observe the impact on the classification outcome, which aggregated to compute the CaCE (Eq. 3) and PNS Eq. 4p. In the example, the classifier M recognizes images with 'red metal object', and we observe, as expected, a partial cansal effect: removing red objects impacts the outcome, but inserting non-metal red ones does not.
Each pair xx,x²) provides a faithful, instance-specific view of the local visual change responsible for decision flip. These local counterfactuals constitute the atomic evidence from which subsequent stages of GIFT will infer higher-level, global explanations.
3.2 Stage 2: From visual counterfactuals to text
For each counterfactual pair, GIFT creates change caption, translating the visual changes from the original image to the counterfactual into simple, descriptive natural language.
Visual counterfactuals, per se, require tedious and subjective manual comparison to become interpretable. Moreover, natural language is often more accessible to humans than low-level visual cues, such as saliency maps or feature vectors (Sammani et al. 2022 Nauta et al. 2023). We use instead vision-language models (VLMs) change-caption (Guo et al. 2022 Qiu al., 2020: Hosseinzadeh Wang 2021 each image pair, i.e., automatically describe the changes between each original image and its counterfactual x'.
Concretely, use Change Captioning model CC that takes as input the pair of images and outputs change caption t CC(x,x'). While more interpretable than visual comparisons, these captions remain local to each image pair and may be less faithful due to the compressed nature of text and potential noise introduced by the VLM. The next stages address these.
3.3 Stage 3: Candidate global explanations
GIFT will now gather all change captions, analyze them for recurrent patterns, and propose candidate global explanations for the target model behavior. Concretoly, we gather the set of all local explanation tuples T = ((CC(r x'), M (x), M (x)))((( E P) and use an LLM to summarize them into a set of candidate global LLM(T) that helps explaining the classifier's behavior. Remark that the LLM has access, for each tuple, to both the change caption and to what happened to the model decision (M M (z')). The LLM has no direct access to the model itself.
explanations
This stage addresses the shortcomings of the local explanations. Tt disambiguates local evidence where given counterfactual change implies multiple plausible explanations. For example, a 'red metal ball became brown' explanation, isolation, could mean that the classifier tracks the absence of 'red objects' (or 'red metal objects') in class or the presence of 'brown objects' (or 'brown metal objects') in class 1. This stage also filters out noise, alleviating linguistic variations and eliminating most irrelevant or inconsistent explanations raised in Stage
3.4 Stage 4: Hypotheses verification
The initial explanations at Stage 1, although local and laborious to interpret, are faithful. By Stage 3, have global interpretable explanations but cannot guarantee their faithfulness, the VLM and LLM

---

Published in Transactions on Machine Learning Research (02/2026)
may introduce noise. This challenge is precisely what motivates Stage 4, that verifies which explanations in & remain faithful to the classifier. Failhfulness in GIFT is quantified causally rather than correlationally. Specifically, a global explanation is retained only if interventions on the identified concept significantly alter the classifier's decision. This ensures that accepted explanations correspond to necessary and sufficient causal factors. For each explanation the underlying concept ce (e.g., 'red object") and apply a two-step verification using these concepts.
(e.g. 'class implies the presence a red object"), we extract
Coarse filter. This first step measures the corvelation between the model's decision and the underlying concopt Co of each candidate explanation e E E. We use the Directed Information (DI) (Gallager 1968) [eaww
between the concept and the predicted class label as the correlation metric DI(ce, = H(c.)
where
[(G.y)) is the mutual information between the concept presence and the classifier output, and H(c.) is the entropy of the concept presence. DI quantifies how much the presence of concept ce explains the classifier's decision assessing the directional influence of the underlying explanation on the classification outcome. To estimate Ce 's presence in an image z, we use Visual Question Answering model VQA (z, ce) - 10,
We measure the DI for each explanation, rank them, and keep the most promising ones. This pre-filter is motivated by the fact that applying the VQA model provides a correlation metric that is much easier and computationally cheaper to obtain than causal metrics requiring image interventions.
Fine filter. We evaluate high-correlation explanations E & that pass the coarse filter on their causal effect on the classifier's decision by measuring the causal association (Pearl) 2009 Goyal et al.] 2019a) between their underlying concept Ce and the model decision y. We partition the samples of a validation set of images according to Ce, intervene those images by inserting removing ce and verify the impact of those interventions on the model decision, illustrated in Fig.
Concretely, we use VQA(#, ce) to partition a validation set X' into images where ce is present (Xa.=1) or absent (X,,=o). We then use a text-guided imagc-oditing model EDIT to intervene and edit x into क = (DIT(>, ce). flipping the presence ce in the image. Specifically, we create images +c := EDIT ce) for each æ E X=.== by adding ce, (similarly, Var X2,=1, == (a, Image editing falls short of the unfeasible ideal of modifying physical reality past image acquisition and assumes other concepts will be left relatively undisturbed. Nevertheless, this practical procedure has shown good results in estimating the actual causal effect (Goyal et al. 2019a). We also assume that edited images remain in-distribution for the classifier M. We now present two metrics to evaluate the causal effect of the interventions.
Causal Concept Effect (CaCE): Goyal et al. 2019a) defines CaCE,,, for a given concept c, as:
CaCE., E[M(x)lc. = 1| [x](())c. 0].
(2)
CaCE,, measures how the addition or removal of 6, induces any class change. Specifically, for binary classifiers or multi-class classifiers evaluated in a one-versus-all manner, considering (x) E (0, 1}, we have CaCE,, E [[,,,]], which can be estimated by:
CaCFa = [
1
Σ RE
M(x) - M0-
Σ M(xtt.) - M(r)) =ENr,
(3)
with the following interpretation:
If CaCE,, is sufficiently positive (resp. sufficiently negative), then the presence of ca 'entails' class (resp. 0) and its absence 'entails' class 0 (resp. 1),
Otherwise, ce, in isolation, has a negligible impact on the classification decision.
Probability of Necessary and Sufficient Cause (PNS): Based on Tian & Pearl (2000), we estimate the prob- ability PNS of ca to be both necessary and sufficient cause for the class y. As shown in Sec. C.2 it

---

Published in Transactions on Machine Learning Research (02/2026)
can be estimated by:
PNS
Σ ) = y/+ |28 xEd
1
Σ
= +1))
(4)
2 ZERcc=0
In addition, for {0,1ss settings, we have: CaCEa, = .N PNS G..= Thus, CaCEe, and PNS..,y=1 (or equivalently CaCE, and .N bring complementary information as PN bet- ter evaluates the impact of on = 1, while CaCEa,, depending on its closeness to PNSS...= reveals how unidirectional the causal effect of C. is on y = (details in Sec. C.3 In the remainder of the paper, we write PNS for PNSS and CaCE for CaCE..
3.5 Further exploration of the explanation space.
Ideally, a single explanation with a very high CaCE and PNS would fully capture the model's decision rule, However, for complex classifiers, single explanations rarely suflice due entangled feature interactions. GIFT allows end-users to explore and refine explanations by proposing new ones combining them with those identified by the framework. These refined explanations are then re-evaluated with CaCE and PNS assess their causal impact the model's output. As GIFT's intermediate stages are fully interpretable, users stay in the loop, enabling an iteralive process to deepen their understanding of the model's decision-making.
4 Experiments
We evaluate GIFT on three use cases of increasing complexity. In Sec. 4.1 we test GIFT's ability to uncover binary classification rules in a controlled setting with the CLEVR dataset. In Sec. 4.2, we show how GIFT finds fine-grained explanations for a binary classifier trained on CelebA. Finally, in Sec. 4.3, we use GIFT reveal biases in a model trained on BDD-OIA, a dataset of complex driving scenes.
4.1 Uncovering classification rules on CLEVR
Here, the goal is uncover the meaning of obfuscaled labels ('O' or 'I') with GIFT explanations.
Data and target classifiers. We use the CLEVR dataset (Johnson et al. 2017 which presents synthetic but photorealistic arrangements objects with various shapes, colors, and textures on a neutral background. We train various target classifiers on binary tasks recognize a specific visual rule such as 'cyan object present" or 'yellow rubber object present'. CLEVR images hide complex compositional potential behind their minimalist appearance, with diverse colors, lextures, and shapes. Uncovering the underlying rule in each classifier from local examples is challenging, even for humans (see Sec. F.1.. We include both ViT- Small-Patch16-22 1--224 (Dosovitskly et al. 2021) and ResNet-34 (He et al. 2016) architectures. Our evaluation comprises 12 unique combinations of visual rule X architecture.
Instantiation of GIFT. The components for each stage are based on the dataset's specific characteristics. For the counterfactual generation (Stage 1), use OCTET (Zemni et al.) 2023) with BlobGAN (Esstein et al. 2022). as it is specifically designed for compositional scenes. CL.IP4IDC (Guo et al.] 2022 handles
changc-captioning (Stage 2) as it was trained on the same visual domain (Park et al. 2201)) J. For Stage
3, we experiment with either ChaiGPT4 [OpenA] 2023) or (iwen2.5-72B-Insttee
(Yang et al. 2024).
Interventions (Stage 4) use a Stable Diffusion model (Rombach et al. 2022) adapted to CLEVR for targeted addition and removal of objects (Cho al. 2024). The VQA model (Stage 4) MiniCPM Yao et al. 2024
Main results. Tab. 2 reports the trials of GIFT uncovering the hidden visual rule underlying the target classifiers after Stages I 4. The true visual rule was uncovered in 11/12 cases, with both ChatGPT or Qwen used in Stage 3. We consider the trial successful when the true rule appears on lop after Stage and ranking by the causal metrics.
We provide a detailed qualitative analysis of the 'Red metal object' (ViT) trial in Fig. 1. (Stage 1:) Counter- factual image generation flips the model's decision by modifying objects' presence, location, or appearance.
7

---

Published in Transactions on Machine Learning Research (02/2026)
Rule (Ground Truth)
Arch.
GIFT
GIFT Qwen
Abl.
CaCE
PNS
Table Classification rule reverse engincering on CLEVR. Success criterion is to have the hidden true rule top after ranking them by the
GPT
Cyan object
ResNet
62.2
62.2
Purple object
ResNet
71.2
71.2
Metal object
ResNet
/
V
24.2
24.2
Rubber object
ResNet
X
29.8
29.8
CaCE and PÑS causal metrics (Eq- 3 4 after Stages 1 1-4. GIFT succeeds for all but one case with both metrics. An ablation (Abl.) with indepen- dent captions instead of change captions misses most rules (see
Red metal object
ResNet
X
N/A
N/A
Yollow rubber object
ResNet
X
67.2
67.7
Cyan object
VIT
X
63.1
63.1
Purple object
VIT
V
70.2
70.2
Metal object
VIT
37.4
37.4
Rubber object
VIT
X
30.8
31.3
App. F.4. Metrics in %.
Red metal object
VIT
70.7
70.7
Yellow rubber object
VIT
71.7
71.7
Concepts associated to the class 'Old'
DI
CaCE
PNS
after Stage 9
Receding Hairline
29.9
2.0
5.0
Neck Wrinkles
28.1
3.0
3.5
Wrinkles on Forehead
26.4
6.5
7.0
Wrinkles around Eyes
20.7
6.5
7.0
Table 3: Output of GIFT on the CelebA 'Old' clas- sifter. Concepts are evalu- ated with causal metrics when Z 15%. The full table is given in App. DI, CaCE and PNS in %.
Glasses
16.5
11.5
16.0
Drooping Eyelids
15.4
2.5
4.0
Detailed Background
0.2
Low Camera Angle
0.1
after manual combinations of concepts with PNS 2 5%
Sec.
3.5
Hairline +
Wrinkles on Forehead
47.8
8.2
8.2
Glasses + Wrinkles around Eyes
60.4
21.7
23.6
Glasses Hairline
61.4
24.6
25.4
Glasses Hairline Wrinkles Forehead
78.2
26.0
27.1
Glasses Wrinkles Forehcad
59.6
24.1
27.6
Glasses Wrinkles on Forehead + Eyes
65.6
26.8
28.9
(Stage 2:) Because each counterfactual pair provides very local and ambiguous explanation, cach change caption extracted has weak evidence. There is also some linguistic variation. (Stage 3:) The sunmarization has to filter out the noise and recover robust trends the whole sample of captions and proposes several potential explanations. Explanation verification (Stage 4) turns out to be critical to uncover the true rule. Without Stage and ranking by causal metric (Sec. 3.4) the user has no guidance to distinguish them.
The of two causal metrics is informative since close results indicate that the causal effect is unidirectional. meaning that the evaluated rule is either the true one metal object' instead of 'Red object'). Indeed, in this use case, the true TTb.. 2) and the overspecified Tab. in Supp. Mat.) rules are always such that [PNS CaCE 0.5% which suggests that appropriate thresholding can reduce the amount of candidate rules.
overspecification of the true one (such as 'Red
The 'Red metal object' (ResNet-34) failure case illustrates how users can interact with GIFT. As shown in Tab. 6 (in Supp. Mat.), the LLM at Stage 3 generates partial rules 'Red object' and 'Metal object', which
Stage tags with CaCE of 42.9% (vs PNS 43.9%) and 11.6% (vs PNS 13.6%) respectively. When manually try the combination of these two partial rules, we obtain the true rule 'Red Metal Object (CaCF=PNS-661.68) Full details in Sec. F.4
8

---

Published in Transactions on Machine Learning Research (02/2026)
M+ - Old
M
Young
M
Young
M Old
4
4
Querl Image x
Edited image
Query image
Edited image to
Figure Samples from the intervention study on the CelebA-'Old classifter. The combined concepts under scrutiny are: 'Glasses', 'Wrinkles on Forehead', 'Wrinkles around Eyes!. Each pair has the query image on the left and the edition on the right.
4.2 Exploring detailed explanations on CelebA
Data and target classifier. We evaluate GIFT on CelebA Liu et al. 2015 a dataset of human faces
that introduces real-world challenges. Following Singla et al. p0200; Jacob et al. (0022) the target classifier is a DenseNet (Huang et al. 2017 trained to predict whether a face is 'Old' or 'Young.
Instantiation of GIFT. Unlike in the CLEVR use-case, where change-captioning and image-editing mod- els were specifically tailored to the dataset, this usc-case employs more general-purpose models. Counterfac- tual generation (Stage 1) uses the diffusion-based model ACE (Jeanneret et al. 2023). Change captioning (Stage 2) employs the Pixtral-12B VLM (Agrawal et al. 2024), guided by a one-shot in-prompl annotated sample as guidance. Concept identification (Stage 3) performed zero-shot using Qwen 2.5 72B Instruct, quantized to bits (Yang al. | 20024). Interventions (Stage 4) use combination an image-conditioned version of Stable Diffusion 3 (Esser al. 2024] and in-painting version of Stable Diffusion 2 (Rombach)
al. 2022 relying masks obtained with Florence-2 (Xiao et al. 2024). Details in App. G Lastly,
causal metrics provide different rankings on this dataset, we favor PNS.
Main results. Tab. 3 shows a subset of explanatory concepts proposed by GIFT. It includes reasonable attributes, such as "Neck Wrinkles' and 'Receding Hairline', and more unexpected ones, such as 'detailed background'. Directed information (DI) scores provide an initial filter to remove explanations whose concepts have no significant relationship to the classifier's output, allowing us to focus on more promising hypotheses. From that refined set, we conduct intervention studies to measure causal effects. Overall, concepts derived from Stage 3 have low causal metrics, despite high DI scores: intervening on any of these attributes alone does not change the classificr's decision. This suggests that the classifier is robust and not overly reliant single attributes. Yet, 'Glasses'
exception, with 16% probability of being necessary and sufficient cause to the 'Old' class, likely reflecting training data bias, as 'Glasses' correlates with 'Old' over 20%.
As described in Sec.. 3.5, we further explore the explanations by combining and reevaluating concepts with PNS 2 5% Tab. 3 rows 'after manual combination ... '). Some concepts that, individually, have low causal metrics yield much higher values after combination, indicating
increased understanding of the classifier. For example, 'Hairline' and 'Glasses' have respective PNS of 5 and 16% but reach 25.4% when combined. Fig. Jillustrates the targeted intervention for the combined attributes 'Glasses', 'Wrinkles Forehead', and 'Wrinkles around Eyes', which is sufficient to flip the model's decision in the two query images while leaving the rest of the face mostly unchanged.
4.3 Discovering classifier bias on BDD-OIA
Here, the task is to identify any classifier's rules, including spurious unexpected ones.
Data and target classifier. We evaluate GIFT on BDD-OLA (Xu et al., 2020). a dataset front- camera driving scenes in complex urban environments annotated with admissible car actions (turn left, go ahead, turn right, stop/ddcclerrate.
Widely used to benchmark explanation methods (Jacob et al. 2022) Jeanneret al. 2024), BDD-OTA is paired with biased binary classifier for the "turn
Zemni et al. 2023
9

---

Published in Transactions on Machine Learning Research (02/2026)
Can turn right
Change caption
Concepts that may explain 'Cannot tumn right"
Directed Information
Causal Concept
PÑS
road >> leftmost lane >> had fower vehidls,, new has more vehicles including
CaCE
Dense Traffic
7.7
51
road -> center -> traffic lights are octive: green light
Dense Traffic in left lane
10.3
45
47
Dense Traffic middle lane
0.3 -x
road rightmoot lane had bus,
Dense traffiic close ego
14,0
27
29
Query image 27
road middle more Illmiinted by surrounding lights
now has more
Stopped vehicles
Red traffic lights
1.4
0.8
M
Cannot tum right
Ego-car dashboard is bright
0.0 -X
vehice >> leftmost >> doser and sligtlly smaller
Wet road
0.0
Dark road
0.4 -X
vehicle center changed to different type: more iluninated
Many buildings
0.1
vehide -> nighimost >> closer and sightly smaller
Many streetlights
0.3
Pedestrians road sidewalks Objects on the road or sidewalks
0.0 -XX
building -> rightmostt> sightl brighter
1.1 --X
Counterfactual explanation
1
GIFT output for the biased classifier on BDD-OIA (Xu et al. 2020). The model M classifying images into Caan/Canmot turn right' is intentionally biased for vehicles in the left lane to yicld 'Cannot turn right'. We illustrate the output of Stages 1 and 2 for single randomly selected input and the global output of Stages 3 and 4. Causal metrics used when DI 2 5%. DI, CaCE, PNS in %.
Figure 4:
right' action introduced by Zemni et al. (2023). The classifier was intentionally trained with dataset bias, associating images with vehicles the left side to the 'cannot turn right' class, in addition to their normal labels. The setup aims to test whether GIFT can reveal this injected bias in the model's decision-making.
Instantiation of GIFT. The instantiation follows the sctup described in Soc. 4.2] building on general-
purpose models. For counterfactual generation (Stage 1), we use OCTET (Zemni et al. 2023), which is better-suited for complex driving scenes and to enable fair comparison to baselines. Details in App. H
Results. Fig.
shows results for each stage of GIFT, with (1) one sample pair of counterfactual images (of the many generated), (2) the change-caption extracted for it, (3) the relevant concepts that the LLM obtains over many change captions, and (4) the correlation and causal metrics computed for verification.
In the sample shown in the figure, the counterfactual image presents several changes, including new vehicle in the left lane. Due to the complexity of the scenes, individual change captions in this dataset tend to be long and noisy. Again, Stage 3 is critical to condense a large collection of weak signals into a small list of meaningful hypotheses, which, in Stage 4, can be pre-filtered with the correlation metric (DI) and then ranked by the causal metrics. Fig. 16 (in Supp. Mat.) illustrates interventions used to confirm these rules.
We finish with three strong explanations for being unable to turn right: 'dense traffic', 'dense traffic in left-lane' (the bias!), and 'dense traffic close to ego-vehicle', with, respectively, CaCE of 51, 45, and 27% and PNS of 53, 47 and 29%. The first and third explanations capture pertinent causes preventing the model yield 'turn right'. Importantly, the second explanation uncovered by GIFT should immediately call the attention of anyone evaluating the model.
Comparison with State of the Art and Ablations. Tab. compares GIFT with existing explanation methods and ablations. The key insights are:
Human inspection insufficient (Ablation of Stages 1-3): In the user study by Zemni et al. (2023) partici- pants analyzed several images and corresponding classifier outputs but failed to identify the lell-lane-vehicle bias. This shows that manual hypothesis generation is impractical for complex models with unsuspected biases and highlights the need for automated methods.
Seeding from failhful explanations is crilical (Ablation of Slages 1 2): Several met rely on LLM- generated hypotheses to detect biases (Csurka et al.) 2024 Prabhu et al.) 2023 D'Incà et al.) 2024). However, with the biasod-BDD-OIA classifier, those methods fail because the set of LLM-generated hypotheses does not contain the bias, which shows that without counterfactual guidance (Stage 1) and visual difference
captioning (Stage 2), LLMs cannot uncover unforeseen biases. Details in App. H.2
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Left-lane vehicle bias identification on BDD-OIA. Only GIFT successfully identifies the left- leniclehicle bias. Methods relying on GPT4 g-geerreee e.g., (Prabhu et al. 2023 Csurka et al. 2024: D D'Incà et al., 2024) or human hypotheses (Zemni et al., 2023), fail detect the bias. Besides, using independent captions in Stage 2, instead of contrastive ones, make the LLM miss the bias. Counterfactuals alone only help 65% of humans to detect the bias (Zemni et al. 2023).
Explanation Method
Ablated stage
Bias Detected?
LLM-generated expl. e.g., Csurka et al. (024)
1,2
No No
Human-proposed expl. Zemnii el al. 2023
1,2,3
GIFT w/o change captions Counterfactual expl. Zemni et al. [0023
2
No
2,3,4
65% of users
GIFT
Yes
Contrastive image analysis is key (Ablation of Slage 2): In this ablation, we replace change captions with independent captions for each image in counterfactual pair. Those independent captions do not highlight key pairwise differences, and the LLM in Stage 3 thus fails to identify the bias. This shows the importance of change captioning, which emphasizes the counterfactual signal and explicitly captures differences. Details in Sec. for BDD and in Sec. F.4 for CLEVR.
Automated analysis reduces human biases (Ablation of Stages 2 4): A simple visual inspection of Stage I's counterfactuals led 35% of the user-study's participants to miss the bias (Zemni et al.) 2023), showing that humans can overlook non-intuitive biases when they appear clearly. GIFT automatically converts counterfactuals into explicit textual explanations with causal scores, making biases more apparent.
Overall, these findings validate the design of GIFT for uncovering non non-intuitive model biases. Unlike existing methods, GIFT requires no user-defined concepts about potential biases and reveals them in plain text.
5 Limitations and Future Work
As GIFT builds on various models from recent literature, inherits the limitations of its components. Below, we discuss these limitations and propose potential ways to address them in future work.
Applicability to new domains. The applicability of GIFT to new domains is intrinsically linked to the capabilities of its underlying components. Our current validation focuses binary classification in natural image domains where off-the-shelf generative models are most robust. Extending the framework to specialized fields (e.g., medical imaging) may require domain-adapted models at three key stages.
The scope of counterfactual explanations may be constrained by the generative model underlying the counterfactual method. Methods like STEEX [Jacob et al., 2022) and ACF (Jeanneret et al. 2023) limit the search space and, for instance, cannot assess the importance of object positions. OCTET (Zemni et al. 2023). which uses BlobGAN (Epstcin al.) 2022), offers partial solution by enabling disentangled representations of scene layout and semantics this motivates our choice of OCTET for object-cent ric datasets (CLEVV in Sec. 4.1 and BDD in Sec. 4.3). To mitigate the potentially limiting scope, combining multiple counterfactual methods could improve the diversity and completeness of the explanations. Despite this, our results show that grounding the explanations in local counterfactuals reveals unexpected biases and insights that human- or L.M--eenerated hypotheses might miss.
Image change captioning in complex domains (Stage 2) relies on an image change captioning model, which face challenges in complex domains, such as driving scenes. These models often lack the training interpret domain-specific or fino-grained changes, such as precise object positions or subtle attributes, which require an advanced understanding of compositional scene structure (Ma et al. 2023 Ray et al. 2023). A potential
solution is to generale synthetic perturbations (e.g,, with models like Instruct Pix2Pix (Brooks el al. 2023
that can be used to fined-tune VLMs for more accurate change captioning.
Image editing (Stage 4) can produce unrealistic outputs, particularly for complex scenes or compositional queries like 'Shift the red the right of the black truck to left' (Ma et al. 2023). Consequently, the
11

---

Published in Transactions on Machine Learning Research (02/2026)
faithfulness of our verification stage is bounded by the performance of these intervention tools: if the editor fails to insert a concept correctly (e-g., fails to add 'glasses)) the classifier will likely not change its decision, leading to a low causal CaCe score. This results in false negative where a valid rule is rejected, reflecting a conservative design choice to prioritize precision over recall. Conversely, the introduction of confounding features (false positives) is mitigated by employing highly specifie text prompts designed to isolate the causal effect of the target concept during editing. While recent advances in compositional understanding (Esser et al. 2024 show promise, further progress is needed to handle such detailed queries reliably.
Reasoning capabilities of the LLM (Stage 3). Stage 3 leverages an LLM to identify global explanations from noisy change captions. This requires the LLM to disambiguate local evidence and infer common explanations across multiple counterfactual changes (as explained in Sec. 3.3). Although our experiments show that LLMs perform reasonably well in this task, it remains an unconventional use case for the LLM, Future work could explore fine-tuning LLMs specifically for this task to improve their ability to reason over aggregated counterfactual signals and handle more complex scenarios. Moreover, achieving high recall for hypotheses at this stage is crucial, as errors can be addressed during Stage 4. Accordingly, prompt- engineering can be used to encourage the LLM to generate a large amount of hypotheses.
Automated exploration of the explanation space. After Stage 4, GIFT allows users to explore and combine concepts with observed correlation causal signals or to test their own hypotheses. While this human-in-the-loopr approach is desirable as it implies end-users in the process, fully automating the exploration process could enhance scalability. One promising direction involves using the Stage 3 LLM to iteratively reason over counterfactual descriptions while incorporating feedback from correlation (DI) and causal measurements (CaCE). This could follow the 'visual programming' paradigm (Gupta & Kembhavi, 2023), where the LLM utilizes tools like Stage 4 verification in a chain-of-thought manner (Wei et al., 2002).. However, our initial experiments in this direction were unsatisfactory, highlighting the current limitations of LLM reasoning for such ambitious goals.
Pipeline complexity. The computational cost of GIFT (see App. I is dominated by Stage 1, where counterfactuals are generated via per-image optimization. The remaining stages zero-shoo change cap- tioning, hypothosis induction, and causal ieification-require only forward passes and are comparatively lightweight. Computing causal metrics adds overhead due to the many image edits, but remains less costly than counterfactual generation.
Runtime was not our primary focus, GIFT targets high-quality, testable global explanations rather than fast local attributions. Its more ambitious goals naturally entail higher cost than traditional saliency- methods. Nonetheless, we find the trade-off acceptable: generating the hypothesis set takes about 13 minutes, with verification adding 6 minutes per hypothesis. This remains efficient relative to approaches such as Zemni et al. (023), which rely on trained human evaluators to identify biases less systematically (see Sece. 4.3).
Regarding scalability to multi-class settings, the computational cost scales linearly with the number of relationships one wishes to analyze, rather than the total number of classes. GIFT is designed to explain specific decision behaviors (e.g., Why Class A and not Class B?" or Why Class A and not the rest?") rather than processing all classes simultaneously. While our experiments focus on binary classification to isolate specific docision boundarics, in a multi-class sctting, the framework operates in a onc-versus-rest or one- versus-one manner. Consequently, explaining a specific transition (e.g. the Top-1 prediction against. the Runner-up) requires only a single run of the pipeline, making the approach viable without modification for classifiers with large label spaces.
6 Conclusion
GIFT offers conceptual and grounded approach for generating global explanations of deep vision models by systematically building from local insights. Starting with instance-level counterfactual explanations, GIFT translates those findings to natural language and aggregates them to identify recurring patterns, allowing global explanations to emerge. Those explanations are then doubly checked, with VQA for high correlation and image interventions for causal effect on the model decisions, supporting the faithfulness of identified patterns to the model's 's decision-making. A key strength of GIFT is its flexibility. By validating the
12

---

Published in Transactions on Machine Learning Research (02/2026)
framework with diverse models at each stage, demonstrate its robustness natural image domains and several use-cases, while acknowledging that its immediate applicability to specialized fields remains bounded by the capabilities of the underlying generative components. In future work, we will explore automatic hypothesis refinement, for example by foeding back causal scores to the LLM to form an iterative loop. This would preserve GIFT's causal verification rigor scaling to even more complex vision models. Besides, leveraging 'reasoning' inference abilities of LLMs (Snell al. 2024) offers promising direction refine Stage 3.
Acknowledgments
This work was partially supported by the ANR MultiTraass project (ANR-21-CE23-0032)
References
Abubakar Abid, Mert Yülkscköönil, and James Zou. Mcaningfully debugging model mistakes using concep- tual counterfactual explanations. In IOML, 2022.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, et al. Pixtral 12b. arXiv preprint AXXi:2410.07070: 2024.
Maximilian Augustin, Valentyn Boreiko, Francesco Croce, and Matthias Hein. Diffusion visual counterfactual explanations. In NewrIPS, 2022.
Aziz Bacha and Thomas George. Training feature attribution for vision models. CoRR, abs/ 251/29500 2025.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Tn CVPR, 2017.
Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain CNN activations better than state-of-the-art feature visualization. In ICLR, 2021.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. In NewrIPS, 2019.
Jaemin Cho, Linjie Li, Zhengyuan Yang, Zhe Gan, Lijuan Wang, and Mohit Bansal. Diagnostic benchmark and iterative inpainting for layout-guided image generation. In CVPR Workshop, 2024.
Gabriela Csurka, Tyler L Hayes, Diane Larlus, and Riccardo Volpi. What could go wrong? discovering and describing failure modes in computer vision. In ECCV eXCV workshop, 2024.
Sanjoy Dasgupta, Nave Frost, and Michal Moshkovitz. Framework for evaluating faithfulness of local expla- nations. In ICML, 2022.
Moreno D'Incå, Elia Peruzzo, Massimiliano Mancini, Dejia Xu, Vidit Goel, Xingqian Xu, Zhangyang Wang, Humphrey Shi, and Nicu Sebe. Openbias: Open-set bias detection in text-to-in text-to-image generative models. CVPR, 2024.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Andrew Engel, Zhichao Wang, Natalie Frank, Ioana Dumitriu, Sutanay Choudhury, Anand D. Sarwate, and Tony Chiang. Faithful and efficient explanations for neural networks via neural tangent kernel surrogate models. In ICLR, 2024.
Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A. Efros. Blobgan: disentangled scene representations. In ECCV, 2022.
Spatially
Patrick Essor, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICMI, 2024.
Sabri Eyuboglu, Maya Varma, Khaled Kamal Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dummmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross-modal embeddings. In ICLR, 2022.
Thomas Fel, Rémi Cadène, Mathicu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look al the variance! efficient black-box explanations with sobol-based sensitivity analysis. In NeurIPS, 2021.
Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom Rousseau, Rémi Cadène, Lore Goetschalckx, Laurent Gardes, and Thomas Serre. Unlocking feature visualization for deep network with magnitude constrained optimization. In NewrIPS, 2023a.
Thomas Fel, Agustin Martin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Sorre. CRAFT: concept recursive activation factorization for explainability. CVPR, 2023b.
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In ICCV, 2017.
Timo Freiesleben. The intriguing relation between counterfactual explanations and adversarial examples.
Minds Mach., 32(1):77 109, 2022.
Nicholas Frosst and Geoffrey E. Hinton. Distilling a neural network into a soft decision tree. In Workshop on Comprehensibility and Explanation in Al and ML @AI*IA 2017, 2017.
Robert G Gallager. Information theory and reliable communication, volume 588. Springer, 1968.
Amirata Ghorbani, James Wexler, James Y. Zou, and Been Kim. Towards antomatic concept-based expla- nations. In NewrIPS, 2019.
Yash Goyal, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). CoRR, abs/1990.07713, 2019a.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lec. explanations. In ICML, 2019b.
Counterfactual visual
Zixin Guo, Tzu-Jui Julius Wang, and Jorma Laaksonen. CLIP4IDC: CLIP for image difference captioning. In AACL/IJCNTI 2022.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training, In CVPR, 2023.
Michael Harradon, Jeff Druce, and Brian E. Ruttenberg. Causal learning and explanation of deep neural networks via autocncoded activations. CoRR, abs/1802 abs/1802.00541, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In CVPR, 2016.
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In ECCV, 2016.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual explanations. In ECCV, 2018.
Mehrdad Hosseinzadeh and Yang Wang. Image change captioning by learning from an auxiliary task. In CVPR, 2021.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weiiberger. Densely connected convolu- tional networks. In CVPR, 2017.
Paul Jacob, Éloi Zablocki, Hédi Ben-Younes, Mickaöl Chen, Patrick Pérez, and Matthicu Cord. STEEX: steering counterfactual explanations with semantios. In ECCV, 2022.
Guillaume Jeanneret, Loic Simon, and Frédéric Jurie. Diffusion models for counterfactual explanations. In ACCV, 2022.
Guillaume Jeanneret, Loic Simon, and Frédéric Jurie. Adversarial counterfactual visual explanations. In CVPR, 2023.
Guillaume Jeanneret, Loie Simon, and Frédéric Jurie. Text-to-image models for counterfactual explanations: a black-box approach. In WACV, 2024.
Justin Johnson, Bharath Hariharan, Laurens der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dalaset for compositional language and elementary visual reasoning. In CVPR, 2017.
Vidhya Kamakshi, Uday Gupta, and Narayanan C. Krishnan. PACE: posthoc architecture-agnostic concept extractor for explaining cnns. In IJCNN. IEEE, 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In ICML, 2018.
Lukas Kuhn, Sari Sadiya, Jörg Schlötterer, Christin Seifert, and Gemma Roig. Efficient unsupervised shorteut learning detection and mitigation in transformers. In ICCV, 2025.
Ashish Kumar, Karan Sehgal, Prerna Garg, Vidhya Kamakshi, and Narayanan Chatapuram Krishnan. MACE: model agnostic concept extractor for explaining image classification networks. IEEE Trans. Artif. Intell., 266):574-583, 2021.
Jae Hee Lee, Sergio Lanza, and Stefan Wermter. From neural activations to concepts: A survey on explaining concepts in neural networks. CoRR, abs/2330.11884 2023.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, 2015.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NewrIPS, 2017.
Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Trena Gao, and Ranjay Krishna. CREPE: can vision-language foundation models reason compositionally? In CVPR, 2023.
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable AI. ACM Comput. Surv., (5(113):295:11295 -29::42, 2023.
Daniel Omeiza, Helena Webb, Marina Jirotka, and Lars Kunze. Explanations in aulonomous driving: A survey. IEEE Trans. Intell. Transp. Syst., 2022.
OpenAl. GPT-4 technical report. CoRR, abs/ 2300.300777744 2023.
Andrés Páez. Understanding with toy surrogate models in machine learning. Minds Mach., 34(4):45, 2024.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In ICCV, 2019.
Judea Pearl. Causality. Cambridge university press, 2009.
Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, and Judy Hoffman. LANCE: stress-testing visual models by generating language-guided counterfactual images. NeurIPS, 2023.
Yue Qiu, Yutaka Satoh, Ryota Suzuki, Kenji Iwata, and Hirokatsu Kataoka. 3d-aware scene change cap- tioning from multiview images. IEEE Robolics Aulom. Lelt., 5(3):4743 4750, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Arijit Ray, Filip Radenovie, Abhimanyu Dubey, Bryan A. Plummer, Ranjay Krishna, and Kate Saenko. Cola: A benchmark for compositional text-to-image retrieval. In Newr/PS, 2023.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 'why should I trust you?": Explaining the predictions of any classifier. In SIGKDD, 2016.
Pau Rodríguez, Massimo Caccia, Alexandre Lacoste, Lee Zamparo, Issam H. Laradji, Laurent Charlin, and David Vázquez. Beyond trivial counterfactual explanations with diverse valuable explanations. In ICCV, 2021.
Robin Rombach, Andreas Blattmann, Dominík Torenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use inter- pretable models instead. Nat. Mach. Intell., 1(5):206 215, 2019.
Fawaz Sammani, Tanmoy Mukherjee, and Nikos Deligiannis. NLX-GPT: A model for natural language explanations in vision and vision-language tasks. In CVPR, 2022.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In TCCV, 2017.
Tamar Rott Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob An- dreas, and Antonio Torralba. A multimodal automated interpretability agent. In ICML, 2024.
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. In ICLR, 2020.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/240s, 224080033144 2024.
Bartlomiej Sobieski and Przemyslaw Biecek. Global counterfactual directions. In ECCV, 2024.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In CVPR, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Jin Tian and Judea Pearl. Probabilities of causation: Bounds and identification. In UAI, 2000.
Erico Tjoa and Cuntai Guan. A survey explainable artificial intelligence (XAI): towards medical XAI. CoRR, abs/1907.07374, 2019.
Simon Vandenhende, Dhruv Kumar Mahajan, Filip Radenovié, and Deepti Ghadiyaram. Making heads tails: Towards semantically consistent visual counterfactuals. In ECCV, 2022.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counteraactull explanations without opening the black box: Automated decisions and the gdpr. Harvard Journal of Law & Technology, 2017.
Jörg Wagner, Jan Mathias Köhler, Tobias Gindele, Leon Hetzel, Jakob Thaddäus Wiedemer, and Sven Behnke. Interpretable and fino-grained visual explanations for convolutional neural networks. In CVPR, 2019.
Yinong Oliver Wang, Eileen Li, Jinqi Luo, Zhaoning Wang, and Fernando De la Torre. Unsupervised model diagnosis. arXiv preprint arXim:24 10. 06243, 2024a.
Yixin Wang and Michael 1. Jordan. Desiderata for representation learning: A causal perspective. CoRR, abs/2109.03795, 2021.
Yongjie Wang, Tong Zhang, Xu Guo, and Zhiqi Shen. Gradient based feature attribution in explainable Al: A technical review. CoRR, abs/2403, 10415, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thoughy prompting elicits reasoning in large language models. In NewrIPS, 2022.
Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Discovering bugs in vision models using off- aff-the-shelf image generation and captioning. CoRR, abs/2208.088811 2022.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In CVPR, 2024.
Yiran Xu, Xisoyin Yang, Lihang Gong, Hsuan-Chu Lin, TinY Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable abject-induced action decision for autonomous vehicles. In CVPR, 2020.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mci Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412.15115, 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A GPT-4V level MILM on your phone. CoRR, Abs/2240.001 2024.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
Éloi Zablocki, Hódi Ben- Youncs, Patrick Pérez, and Matthicu Cord. Explainability of doep vision-basod autonomous driving systems: Review and challenges. IJCV, 2022.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Mehdi Zemni, Mickaël Chen, Ėloi Zablocki, Hédi Ben-Younes Patrick Pérez, and Matthieu Cord. OCTET: object-aware counterfactual explanations. In CVPR, 2023.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In CVPR, 2018.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin P. Rubinstein. Invertible concept-based explanations for CNN models with non-negative concept activation vectors. In AAAT, 2021.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Yang Zhang, Yawei Li, Hannah Brown, Mina Rezaci, Bernd Bischl, Philip H. S. Torr, Ashkan Khakzar, and Kenji Kawaguchi. Attributionlab: Faithfulness of feature attribution under controllable environments. CoRR, abs/2310.065// 2023.
Jan Ruben Zilke, Eneldo Loza Mencia, and Frederik Janssen. Deepred rule extraction from deep neural networks. In DS, 2016.
18

---

Published in Transactions on Machine Learning Research (02/2026)
A Broader impact statement
By enhancing the understanding of deep learning models, particularly in vision classification, this work contributes to the responsible deployment of Al systems in safety-critical applications. While the potential societal consequences this work are generally aligned with the positive progression of machine learning, we acknowledge that enhanced interpretability could be used in ways that are not constructive, such identify and exploit weaknesses in deployed systems. However, we believe that the overall impact of this research will be to improve the fairness and trustworthiness of AT systems. We will release our code and data to promote transparency and collaboration in this field, which hope will mitigate any potential risks and will allow further research into the ethical considerations arising from model interpretation. We encourage other researchers lo use and extend the GIFT framework in ways that lead to more reliable and trustworthy applications of Al.
Further discussion on related works
Some prior work produces global and sometimes textual explanations. However, these approaches differ fundamentally from GIFT in both scope and capabilitics, beyond the faithful (causal) criterion. Tab. 5 summarizes key differences.
Classical concept-based methods such TCAV Kim et al. 2018 require manually predefined concepts. This is strong limitation because they can only evaluate what users anticipate. As result, these methods may miss unexpected model biases. The OCTET Zemni et al. (2023) user study shows this clearly (Sec. 4.3) no human anticipated the BDD bias even after inspecting input images and model predictions. In contrast, GIFT automatically discovers such unexpected factors without requiring any predefined concepts.
Unsupervised concept discovery methods such ACE Ghorbann et al. (2019) and CRAFT Fel 2023b) avoid predefined concepts but architecture-speeffic (CNN-only) and produce concept vectors that require manual interpretation through visualization (e.g., (Fel et al. 2023a prototype inspection. This makes them unsuitable for holistic, model-agnostic explainability.
GIFT, instead, directly outputs interpretable natural language explanations and applies to any differentiable classifier, providing causal verification of global explanations.
C Details on hypotheses verification (Stage 4)
C.1 Directed information (DI)
We provide detailed steps for computing Directed Information (DI), described in Sec. 3.4 It evaluates the correlation between concepts c, and class label y. DI is computed
1(ce;))
DI(ce, 9) H(c,))
where 1(ce;g) is the mutual information and H (ce) is the entropy. Below, we explain each term and outline how they are computed.
Mutual Information, I(c,;)) Mutual information quantifies the amount of information the presence of concept ce provides about the predicted class label It is defined as:
I(ce v) Ep(ce y) log p(ca: v) ( p(ce)p(v)
Ce.9
where:
p(Ces v) is the joint probability of the concept ea and class label y,
19

---

Published in Transactions on Machine Learning Research (02/2026)
Table 5: Comparison of global explainability methods. GIFT differs by producing faithful (causal) textual explanations without requiring predefined concepts, human interpretation, or architeeture-specifie compo- nents.
Method
Scope
Interpretability
Predefined Concepts
Interpretation Architecture Faithfulness
Needed
Specific
GIFT
Global
High (text)
No
No
No
Yes
(causality) No (correla- tions)
Classical
Global
High (text)
Yes (manually
No
No
concept-based
defined)
[e.g.. TCAV Kim
al.
2018
Unsupervised
Global
Low (vectors)
No
Yes (manual visualization)
Yes (CNN-
No (correla- tions)
concept-base
specific)
(e-g->
ACE Ghorbani
al.
2019
CRAF
[e et al.
2023b
p(ce) is the marginal probability of ce:
p(y) is the marginal probability of y.
We use the outputs of a Visual Question Answering (VQA) model to estimate p(ce, v). The VQA model predicts whether the concept ce is present in an input image x. For a dataset of images X, we calculate:
ple. v)
Σ 1[VQA(z,c)
1A)(2) = y),
where 1|-] is the indicator function, and |/| the cardinal of the set X. The marginal p(c)) and p(y) are derived from the joint probabilities:
p(ce) = Ep(cesw), p(y) = Ep(cesv).
Entropy of the Concept Presence, ((c.) Entropy measures the uncertainty the presence of concept Ce across the dataset. is defined as:
H(c,)= = (o(c.)lop(e))
We use the marginal probability p(c.) estimated above.
C.2 Probability of Necessary and Sufficient Cause (PNS)
We consider Y, a random variable that encodes the values of the label of a data point E X. Following Tian & Pearl (2000), Wang & Jordan (2021) define, for a given class value y and data representation Z, the probability the given representation to be a necessary cause (PNzy) and to be a sufficient cause (PSz.v) of Y y. We use a similar thcoretical framework in this study but use the notion of concept ce, being present (c. 1) absent. (Ce = 0) in the data point, instead of representation. The formula, for PN and its empirical estimate PN are given below:
PN P(M(x (l)((²) y. Xx,=1)
(5)
20

---

Published in Transactions on Machine Learning Research (02/2026)
Each one of the descriptions below (separated by '-) details the changes pair of images underwent part of counterfactual analysis for machine-learning binary classifier. The described changes caused the classifier to change its prediction either from class 0 to class or from class 1 to class 0. Your task is to consider ALL the descriplions and summarize the main factors leading the classifier to choose class 0 or 1. The counterfactual analysis is noisy, and the descriptions may contain irrelevant or contradictory information. Your task is to focus on the most important factors that appear consistently across many instances
Find testable factors that can be observed and measured in the images (e.g., objects' presence, appearance, arrangement, etc.)
Present your factors bulleted list
From class 0 to class 1:
left car - closer; more visible headlights
ego-car + dashboard + brighter
From class 0 to class 1:
traffic + leftmost lane clear, taxi is present
traffic + rightmost lane > was clear, now is occupied by cars
Figure 5: Prompt used for the LLM in Stage 3. This prompt is utilized across all experimental use cases. The ellipses ....' are placeholders, replaced with the concatenation all change captions gathered
stage 2. Note that the class labels are provided; instead, they are represented generically as or 1.
L[M( -e (b))))
xC
IF Σ 1[M(x) = ul 0
PNe..
PN Cery
Σ
[
(6)
DENce-1
otherwise.
Similarly, for PS and its empirical estimate PS
PS
P(M ylM(x) + V, Xo,=0)
(7)
Σ
I[M(a M())))
Σ
1lM(x) vl #
PS
1[(()
(8)
=
One can then define the probability PNS c..y of ce being both a necessary and a sufficient cause of Y = y, as the following weighted combination of PN and PS...y:
PNS = (x) = =UI Xe==1) PN + P(M(I) # y. E Xo-=o) PS
(9)
We approximate PNS using again empirical estimates, to obtain our metric
Eq. 4
Σ
([M(x) = yl
Σ
[M(()) + vl
PNS
zENo,
NE m
PS
X
X
(10)
1M AyA
Σ
1[M +0)==t)))))))) = (M + +11
X
ZEN
NENc,
21

---

Published in Transactions on Machine Learning Research (02/2026)
A first quality of PNS...v metric is that, contrarily to CaCEe,, it is not only specific to given concept but also to given class. Thus, it can be freely extended to settings beyond binary and exclusive classes, while still being interpretable.
Furthermore, PNSc..y can lower bounded using the difference of two intervention distributions Tian Pearl 2000 This bound, given below, can also be used to get further causal signal evidence.
PNS 2 (x) = y/do(x e xc,=1)) (z) = y/do(2 e Xc.=o)).
(11)
C.3 Relation between CaCE,, and PNS ce
Let us prove that for [0.1)-class scttings, we have CaCEa, = PNS 6. PNS
We use
Eq. 3 that
gives us CaCEa (Goyal et al., 2019a) as follows:
CaCEa.
Σ
M(x) - M(x + E
M(x+ee) -M(x))
(12)
X
NEXoc=1
Using the (0.1)-clas setting, we can rewrite CaCFa, 's equation using the indicator function (1[-):
CaCEa.
Σ
A - -1)
ZEN
= 0f 1IM = - 0 =1])
=
Σ
1[M(x)= = (.) 0 E
[[M(xto)) = 1A ((x))) -(]])
(13)
X
ZE
Σ
=1
1|+ 1[M(zto) = 0 M() =1]
X
xEX
PNS
PNS
Eq. 13 shows that CaCE,, PNNS (resp. CaCEa, PNS... if and only if PNS 0 (resp. PNS y=1 = 0), i... and only if the causal effect purcly unidirectional with ce causing = and never = (resp. = and never y = 1).
D LLM prompts (Stage 3).
Default LLM prompt for Stage 3. We report in Fig. the prompt that to summarize change captions and make global hypotheses emerge. The prompt is exemplified for the BDD experiment but is identical for CLEVR and CelebA experiments.
LLM prompt (Stage 3) for the ablation of Stage 2. In the baseline, described in Sec. 4.3 ( ('Ablation of Stage 2;), instead of feeding the LLM with change captions, provide simple captions for all images and counterfactuals. To generate the captions for each image, we use the recent Florence-2 (Xiao et al. 2024), prompted with the 'More detailed caption' lask mode. Then, we feed all the captions, along with the classification yielded by the target model M, to the LLM, with a prompt shown in Fig. 14] LLM fails to hypothesize that Class '1' is confounded by the presence of vehicles the left-lanc. This ablation highlights the importance of fine-grained pairwise comparisons between an image and its counterfactual explanation, provided by Stage? 2. They crucial for the effectiveness of our method.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Images classified in class "0" by the model M
Images classified in class "I" by the model M
Figure 6: Can you find the class meaning? We propose challenge to the reader. The classifier M has been trained to recognize specific visual rule, e-g. 'images in class 1 contain a yellow rubber object', with the experiments of Sec. 4.1 Can you guess what is the rule simply by observing the images and their classification given by the model M? This challenge illustrates the difficulty of the CLEVR data domain: CLEVR images hide complex compositional potential behind their minimalist appearance, with diverse colors, textures, and shapes. The answer is given in the footnote
LLM prompt (Stage 3) for the ablation of Stages 1 and 2. In the baseline, described in Sec. 4.3 ('A ('Ablation of Stages and 2'), we prompt the LLM of Stage 3 to imagine possible biases that the target classifier may have. There are thus no counterfactual explanations generation and the LLM does not depend on any information given by the classifier, except for the class meaning (=labels). We show the prompt and output Fig. 15. As there are no dependencics anymorc on the target classifier, the LLM can only imaginc generic biases and fails to find the unexpected left-lane bias. As LLM-generated hypotheses fail to raise the veft-lane-vehicle bias, it shows the importance of the counterfactual guidance (Stages and 2) in the GIFT framework.
E VLM prompts (Stage 2)
We report in Fig. 12 and Fig. 13 the prompts that we use to prompt Pixtral for image change captioning, for BDD and CelebA experiments respectively. We use a one-shot in-prompt annotated sample as guidance, where we manually describe the differences between an image (randomly selected) and the obtained counter- factual explanation after Stage 1. We qualitatively find that this guides the VL.M to produce more accurate change captions.
In the case of CLEVR, we use CLIP4IDC which is trained for the image change captioning task on the CLEVR domain, with the annotated CLEVVR-Change (Park et al. 2019) dataset.
F
Experimental Details on CLEVR
F.1 CLEVR: a challenging compositional domain
Fig. 6 presents a challenge to the reader to uncover the underlying visual rule Icarned by the classifier M. In this example, M has been trained on a specific rule, such as 'images in class contain a yellow rubber object' or 'images in class 0 contain a red object', similar to the setup described in Secc. 4.1 By observing the images and the model's prodictions, the reader is invited to deduce the rule. This task highlights the inherent complexity of the CLEVR dataset, where simple appearances mask intricate combinations of colors, textures, and shapes. The solution to the challenge is provided in the footnote at the end of the figure caption.
23

---

Published in Transactions on Machine Learning Research (02/2026)
F.2 CLEVR Classifiers
We use CLEVR-Hans Stammer et al. (2021) to generate six binary classification datasets with BLENDER. The images are then resized and center-cropped to 128x128 to match the resolution of the BlobGAN gen- eralive model. The training and validation sets contain 3,000 and 300 samples, respectively, with balanced labels. We train 12 classifiers (listed in Tab. 2 with either ViT or ResNet34 backbones these datasets, selecting the checkpoint with the best validation accuracy from the first three epochs. For all these classifiers, we report near-perfect validation accuracies, ranging from 96.3% to 99.3%.
F.3 GIFT Instantiation
Stage 1. To instantiate GIFT on the CLEVR domain, first require method for counterfactual ex- planation (Stage 1). For this, we employ OCTET (Zemni et al. 2023 which produces 'object-aware' explanations and is well-suited for the object-centric nature of CLEVR. However, since the official OCTET implementation is not compatible with CLEVR, we reimplemented it for this domain. A key component of this adaptation is BlobGAN (Epstein et al. 2022), a compositional generative model capable of editing, inserting, and removing objects in diffcrentiable manner. We trained BlobGAN on the original CLEVR dataset. The images were resized and center-cropped to 128x 128 resolution. We used K 15 blobs, slightly exceeding the maximum number of objects (10) in CLEVR scene. The BlobGAN training was conducted for 181 epochs. Notably, we did not need to retrain image encoder. Instead, directly sample within BlobGAN's latent space to generate the original query image. This approach not only produced satisfy- ing results but also circumvented the need for computationally intensive image reconstructions, which are typically required in the first stop of OCTET optimization.
Stage 2. For the image change captioning (Stage 2), we use CLIP4IDC (Goo et al. 2022 which is specifically designed for the CLEVR dataset. This model is trained on BLEVR-Chaage Park et al. 2019), a dataset that inchudes image pairs and textual change captions describing the differences between the images. We note that since the CLEVR-Change dataset does include shape or size changes (e.g., a sphere turning into a cylinder, or a small sphere now a big sphere), CLIP4IDC unable to describe such transformations. As a result, we only consider rules that do not rely on shape or size, as counterfactual explanations produced with OCTET would inchude such changes that CLIP cannot translate into natural language. In contrast, for the other use-cases with CelebA and BDD-OIA, we do not face this limitation as we use generalist VLMs. However, these models are somewhat noisier since they were not specifically trained for this task.
Stage 4. In Stage 4, hypothesis verification was conducted using 200 images: 100 classified as 1 by the model and 100 classified as 0. Importantly, we never accessed the ground-truth labels. For concept verifica- tion in Stage 4 and the DI computation, we utilized the MiniCPM (Yao et al. 2024) model. Interventions in Stage 4 involved object additions and removals:
Object addition: We identified empty regions in the image by checking if the mean pixel value and standard deviation within a selected area were close to the background color and zero, respectively, This produced mask that was passed to Stable Diffusion (Esser al.) 2024) alongside the descrip- tion of the object to add, e.g., 'a cyan object?. The model then placed the object in the specified region.
Object removal: We identified the bounding boxes of the objects to remove using Florence-2 Xiao
et. al.]
2024) in 'CAPTION TO PHRASE GROUNDING' mode. Subsequently, Stable Diffusion
Esser et al. 2024) was used to inpaint these regions with the word 'background', effectively removing
the objects.
Coutaby a cran
of Llss S sonanee
metallic object,
Auswsr soneem,
24

---

Published in Transactions on Machine Learning Research (02/2026)
The target model M is a VIT that outputs 1 the image contains cyan object
The target model M ResNet that outputs if the image contains yellow rubber object
Target Model M
Target Model M
Target Model M 0
Target Model M
CEX: Counterfactual
CEX: Counterfactual
explanation generation
explanation
generation
Query image x
Counterfact. explanation
Query image
Counterfctt explanation
CC: Image Change Captioning
2
CC: Image Change Captioning
the tiny blue matte block that in front of the tiny yellow metal object became cyan
the tiny yellow matte ball behind the red matte thing has been newly placed
Target Model . +
Target Model M
Target Model M 0
Target Model M
CEX: Counterfactual
CEX: Counterfactual
explanation generation
explanation
generation
Query image it
Counteffact. explanation
Query Image
Counteract: explanation
CO: Image Change Captioning
CO: Image Change Captioning
the tiny cyan matte sphere that is right of the tiny brown rubber cylinder is no longer there
the tiny brown matte ball that is behind tiny yellow metal thing became yellow
Target Model M + 1
Target Model M 0
Target Model M 1
Target Model M 0
CEX: Counterfactual
CIE:: Counterfactual
explanation generation
explanation generation
Query mage it
Counterfact. explonation
Query image it
Counterfact explanation
CO: Image Change Captioning
CO: Image Change Captioning
the tiny cyan matte ball that is right of the big purple matte object is missing
the tiny yellow matte ball that is in front of the tiny green matte thing became metallic
Figure Visualization of the Stages and 2 for classifiers trained CLEVR data. CEX generates counterfactual explanation and CC describes differences in plain text. Examples the left block are obtained for a model M that outputs '1' if the image contains a cyan object. Examples on the right block are oblained for model M that outputs '1' is the image contains a yellow rubber object. The counterfactual explanations exhibit notable diversity, encompassing changes in object colors, appearances, disappearances, and textures. Furthermore, the CLIP4IDC model, specifically trained for this task, excels at translating visual changes into natural language descriptions. Its outputs are consistently accurate, with minimal noise and rare errors.
25

---

Published in Transactions on Machine Learning Research (02/2026)
Table 6: Complete table of experiments run in Sec. 4.1, The explanations that obtained the highest scores for each classifier and metric are shown in bold. In all cases except one, the rule used train the target classifier was proposed by the LLM after Stage The rule always corresponds to the hypothesis with
the highest CaCE and PNS measurement. The use of italics for 'red metal object' indicates that the rule was manually created by recombining the first rules found in Stage
Output of Stage 3
Output of Stage
Rule (Ground Truth)
Architecture
Explanation:
'Class contains a...
DI (%)
CaCE (%)
PNS (%)
cyan object
96.8
62.2
62.2
cyan object
ResNet34
cyan metal object
26.0
50.9
50.9
cyan rubber object
31.2
47.6
47.6
purple object
ResNet33
purple object red object metal object
92.9
71.2
71.2
0.2
8.6
9.1
metal object
ResNet34
88.9
24.2
24.2
rubber object
ResNet34
rubber object
66.5
29.8
29.8
yellow rubber object
44.3
67.2
67.7
yellow rubber object
ResNet34
small yellow rubber sphere
14.1
57.6
57.6
ycllow object
31.0
38.4
39.9
red object
46.8
42.9
43.9
red metal object
ResNet34
metal object
3.6
11.6
13.6
red metal object
65.5
61.6
61.6
cyan object cyan sphere cyan cube
92.9
63.1
63.1
cyan object
ViT
24.4
50.3
50.5
28.0
42.4
43.4
purple sphere purple cylinder
17.2
52.5
52.5
purple object
VIT
29.6
53.0
53.0
purple object
92.9
70.2
70.2
metal object rubber object
83.4
37.4
37.4
16.3
5.6
6.6
red metal object
14.7
36.4
36.9
metal object
ViT
ycllow metal object
14.7
3488
35.4
green metal object
10.8
29.3
29.3
rubber object blue object cyan object
62.0
30.8
31.3
rubber object
ViT
0.0
8.6
9.6
0.6
7.6
8.1
yellow rubber object
51.2
71.7
71.7
yellow rubber object
ViT
yellow object
42.4
49.0
50.0
metal object red metal object
0.3
12.6
14.1
63.2
70.7
70.7
red object rubber object
40.1
43.4
43.9
red metal object
VIT
1.3
8.6
10.6
cube
0.2
11.1
13.6
sphere
0.0
9.6
11.1
F.4 Results
Qualitative samples after Stages and 2. 2, highlighting the outputs of the counterfactual generator (CEX) and the change captioner (CC). The
Fig. provides qualitative examples from Stages and
26

---

Published in Transactions on Machine Learning Research (02/2026)
counterfactual explanation optimization depends three inputs: the original image, the model decision to be flipped, and the model itself (illustrated with blue boxes and arrows). In contrast, the image change captioning stage relies solely on the pair of imput images (indicated by the green box). The counterfactual explanations exhibit notable diversity, encompassing changes in object colors, appearances, disappearances, and lext ures. Furthermore, the CLIP4IDC model, specifically trained for this lask, excels al translating visual changes into natural language descriptions. Its outputs are consistently accurate, with minimal noise and rare errors.
Complete results. Tab. summarizes the results of Stages and 4 across the classifiers tested on the CLEVR. dataset. Notably, in all but one case, the rule that governed the target classifier was correctly proposed by the large language model (T.M) after Stage 3. Stage 3 outputs reveal that the LLM generates small set of hypotheses (between and 5). In certain instances, only a single hypothesis is produced, which happens to be correct. Still, achieving high recall for hypotheses at this stage is crucial, as errors can be addressed during Stage 4. Importantly, the rule always corresponds to the hypothesis with the highest CaCE measurement, ensuring a robust and systematic selection process.
presents the results of Stage 3 for the experiment in which we ablate the change captioning component (Stage 2). In this experiment, the change captions are replacod with independently acquired caplions for the images and their counterfactuals, using a prompt provided in Fng. 14, This modification significantly reduces performance: the rule is correctly identified for only 4 out of the classifiers tested, as shown in [Tb. 2 These results highlight the critical importance of fine-grained pairwise comparisons between an image and its counterfactual explanation. Such comparisons, facilitated by change captioning, are essential for the success of our method.
Ablation of Stage 2. Tab. 7
Stage 3 adaptability to the captions' information. In our main experiment, we used large number of counterfactuals, approximately N 100 image pairs, reported in the paper. This was our initial attempt, designed to ensure fairness without tuning the number of pairs. In the ablation study, we find that as few as three pairs (N = =3,, and occasionally even two, are sulficient for the LLM to hypothesize the correct rule. This is because the change captioning stage (Stage 2) is minimally noisy, as it uses CLIP4IDC, which is specifically trained for the image change captioning task. Additionally, the simplicity the rule further reduces the need for a large number of counterfactual explanations.
Stage 4 importance for selecting the best rules. Stage 4 is critical to select and rank the hypotheses after Stage 3. As mentioned Sec. 4.1 each trial on CLEVR raised 2 to viable explanations, which Stage 4 then correctly ranked 11 out of 12 times (Tb.. 2). In Sec. 4.2) we find dozens of candidate concepts, which Stage 4 reduces to handful of promising ones Tab. 3).
G
Experimental Details on CelebA
G.1 GIFT Instantiation
The target model M is a DenseNet121 (Huang et al. 2017). trained to classify face images as either 'old' or 'young,' following prior work (Jacob et al. 2022 Jeanneret et al. 2023
For counterfactual generation (Stage 1), we leverage the diffusion-basod model ACE (Jeanneret et al, 2023
using its official implementation.
In the image change captioning stage (Stage 2), we employ Pixtral-12B VLM (Agrawal et al. 2024h. The prompt used for this stage is shown in Fig. 13 To guide the VLM, we include onc-shot annotated sample within the prompt, where we manually describe the differences between a randomly selected image and its corresponding counterfactual explanation from Stage 1. This guidance qualitatively improves the accuracy of the generated change captions.
Global explanation identification (Stage 3) is performed zero-shot using Qwen 2.5, 72B parameter model, quantized to 8 bits (Yang et al. 2024). The prompt used for this stage is provided in Fig.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Table 7: Ablation of Stage 2: Complete Output. We replace the change captions with simple indepen- dently acquired captions for images and counterfactuals, using the prompt shown Fig. 14 The column "Correct hypothesis' indicates whether the hypothesis corresponds to the rule used train the classifter, The rule is found for only 4 out of the 12 classifiers tested. This ablation highlights the importance in method of fine- fine-grained pairwise comparisons between image and its counterfactual explanation
Rule
Architecture
Explanation: 'Class 1 contains a...' Correct hypothesis
yellow rubber object
VIT
metal object metal object
metal object metal object
VIT
green object
metal object
ResNet34
red object and yellow object sphere and eube
blue sphere
cyan object
ResNet34
yellow or gray object
gray sphere yellow object
eyan object
VIT
purple object
VIT
purple object
purple object purple sphere
purple object
ResNet34
green object and purple object
red object
red metal object
ResNet34
yellow object
red object and yellow object
red object
red metal object
VIT
metal object
red object and metal object
yellow object green object
rubber object
ResNet34
metal object
blue object
purple object
rubber object
VIT
yellow object and purple object
red object and yellow object
For hypothesis verification and interventions in Stage 4, 200 images: 100 classified 'young' and 100 classified as 'old' by the target classifier M.
The VQA model for this stage MiniCPM Yao al. 2024
Global interventions: For concepts that require global adjustments, such as increasing lighting, we use image-conditioned Stable Diffusion 3 Esser et al. 2024
Local interventions: For concepts requiring localized changes, we first identify the region to be edited using Florence-2 (Xiao al. 2024) for open vocabulary detection. This provides the necessary masks, which are then passed Stable Diffusion 2 (Rombach et al. (022),, along with prompt stating to add remove the concept, resulting in the post-intervention image.
28

---

Published in Transactions on Machine Learning Research (02/2026)
M
Young
M
Young
M
Old
M
Young
M
Old
e
M
Old
M
Old
M
Young
M
Old
M
Young
1
e
2
forehead lines appeated eyebrows lifled and more arched eyes > more open
eyes lighter; less naticeable
lighting natural
forehead wrinkies
eyebrows darker eyelids droopier; lower periorbital edema appeared
hair more voluminous;
became more pronounced
less pronounced; thinner cheeks
lighter highlights forehead smoother;
arched
eyes lired appearance; droopier eyellds lower eyelld became droopier
checks less full; lines appeared nasolablal folds less
eyelids > pronounced
elevaled;
wrinkles less visible
cheeks >> sunken
lines appeared
eysbrows - more defined;
nasolabial folds became
nasolabial folds became
darker eyes more alert; brighter
deep
apperent mouth -> fuller
apperent
cheeks sunken; more pronounced
lips fuller: redder ears more visible ears
lips >> darker, thinner
eyelids -> less droopy
lips -> plumper, more prominent
hair > lighter. less blonde background >> different pose straight-on angled
cheeks higher; less
nasolablal folds more pronounced slightly more closed lower lip chin > more pronounced jowi; sagged
stuck out slightly
sagging chin defined:
hair more white
jawline more
pose facing slightly
hair > more voluminous lighting brighter
wrinkles less visible
facing straight
slighlly more ferward
camera angle slightly below >> slighlly above lighting brighter bright: shadows more pronounced
background > different slightly
head more to the side more
hairline defined hair parting > moved to the
facial cheerful and confident
skin - smoother; less wrinkles
age appeaars younger
Figure Visualization of Stages and 2 for the CelebA 'Young/Old Classifier. The top row shows the original images. The second row displays the corresponding counterfactual explanations generated in Stage using the diffusion-based ACE method (Jeanneret et al. 2023). The third row presents the change captions produced in Stage 2 by the Pixtral model, which describe diflerences between the original images and their counterfactual explanations.
29

---

Published in Transactions on Machine Learning Research (02/2026)
No change
Light change
Modate change
Query image
Edited image x +c
Query image
Edited image x
Query image
Edited image x
DF
NAT
WAT
ITION
ATION
Query image æ
Edited image x -
Query image æ
Edited image x c
Query image
Edited image æ -c
Examples of adding (top) or removing (bottom) neck wrinkles, illustrating the three manual categories: no change, light change, and moderate change.
Figure 9:
G.2 Results
Qualitative samples after Stages and Fig. provides qualitative examples from Stages 1 and 2 in the analysis of the CelebA 'Young/Olc" classifier. The counterfactual explanations generated by ACE (Jeanneret et al. 2023) highlight subtle changes, such adding facial wrinkles smoothing the skin. In Stage the Pixtral model produces detailed change captions that describe point-by-point differences between the original images and their corresponding counterfactual explanations. While most of the caption content is accurate and meaningful, we observe occasional noise and hallucinations from the VLM, where it describes changes that are not present in the images.
Noise from Stage 2. We observe noticeably higher noise levels by using the zero-shot Pixtral in Stage 2, rather than the domain specific CLIP4IDC in the CLEVR experiments. To quantify this, conducted small diagnostic study five randomly selected counterfactual pairs for the Young/Ola classifier. Across these samples, the VLM produced 18 incorrect or hallucinated change descriptions out of 55 total described changes (e.g., spurious changes in pose, background, or hair volume), corresponding to an estimated hallu- cination rate of approximately 33%.
Image editing model assumptions. Our causal verification relies the assumption that interventions introduce or remove only the target concept while leaving other attributes unchanged. This assumption is standard in causal-intervention work (e.g., Goyal et al. (2019a)) ). To make it hold as well as possible in practice, we design the editing procedure to be highly localized: we use spatial masks together with mask-guided text-based editing, which restricts the modification to a specific region and largely prevents unintended global changes.
To evaluate the quality of these interventions, we manually inspected edits on CelebA for three fine-grained attributes. For each concept, and for both adding and removing operations, reviewed 50 edited images and assigned outcomes to one of three categories: no change, light change, and moderate change. Fig. 9 and Fig. 10 illustrate typical examples from each category for the 'neck wrinkles' and 'forehead wrinkles' attributes respectively. Tab. shows the results of this analysis.
Overall, odits remain well targetod: most interventions produce either no side effeets or only light ones, particularly when removing concepts. When secondary changes appear, they are not systematic, e.g., adding concept may occasionally co-occur with unrelated attributes, but these side effects differ across samples
30

---

Published in Transactions on Machine Learning Research (02/2026)
No change
Light change
Modate change
Query image
Edited image x +c
Query image
Edited image x
Query image
Edited image x
Query image
Edited image
Query image
Edited image
Query image
Edited image
Figure 10: Examples of adding (top) or removing (bottom) forehead wrinkles, covering the three quality calegories.
Table 8: Intervention quality on CclebA. Porcentage of inspected samples (50 per sctting) falling into each calegory when adding or removing a concept.
Adding concept
Removing concept
Concept
No change
Light
Moderate
No change
Light
Moderate
Receding hairline
54
30
16
82
18
Neck wrinkles
52
24
24
52
26
22
Forchead wrinkles
48
30
22
68
22
10
and do not correlate with the manipulated concept. This lack consistency prevents them from influencing downstream causal inference.
Complete results. Tab. 9 lists all the explanations associated with the class 'Old'' as generated by GIFT, along with the corresponding DI and CaCE measurements. The identified attributes include plausible ones, such as 'wrinkles on the forehead' and 'receding hairline,' as well as more unexpected ones, such as 'detailed background' 'low camera angle.'
Experimental Details on BDD-OIA bias discovery
H.1 GIFT Instantiation
The instantiation of GIFT for use-case 3, with the bias discovery, follows the same process as in the CelebA usc-case, with the exception of counterfactual explanation gencration. For this, we use OCTET (Zemni et al. 2023), which comes with the BlobGAN (Epstein et al. 2022) generative model for the BDD image domain. The biased 'Turn right' classifier used in OCTET serves as the target classifier for this use case.
H.2 Results
Qualitative samples after Stages 1 and 2. Fig. provides qualitative examples from Stages and the analysis of the BDD-OIA 'Can/Cannot turn right' classifier. The counterfactual explanations generated OCTET (Zemni et. al. 2023) highlight sparse semantic changes, despite significant, pixel value changes.
31

---

Published in Transactions on Machine Learning Research (02/2026)
M
Can turn right
M
Cannot turn right
M
Can turn right
Query image œ.
Query image
Query image ær
M
Cannot right
M
Can turn right
M
Cannot turn right
Counterfactual image x'
Counterfactual image x''
Counterfactual image xr'
slighly darker
leflmost
disappeared
road >> middle lane
parked
stopped
previcusly
appeers slightly
cars laftmost parked appeared cars rightmest parked appeared buildigs > leftmosl > siihtly doser building fightmosl > slighlyy doser streetights laftmost
street de has muliple stationary vehicles on the left side, previouely had fewer previcusly moving towards the right side, side, previously
road leftmoet sidewalk de people disappeared
traffic signal i leftmost > turned green
street io
ego-car >> slighly shifted right
visibl;; streotights rightmeat more
street more
closer
fewer
sky -h color less bright: sighily more doudy ago-car -a deshboard -> brighter
streat road surface appears darkar, more reflective streetights appear brighter, more numerous buildings
visible
ego-car
Visualization of the Stages and 2 for the biased turn-right classifier trained on BDD-OIA data. The top row show original images. The second row show corresponding counterfact ual explanations, obtained after Stage 1 with OCTET (Zemni al. 2023).. The third row shows change captions that we obtain after Stage 2, with the Pixtral change captioning.
Figure 11:
32

---

Published in Transactions on Machine Learning Research (02/2026)
Table 9: Complete output of GIFT on the CelebA 'Old' classifier. Evaluation with causal metrics when DI 2 15%.
Concepts associated to the class 'Old'
DI (%)
CaCE (%)
PNS (%)
after Slage
Receding Hairline
29.9
2.0
5.0
Neck Wrinkles
28.1
3.0
3.5
Wrinkles on Forehead
26.4
6.5
7.0
Wrinkles around Eyes
20.7
6.5
7.0
Glasses
16.5
11.5
16.0
Drooping Eyelids
15.4
2.5
4.0
Long and Voluminous Hair
14.9
Tired-looking Eyes
13.4
Gray Hair
12.3
Wrinkles on Cheeks
11.9
Small Eyes
3.6
Thick Eyebrow
0.0
Eyebrow Shape
2.1
Prominent Checkbones
0.7
Visible Nasolabial Folds
0.6
Prominent Jawline
0.6
Pale Skin
0.8
Scrions Exprcssion
0.1
Dark Hair Color
0.0
Soft Lighting
0.2
Tilted Head
0.3
Prominent Ears
0.1
Dark Facial Hair
2.8
Pronounced Makeup
8.0
Thin and Downturned Lips
5.0
Detailed Background
0.2
Low Camera Angle
0.1
after manual combinations of concepts with PNS > 5% Sec. 3.5
Hairline Wrinkles Forchcad
47.8
8.2
8.2
Glasses Wrinkles around Eyes
60.4
21.7
23.6
Glasses Hairline
61.4
24.6
25.4
Glasses + Hairline + Wrinkles on Forehead
78.2
26.0
27.1
Glasses Wrinkles on Forehead
59.6
24.1
27.6
Glasses Wrinkles on Forehead + Eyes
65.6
26.8
28.9
This is due the object-aware generative backbone, which manipulates disentangled representations of objects. For example, blocking lanes that were previously free. In Stage 2, the VLM produces detailed change captions that describe the point-by-point differences between the original images and their corresponding counterfactual explanations. While most of the caption content is accurate, we observe occasional noise and hallucinations from the VLM, where it describes changes that are not actually present in the images.
may appear disappear (left and middle images) and shift their position,
Ablation of Stage 2. In the baseline, described in Sec. 4.3 ('Ablation of Stage 2'), instead of feeding the LLM with change captions, we provide simple captions for all images and counterfactuals. To generate the captions for cach image, we use the recent Florenco-2 (Xiao et al.] (2024), prompted with the 'More detailed caption' task mode. Then, we feed all the captions, along with the classification yielded by the target model M, to the LLM, with a prompt shown in Fig. 14 LLM fails to hypothesize that Class 'I' is confounded
33

---

Published in Transactions on Machine Learning Research (02/2026)
You are an expert in street/road traffic analysis and driving- -cenne analysis.

 List how Image-2 differs from Image-1:
Consider how each element of the image (objects, people, animals, etc.) has changed, considering their
sizes, materials, colors, textures, poses, positions, and relative positions
Consider elements that might have appeared or disappeared in the transition to Image-2 from Image-1 Consider whether the image or scene as a whole has changed, and if that's the case, explain how
Omit commonalities; focus only on the differences
BE RIGOROUS: DO include comparisons that are clearly visible both images; DO NOT guess or infer
details
Present the changes plain-text list, one change per line, in the format <element> + <specifies>> +]
<change> 

Image-1: IMAGE TOKEN
Imago-2:
CHANGES:
road + left lane was clear, now blocked by car in the same direction
road - right lane - had an incoming car, now has car in the same direction road + middle lane - surface more reflective
road + pedestrian crossing + less visible; more faded traffic lights leftmost > appeared; green light middle car brighter; color more saturated rightmost streetlight disappeared ego-car dashboard brighter
Image-1: 1: IMAGE TOKEN
Image-2: 2: IMAGE TOKEN
CHANGES:
Figure 12: Prompts used for stage 2 for BDD. We prompt the Pixtral VLM for the image change captioning stage. The special tokens 'MAGE TOKEN' TOKEN are replaced by the tokenized images. We use a one-shot in-prompt annotated sample as guidance, where we manually describe the differences between an image (randomly sclected) and the obtained counterfactual explanation after Stage 1.
by the presence of vehicles in the left lane. This ablation highlights the importance of fine-grained pairwise comparisons between image and its counterfactual explanation, provided by Stage 2, are crucial for the effectiveness of our method.
Noise from Stage 2. As we rely on a zcro-shot VLM (Pixtral) for chango-captioning in Stage 2, we observe substantially higher noise levels compared to CLEVR, where a domain-specific captioning model was available. To quantify this, we ran a small diagnostic study on five randomly selected counterfactual pairs for the biased classifier. Across these samples, manually counted 14 incorrect change descriptions out of the 35 total described changes (c.g., spurious mentions of changes in streetlight visibility, traffic light color, or added rremoved pedestrians), yielding an estimated hallucination rate of approximately 40%.
In the baseline, described in Sec. 4.3 ('Ablation of Stages 1 and 2'), we prompt the LLM of Stage imagine possible biases that the target classifier may suffer. There are thus no counterfactual explanations generation and the LLM does not depend on any information given by the classifier, except for the class meaning (=labels). We show the prompt and output in Fig. 15 As there
Ablation of Stages and 2.
34

---

Published in Transactions on Machine Learning Research (02/2026)
You expert in image analysis and forensic facial comparison.
[LNST] List how Image-2 differs from Image-1:
Consider how cach facial feature has changed, including the eyes, nose, mouth, cars, hair, etc
Consider whether the facial expression has changed and how this has affected the facial appearance
Consider whether the pose and camera angle have changed
Consider whether the lighting, background, and other environmental factors have changed
Omit commonalitics; focus only on the differences
BE RIGOROUS: DO include comparisons that clearly visible in both images; DO NOT guess infer
details
Present the changes plain-text list, one change line, in the format <element> - <<specific>> +] <change> 

Image-1: 1: IMAGE TOKEN
Image-2: IMAGE 1
CHANGES:
forehead + lines appeared
eyes + darker; less shiny
eyelids + lower + periorbital edema appeared
cheeks - more prominent; lines appeared
nose + slightly wider
nasolabial folds > became apparent; very marked lips thinner; darker; less red
Image-1: 1: IMAGE TOKEN
Imago-2: MAGE_TOKES
CHANGES:
Figure 13: Prompts used for stage 2 for CelebA. We prompt the Pixtral VLM for the image change captioning stage. The special tokens one-shot in-prompt annolated sample as guidance, where we manually describe the differences between an image (randomly selected) and the obtained counterfactual explanation after Stage 1.
are replaced by the tokenized images. We use a
no dependencies anymore on the target classifier, the LLM can only imagine generic biases and fails to find the unexpected left-lanc bias. As LLM-genorated hypotheses fail raise the left-lane-vehicle bias, shows the importance of the counterfactual guidance (Stages 1 and 2) in the GIFT framework
Stage 3 adaptability to the captions' information. Our main experiment used large number of counterfactuals, approximately 150 image pairs, reported in the paper. This was initial attempt, without tweaking this number, to ensure fairness. We then conducted an ablation study to determine the minimum number of pairs needed to detect the bias. The results showed that at least 20 pairs are required to reliably identify the bias. The key insights are: using a large number pairs works well, as the LLM is able reason effectively and filter out noise. We hypothesize that a smaller LLM might struggle in this scenario. Additionally, unlike CLEVR, where only three pairs were sufficient, the noisier change captions this case require more pairs to allow the LLM to distinguish the true signal from the noise.
H.2.1 Stage 4 qualitative samples
16 shows image intervention for the Dense Traffic in the Left Lane' explanation.
35

---

Published in Transactions on Machine Learning Research (02/2026)
Computational footprint.
In our runtime breakdown, on an A100, for either CclebA or BDD classifier, Stage (OCTET) takes 4.28/imang /s/iaage (batch of 16), Stage 2 (Pixtral) takes 3.5/ 5//maage and Stage 3 takes 20s (total). Stage takes 98/ /image (batch of 32).
36

---

Published in Transactions on Machine Learning Research (02/2026)
Each one of the descriptions below (separated by "-)) details the content of an image. The descriptions are grouped by the prediction of binary classifier their corresponding images: images classified O and images classified as 1. Your lask is to consider ALL the descriptions and identify the main factors that could lead the classifier to choose class 0 over class 1, and vice-versa.
The set description is noisy, and the descriptions may contain irrelevant or contradictory information. Your task is to focus on the most important factors that appear consistently across many instances -Find testable factors that can be observed and measured in the images (e.g., objects' presence, appearance, arrangement, etc.)
-Present your factors bulleted list
Description of images classified as O:
The image is of a highway at night. The sky dark and the street lights are on. There are two yellow lines painted on the road. On both sides of the road there are rows of parked cars. The cars are all black in color. In the front of the image there windshield of car. The headlights of the are on and they are all turned on.
Cars are driving on a street at night. The street has white lines on it. There are buildings on both sides of the street. The sky is dark and there are street lights on.
Description of images classified as 1:
A car is driving down the street. There are several cars parked the side of the street next to tall buildings. There is red car in the middle of the road.
This is an image of highway at night. The sky is completely black. The street lights are on There are two yellow lines painted on the road. There a red car in the middle of the street. The headlights of the car are on and it dark outside. The car is driving on the right side of the road and there are several other cars on the left side.
A car is driving down the street at night. There are while lines painted on the street. There is green light in the distance on the road.
[A nswerr
Street Lines and Markings: More detailed and frequent in class 0.
Parked Cars: More common in class O.
Building Appearance: More detailed and diverse in class 1.
Lighting and Reflections: Brighter and more reflective in class 1.
Time of Day: More night time in class more varied in class 1.
Traffic Density: Higher in class 1.
Perspective: More inside car in class 0, more varied in class
Ablation of Stage 2: Prompt and LLM Response in Stage 3. The top section shows the prompt used to query the LLM, containing Florence-2-generatee captions for each image and its coun- terfactual, grouped by the target model's predictions. Class '0' denotes Cannot turn right, and Class '1' denotes Can turn right. The bottom shows the LLM'' response, which notably fails identify that Class '1' is confounded by vehicles in the left lane. This ablation underscores the importance of fine-grained, pairwise comparisons for our method's effectiveness.
Figure 14:
37

---

Published in Transactions on Machine Learning Research (02/2026)
[Prompt]:
You are an expert in analyzing and explaining the behavior of machine learning classifiers. Consider binary classifier designed to predict whether vehicle in driving scene turn right. This classifier is potentially imperfect. What possible biases or spurious correlations might this classifier have regarding the decision to turn right?
[Answer by Qwen 2.5]
Time of Day: The classifier might learn that certain times of day (e-g., rush hour) more likely to have restrictions turning right due to increased traffic.
Day of the Week: Similar to time of day, the classifier might pick up on patterns where certain days (e.g..
weekends vs. weekdays) have different traffic conditions that affect the ability to turn right.
Urban vs. Rural Areas: The classifier might learn that turning right is more common or less restricted in rural areas compared to urban areas due to differences in traffic density and infrastructure.
Specific Locations: The classifier might overfit specific intersections or regions where turning right is either very common very rare, leading to poor generalization to new locations.
Weather Patterns: The classifier might learn that certain weather conditions (e.g., rain, snow) make it less likely for vehicles to turn right due to reduced visibility or road conditions.
Type of Vehicle: The classifier might pick up correlations between the type of vehicle (e.g., trucks,
cars, motorcycles) and the likelihood of turning right. For example, larger vehicles might be less likely to turn right in tight spaces.
Traffic Light States: The classifier might learn that certain states of traffic lights (e.g., red light) are associated with not being able to turn right.
Signage: The presence or absence of specific traffic signs (e.g., "No Right Turn' signs) might be overem- phasized by the classifier.
Driver Demographics: If the dataset includes information about the driver, the classifier might pick up on spurious correlations related to the age, gender, or experience level of the driver.
Driver Intent: The classifier might infer the driver's intent to turn right based on subtle cues that are not directly related to the feasibility of the turn (e.g., the position of the turn signal).
Dataset Imbalance: If the dataset contains a disproportionate number of examples where turning right is allowed or prohibited, the classifier might be biased towards the more frequent class.
Sensor Quality: Differences in the quality or type of sensors used to collect data (e.g., cameras, LIDAR) might introduce biases if the classifier learns to rely artifacts specific to certain sensor types.
Overfitting to Training Data: The classifier might overfit to the specific characteristics of the training data, leading poor performance on unseon data. This can manifest as spurious correlations that do not generalize well.
Pedestrian Presence: The classifier might learn that the presence of pedestrians near the intersection makes it less likely to turn right.
Bicycle Lancs: The presence of bicycle lanes might be correlated with restrictions turning right, especially in areas with high bicycle traffic.
Figure 15: Ablation of stages 1 and on BDD-OIA bias discovery. In this experiment, we prompt the LLM in Stage to imagine possible biases that the largel classifier may have (prompt shown al the top), and display the LLM's response (bottom). Without relying on the target classifier, the LLM only generates generic biases and fails to detect the unexpected left-lanc bias. This demonstrates the critical role of counterfactual guidance (stages and 2) in the GIFT framework.
38

---

Published in Transactions on Machine Learning Research (02/2026)
M
Can turn right
M
Cannot turn right
Query image 2
Edited image 2c to
M
Cannot turn right
Can turn right
Query image T
Edited image x
Figure 16: Image intervention for the 'Dense Traffic in Left Lane' explanation. The model is biased for vehicles in the left lanes to yield the 'Cannot turn right' output. Left: query images; Right: images t and = with the added and removed concept.
39


Reviews:
[{"review_id": "asH7Zp1Oof", "reviewer": "Reviewer_ZM9U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "GIFT proposes a 4-stage framework for explaining vision classifiers that combines: (1) local counterfactual generation, (2) vision-language model translation to text, (3) LLM aggregation into global hypotheses, and (4) causal verification through image interventions. The framework aims to produce explanations that are simultaneously global, interpretable, faithful, and textual—addressing limitations of existing methods. Evaluation on CLEVR, CelebA, and BDD-OIA demonstrates the framework's ability to uncover classification rules and biases.", "claims_and_evidence_yn": "No", "claims_explanation": "Mainly yes.\nThe claim of \"Causal\" verification rests on the assumption that the image editing model (Stage 4) changes only the target concept and nothing else. Can you verify this is actually the case, eg by checking intervention quality with a VQA model? Also, do you make sure edited images remain in-distribution. This seems an implicit assumption.", "audience_yn": "Yes", "audience_explanation": "The authors address an important application", "requested_changes": "Experiments have quite a limited scope. Can you add a data set with multi-class classification and a more complex real world task, e.g. the knee radiography daatset used in [1-3]. I would also be interested in how [3] performs as a related baseline from the shortcut detection literature (which should be discussed as related work).\n\nYou mention \"occasional noise and hallucinations from the VLM, where it describes changes that are not actually present in the images\" - can you quantify how often this occurs and the effect of performance?\n\nCan you provide quantitative metrics for explanation completeness and accuracy?\n\n[1]  DeGrave, Alex et al.. AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610–619, 2021. \n\n[2] Adebayo, Julius et al. Post hoc explanations may be ineffective for detecting unknown spurious correlation. In International Conference on Learning Representations, 2022\n\n[3] Kuhn, Lukas, et al. \"Efficient unsupervised shortcut learning detection and mitigation in transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vjCc1lNrus", "reviewer": "Reviewer_2BHc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces GIFT, a framework which proposes and verifies\ntextual explanations for global classification decisions.\nThe framework makes use of multiple large models, including \n(1) a contrastive explanation model to generate local contrastive examples,\n(2) a caption-change model (or VLM), which generates textual descriptions of\nthe local changes introduced by the model in (1),\n(3) an LLM, which summarizes all textual descriptions in (3) to obtain global textual descriptions, and\n(4) a VQA model to identify examples where the textual description of (3)\napplies, and a text-guided image-editing model to add/remove the property\ndescribed by the textual description, the two of which are used to verify and\nquantify whether the textual descriptions are faithful explanations of the model prediction strategy.\n\nThe proposed framework is empirically validated on toy dataset (CLEVR),\nas well as two realistic datasets (CelebA and BDD-OIA) of increasing\ncomplexity.\n\n\n### Strengths\n\n- The work discusses all the limitations inherited by the used models.\n- Three datasets of increasing complexity are analyzed.\n- Key observations are clearly presented (4.3).\n- The verification in Stage 4 is a good approach.\n- Figure 1 describes the pipeline very well.\n\n### Weaknesses\n\n- The use of multiple possibly biased and/or error-prone models in the proposed\n  framework can lead to error accumulation.\n- The required performance for a successful recovery of the pipeline (i.e.,\n  failure cases) of the individual stages is not evaluated.\n- The computational requirement of the proposed framework seems immense.\n- The framework heavily relies on the local contrastive examples generated in\n  Stage 1, which are unlikely to capture all data properties, and may require \n  an infeasible number of examples.\n- ChatGPT 4, a proprietary model, is used in one of the experiments, leading to\n  non-reproducible results.", "claims_and_evidence_yn": "No", "claims_explanation": "The work demonstrates the recovery of textual classification rules\nin an empirical analysis for each of the three analyzed data settings.\nHowever, each of these setups are only presented with specific models.\nThe model requirements for this analysis to succeed are not analyzed at all.\nFor instance, what is the cheapest setup one could use to recover\nthe classification rules for CLEVR?\nFurthermore, it is unclear whether the claim that \"no prior knowledge is\nrequired\" holds, given that the counterfactual models have been trained for specific semantic concepts, and given that the \"true\" rule seemingly requires\nmanual combination of more specific rules.", "audience_yn": "Yes", "audience_explanation": "The work combines several large models to test whether our current\nstate-of-the-art allows a reasonably automated way to provide global\nprediction strategies of vision classifiers as textual descriptions.\nThus, it provides an important down-stream task for these models,\nwhich also serves as a benchmark.", "requested_changes": "- 1. (critical) For Section 4.1, you use ChatGPT 4.\n  ChatGPT 4 is proprietary and has undocumented changes all the time.\n  Therefore, it is safe to assume that any experiment based on ChatGPT\n  cannot be reproduced in the future.\n  Even if it may be the most performant model,\n  I do not think it is a good idea to use a remote service with a model that\n  is prone to change without notice, and may not be available in the near\n  future for empirical experiments. Would it be possible to use\n  some local LLM? Why have you not used Qwen2.5 72B, like in the CelebA\n  setting?\n\n- 2. (critical) How do you make sure the counterfactuals cover all semantic\n  concepts? How do you make sure to capture \"non-semantic\" counterfactuals? It\n  could be very likely that some secondary attributes of the image, e.g, focus,\n  pixel density, camera model, could be used by the model to classify. Unless\n  the counterfactual explanation generator is exhaustive, there is a good\n  chance that some unintuitive prediction strategies are missed.\n  - Could this be the reason GIFT fails in Table 2 for the red metal object on\n    ResNet, while it succeeds for ViT?\n\n- 3. (critical) The last paragraph of 4.3 reads: \"GIFT requires no prior\n   assumptions about potential biases\". I do not think this is entirely true,\n   as Stage 1 *does* require prior knowledge (of the counterfactual model)\n   to generate counterfactual explanations. If counterfactuals are not\n   generated for certain attributes in Stage 1, then GIFT also has no way of\n   identifying these.\n   - This can simply be addressed by changing the wording to weaken the claim.\n\n- 4. (critical) The number of counterfactual examples required to exhaustively\n  discover all relevant rules could be extremely large, depending on the\n  complexity of the task. And even if one does generate a lot of examples, the\n  contrastive example generator might be biased and miss some important\n  changes. As also discussed in 4.3, the Stage 2 might also fail for complex\n  data examples, or might also be biased.\n  - I think one way to at least partly address this concern would be to\n    show how the quality of the identification of rules scales with the number\n    of contrastive examples generated.\n\n- 5. (critical) As the quality of the analysis can greatly vary with the models\n  used, it would be good to compare different models in the different stages in\n  4.1. While we can see that the rules of ResNet seem harder to discover, it\n  would also be interesting to see how good the models need to be in order for\n  the pipeline to succeed.\n  - The conclusion claims that the framework was validated \"with diverse models\n    at each stage\", but there is no direct comparison of models. This would be\n    addressed by the suggestion above.\n\n- 6. (critical) The combination of the different attributes makes a lot of sense.\n  Is this done only manually, or automatically? If not automatically, do you\n  think there is a way to do so?\n  - Is the counterfactual generator able to create \"combined\" attributes?\n\n- 7. (stronger) Related work: GIFT is not compared to counterfactual explanations\n  or feature attributions. A simple comment on use of counterfactual\n  explanations, and differences to feature attributions regarding GIFT would\n  suffices.\n\n- 8. (stronger) Stage 4: The work explains that the most promising descriptions\n  are kept based on the PNS. It would be helpful to explain how a good\n  threshold is chosen, as you also mention in 4.1 that \"appropriate\n  thresholding\" can help. Additionally, the PNS seems low in 4.2, but it is not\n  clear to me at which point a rule can be interpreted as \"true\".\n\n- 9. (stronger) I really welcome the limitations in Appendix J. Given that this\n  pipeline setup seems fragile and the limitations are extremely important for\n  this setup, I think it would be much better to include these in the main\n  manuscript, before the conclusion. This would also still be well within the\n  12 page limit.\n\n- 10. (stronger) While the model succeeds in identifying known biases,\n  it would strengthen the work a lot if it could be shown\n  to identify previously unknown biases. However, I also understand\n  that this might be quite challenging to pull off.\n\n### Minor\n\n- Please add a reference for model-agnostic methods: \"reduced fidelity and\n  limit the reliability of explanations for complex, high-dimensional data.\"\n\n- \"acting as mere edge detectors\" -> \"resembling edge-detectors on CNNs\"\n    - they are not edge detectors, only look like it\n- input attribution -> feature attribution\n- model agnostic is too broad, and I would not call surrogates \"agnostic\"\n- counterfactual explanations are not just images\n\n- no recent work cited on feature attribution\n- no recent work cited on surrogate model attribution\n\n- Eq. (2) the $\\forall$ should be an \"if\"\n\n- Why is Stage 1 linked in the PDF, but not the other stages?\n\n- Fine filter Pearl (2009); Goyal et al. should be citep, not citet\n\n- (Freisleben, 2020) reference is published:\n```\nFreiesleben, Timo. 2022. “The Intriguing Relation Between Counterfactual Explanations and Adversarial Examples.” Minds and Machines 32 (1): 77–109. https://doi.org/10.1007/s11023-021-09580-9.\n```\n\n- M is not well defined. Does it encompass *any* class change?\n  - I am assuming the \"M(x)\" produces the predicted label.\n  - For the image pairs on page 4, the \"model prediction\" can be interpreted as\n    the class label, same for (3).\n  - In (2), the effect on a specific class is measured by an explicit\n    difference of the labels, which is counter-intuitive. It works better when\n    using the same notation as in (3), which explicitly checks the prediction\n    for some class y, which is the true class.\n\n- in the paragraph before (1), maybe use `:=` rather than `=` to define the\n  edited samples\n\n- the image pairs on page 4 are missing an equation number\n\n- use a wide-hat for PNS? It looks like only the N has a hat.\n\n- maybe box figure 5, as it looks like the text is continuing from the last page\n\n- ChatGPT4: reference should be citep, not citet\n\n- 4.1 Main results: The true visual rule was uncovered [in] 11/12 cases.\n\n- 4.1 second to last paragraph: (such as 'Red ~~metal~~ object instead of 'Red [metal] object'\n\n- Qwen2.5 citation \"Team\" is odd\n\n- Figure 4 is very far away from the paragraph where it is first mentioned.\n\n- Table 6 is mentioned in the manuscript, but it is not mentioned that this\n  table is in the appendix. Same for Figure 14.\n\n- Figure 6 was a fun addition, my first quick guess was correct", "broader_impact_concerns": "I do not see any ethical implications of this work that would require a broader\nimpact statement.", "additional_comments": "- I think the comparison with state of the art and ablation studies documented\n  in the last paragraphs of 4.3 are quite helpful, and summarize the\n  observations well. The caption-change ablation is especially interesting.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fp6JRtdtOg", "reviewer": "Reviewer_kjB2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces GIFT, a novel framework designed to generate textual global explanations by leveraging off-the-shelf components: a counterfactual generator, a large language model (LLM), and an image editing model. The proposed pipeline operates in four key steps: (1) generating a set of counterfactual examples, (2) producing captions for these counterfactuals, (3) deriving candidate global explanations from the captions, and (4) validating the explanations by probing the target model’s behavior under generated interventions. To evaluate the effectiveness of GIFT, the authors conduct comprehensive experiments that showcase that she explanations are able to undercover induced classifications rules.\n\n**Strengths**:\n- Novelty and Potential Impact: The paper introduces a unique approach to generating global, textual, and counterfactual-based explanations, which could serve as a foundation for new XAI methods. Beyond interpretability, the flexibility of GIFT opens avenues for applications in reverse engineering, as demonstrated in the experiments.\n\n- Clarity and Rigor: The paper is well-written, with a clear and structured presentation. The related work section is thorough, and the ablation studies address key questions a reader might have. The inclusion of a dedicated limitations section further strengthens the work.\n\n- Explanation Quality Assurance: A major strength is the emphasis on verifying explanation quality, an aspect often neglected in prior work on textual explanations. This rigorous validation enhances the reliability and trustworthiness of the proposed method.\n\n- Scalability and Future-Proofing: GIFT’s instance-based architecture makes it scalable, allowing seamless integration with future advancements in LLMs and image generation models. This adaptability ensures its relevance as underlying technologies evolve.\n\n**Weaknesses**:\n\n- Limited Experimental Scope: The experiments focus exclusively on binary classification, leaving open questions about how GIFT scales to multi-class settings. While computational costs are currently reasonable, the method’s efficiency and effectiveness may degrade as the number of classes increases.\n\n- Overstated Claims: The assertion that GIFT is the \"first framework for global, faithful, interpretable explanations for vision classifiers\" may be overstated. Prior works [1] have explored similar axioms using discrete representations, and claiming full interpretability and faithfulness is ambitious, given the inherent limitations of counterfactual generation and LLM-based reasoning.\n\n- Faithfulness Concerns: The fourth stage—validating explanations via model probing—does not fully guarantee faithfulness, as it relies on an intervention generation process that may itself be prone to errors or inaccuracies. If the image editing or counterfactual generation fails, the resulting explanations could be misleading. A more nuanced discussion of these dependencies would strengthen the claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "While I acknowledge that some claims in the paper—particularly regarding faithfulness and interpretability—may be ambitious, I recognize that these notions are inherently subjective and open to interpretation. If the authors ground their assertions in supporting literature, their perspective can be reasonably justified.\nRegarding the experiments, although the relatively low $\\hat{PNS}$ scores raise questions about performance, the diversity of tasks and datasets (three distinct datasets were used) provides stronger empirical evidence and helps mitigate these concerns. This breadth enhances the robustness and generalizability of the findings.", "audience_yn": "Yes", "audience_explanation": "The proposed approach introduces a novel class of explanations with, to my knowledge, no direct equivalents in existing literature. The emphasis on hypothesis verification addresses a critical gap in current XAI research, where rigorous validation of explanations remains underexplored. Moreover, the framework’s design opens the door to a wide range of emergent applications, making it a promising contribution to both interpretability and broader model analysis.", "requested_changes": "- Potential addition in related works: [2]\n- To substantiate the claims, the authors should either:\n     -  Support with evidence: Provide studies demonstrating that textual explanations are more interpretable than alternatives listed in Table 1.\nClarify why counterfactual-based methods should be considered fully faithful, referencing existing literature or empirical validation.\n     - Mitigate the claims: Revise statements like \"We introduce the first framework for global, faithful, interpretable explanations for vision classifiers\" to a more precise formulation, such as: \"We introduce the first framework for global, textual, and counterfactual explanations for vision classifiers. The faithfulness of the explanations is supported by causality quantification.\"\n\n\n- Discuss the potential impact of increasing the number of labels on the method’s computational efficiency, explanation quality, and overall viability. Address whether the current approach can scale effectively or if adaptations would be necessary for multi-class settings.", "broader_impact_concerns": "N/A", "additional_comments": "References: \n\n[1] Nauta, Meike, et al. \"From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai.\" ACM Computing Surveys 55.13s (2023): 1-42.\n\n[2] Shaham, Tamar Rott, et al. \"A multimodal automated interpretability agent.\" Forty-first International Conference on Machine Learning. 2024.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)