Salvage: Shapley-distribution Approximation Learning Via Attribution Guided Exploration for Explainable Image Classification
Authors: Mehdi Naouar
Hanne Raum
Jens Rahnfeld
Yannick Vogt
Joschka Boedecker
Gabriel Kalweit
Maria Kalweit
Venue: ICLR 2025 Poster (2025)
Keywords: Explainability
XAI
feature attribution
Primary area: interpretability and explainable AI

Abstract: The integration of deep learning into critical vision application areas has given rise to a necessity for techniques that can explain the rationale behind predictions. In this paper, we address this need by introducing Salvage, a novel removal-based explainability method for image classification. Our approach involves training an explainer model that learns the prediction distribution of the classifier on masked images. We first introduce the concept of Shapley-distributions, which offers a more accurate approximation of classification probability distributions than existing methods. Furthermore, we address the issue of unbalanced important and unimportant features. In such settings, naive uniform sampling of feature subsets often results in a highly unbalanced ratio of samples with high and low prediction likelihoods, which can hinder effective learning. To mitigate this, we propose an informed sampling strategy that leverages approximated feature importance scores, thereby reducing imbalance and facilitating the estimation of underrepresented features. After incorporating these two principles into our method, we conducted an extensive analysis on the ImageNette, MURA, WBC, and Pet datasets. The results show that Salvage outperforms various baseline explainability methods, including attention-, gradient-, and removal-based approaches, both qualitatively and quantitatively. Furthermore, we demonstrate that our explainer model can serve as a fully explainable classifier without a major decrease in classification performance, paving the way for fully explainable image classification.

Published as a conference paper at ICLR 2025
SALVAGE: SHAPLEY-DISTRIBUTION APPROXIMATION LEARNING VIA ATTRIBUTION GUIDED EXPLORATION FOR EXPLAINABLE IMAGE CLASSIFICATION
Mehdi Naouart.2, Hanne Raum', Jens Rahnfeld', Yannick Vogth.2, Joschka Boedecker Gabriel Ka Maria Kalweit University of Freiburg, ?CRION, Brainl.inks Brrainlinks-BrrainTToo) Brain'Tools na naouarmêcs uni cail-freibug.c.
ABSTRACT
The integration of deep learning into critical vision application areas has given rise to necessity for techniques that can explain the rationale behind predictions. In this paper, we address this need by introducing Salvage, novel removal based explainability method for image classification. Our approach involves training an explainer model that learns the prediction distribution of the classifier on masked images. We first introduce the concept of Shapley-distribaiions which offers a more accurate approximation of classification probability distributions than ex- isting methods. Furthermore, we address the issue of unbalanced important and unimportant features. In such settings, naive uniform sampling of feature subsets often results in highly unbalanced ratio of samples with high and low predic- tion likelihoods, which can hinder effective learning. To mitigate this, we propose informed sampling strategy that leverages approximated feature importance scores, thereby reducing imbalance and facilitating the estimation of underrepre- sented features. After incorporating these two principles into our method, we con- ducted an extensive analysis on the ImageNette, MURA, WBC, and Pet datasets. The results show that Salvage outperforms various baseline explainability meth- ods, including attention-, gradient-, and removal-basee approaches, both qualita- tively and quantitatively. Furthermore, we demonstrate that our explainer model can serve as fully explainable classifier without major decrease in classification performance, paving the way for fully explainable image classification.
INTRODUCTION
In recent years, the expansion of artificial intelligence (AI) techniques, particularly the field of computer vision, has revolutionized numerous industries and societal domains, ranging from health- care to autonomous vehicles. Notably, the emergence of Vision Transformers (ViT), (Dosovitskiy et al. 202T) has lately significantly impacted the field of computer vision, establishing dard for image classification. Their ability to leverage self-attention mechanisms and effectively model long-range dependencies has positioned them the forefront of research and applications across diverse domains. However, despite their remarkable performance, a critical challenge per- sists: the incompatibility of existing explainability techniques with vision transformers. The ma- jority of explainability methods developed and refined for Convolutional Neural Networks (CNNs) often prove inadequate when applied to vision transformers. This difference highlights the need for architecture-iddeeenden explainability methods.
stan-
Among such methods, removal-based techniques work by iteratively masking portions of an input image to observe the resulting changes in the model's predictions. If the removal of particular region significantly affects the model's prediction, suggests that the region is important for the decision. Conversely, if removing region has little to no effect, it is deemed less relevant to the model's decision. Recently, VIT Shapley (Covert et al. 2023) has been introduced, merging the principles of removal-basee based methods with game-theoretie foundation. The method involves training an explainer model to estimate the Shapley values (Shapley) [952) of the image patches, quantifying their contribution to the classifier's prediction. The Shapley value of patch is estimated

---

Published as a conference paper at ICLR 2025
as the average change in the model's prediction when the patch is added to the image, calculated across all possible masked variations. Given that the number of possible mask combinations grows exponentially with the number of patches, ViT-Shapley leverages the FastShap (Jethani et al., 2022) algorithm, approximating the Shapley values via a least squares objective and stochastic gradient descent randomly sampled masks, efficiently handling the computational complexity.
We propose a similar removal-based approach but with a key difference. Instead of approximating the Shapley value during training, we train an explainer model to learn a representation of the classi- fier's prediction distribution on masked images. At test time, the Shapley value for each patch is then derived from the learned representation. Moreover, we address two weaknesses of ViT-Shapley:
The ViT-Shapley method often falls short by treating differences in prediction probabilities as linear scores and optimizing them using least squares, which does not adequately capture the probabilistic nature of the classifier's outputs. We address this limitation using a more conven- tional approach for probability distribution approximations, minimizing the divergence between the explainer's 's estimated distributions and the classifier's 's actual prediction distributions.
The random mask sampling employed by ViT-Shapley sample-inefficient, especially when deal- ing with heavy unbalanced ratios of important and unimportant patches, which can hinder effec- tive learning. To mitigate this issue, we adopt an informative sampling strategy to enhance sample efficiency throughout training. By integrating the estimated attribution scores into the sampling process, we are able to achieve a more balanced distribution of masks with low and high prediction likelihoods, thereby facilitating the estimation of underrepresented features.
After incorporating the two proposed optimizations into a method, we refer to as Salvage (Shapley- distribution Approximation Learning Via Attribution Guided Exploration), we performed an evalua- tion on four datasets (ImageNette, WBC, Pet, MURA) and observed that Salvage outperforms vari- ous baseline explainability methods, including attention- gradient-, and removal-basee based approaches, both qualitatively and quantitatively. Moreover, we introduce a novel concept, classifying by ex- plaining which shifts the focus from explaining a classifier's behavior to aggregating the explainer's estimated feature importance scores into a classification prediction. By doing so we can guarantee the consistency between the predictions and explanations of the model. Our results demonstrate that our explainer can serve as a fully explainable classifier without a major decline in classification performance, advancing the development of more trustworthy image classifiers.
2 RELATED WORK
With the increasing demand for explainable AI, variety of different attribution methods have been explored. These fall into five main categories.
Class Activation Maps: Convolutional Neural Networks (CNNs) have inspired the development of Class Activation Mapping (CAM) techniques to highlight important features in visual tasks. The original CAM (Zhou et al. 2015) method works for CNNs with Global Average Pooling (GAP) layers by generating attribution maps based on weighted feature maps. However, this method is limited to architectures with GAP layers. Grad-CAM (Selvaraju al. (209)) improves upon this by using backpropagated gradients to compute feature map weights, making it more flexible. Vari- ants like Grad-CAM++ (Chattopadhay et al. 20188. Eigen-CAM (Muhammad & Yeasin [2020), and Ablation-CAM (Desai & Ramaswamy (0200) explore different ways of refining these weights. Despite these advancements, CAM-based techniques were primarily designed for CNNs and often underperform when applied to transformer architectures (Covert et al. 2023).
Attention-based Methods: The attention mechanism of transformer models naturally allows in- sights into the information flow within the network. A straightforward method to assess importance by analyzing the attention scores between the class token and input tokens at a specific layer (Clark et al., (219). However, this approach gives limited insight into the overall information flow since different layers may focus on different regions, and the final output is shaped by the inter- action across all layers. (Abnar & Zuidema, 2020) tackles this by modeling the information flow as a directed acyclic graph, using attention scores as edge weights. They propose two methods to extract input token relevance: attention rollout, which traces attention weights from the class token back to the input tokens, and attention flow, which estimates information flow using maximum flow computations in the graph. However, attention mechanisms often exhibit issues like high attention
2

---

Published as a conference paper at ICLR 2025
scores focusing on low-informative background regions (Covert al. 2023 Dareet et al. 2023 (Darcet et al., (023) suggest this problem stems from the use of random tokens as intermediaries for internal computations and address it by adding supplementary tokens. While attention scores can be useful in some cases, recent studies question their reliability as explanations, arguing they may not always reflect model's true reliance on each token (Jain & Wallace 2019; [Serrano & Smith 2019). Moreover, attention-base based methods are class-agnostic, providing a single explanation per prediction and lacking class-specific insights.
Saliency maps (Simonyan et al. 2014) are an early method that extracts the gradient of the class score with respect to the input image. However, these gradients can be highly sensitive to small input perturbations, resulting in significant fluctuations (Smilkov et al., (017). To mitigate this, SmoothGrad (Smilkov et al. 2017) averages the gradients over multiple noisy versions of the input image, effectively smoothing the gradients and reducing volatility. Integrated Gradients (Sundararajan et al., 2017) further improves on this by integrating gradients along the path between baseline image (typically black image) and the target image.
Gradient-based Methods:
Treating neural networks as complete black box function, removal- based Methods measure fluctuations in the predicted class probabilities under partial information. The estimation of the prediction under partial information is achieved by inferring the classifier on masked images. RISE (Petsiuk et al. 2018) suggests measuring the contribution of each part of the image by sampling a large number of masks, computing the prediction of the network on each masked image, and finally summing up the averaged product of the masks with their corresponding predictions. However, as the number of possible masks grows exponentially in the number of image patches, large amount of masks is required
Removal-based Methods:
obtain decent estimation for each single region. To address this issue, FaStSHAP (Jethani et al. (022) proposes a game theory approach, training an explainer model estimate the Shapley value of the image patches, which consists of the average change in the model's 's prediction when the patch is added to the image. Building upon this concept, ViT-Shapley (Covert et al. 2023) further extends this method by adopting a vision transformer-based architecture for the explainer model. While this method aims to train a model to approximate the Shapley value directly, our approach learns a representation of the classifier's 's prediction distribution from which the Shapley value can be extracted.
LRP-based Methods:
Layer-Wiss Relevance Propagation (LRP) (Lapuschkin et al. 2015), based on Deep Taylor Decomposition (DTD) (Montavon et al. 2017), explains model predictions by prop- agating the output back to the input using specific decomposition rules. While LRP has shown good results CNNs, applying it to transformer architectures has led unstable explanations. (Chefer let al. 2020) attributes these instabilities to skip connections and attention layers, and the authors propose alternative propagation rules for these operations, particularly combining LRP relevance with gradient-based based attention rollout for attention layers. Similarly, (Ali et al. (022) attempts to address this issue by proposing more stable rules for the self-attention and LayerNorm operations.
BACKGROUND
In this section, we introduce Shapley values (Shapley 1952), which serve as the foundation of our method. Originating from cooperative game theory, Shapley values are used to fairly distribute payoffs among players based on their individual contributions to the total value. In the context of machine learning, they quantify the impact of each feature on model's 's prediction by measuring the average change in prediction when the feature is included in an input subset. We begin by presenting the formal definition of Shapley values, followed by a rearrangement that enables their approximation without requiring their marginal contributions.
3.1 SHAPLEY VALUES
Let N be a set of features and v(S) the prediction outcome given feature subset S C N. The Shapley value di of feature i is obtained follows:
[[N
[S| - 1)! (v(SU (1)) (()))
6
Σ SCN(()
(1)
WS
3

---

Published as a conference paper at ICLR 2025
The computation of Shapley values for all features requires evaluating the predictions for every pos- sible subset of features. However, as the number of features increases, the number of subsets grows exponentially, making the computation of the exact Shapley values infeasible for very large numbers of features. To address this challenge, FastShap (Jethani et al. (0022) proposes an approximation of the Shapley values using least-squares objective over the feature subset distribution pu (S) to sampled proportionally to ws:
Ep_ (s)(((())))) Σ o.) IES
However, using the Mean Squared Error (MSE) loss to approximate probabilistic model outputs in- troduces significant limitations. MSE is primarily designed for comparing scalar values and is not well-suited for capturing the complexities of probability distributions, such as their inherent uncer- tainty, variance, and multimodal characteristics. Additionally, MSE is sensitive scale and does not enforce the necessary constraints of probability measures, like non-negativity and normalization. As a result, MSE often yields invalid or suboptimal approximations when applied to distributions.
3.2 APPROXIMATING SHAPLEY VALUES WITHOUT MARGINAL CONTRIBUTIONS
Kolpaczki et al. (2024) suggests rearrangement the Shapley value formula. Instead of expressing it as the weighted average of marginal contributions, it can be viewed as the difference between the weighted average of prediction outcome when feature is included and the weighted average of prediction outcome when feature is excluded:
Φi =
Σ SCNYO
Ws ((S -
Σ Wg vSS)
(3)
SCNVE
The positive and the negative Shapley values can be seen as the expected values o7 = E[e(S Ui))
and o; = E[((S)), over the set distribution (S) x for S C N (i).
4
APPROACH
4.1 SHAPLEY DISTRIBUTION ESTIMATION
We build upon the concept of optimizing Shapley values without relying on marginal contributions by training an explainer model learn both positive and negative Shapley values. During training. we sample masked images from the distribution Pw (S) о ws. For each sampled masked image, we update the estimated positive Shapley values o, for all visible image patches E S, and the negative Shapley values o for all masked patches j E S. This is achieved by minimizing the difference between the sum EiEs ot + Eigs o and the actual prediction outcome u(s). As mentioned in section 3.1 using the mean squared error (MSE) to approximate the target distribution v(s) would yield suboptimal results because the probabilistic nature of the classifier's output. Therefore, we propose mapping the summed term into probability distribution u(S), which refer to as Shapley probability distribution:
u(S) = 0t
ΣØ,1 )7)
where a denotes the softmax function in a multiclass classification setting or the sigmoid function for binary classification. The Shapley distribution of the masked image is then optimized by minimizing the Bensen-Shannen (JS) (Lin, 1991) divergence hetween the classifier's prediction v(S) and its corresponding estimated probability distribution u(S):
arg min Ep.. Is ( (((S)[(((S))
(5)
.6-
At test time, the feature importance scores each feature (image patch) are given by their estimated Shapley value dh ot o;-
4

---

Published as a conference paper at ICLR 2025
4.2 ATTRIBUTION GUIDED SAMPLING
In our experiments, we observed that sampling from the random mask distribution often results in disproportionate number of masked images having either high low likelihoods of the pre- dicted class. This imbalance severely affects the minority class estimation, since a large number of samples are required to estimate the values of its members. This finding motivated us address this imbalance through an alternative mask distribution, thereby enhancing the sampling efficiency during training. Thus, we propose exploiting the current estimates of feature importance scores to rebalance the ratio of masks, targeting the most and least informative regions of the image. Our proposed informative sampling distribution, pa (S) x Φ, operates in two stages:
Sampling the number masked patches: First, the number masked patches within an image is sampled from uniform distribution (1, where represents the total number patches in the image.
2. Selecting patches: Next, patches are selected without replacement, using the estimated feature importance scores & of the target class, sampling weights. For instance, if three patches have importance scores of 0, 1, and 3, their corresponding probabilities of being sampled would be 0, 0.25, and 0.75, respectively.
We then generate two equal mask subsets: the first, prioritizing the most informative regions from the image, and the second by masking them to target the least informative regions. A detailed description of the sampling process is provided as pseudo-code in Algorithm 3 Compared to random sampling, (S) yields mask distribution with more balanced prediction likelihoods (see Figure 6), thereby enhancing sample efficiency during training.
4.3
FROM AN EXPLAINER TO FULLY EXPLAINABLE CLASSIFIER
Since the classifier and the explainer are two decoupled models, the explainer merely approximates the behavior of the classifier. Thus, there is no guarantee of consistency between the classifier's predictions and the explainer's explanations, especially under domain shift. We suggest addressing this issue by using the explainer as unified model for both classification and explanation.
Recall that Salvage is trained to minimize the divergence between its Shapley distribution u( ((S) and the corresponding classifier prediction o(S). By setting S to the full (unmasked) image N in eq. (4). obtain the explainer's approximation for the classifier's prediction for the complete image:
(N) = o Σ Φt) ~ u(N)
(6)
iEN
In addition to ensuring consistency between the classification prediction and its explanation, using the explainer as classifier offers a unique advantage. By aggregating the importance of each image region, we obtain a precise understanding of how each region contributes to the overall prediction. This approach results in classifier that is fully transparent and explainable.
EXPERIMENTS
In this section, first describe our experimental setup. We then conduct both qualitative and quantitative analysis of our method and several baselines. Next, we conduct an ablation study, show- ing the advantage of informative sampling. Finally, we evaluate the classification and explanation of Salvage as an explainable classification method.
5.1 EXPERIMENTAL SETUP
We adopt the experimental setting from ViT-Shapley (Covert et al. 2023 evaluating the explana- tion performance of our method across the ImageNette (Howard & Gugger 2020) WBC (bodzás)
et al.) 2023), and Pet (Parkhi et al.) 2012) datasets for multi-class classification and the MURA RRajuurkar et al. 2018) dataset for binary classification. For the target model, use Vision Transformer base model (ViT-B) with a patch size of 16 and registers (Darcet et al. 2023). lever- aging the implementation and pre-trained weights provided by DINOv2 (Oquab al. 2024. The
5

---

Published as a conference paper at ICLR 2025
classifier model is trained for 25 epochs and fine-tuned on masked images for 50 epochs. We adopted the Segmenter (Strudel et al. 2021] segmentation architecture for our explainer models and trained them using 32 masks per image for 18k iterations (corresponding to 50 epochs for ImageNette, WBC, and MURA, and 100 epochs for Pet). All models were trained with a batch size of 64, an AdamW (Loshchilov & Hutter 2019) optimizer with learning rate of le-5 and weight decay of le-5, except the explainer models of ViT-Shapley which were trained with Ir lc-4. We compare our method to 10 different baselines; GradCam (Selvaraju et al. 2019), EigenCam (Muhammad
& Yeasin 2020). Attention scores from the last layer with registers (Clark et al., 2019 Darcet
al.
2023) (Attn. last). Attention Rollout with registers (Abnar & Zuidemal 2020 Darcet al.,
2023
VIT-CX (Xie et al. 2023), Saliency maps (Simonyan et 2014)) Integrated gradients (Sun-
dararajan et al. 2017) 7). LRP beyond attention (Chefer al., 2020). RISE (Petsiuk et al. 2018), and
ViT-Shapley (Covert et al. 2023). As reference, we additionally evaluate the metric scores on ran-
domly generated maps (Random) The implementation of the CAM-base methods is adapted from
Gildenblat & contributors 2021 while the other baselines use their original implementations.
5.2
QUALITATIVE ANALYSIS
Srccon
2K MAU
VIQ
UPP
VDL LUT
A2I
STOPsee
Figure 1: Qualitative examples computed ImageNette, Pet and MURA.

---

Published as a conference paper at ICLR 2025
We conduct qualitative analysis comparing method with the baselines. We present 6 examples from ImageNette, from Pet, and from MURA in Figure Saliency maps, Vit-CX, and LRP seem unreliable their attribution maps often focus random parts of the background. Despite its solid quantitative results, the attribution maps generated by RISH are highly noisy. Attention rollout (with registers) and ViT-Shapley yield decent results most images with relatively low scores outside the informative regions. Solely Salvage excels at highlighting the entire relevant region of interest. Further qualitative examples from ImageNette, Pet and MURA, well as qualitative examples from WBC, can be found in Appendix A.5
5.3
QUANTITATIVE ANALYSIS
Table 1: Quantitative results computed on the Pet, ImageNette, WBC, and MURA datasets. The performance of the 10 compared methods is measured in terms of SRG, R-SRG, RMA, and RRA.
Pet
ImageNette
MURA
WBC
R-
R- SRG
R-
R-
Method
SRG
RMA
RRA
SRG
SRG
SRG
SRG
SRG
SRG
GradCam
10.6
3.5
48.1
42.7
-1.9
-3.3
16.2
10.1
-18.5
-20.2
EigenCam
27.4
3.2
48.9
62.9
13.2
-3.1
0.1
-4.5
22.9
-7.0
Attn. last
47.9
9.6
61.1
70.1
27.0
3.0
22.4
7.0
42.2
1.6
Attn. Roll.
52.0
11.2
51.5
74.6
32.0
3.4
17.6
6.3
48.1
2.5
ViT-CX
50.2
17.6
30.6
67.5
29.9
7.5
19.8
9.1
41.6
7.3
Sal. Maps
51.1
10.8
52.7
76.3
27.7
2.8
25.3
8.5
42.8
2.1
IntGrad
27.4
7.9
51.5
58.8
11.0
2.2
13.9
6.1
11.8
1.6
LRP
49.5
9.2
63.9
71.8
27.9
3.0
19.3
6.8
37.0
1.7
RISE
63.7
18.5
30.1
47.8
22.9
5.4
56.5
22.1
20.7
3.5
ViT-Shap
61.1
14.7
52.7
69.0
40.3
6.2
65.3
20.6
57.4
7.3
Salvage
68.5
26.3
64.9
73.5
51.3
14.9
68.6
25.3
69.7
22.6
Random
0.0
0.0
30.0
29.4
0.0
0.0
0.0
0.0
0.0
0.0
Based on our baselines, we employed various metrics to assess Salvage's performance. Since the true importance of features is unknown beforehand, evaluating explanation accuracy poses chal- lenge. The metrics used to evaluate our method include Most and Least Influential First (MIF, LIF) (Petsiuk et al., 2018), Symmetric Relevance Gain (SRG) (Blücher al.) 2024), Relevance Rank Ac- curacy (RRA) and Relevance Mass Accuracy (RMA) (Arras al.) 2022 MIF and LIF (also known as Deletion and Insertion) measure performance by progressively removing image patches based on their importance scores, with the goal of minimizing MIF (removing most important features first) and maximizing LIF (removing least important features first). SRG improves upon these metrics by addressing their sensitivity to masking strategies and caleulating the difference between the MIF and LIF scores to provide more consistent performance rankings. Additionally, RRA and RMA assess the alignment of feature importance scores with human-annotated regions of interest. RRA evaluates how well the top-k important patches match the annotated region, while RMA measures the proportion of attributions within the annotated area, reflecting the focus relevant regions.
It is important to note that MIF, LIF, and SRG offer only limited insights into the quality of expla- nations, as they focus solely on the ranking of features while disregarding their relative differences. To address this limitation, we extend our metric selection by further including three new metrics to assess the relative score differences within an attribution map. Specifically, we extend the MIF and LIF metrics into R-MIF (Relatively Most Influential First) and R-LIF (Relatively Less Influential First). Instead of selecting patches purely based on their rank, we use estimated importance scores as weights in a sampling process to determine which patches to add first. To reduce sampling vari- ance, we generate 128 masks for each size of feature subset. Analogously to SRG, we define R-SRG the difference between R-LIE and R-MIF. These proposed metries provide deeper insight into the relative importance of feature attribution scores and their influence on the model predictions. A pseudo-code and detailed description of the metrics is provided in appendix A.

---

Published as a conference paper at ICLR 2025
Our results, shown in Table 1 and Figure clearly demonstrate that our method outperforms all baselines across both cank-based and velative-based metrics on all three datasets. We have observed in Figure 2 that all top-3 methods (Rise, ViT-Shapley, and Salvage) reach similar LIF and R-LIF scores hinting, that all three methods highlight a small subset of important features which is suf- ficient for the model to recognize the classified object. However, we observed that Salvage reaches significantly better MIF and R-MIF scores, which suggests that our method is more effective at identifying a larger portion of the important features compared to the other methods. Disregarding the top-3 methods, our results are in line with Covert et al. (203)) showing poor performance of the CAM-based, attention- based, gradient- based, and LRP- methods. The LIF, MIF, R-LIF, and
R-MIF scores of all methods have been included in appendix A.3 for the sake of completeness.
Moreover, we evaluate the RMA and RRA metrics on the Pet dataset, for which aanan-andotated regions of interest were provided. The results (ef. Table TT showed that Salvage achieves the best scores in terms of RMA. We additionally observe that Sal. Maps and Attn. Roll. reach slightly higher RRA scores showing strong alignment in ranking, meaning their top attribution scores lie within the object of interest. However, these methods also show a lower attribution mass within the object (RMA) and exhibit weaker SRG and R-SRG scores, suggesting they may not focus the most relevant parts of the object for the prediction, potentially limiting their effectiveness.
VIT Shapley
Salvage
Rise
MF
SRG
Imagsfattle
MURA
WBC
Trobabiity
152
275
de 25
Candinalty
Cardinality
Cardinality
Candinolity
Vil-Shaplay
Salvage
Rise
RAI
R-SRG
Imagelette
MURA
Deddicten Protebblity
Candinalty
Cardinalty
Cardiollity
Candinaitty
Figure 2: An Illustration of the MIF, LIF, SRG, R-MIR, R-LIF, and R-SRG across the different car- dinality values (number of masked patches) for the top-3 performing methods (RISE, ViT-Shapley, and Salvage) computed on the Pet, ImageNette, WBC, and MURA datasets.
5.4 ABLATION STUDIES
In this section, investigate the individual contribution of each core principle in our method, Shapley distribution estimation and informative sampling, through two ablation studies. The first study compares the performance of Shapley distribution estimation the MSE-base approximation (ViT-Shapley) without informative sampling. The second study investigates the effect of informative sampling on the performance Salvage by comparing its performance with and without the use of informative sampling.
The results of our studies, illustrated in Figure 3 reveal the following: (a) Shapley distribution estimation outperforms MSE-based approximation (ViT-Shapley) on ImageNette, WBC, and Pet datasets, but shows a slightly lower performance on MURA. However, we believe the latter could benefit from adjusting the temperature parameter in the sigmoid function used during training. (b) Informative sampling clearly improves performance on MURA and ImageNette, while achieving modest gains in R-SRG of 0.68 and 0.26 for WBC and Pet, respectively.
8

---

Published as conference paper at ICLR 2025
R MIF
RIFI
R-SRG
R-MIF
R-LIF
R-SRG
ImageNette
MURA
WBC
ImageNette
MURA
WBC
1.0
Prediction Probability
Prediction Probability
0.9
0.8-
0.7
MSE
Shopley dist.
MSE
Shapley 133.
MSE
Shapley 19%
MSE
Shapley dSt.
inf. With Inf. 5ompl.
weo inf. sampl.
Withint. sompl.
wio Int sampl
withint sompl.
we int
With
5ampp
sompl
sampl.
(a)
(b)
Figure 3: (a) Ablation study quantifying the performance gain of using Shapley distribution esti- mation versus MSE-based approximation (ViT-Shapley). (b) Ablation study comparing the perfor- mance of Salvage with and without informative sampling. The performance is reported in terms of R-SRG, which is given by the difference between R-LIF and R-MIF (the larger the better).
5.5 FROM EXPLAINER TO EXPLAINABLE CLASSIFIER
ImageMette
WBC
Salvage Outcome Wrog Correct
98.83%
0.05%
92.11%
1.2%%
68.52%
4.00%
99.63%
0.12%
0.82%
3.81%
2.86%
5.34%
21.94%
VIT Shap Outcome Wooo coneec
0.05%
0.00%
0.00%
0.00%
51.44%
9.73%
0.00%
Correct
Wrong
Wrong
Wrog Classifier Outcome
Outcome
Outcooe
Classifier
Figure 4: Confusion matrices showing the overlap between correctly and incorrectly classified sam- ples for the explainer models (Salvage and ViT-Shapley) and classifier. Each cell indicates the percentage of samples classified correctly or incorrectly by both models, with rows representing the explainer's outcomes and columns representing the classifier's outcomes.
In this subsection, we assess the benefits of using Salvage as an explainable classifier. By directly deriving predictions from the feature importance scores, the exact contribution of each image re- gion to the classification outcome is made explicit. This makes Salvage particularly well-suited for applications where explainability is as critical as classification accuracy.
We start by analyzing the overlap between correctly and incorrectly classified samples for the ex- plainer and classifier in Figure 4 The results suggest that Salvage demonstrates no major drop in performance relative to the original classifier and high agreement with the predictions of the classifier. In contrast to Salvage, we observe that the ViT-Shapley explainer yields poor classifica- tion performance and low overlap with the predictions of the classifier. A more detailed qualitative analysis of the classification performance of Salvage is presented in Table
Next, we analyze the explanations of Salvage in cases where its classification prediction is inaccu- rate. In Figure 5 we present examples where Salvage failed to produce the correct classification prediction and provide its attribution maps for both the predicted class (second row) and ground truth classes (third row). the second row of the figure, we can see that Salvage provides clear, understandable explanations for its misclassifications. For example, it can explain errors such as

---

Published as a conference paper at ICLR 2025
mistaking monument for church, construction truck for garbage truck, or mouse for an English Springer Spaniel. Notably, the last row, even in these failure cases, the attribution map corresponding the ground truth class of the image still accurately highlights the ground truth object (parachute, french horn...). The ability to provide reliable explanations, even in cases of classification failure, underscores Salvage's strong potential to serve fully explainable classifier.
Image
Church
English Springer
Garbage Truck
Garbage Truck
Predicted
class
Parachute
French Horn
Gas Pump
Chain Saw
Ground-truth
Class
Figure 5: Examples of test images from ImageNette where Salvage fails to make a correct classi- fication prediction. In the second row, present the Salvage's attribution map for the (wrongly) predicted class, in the last row its attribution map for the ground truth class of the image.
CONCLUSION
By incorporating novel methodology for attribution score estimation and informative sampling, we have developed removal-based explanation method for image classification called Salvage. Our experiments demonstrate that Salvage outperforms all 10 evaluated baselines, both qualitatively and quantitatively, by delivering higher-quality explanations and clearly distinguishing relevant from irrelevant image regions. Beyond its strong explanation performance, we have also established Salvage's potential as fully explainable classifier. While its classification accuracy is comparable to that of a classifier model, Salvage consistently provides detailed and interpretable explanations, even for images that are misclassified. This capability not only highlights the regions contributing to the predictions but also helps users understand the underlying factors leading to errors. Overall, these features underscore Salvage's strong potential serve a fully explainable classifier in applications where explainability is as critical as classification accuracy.
7 LIMITATIONS AND FUTURE WORK
Our work offers several promising avenues for further advancement. Future optimizations of method could involve refining the neural architecture of the explainer model, as improved segmen- tation architectures may enhance its ability capture spatial relationships and accurately estimate attribution scores for each superpixel-class pair. Moreover, introducing a temperature parameter in the softmax or sigmoid functions during the approximation of the classifier's distribution may be valuable and could offer a better alignment of the magnitude of the approximated values with the output logits of the classifier. Additionally, since Salvage adopts the principles of Shapley's additive explanation, it relies on the assumption that all features are linearly independent-aa assumption that may be overly restrictive in practice. promising direction for future work could involve ex- tending Salvage to account for feature interactions. Motivated by the quality of the explanation maps produced by Salvage, we plan to explore its potential on different tasks and data modalities, as well unsupervised segmentation model in future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project was funded by the Mertelsmann Foundation and the Deutsche Forschungsgemeinnccaatt
(DFG, German Research Foundation) Project-ID 499552394 SFB 1597. This work is part of BrainLinks- Boolss which is funded by the Federal Ministry of Economics, Science and Arts of Baden-Württemberg within the sustainability program for projects of the excellence initiative II.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers, 2020.
Ameee Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Alaus-Robert Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation, 2022.
Leila Arras, Ahmed Osman, and Wojeiech Samek. Clevr-xai: A benchmark dataset for
ground truth evaluation of neural network explanations. Information Fusion, 81:14-40, 2022. ISSN 1566-2535. doi: http:///c..iiieaa....aaaa//////a sciencedirect. wwt.
http://0.....i..ifffu.000000...0 URL https: //www.
Stefan Blücher, Johanna Vielhaben, and Nils Strodthoff. Decoupling pixel flipping and occlu-
sion strategy for consistent xai benchmarks, 2024.
06654
Alexandra bodzás, Pavel Kodytek, and Jan Zidek. A high-resolution large-scale dataset of pathological and normal white blood cells. Scientific Data, 10, 2023. doi: 10.1038/
541597-023-02378-7
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian, Grad-
cam++: Generalized gradient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEBE, March 2018. doi: WACV. 2015 8 00097
10. 10.1109/wacv.21 8. 8.00097.
dol. http::/:/..o.ooo.or///..0..111
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interprraiityy beyond attention visualization, 2020. BRLLttps: / api. semant wwwpaticccccccacoooooooooa.o. : 229297908
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D). Manning. What does bert look at? an analysis of bert's attention, 2019.
Ian Covert, Chanwoo Kim, and Su-In Lee. Learning to estimate shapley values with vision trans- formers, 2023.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023.
Saurabh Satish Desai and H. G. Ramaswamy, Ablation-cam: Visual explanations for deep convo- lutional network via gradient-free localization. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 972-980, 2020. URL https:: semant cscholar.
Sog//oorpusID:21460400
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale, 2021.
Jacob Gildenblat and contributors. Pytorch library for cam methods. 2021.
https://gitthut com/
Jeremy Howard and Sylvain Gugger. Fastai: A layered api for deep learning. Information, 11(2): 108. February 2020. ISSN 2078-2489, doi: 10.3390/mfol 1020100.00000 URL btLp://: doi. org /1. 200.3333//nfo112200000
Sarthak Jain and Byron C. Wallace. Attention is explanation, 2019.

---

Published as a conference paper at ICLR 2025
Neil Jethani, Mukund Sudarshan, Ian Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation, 2022.
Patrick Kolpaczki, Viktor Bengs, Maximilian Muschalik, and Eyke Hüillermeier. Approximating the shapley value without marginal contributions, 2024. URL https://arxiv..rgg/
2302. 00736
Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10:c0130140, 07 2015, doi: 10.137 1.jorn/jouuall.poo 0130140.
J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(1):1445-15 151, 1991. doi: 10/
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
//arxiv.orq/////a..vv.... hatg//b//111 wttp://rrr/.aaa/1....55.. 05101
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojeiech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:211 222, May 2017. ISSN 0031-3203, doi: 10. patcog. 1.008. URL
http://dx. doi. org. 10. 1016 patcog 2016.11.000 11.008
Mohammed Bany Muhammad and Mohammed Yeasin. Eigen-cam: Class activation map using principal components. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, July 2020. doi: 10.1 10910/9cn488600.... 2020.9266222 URL http:://aae http//xx.doo doi. www....ooo..gg.....o. 10//jcam/885552222226666666
I109/IJCNN486033 2020. 9206626
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nico- las Ballas, Wojciech Galuba, Russell Howes, Po-Ya Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. /R/.ttte://aaxv...o///////////// / 07193
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1498-3505, 2012. doi: 10.11099CVPR.2212..2442
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models, 2018
Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang, Kaylie Zhu, Dillon Laird, Robyn Ball, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, and Andrew Y. Ng. Mura: Large dataset for abnormality detection in musculoskeletal radiographs, 2018 URL
06957
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based lo- calization. International Journal of Computer Vision, 128(2):336-35,, October 2019. ISSN 1573-1405. doi: 111.10077/11223309--00223
URL http: http://xxxcci.ooo wtt.dxx.....ooo///.0/ 10077
s 111263-019-01220
Sofia Serrano and Noah A. Smith. Is attention interpretable?, 2019.
Lloyd S. Shapley. A Value for N-Person Games. RAND Corporation, Santa Monica, CA, 1952. doi: 10.7249/P0295.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Vićgas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise, 2017.
12

---

Published as a conference paper at ICLR 2025
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation, 2021.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017.
Weiyan Xie, Xiao-Hui Li, Caleb Chen Cao, and Nevin L.. Zhang. Vit-ex: Causal explanation of vision transformers, 2023.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. features for discriminative localization, 2015.
Learning deep
A APPENDIX
A.1 METRICS OVERVIEW
Least influential First (LIF) and Most influential First (MIF) also known Insertion and Deletion (Petsiuk et al. 2018) repeatedly mask the images by removing image patches based on their ascending/descending ranking from the least important the most important attribution scores of the method being evaluated. The area under the resulting curve (predictions/number then computed as the performance score illustrated in Algorithm [ The LIF score should be maximized, which is achieved by removing the least important features first, in order to get high predictions with as few patches possible. Conversely, the MIF score should be minimized by removing the most influential patches first. It is important to note that MIF and LIF offer only limited insights into the quality explanations, they focus solely on the ranking of features while disregarding their relative differences.
patches)
Algorithm Most Informative First (MIF) and Least Informative First (LIF)
Require: N: Set of all features, v: Prediction outcome function, @: Importance scores for each feature, metric: "MIF" or 'LIF"
Ensure: (averaged) MIF or LIF score
1: S 5-0
Initialize empty subset b. List to store prediction outcomes
2: preds <
3: if metric 'LIF" then
4: forted,features + Sort features in descending order of importance based on ø
5: else
6:
sorted,features + Sort features in ascending order of importance based on Ø
7: end if
8: for i = to [N] do
Add sorted features[]] to S 10: pred 4- v(S)
9:
D Evaluate the prediction outcome with the current subset
11: Append pred to preds
12: end for
13: final-score, <- Σ preds
> Compute the average over all subset sizes
|N]
14: return final sccre
Symmetric Relevance Gain (SRG): In a recent study, Blücher et al. (2024) demonstrated the in- consistency of the MIF and LIF metrics while using different masking strategies, these can lead to different performance rankings depending on the robustness of the masking strategy. In order to address this issue, the authors presented a simple, yet effective metric named SRG, which is given by the difference between the MIF and LIF scores:
SRG = LFF MTF
The authors have shown 40 different masking strategies that this metric breaks the inherent nection to the underlying occlusion strategy and leads to consistent rankings.
Relatively Most influential First (R-SRG), Relatively least influential First (R-LIF) MIF, LIF, and SRG provide limited insights into the quality of explanations, as they focus exclusively on fea- ture ranking without accounting for the relative difference scores between the features. To address
13

---

Published as a conference paper at ICLR 2025
this limitation, we propose extension of these metrics aimed at capturing the faithfulness of the relative differences in importance scores across different features. While MIF and LIP evaluate the prediction model by selecting the patches to be removed purely based on their ranking, selecting n features through a weighted sampling process. This process uses feature importance scores sampling weights, ensuring that features portance. By employing this method, we can assess the faithfulness of the relative differences in attribution scores, which are integrated into the sampling process. Analogous to MIF and LIF, R- MIF aims to sample the most relevant features, while R-LIF aims to sample the least relevant ones. In our experiments, we repeated the sampling process for each mask size 128 times to minimize variance in performance scores resulting from the sampling process (1im asks 128). For a more detailed overview, we present the pseudo-code for the computation of the R-MIF and R-LIF scores in Algorithm
propose
sampled in proportion to their estimated im-
Algorithm 2 R-MIF and R-LIF
Require: N: Set of all features, Prediction outcome function, d: Importance scores for each feature, metric: 'R-MIF" or 'R-LIF', n namasss: Number of subsets to sample per cardinality
Ensure: (averaged) R-MIF R-LIF score
1: S 5-0
Initialize empty subset > List to store averaged prediction outcomes > Find minimum feature importance score Find maximum feature importance score Min-max normalization ensuring positive sampling weights
2: preds t-
3: Ó_min + min(o)
4: Φ_mar + max
5: o_norm Hmnoz m.min
6: if metric = RR-LIF" then
7: weights + ø_norm
> Use the normalized scores as weights for R-LIF
8: else
9:
weights t 1 - norm
use minus the normalized scores as weights for R-MIF
10: end if
11: for = 1 to | do
12: subset_preed +
b> Store prediction outcomes for this step
for j to n_masks do
14:
S₁ + draw i features from N using weights as sampling weights, without replacement Evaluate the prediction outcome with the sampled subset
pred + v(S,)
16:
Append pred to subset preds 17: end for
18: aug-pred <- subset--pre preds
Average the prediction outcomes over all samples
n.nasks
19: Append avg oug-pred to preds
20: end for
21: final-score +- Σ preds |N]
b Compute the final average score over all subset sizes
22:
return final_score
Relative Symmetric Relevance Gain (R-SRG) Analogously to SRG, we combine the R-MIF and
R-LIF scores by defining:
R-SRG = R-LIF R-MIF
Relevance Rank Accuracy (RRA) Relevance Mass Accuracy (RMA) Arras et al. (2022) pre- sented two metrics measuring the consistency of the feature importance scores with target region of interests provided by human annotations. For relevance rank accuracy, image patches are ordered based on their importance scores, and the number of top-k pixels within the ground truth mask is measured, with k set to be the number of pixels in the ground truth mask. A high relevance rank score is indicative a strong alignment between the explanation and the human annotation. For relevance mass accuracy, the ratio of positive attributions within the ground truth mask to the sum of all positive attributions calculated. A high relevance mass score indicates that significant attention placed on the same region as the human annotation, with little attention directed other regions.
A.2 SAMPLING DISTRIBUTION
For the sake of completeness, we present a pseudo code of the informative sampling procedure in Algorithm Moreover, we computed the average prediction of the sampled subsets once using ran-
14

---

Published as a conference paper at ICLR 2025
dom sampling and once using attribution -nnfoomeed sampling. The results illustrated in appendix A.2 A.2 demonstrate that the average prediction likelihood of our informative sampling technique yields more balanced distribution than random sampling.
Additionally, we evaluated the effect of addressing this unbalance on sample efficiency. To do so, we illustrate in Figure the effect of informative sampling sample efficiency by comparing the SRG metric (higher is better) during training for models trained with and without the proposed informative sampling method. The results demonstrate that informative sampling consistently improves the SRG metric across the training process, indicating enhanced sample efficiency.
Algorithm Feature Subset Sampling Using Importance Scores
Require: N: Set of all features, Φ: Importance scores for each feature, n_masks: Total number of
subsets to sample
Ensure: List of sampled subsets
1: O_min min(
Find minimum feature importance score
2: Φ_mar -- max (Ø)
> Find maximum feature importance score
3: o_norm G_maz ommin
> Min-max normalization ensuring positive sampling weights List to store sampled subsets
4: samples
5: for k = 1 to n.masks do
2
6: m + -Sample from (1,
Sample subset size from uniform distribution 7: So +- draw features from N using o nunorm sampling weights, without replacement
8: Append Sg to samples
9:
end for
10: for k = to n mosks do
Sample subset size from uniform distribution 12: S1-6 t- draw m features from N using (1 Φ_norm) as sampling weights, without repl.
m Sample from UNI,NN)
Append S1-ø to samples
14: end for
return samples
Sampling
Anerge Tretetionn Liothoor
Imagshwile
Par.
MURA
WBC
1.6
1.6
53-
Cordiailty
180
Cardinlity
11
Cordiartty
08
Figure 6: Average prediction likelihood of the ground truth class using informative sampling versus random sampling, computed across various mask sizes.
15

---

Published as a conference paper at ICLR 2025
Sa vage Uninteemed
Salvage informed
55
15
30
30
30 33 155
15
30 20
25
15
Figure 7: Comparison of the SRG metric during training of the Salvage explainer model. The SRG metric (higher is better) is evaluated for models trained with informative sampling (Salvage Informed) and without (Salvage Uninformed) across 25%, 50%, and 100% of the overall training duration.
A.3 FURTHER QUANTITATIVE ANALYSIS
In this section, we present the averaged LIF and MIF scores for all baselines in Table along with the R-LIF and R-MIF scores in Table The results indicate that while the top three methods achieve relatively high LIF and L-LIF scores, our method demonstrates superior performance in the MIF and R-MIF metries. This suggests that our model is better at identifying larger portion of the important features, resulting in a clearer distinction between the relevant and irrelevant regions of the image. This observation is in line with the findings of our qualitative results.
Table 2: Quantitative results computed on the dataset Pet, ImageNette, WBC, and MURA. The performance of the 12 compared methods is measured in terms of MIF and LIF.
Pet
ImageNette
MURA
WBC
Method
LIF f
MIF $
LIF 1
MIF +
LIF
MIF
LIF
MIF
GradCam
81.08
70.47
88.94
90.89
78.23
62.07
76.22
94.70
EigenCam
76.47
49.05
87.70
74.45
65.59
65.48
85.64
62.76
Attn. last 89.44
41.55
96.76
68.74
74.69
52.28
96.46
54.27
Attn. Roll. 89.62
37.65
96.95
64.93
73.60
56.02
97.19
49.13
VIT-CX 88.19
37.96
96.29
66.37
74.44
54.60
93.51
51.88
Sal. Maps
89.38
38,26
96.09
68.34
74.57
49.31
96.36
53.58
IntGrad
89.45
62.01
96.76
85.80
75.32
61.45
97.32
85.47
LRP 89.64
40.09
97.05
69.17
75.14
55.85
97.01
60.02
RISE
95.73
32.03
98.30
75.38
93.20
36.72
98.47
77.76
ViT-Shap
93.52
32.45
98.71
58.38
91.25
25.91
98.81
41.38
Salvage
93.21
24.75
98.44
47.09
91.94
23.38
98.33
28.63
Random
83.29
83.67
95.13
95.07
71.24
70.78
96.55
96.55
16

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative results computed on the dataset Pet, ImageNette, WBC, and MURA. The performance of the 12 compared methods is measured in terms of R-MIF and R-LIF.
Pet
ImageNette
MURA
WBC
R-LIF
R-MIF
R-LIF
R-MIF
R-LIF 1
R-MIF
R-LIF
R-MIF
Method
4
t
4
+
GradCam
85.19
81.68
92.59
95.90
78.27
68.18
77.93
98.12
EigenCam
79.15
75.99
90.03
93.10
66.30
70.84
85.64
92.60
Attn. last
91.40
81.78
98.47
95,44
75.36
68.36
98.95
97.34
Attn. Roll. 90.83
79.61
98.29
94.83
74.79
68.53
99.07
96.53
ViT-CX 89.08
71.45
97.98
90.49
74.37
65.28
98.78
91.49
Sal. Maps
90.84
80.06
98.09
95.25
75.19
66.69
98.94
96.83
IntGrad
90.62
82.74
98.20
96.03
75.38
69.29
99.09
97.52
LRP
91.32
82.09
98.52
95.47
75.66
68.82
99.08
97.35
RISE
91.62
73.12
98.26
92.89
81.91
59.84
98.99
95.46
ViT-Shap
91.66
76.97
98.89
92.67
81.05
60.47
99.55
92.28
Salvage
91.60
65.31
98.79
83.92
81.77
56.45
99.50
76.88
Random
85.20
84.91
96.66
96.68
71.53
71.79
98.15
98.15
A.4 CLASSIFICATION ANALYSIS
In this section, first asses the classification performance of Salvage quantitatively in Table 4 by comparing it to baseline classifier and the ViT-Shapley explainer model. Salvage demonstrates no major drop in performance relative to the original classifier. In contrast, the ViT-Shapley explainer model performs poorly in classification. This issue can be attributed to the explainer's 's attribution scores for different classes being decoupled during training due to the use of additive normalization (Covert et al. 2023).
Table 4: An overview of the classification performance of the original classifier, ViT-Shapley, and Salvage computed Pet, ImageNette, WBC and MURA.
Model
Pet Accuracy
ImageNette Accuracy
WBC Accuracy
MURA
Precision
Recall
Fl-score
MCC
Classifier
95.91%
99.64%
99.75%
84.64%
78.88%
81.66%
0.66
ViT-Shapley
0.00%
0.05%
0.69%
59.03%
92.74%
72.14%
0.39
Salvage 93.61%
98.88%
99.75%
80.31%
80.52%
80.41%
0.62
A.5
QUALITATIVE EXAMPLES
We present further qualitative examples comparing Salvage the baseline methods in Figure 9 The figure includes examples from ImageNette, 5 examples from Pet, and 4 examples from MURA. Additionally present 8 examples from the WBC Dataset (one example per cell type) in Figure
17

---

Published as a conference paper at ICLR 2025
Nezursk
Figure Qualitative examples computed on the WBC dataset.
18

---

Published as a conference paper at ICLR 2025
Pundra
Figure 9: Qualitative examples computed on ImtteNette, Pet and MURA.
19


Reviews:
[{"review_id": "xpLn6snkn7", "reviewer": "Reviewer_d1Q4", "summary": "The authors highlight the shortcomings of existing methods for existing Shapley-based explainers and propose a method called Salvage, which effectively learns and samples based on the Shapley value distribution.", "questions": "1) What exactly does Figure 3 illustrate?\n2) Since the goal is to derive an explainer model for a specific classification model, I am curious not only about the classification performance but also about how well the predictions align with those of the existing classification model (as seen in Table 2).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The issues identified with existing methods appear valid and relevant.\n2) The authors demonstrate an improvement in explanation accuracy compared to existing methods across various datasets.", "weaknesses": "1) The explainer model is essentially an estimation model for interpreting the behavior of the target classification model; however, this paper does not clearly define what the target model is. Furthermore, there is insufficient evidence to show that the method operates effectively across different target models.\n2) Some results seem incomplete, as suggested by Figure 1.\n3) The ablation study is lacking. While the proposed method focuses on effectively learning the distribution and sampling, there is no analysis of which aspect is more critical to the overall success of the method.\n4) The problems identified with existing methods are described conceptually but lack empirical validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttVgxEfAXe", "reviewer": "Reviewer_sz6D", "summary": "This paper introduces a novel explainability method for image classification known as Salvage. The paper employs a removal-based technique coupled with the concept of Shapley-distributions. These techniques are used to train an explainer model that learns the prediction distribution of the classifier on masked images. The authors address the imbalance between important and unimportant features by devising an informed sampling strategy. This strategy facilitates better approximation of the classifier’s distribution and helps the estimation of underrepresented features. The effectiveness of Salvage is validated through experiments on the ImageNette, MURA, and Pet datasets. The study illustrates that Salvage outdoes various baseline explainability methods and can additionally be used as a fully explainable classifier without a considerable fall in classification performance. The paper concludes by pointing out future optimizations and improvement possibilities for Salvage.", "questions": "1. Could you please elaborate more on why the informative sampling's improvement in performance is less noticeable in the Pet dataset compared to the other datasets?\n2. How would different neural architecture designs for the explainer model impact the performance of Salvage?\n3. Given that the model has been tested on a limited number of datasets, have you considered testing Salvage on a wider array of datasets, particularly more complex or diverse ones, to evaluate its broad applicability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents \"Salvage,\" a novel removal-based explainability method for image classification that tackles unbalanced important features via an informed sampling strategy. The invention of Shapley-distributions for a more accurate approximation of classification probability distributions is impressive. The paper's comprehensive and clear presentation, alongside robust experimental evaluation, underlines the method's potential for explainable classification, with comparable accuracy to standard classifiers.", "weaknesses": "1. Although the paper demonstrates a performance comparison with a couple of explainability methods like ViT-Shapley and RISE, more extensive comparison with a wider array of contemporary removal-based explainability methods could provide a more robust evaluation of the Salvage algorithm's efficacy.\n2. It would be beneficial to see Salvage's performance with other types of data or in other domains, beyond the ones mentioned in the paper (ImageNette, MURA, and Pet datasets). This would help in evaluating the broad applicability and versatility of the approach.\n3. The impacts of temperature parameter changes in the softmax or sigmoid functions during the approximation of the classifier’s distribution could have been explored in more depth. More extensive experimental study in this aspect could enhance the robustness of Salvage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dBFaIgADjm", "reviewer": "Reviewer_7jFJ", "summary": "This paper introduces Salvage,a removal-based explainability method for image classification. It includes a concept of Shapley-distributions,which offers a more accurate approximation of classification probability distributions and an informed sampling strategy that leverages approximated feature importance scores to reduce imbalance and facilitate the estimation of underrepresented features.", "questions": "1.How is the experimental performance when using SHAPLEY DISTRIBUTION ESTIMATION alone?\n\n2. Is this method applicable to other types of tasks (such as object detection or segmentation)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. A  new concept of Shapley-distributions,which offers a\n more accurate approximation of classification probability distributions, is introduced.\n2. The comparison methods, datasets, and metrics are quite comprehensive. On some metrics, it has a clear advantage.", "weaknesses": "**The presentation should be improved a lot.**\n\n1. The explanation of symbols in the formulas is not clear enough, causing difficulty in understanding, such as Eq.2.\n2. In Table 1, there are several obvious typographical errors of the experimental results, for example “68,56”. It should be “68.56”.\n3. The bottom line of the Table 2 is not drawn!\n\n**Soundness**\n1. There is no theoretical  proof or experimental results can demonstrate that the INFORMATIVE SAMPLING has improved efficiency.\n2. In Table 2, on PET, in terms of RRA, Salvage underperforms about 3 point compared with SOTA. There should be appropriate analysis and discussion regarding this.\n3. Only make ablation study on INFORMATIVE SAMPLING.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oh8z9vj9dw", "reviewer": "Reviewer_Hzto", "summary": "This article proposes a new explainability method for image classification. Most of the explainability methods target CNNs; instead, Salvage, the proposed method is architecture agnostic. Salvage is a removal-based approach based on ViT-Shapely with further improvements. The method surpasses the current SOTA.", "questions": "I find the paper good as is.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is clear and well-explained. The methodology is adequate. The method is well evaluated. The method, despite not being completely novel, builds on SOTA methods (ViT Shapely) and improves their shortcomings.", "weaknesses": "The method is incremental with respect to ViT-Shapely but better. So no much concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)