Learning to Estimate Shapley Values with Vision Transformers
Authors: Ian Connick Covert
Chanwoo Kim
Su-In Lee
Venue: ICLR 2023 notable top 25% (2023)
Keywords: ViTs
Shapley values
amortization
explainability

Abstract: Transformers have become a default architecture in computer vision, but understanding what drives their predictions remains a challenging problem. Current explanation approaches rely on attention values or input gradients, but these provide a limited view of a model’s dependencies. Shapley values offer a theoretically sound alternative, but their computational cost makes them impractical for large, high-dimensional models. In this work, we aim to make Shapley values practical for vision transformers (ViTs). To do so, we first leverage an attention masking approach to evaluate ViTs with partial information, and we then develop a procedure to generate Shapley value explanations via a separate, learned explainer model. Our experiments compare Shapley values to many baseline methods (e.g., attention rollout, GradCAM, LRP), and we find that our approach provides more accurate explanations than existing methods for ViTs.

Published as a conference paper at ICLR 2023
LEARNING TO ESTIMATE SHAPLEY VALUES WITH VISION TRANSFORMERS
Ian Covert; Chanwoo Kim & Su-In Lee
Paul G. Allen School of Computer Science & Engineering
University of Washington {i covert, chanwkim, suinlee] (cs. @cs.wwwhington.cco
ABSTRACT
Transformers have become default architecture in computer vision, but under- standing what drives their predictions remains a challenging problem. Current explanation approaches rely on attention values input gradients, but these pro- vide a limited view a model's ' dependencies. Shapley values offer a theoretically sound alternative, but their computational cost makes them impractical for large, high-dimensional models. In this work, we aim to make Shapley values practical for vision transformers (ViTs). To do so, we first leverage an attention masking ap- proach to evaluate ViTs with partial information, and we then develop procedure to generate Shapley value explanations via separate, learned explainer model. Our experiments compare Shapley values to many baseline methods (e.g., attention rollout, GradCAM, LRP), and we find that our approach provides more accurate explanations than existing methods for ViTs.
INTRODUCTION
Transformers (Vaswani et al. 2017 were originally introduced for NLP, but in recent years they have been successfully adapted to a variety of other domains (Wang et al. 2020: Jumper et al.) 2021). In computer vision, transformer- based models now used for problems including image classification, object detection and semantic segmentation (Dosovitskiy et al., 2020 Touvron et al.
2021 Liu et al. 2021). and they achieve state-of-the-ee performance in many tasks (Wortsman et al. 2022). The growing use of transformers in computer vision motivates the question of what drives their predictions: understanding complex model's dependencies is important problem in many applications, but the field has not settled on a solution for the transformer architecture.
Transformers composed of alternating self-attention and fully-connected layers, where the self- attention operation associates attention values with every pair of tokens. In vision transformers (ViT') (Dosovitskiy et al.) 2020), the tokens represent ove-llerlap image patches, typically total of x4 196 patches each size 16 x 16. is intuitive to view attention values as indicators of feature importance (Abnar and Zuidema 2020 Ethayarajh and Jurafsky 2021). but interpreting transformer attention in this way is potentially mis laading Recent work has raised questions about the validity of attention as explanation (Serrano and Smith 2019 Jain and Wallace 2019 Chefer et al.) 202I). arguing that it provides an incomplete picture of model's 's dependence on each token.
If attention is not reliable indicator of feature importance, then what is? We consider the perspective that transformers are no different from any other architecture, and that we can explain their predictions using model-agnoste approaches that currently used for other architectures. Among these methods, Shapley values are a thcoretically compelling approach with feature importance scores that are designed satisfy many desirable properties (Shapley 1953 Lundberg and Lee, 2017).
The main challenge for Shapley values in the transformer context is calculating them efficiently, because a naive calculation has exponential running time in the number of patches. If Shapley values are poorly approximated, they are unlikely to reflect a model's true dependencies, but calculating them with high accuracy is currently too slow to be practical. Thus, our work aims make Shapley values practical for transformers, and for ViTs in particular, Our contributions include:
"Equal contribution.

---

Published as a conference paper at ICLR 2023
1. We investigate several approaches for withholding input features from vision transformers, which is a key operation for computing Shapley values. We find that ViTs can accommodate missing image patches by masking attention values for held-out tokens, and that training with random masking is important for models to properly handle partial information.
2. We develop a learning- -ased approach to estimate Shapley values efficiently and accurately. Our approach involves fine- finettnning an existing ViT using a loss function designed specifically for Shapley values (Jethani et al., 2021b), and we prove that our loss bounds the estimation error without requiring ground truth values during training. Once trained, our explainer model provides significant speedup over methods like KemnelSHAP (Lundberg and Lee 2017).
3. Our experiments compare the Shapley value-based approach to several groups of competing methods: attention-based, gradient- maadient-based and removal- based explanations. We find that our approach provides the best overall performance, correctly identifying influential and non- influential patches for both target and non-target classes. We verify this using three image datasets, and the results are consistent across multiple metrics.
Overall, our work shows that Shapley values can be made practical for vision transformers, and that they provide a compelling alternative to current attention- and gradient-bant -based approaches.
2 RELATED WORK
Understanding neural network predictions is challenging problem that has been actively researched for the last decade (Simonyan et al. 2013 Zeiler and Fergus 2014 Ribeiro et al. 2016). We focus on feature attribution, or identifying the specific input features that influence a prediction, but prior work has also considered other problems (Olah et al.) 2017 Kim et al.. 2018). The various techniques that have been developed can be grouped into several categories, which we describe below.
self-attention to associate weights with each pair of tokens /Vaswani et al. 2017), and a natural idea is to assess which tokens receive the most attention (Clark al. 2019 Rogers et al.] 2020 Vig al., (020). There several versions of this approach, including attention rollout and attention flow (Abnar and Zuidemal 2020), which analyze attention across multiple layers. Attention is popular interpretation tool, but it is only one component in a sequence of nonlinear operations that provides an incomplete picture of a model's 's dependencies (Serrano and Smith 2019; Jain and Wallace [2099] WWegreffe and Pinter 2219), and direct usage of attention weights has not been shown to perform well in vision tasks (Chefer et al. 202T).
Attention-based explanations
Transformers
Gradient-based methods For other deep learning models such as CNNs, gradient-based expla- nations are popular family of approaches. There are many variations on the idea of calculating input gradients, including methods that modify the imput (c.g., SmoothGrad, IntGrad) (Smilkov et al. Sundaaannaa et al.) 2017 Xu et al. 22020), operate at intermediate network layers (GradCAM) (Selvaraju al. [017] ), or design modified backpropagation rules (e.g., LRP, DeepL.ft) (Bach et al.) Shrijuur et al. 2016 Chefer et al., 2021). Although they are efficient to compute for arbitrary network architectures, gradient-based explanations achieve mixed results in quantitative benchmarks, including for object localization and the removal of influential features (Petsiuk et al. 2018 Hooker let al. 2019 Saporta et al. [021 Jethani al. 2021b), and they have been shown to be insensitive to the randomization of model parameters (Adebayo et al. 2018).
Removal-based explanations Finally, removal-based explanations are those that quantify feature importance by explicitly withholding inputs from the model (Covert al. 2021). D. For models that require all features to make predictions, several options for removing features include setting them to default values (Zeiler and Fergus 2014), sampling replacement values (Agarwal and Nguyen 2020) and blurring images (Fong and Vedaldi 2017). These methods work with any model type, but they tend to be slow because they require making many predictions. The simplest approach of removing individual features (known as leave-one-out, Ethayarajh and Jurafsky (0211) is relatively fast, but the computational cost increases we examine more feature subsets.
Shapley values (Shapley 1953) are an influential approach within the removal-based explanation framework. By examining all feature subsets, they provide a nuanced view of each feature's 's influence and satisfy many desirable properties (Lundberg and Lee (201)). They are approximated in practice using methods like TreeSHAP and KernelSHAP (Lundberg et al,, 2020 Covert and Lee 202T). but these approaches either are not applicable or do not scale to large ViT's. Recent work highlighted
2

---

Published as a conference paper at ICLR 2023
connection between attention flow and Shapley values (Ethayarajh and Jurafsky 2021 but this approach is fundamentally different from SHAP (Lundberg and Lee 2017): attention flow treats each feature's influence the model strictly additive, which is computationally convenient but fails to represent feature interactions. Our work instead focuses on the original formulation (undberg and Lee 2017) and aims to make Shapley values based feature removal practical for ViTs.
BACKGROUND
Here, we define notation used throughout the paper and briefly introduce Shapley values.
3.1 NOTATION
Our focus is vision transformers trained for classification tasks, where 2224x224x3 denotes an image and y E (1...., K} denotes the class. We write the image patches as = (xi,...,, Xa), where ViTs typically have x; E 16161163 and 196. The model given by [(x;1) [0. 1] and (x: m) E [0, 1] represents the probability for the yth class. Shapley values involve feature subsets, so use
1 to denote a subset of indices and x, {x, 5; 1} subset of image patches. We also use 0 and E I2 to denote vectors of zeros and ones, and e; E IR is a vector with a one in the ith position and zeros elsewhere. Finally, bold symbols x, y random variables, z, are possible values, and p(x. y) denotes the data distribution.
3.2 SHAPLEY VALUES
Shapley values were developed in game theory for allocating credit in coalitional games (Shapley 1953). A coalitional game is represented by function, and the value for each subset indicates the profit achieved when the corresponding players participate. Given a game with d players, or a set function v : {0, 134 + I., the Shapley values denoted by Φ1 .a..)....)(())) R for each player, and the value (v) for the ith player is defined as follows:
0.(v) = Σ
Σ ::8,=0
d --:1)
-1 (v(s te)s -
Intuitively. (1) represents the change in profit from introducing the ith player, averaged across all possible subsets which can be added. Shapley values are defined in this way to satisfy many reasonable properties: for example, the credits sum to the value when all players participate, players with equivalent contributions receive equal credit, and players with no contribution receive zero credit (Shapley) 1953). These properties make Shapley values attractive in many settings: they have been applied with coalitional games that represent a model's prediction given a subset of features (SHAP) (Strumbelj and Kononenko, 2010 Lundberg and Lee 2017). well as several other use-cases in machine learning (Ghorbani and Zou 2019 2020 Covert et al. 2020).
There are two main challenges when using Shapley values to explain individual predictions (Chen et al. 2022). The first is properly withholding feature information, and we explore how to address this challenge in the VIT context (Section 4). The second is calculating Shapley values efficiently, because their computation scales exponentially with the number of inputs d. Traditionally. they are approximated using sampling-ben based estimators like KemelSHHA (Castro et al., 2009 Strumbelj and Kononenko 20100] Lundberg and Lee 2017), but we build on more efficient learning-basse approach (FastSHAP) recently introduced by Jethani et al. (2021b) (Section 5).
EVALUATING VISION TRANSFORMERS WITH PARTIAL INFORMATION
The basic idea behind Shapley values, as well as other removal- based explanations (Covert et al. 202I), is to evaluate the model with partial feature information and analyze how a prediction changes. Most models need values for all the features to make predictions, so in practice we require a mechanism to represent feature removal. For example, we can set held-out image regions to zero, or we can average the prediction across randomly sampled replacement values.
With vision transformers, the options for removing features are slightly different. Recent work has demonstrated the robustness of ViTs to randomly zeroed pixel values (Naseer et al., 202 but the self-attention operation enables more elegant approach: we can simply ignore tokens for image patches we wish to remove (Jain et al. 2021). We achieve this by masking attention values at each self-attention layer, setting them to large negative value before applying the softmax operation
3

---

Published as a conference paper at ICLR 2023
(see Appendix A). This resembles causal attention masking in transformer language models like GPT-3 (Brown et al. 2020). but we use masking for a different purpose. Alternatively, we could use unique token value in masked language models such as BERT (Devlin al. 2018 which would involve simply setting held-out tokens to the mask value.
Using this attention masking approach, we can evaluate a VIT model f(x; n)) given subsets of image patches, denoted by Xg. However, because these partial inputs represent off-manifold examples, the predictions with partial information may behave as desired. We have two options correct this: 1) we can ensure that the model is trained with random masking, or 2) we can fine-tune the model to encourage sensible behavior with missing patches. The first option is more direct, but it does not allow us to explain models trained without masking. For the latter option, we can create an updated model denoted by g (xx;/) (9) that we fine-tune using the following loss,
min B
Bpt) DKT. (x):)) |I g(xs) ((xx:(8))
(2)
where p(s) is a distribution over subsets. In practice, we sample the cardinality m = I from m O0, d) and then sample m patches uniformly random. Intuitively, jeq. (2) encourages B) to preserve the original model's predictions even with missing features. We use this loss
because it satisfies the desirable property that the optimal model g(xi:8*) outputs the expected prediction given the available information (Covert et al. 202T or
g( 3²*) = =E[f(x;)) 7) x, = =..].
Note that this represents a best-effort prediction, because if f(x; p(y x) then we have g(x:: (8*) p(y x2). Similarly, in the case where f(x;; 1)) is trained directly with random masking, the training process estimates f(x.: 77) R3 p(y xx) (see Appendix B). We refer to the fine-tuned model g(xsi 3) surrogate, following the naming in prior work (rry al. 2020). Whether we use the original model or a version fine-tuned with random masking, our attention masking approach enables us probe how individual predictions change as we remove groups of image patches.
5 LEARNING TO ESTIMATE SHAPLEY VALUES
Given our approach for evaluating ViTs with partial information, we can use Shapley values to identify influential image patches for an input x and class y. This involves evaluating the model with many feature subsets I., we define coalitional game (s) = 9y (x.; 8). Alternatively, if we use a model trained with masking, we can define the coalitional game as Tay (sy
[x(25) 77). Common
Shapley value approximations are based on sampling feature permutations (Castro et al. 2009
Strumbelj and Kononenko 2010) or fitting a weighted least squares model (Lundberg and Lee 2017
Covert and Lee| (021). J, but these can require hundreds or thousands of model evaluations to explain a single prediction Instead, we develop a learning-based estimation approach for ViTs.
Our goal is to obtain an explainer model that estimates Shapley values directly. To do so, we train new vision transformer (x, y; 0) E Ed that outputs approximate Shapley values for input- output pair (I, y) in a single forward pass. Crucially, rather than training the model using a dataset of ground truth Shapley value explanations, we train it by minimizing the following objective,
((0)= ( [1
Uxy (0) s dvvi(xx y; 0))
S.L. (or y: 0) = Uzy (1) Vay (0) 4 (I, v):
< d and Psn (1) = Psn (0) = 0. Intuitively, eq. (4) encourages the explainer model to output feature scores that provide additive approximation for the predictions with partial information, where the predictions are represented by (s) and the additive approximation by Uxy (0) + (x, 0).
where (s) is distribution defined as (s) (1 1) 1 1)! for 0 < 1 s
The loss in eg. was introduced by Jethani et al. (202 and is derived from an optimization-basee characterization of the Shapley value (Charnes et al. 1988). To rigorously justify this training approach, we derive new results that show how this objective controls the Shapley value estimation error. Proofs are in AAppendix D First, we show that the explainer's 'ss loss for a single input is strongly convex in the prediction, a result that implies the existence of unique optimal predictions.
'The number of model evaluations depends on how fast the estimators converge, and we find that KerneISHA requires 100,000 samples converge for ViTs (Appendix H).
4

---

Published as a conference paper at ICLR 2023
Gas pump
Insertion
Deletion
1.0
Probabitty
Probabilty
0.8
True: Gas pump
168
French horn
x1o-2
Insertion
Dele Deletion
Probabitty
Probabilty
0.3
0.4
0.2
0.7
6:0
112
168
True: Abnormal
Abnormal
x10-1
Frobablityy
0.0
28
56
84
112
140
166
196
28
56 84
112
140
168 196
of Inserted Patches
of Deleted Patches
Patches
Attention rollout
VIT Shapley (Durs)
Explanations where our approach identifies relevant information for target and non-target classes. Left: original images from the ImageNette and MURA datasets. Middle left: explanations generated by ViT Shapley for specific classes. Right: probability of the class being explained after the insertion deletion of important patches (higher better for insertion, lower for deletion).
Figure 1:
Lemma 1. For a single input-output pair (I. y), the expected loss under eq. (4) the prediction Ówr(2, y; 9) is p-strongly convex with μ - where H4-1 is the (d 1)th harmonic number.
Next, utilize the strong convexity property from Lemma to prove main result: that the explainer model's loss function upper bounds the distance between the exact and approximated Shapley values. This is notable because do not utilize ground truth values during training.
Theorem For a model (var(x. y: 0) whose predictions satisfy the constraint in eq. (4) the objective value L(0) upper bounds the Shapley value estimation error as follows,
O (x 0) <(vxy))l s ((H))
- -++)
Ep(xy)
where L represents the loss achieved by the exact Shapley values.
This shows that our objective is viable approach for training without exact Shapley values, because optimizing eq. (4) minimizes an upper bound on the estimation error. In other words, if we can iteratively optimize the explainer model so that its loss approaches the optimum obtained by the exact Shapley values (C( + C"), our estimation error will go to zero.
In practice, train the explainer model dvar(x, y; using stochastie gradient descent, and several other steps are important during training. First, we normalize the explainer's unconstrained predictions in order satisfy the objective's 's constraint eq. (4) this ensures that the Shapley value's 's efficiency property holds (Shapley 1953). Next, rather than training the explainer from scratch, we fine-tune an existing model that can either the original classifier
ViT pre-trained on different supervised or self-super learning task (Touvron et al. 2021 He et al.) 2021): ViTs are more difficult to train than convolutional networks, and we find that fine-tuning is important to train the explainer effectively (Table 3). Finally, we simplify the architecture by estimating Shapley values for all classes simultaneously. Our training approach is described in more detail Appendix C
By using VIT to estimate Shapley values, we model the true explanation function and learn rich representations that capture not only which class is represented, but where key information is located. And by fine-tuning an existing model, we allow the explainer to re-use visual features that were informative for other challenging tasks. Ultimately, the explainer cannot guarantee exact Shapley values, but approximation algorithm can; instead, it offers favorable trade-off between accuracy and efficiency, and we find empirically that this approach offers a powerful alternative to the methods currently used for ViTs.

---

Published as a conference paper at ICLR 2023
ImageNette
MURA
2.5
2.0
2.0
1.5
1.5
1.0
1.0
112
56
of Deleted Patches
4 Deleted Patches
Masking
Zeros imput
Masking (Post-softrnax)
Zeros at am embbeddiig
Random imputation
Masking (fine-tuned)
Mask token (Finettuned)
ViT predictions given partial information. We delete patches at random using several removal mechanisms, and then measure the quality of the resulting predictions via the KL divergence relative to the original, full-image predictions (lower is better).
Figure 2:
6 EXPERIMENTS
We now demonstrate the effectiveness of our approach, termed ViT Shapley First, we evaluate attention masking for handling held-out patches in ViTs (Section 6.1). Next, we compare explanations from ViT Shapley to several existing methods (Section 6.2p. Our baselines include attention-, gradient- and removal-based explanations, and we compare these methods via several metrics for explanation quality, including insertion/deletion of important features (Petsiuk et al. 2018), sensitivity-n (Ancona 2019).
al. 2018), faithfulness (Bhatt et al. 2021 and ROAR (Hooker et al.
Our experiments are based on three image datasets: ImageNette, a natural image dataset consisting of ten ImageNet classes (Howard and Gugger 2020 Deng et al., 2200)). MURA, a medical image dataset of musculoskeletal radiographs classified as normal or abnormal (Rajpurkar et al. (2017), and the Oxford IIlT-Pets dataset, which has 37 classes (Parkhi et al., 2012). See T foor example images. The main text shows results for ImageNette and MURA, and Pets results are in|Appendix H We use ViT-Base models (Wightman) 2019) as classifiers for all datasets, unless otherwise specified.
6.1 EVALUATING IMAGE PATCH REMOVAL
Our initial experiments test whether attention masking is effective for handling held-out image patches. We fine-tuned the classifiers for each dataset following the procedure described in Section 4| and also tested several approaches without performing any fine-tuning: attention masking. attention masking applied after the softmax operation (how dropout is often implemented for ViTs, Wightman 2019), setting input patches zero (Naseer et al. (02T), setting token embeddings to zero, and replacing with random patches from the dataset. Finally, we performed identical fine-tuning while replacing input patches with zeros, which is equivalent to introducing fixed mask token.
As measure of how well missing patches are handled, we caleulated the KL divergence relative to the full-image predictions as random patches are removed. This can be interpreted as a divergence measure between the masked predictions and the predictions with patches marginalized out (see Appendix B), or how close we are to correctly removing patch information. The metric calculated with randomly generated patch subsets, and it represents whether the model makes reasonable predictions given partial inputs. Similar results for top-1 accuracy are in Appendix H
Figure shows the results. Most methods perform well with <25% of patches missing, leading to only small increases in KL divergence. This is especially true for ImageNette, where large objects make the model more robust to missing patches. However, the methods with no fine-tuning begin diverge as larger numbers of patches removed and the partial inputs become increasingly off-manifold. Thus, fine-tuning becomes necessary to properly account for partial inputs as more patches are removed.
For all datasets, we find that fine-tuning with either attention masking input patches set provide comparable performance, and that these perform best across all numbers of patches. This means that fine-tuning makes attention masking significantly more effective for marginalizing out missing patches, and these results suggest that training ViTs with held-out tokens may be necessary to enable robustness to partial information. As prior work suggests, properly handling held-out
nleel ///vviiie

---

Published as a conference paper at ICLR 2023
Table 1: Evaluating ViT Shapley using standard explanation metrics, with explanations calculated for the target class only. Methods that fail to outperform the random baseline are shown in gray, and the best results are shown in bold (accounting for 95% confidence intervals).
ImageNette
MURA
Ins. (1)
Del. (4)
Faith. (1)
Ins. (1)
Del. (U)
Faith. (1)
Attention last
0.962 (0.004)
0.793 (0.013)
0.694 (0.015)
0.890 (0.010)
0,592 (0.013)
0.635 (0.016)
Attention rollout
0.938 (0.005)
0.880 (0.010)
0.704 (0.015)
0.845 (0.011)
0.692 (0.014)
0.618 (0.016)
GradCAM
0.914 (0.006)
0.937 (0.008)
0.680 (0.015)
0.899 (0.009)
0.681 (0.015)
0.631 (0.016)
IntGrad
0.967 (0.004)
0.930 (0.008)
0,403 (0.024)
0.897 (0.010)
0.796 (0.015)
0.201 (0.022)
Vanilla
0,950 (0.004)
0.808 (0.013)
0.703 (0.015)
0.890 (0.010)
0.537 (0.014)
0.629 (0.016)
SmoothGrad
0,947 (0.005)
0.942 (0.006)
0.703 (0.015)
0.870(00.010)
0.813 (0.011)
0.617 (0.016)
VarGrad
0,949 (0.005)
0,946 (0.005)
0.700 (0.015)
0.857 (0.011)
0.823 (0.011)
0.615 (0.016)
LRP
0.967 (0.004)
0.779 (0.014)
0.705 (0.015)
0,900 (0.009)
0,551 (0.013)
0.646 (0.016)
Leave-one-out
0.969 (0.002)
0.917 (0.010)
0.140 (0.040)
0.926 (0.008)
0.694 (0.017)
0.308 (0.032)
RISE
0.977 (0.001)
0.860 (0.014)
0.704 (0.015)
0,957 (0.004)
0.573 (0.018)
0.618 (0.016)
VIT Shapley
0.985 (0.002)
0.691 (0.014)
0.711 (0.015)
0.971 (0.002)
0.307 (0.013)
0.707 (0.013)
Random
0.951 (0.005)
0.951 (0.005)
0.849 (0.010)
0.847 (0.010)
information is crucial for generating informative explanations (Frye et al. 2020 Covert et al. 2021 so the remainder of our experiments proceed with the fine-tuned attention masking approach.
6.2 EVALUATING EXPLANATION ACCURACY
Next, we implemented ViT Shapley by training explainer models for both datasets. We used the fine- tuned classifiers from SScction to handle partial information, and we used the ViT-Base architecture with extra output layers to generate Shapley values for all patches. The explainer models were trained by optimizing leq. (4) using stochastic gradient descent (see details in [Appedixx C}, and once trained, the explainer outputs approximate Shapley values in a single forward pass (Figure I).
As comparisons for ViT Shapley, we considered large number of baselines. For attention-based methods, we attention rollout and the last layer's attention directed to the class token (Abnar and Zuidema 2020). Similar to prior work (Chefer et al., 2021), we did not use attention flow due to the computational cost. Next, for gradient-based methods, we use Vanilla Gradients (Simonyan et al. 2013), IntGrad (Sundararajan et al. (2077). SmoothGrad (Smilkov et al., (017)) VarGrad (Hooker et al. 2019), LRP (Chefer et al. (2221) and GradCAM (Selvaraju al. 2217). For removal-based methods, we use the leave-one-out approach (Zeiler and Fergus] 2014) and RISE (Petsiuk et al.,
2018). Appendix F describss the baselines in more detail, including how several were modified to provide patch-level results, and Appendix shsows the running time for each method.
Given our set of baselines, we used several metrics to evaluate ViT Shapley. Evaluating explanation accuracy is difficult when the true importance is not known a priori, so we rely metrics that test how removing (un)important features affects a model's predictions. Intuitively. removing influential features for particular class should reduce the class probability, and removing non-influential features should not affect or even increase the class probability. Removal-basse explanations are implicitly related to such metrics (Covert et al. 202T). but attention- and gradient-based methods may be hoped to provide strong performance with lower computational cost.
First, we implemented the widely used insertion and deletion metrics (Petsiuk et al. 2018). For these, we generate predictions while inserting/removing features in order of most to least important, and we then evaluate the area under the curve of prediction probabilities (see (siuue T). Here, we average the results across 1,000 images for their true class. We use random test set images for ImageNette, and for MURA we use test examples that were classified as abnormal because these are more important in practice. When removing information, we use the fine-tuned classifier because this represents the closest approximation to properly removing information from the model (Section 4).
Table I displays the results, and we find that ViT Shapley offers the best performance on both datasets. RISE and LRP tend to be the most competitive baselines. and perhaps surprisingly, certain other methods fail to outperform a random baseline (GradCAM, SmoothGrad, VarGrad). The basclines are sometimes competitive with ViT Shapley on insertion, but the gap for the deletion metric is larger.
7

---

Published as a conference paper at ICLR 2023
ImageNette
MURA
0.7
0.6
0.6
Comniittio
0.4
Commiston
0.4
0.2
0.2
0.0
0.0
-0.1
-0.1
28
56
84
112
140
160
20
56
04
112
140
168
Patches
SmoothGrad
Leave- out
Attention rollout
IntGrad
VarGrad
RISE
Shapley
Figure 3: Sensitivity-n evaluation for different subset sizes. The metric is generated separately for a range of subset sizes, whereas faithfulness is calculated jointly subsets of all sizes.
Practically, this means that ViT Shapley identifies important features that quickly drive the prediction towards a given class, and that quickly reduce the prediction probability when deleted.
Next, we modified these metrics to address common issue with model explanations: that their results are not specific each class (Rudin) 2019). ViT Shapley produces separate ex- planations for each class, so can identify relevant patches even for non- target classes (see Figure T). [Talle> shows insertion/deletion results av- eraged across non-target classes for ImageNette. The attention-based methods do not produce class-specific explanations, and the remaining base- lines generally provide poor results. Empirically, this is because the planations are often similar across classes (see[Appendix 1b. ViT Shapley performs best, particularly on inser- tion, and RISE is the best- performeronnnng baseline. Appendix H shows results for MURA, as well as the curves used to calculate these results.
Table 2: Evaluating ViT Shapley for explaining non-target classes. Methods that fail to outperform the random baseline are shown gray, and the best results are shown in bold (accounting for 95% confidence intervals).
ImageNette
Ins. (t)
Del. (1)
Faith. (1)
Attention last
Attention rollout
GradCAM
0.021 (0.002)
0.005 (0.000)
-0.672 (0.015)
IniGrad
0.008 (0.001)
0.004 (0.000)
0.294 (0.022)
Vanilla
0.006 (0.001)
0.020 (0.001)
-06822 (0.015)
SmoothGrad
0.006 (0.001)
0.006 (0.001)
-0.683 (0.015)
VarGrad
0,006 (0.001)
0.006 (0.001)
-0,680 (0.015)
LRP
0.004 (0.001)
0.022 (0.001)
-0.680 (0.015)
Leave-one-out
0.013 (0.002)
0.003 (0.000)
-0.017 (0.028)
RISE
0.023 (0.003)
0.002 (0.000)
-0.681 (0.015)
VIT Shapley
0.093 (0.004)
0.001 (0.000)
0.672 (0.014)
Random
0.005 (0.001)
(0.001)
The insertion and deletion metrics only test importance rankings, so we require other metrics to test the specific attribution values. Sensitivity-n (Ancona et al., (2188) was proposed for this purpose, and measures whether attributions correlate with the impact on a model's prediction when a feature removed. The correlation is typically calculated across subsets of fixed size, and then averaged across many predictions. Faithfulness (Bhatt et al., 2021) is similar metric where the correlation is calculated across subsets of all sizes.
show faithfulness results. Among the baselines, RISE and LRP remain most competitive, but VIT Shapley again performs best for both datasets, Figure 3 shows sensitivity-n results caleulated across range of subset sizes. Leave-one-out naturally performs best for large subset sizes, but ViT Shapley performs the best overall, particularly with smaller subsets. The sensitivity-n results focus the target class, but but Appeedix H shows results for non-target classes where VIT Shapley's advantage over many baselines (including LRP) is even larger.
Table and Table 2
Finally, we performed an evaluation inspired by ROAR Hooker et al. 2019), which tests how a model's accuracy degrades as important features are removed. ROAR suggests retraining with masked inputs, but this unnecessary here because the fine-tuned classifier is designed to handle held-out patches. We therefore generated multiple versions of the metric. First, evaluated accuracy while using the tne-tuned classifier to handle masked patches. Second, we repeated the evaluation using a separate evaluator model trained directly with held-out patches, similar to EVAL--X (Jethani et al.
8

---

Published as a conference paper at ICLR 2023
Fine-tuned Classifier
Separate Evaluator
1.0
1.0
Accurat)
0.6
Accuracy
0.6
0.4
0.4
0.2
0.2
0.0
112
196
0.0
56
112
140
Deleted Patches
Deleted Patches
Masked Retraining w/ Positional Embed.
Masked Retraining w/o Positional Embed.
1.0
1.0
0.8
Accursyy
0.6
0.2
0.2
0.0
28
56
84
112
140
160
196
0.0
28
56
04
112
140
168
196
Deleted Patches
Deleted Patches
Attention last
GradCAM
SmoothGrad
Leave-one-out
Random
Attention rollout
IntErad
VarGrad
RISE
Vanilla
Shapley
Figure 4: ImageNette accuracy when removing features order of their importance, run with four evaluation strategies: fine-tuned classifier, separate evaluator, masked retraining, and masked retraining without positional embeddings.
2021a). Third, we performed masked retraining as described by ROAR. The first version represents the original classifier's best-effort prediction, and the second is best-effort prediction disconnected from the original model; masked retraining is similar, but the retrained model can exploit information communicated by the masking, such the shape and position of the removed object.
Figure 4 shows the results when removing important patches. VIT Shapley consistently outperforms the baselines across the first two versions of the metric, yielding faster degradation when important patches are removed. ViT Shapley also performs best when inserting important patches, yielding a faster increase in accuracy (Appendix H). It is outperformed by several baselines with masked retraining in the deletion direction (Figure bottom left), but we find that this is likely due to spatial information leaked by VIT Shapley's 's deleted patches; indeed, when we retrained without positional embeddings, we found that ViT Shapley achieved the fastest degradation with a small number of deleted patches (Figure 4 botton right). Interestingly, positional embeddings in ViTs offer a unique approach to alleviate ROAR's ' known information leakage issue (Jethani et al.) 2021a)
In addition to these experiments, we include many further results in the supplement (Appendix H). First, we observe similar benefits for ViT Shapley when using the Oxford-I1IT Pets dataset. Next, regarding the choice of architecture, we observe consistent results when replacing ViT-Base with ViT-Tiny, -Small or Large. We also replicate our results when using a classifier trained directly with random masking, an approach discussed in prior work to accommodate partial input information (Covert et al. (02T). We then generated metries comparing ViT Shapley's approximation quality with KernelSHAP (Lundberg and Lee| 2017), and we found that ViT Shapley's accuracy is equivalent to running KernelSHAP for roughly 120,000 model evaluations (Appendix H). Lastly, we provide qualitative examples in inpppendix I including comparisons with the baselines. Overall, these results show that ViT Shapley is a practical and effective approach for explaining ViT predictions.
7 CONCLUSION
In this work, we developed learning- based approach to generate Shapley values for VITs. Our approach involves training a separate explainer model using an objective that does not require ground truth supervision, and VIT Shapley outperforms a variety of attention- and gradient-based methods across a range of accuracy metrics. Our experiments provide empirical support for Shapley values as an alternative to attention-basse directions involve extending ViT Shapley to NLP models, operating with arbitrary token groups or superpixels, and accelerating or otherwise improving the explainer model's ''s training.
approaches, which remain popular for transformer models. Future

---

Published as a conference paper at ICLR 2023
ACKNOW
We thank Mukund Sudarshan, Neil Jethani, Chester Holtz and the Lee Lab for helpful discussions. This work funded by NSF DBI-1552309 and DBI-1759487, NIH R35-GM-128638 and R01- NIA-AG-061132.
BIBLIOGRAPHY
Abnar, S. and Zuidema, W. (2020). Quantifying attention flow in transformers. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 4190-4197.
Adebayo. J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. (2018). Sanity checks for saliency maps. Advances in Neural Information Processing Systems, 31.
Agarwal, C. and Nguyen, A. (2020). Explaining image classifiers by removing input features using generative models. In Proceedings of the Asian Conference Computer Vision.
Ancona, M., Ceolini, E., Öztireli, C., and Gross, M. (2018). Towards better understanding of gradient- based attribution methods for deep neural networks. International Conference on Learning Representations.
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One, 10(7):e0130140.
Beyer, L., Zhai, X., and Kolesnikov, A. (2022). Better plain VIT baselines for ImageNet-1k, arXiv preprint ar arXiv:2205.0 500000.........
Bhatt, U., Weller, A., and Moura. M. (2021). Evaluating and aggregating feature- based model explanations. In International Conference on International Joint Conferences on Artificial Intelli- gence.
Boyd, S., Boyd, S. P., and Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 13:1777-1901.
Castro, J., Gómez, D., and Tejada, J. (2009). Polynomial calculation of the Shapley value based on sampling. Computers & Operations Research, 126
Charnes, A., Golany, B., Keane, M., and Rousseau, J. (1988). Extremal principle solutions of games in characteristic function form: core, Chebychev and Shapley value generalizations. In Econometrics o Planning and Efficiency, pages 123-133, Springer.
Chefer, H., Gur, S., and Wolf, L. (2021). Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 782-791.
Chen, H., Covert, I. C., Lundberg, S. M., and Lee, S.-I. (2022). Algorithms to estimate shapley value feature attributions. arXiv preprint arXiv:2207.07600
Clark, K., Khandelwal, U., Levy, O., and Manning, C. D. (2019). What does BERT look at? an analysis of BERT's attention. arXiv preprint 17Xiv:1906.04441
Covert, I. and Lee, S.-I. (2021). Improving KemelSHAA: Practical Shapley value estimation using linear regression. In International Conference on Artificial Intelligence and Statistics, pages 3457-3465. PMLR.
Covert, I., Lundherg. S., and Lee, (2021). Explaining by removing: unified framework for model explanation. Journal of Machine Learning Research, 22(209):1-90.
Covert, I., Lundberg, S. M., and Lee, S.-I. (2020). Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems, 33:17212- 17223.
10

---

Published as a conference paper at ICLR 2023
Deng, J., Dong. W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large- scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248-28882 255. IEEE
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1XX.:1810
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Ethayarajh, K. and Jurafsky, D. (2021). Attention flows are Shapley value explanations. arXiv preprint wwXiv:2105.14652
Fong, R. C. and Vedaldi, A. (2017). Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision, pages 3429-3437.
Frye, C., de Mijolla, D., Begley, T., Cowton, L., Stanley, M., and Feige, (2020). Shapley explainability on the data manifold. In International Conference on Learning Representations.
Ghorbani, A. and Zou, J. (2019). Data Shapley: Equitable valuation of data for machine learning, In International Conference on Machine Learning, pages 2242-2251. PMLR.
Ghorbani, A. and Zou, J. Y. (2020). Neuron Shapley: Discovering the responsible neurons. Advances in Neural Information Processing Systems, 33:5922-5932.
Gildenblat, J. and contributors (2021). PyTorch library for CAM methods. https://qithu:
Com/
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, (2021). Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377
Hooker, S., Erhan, D., Kindermans, P.-., and Kim, B. (2019). A benchmark for interpretability methods deep neural networks. Advances Neural Information Processing Systems, 32.
Howard, J. and Gugger, S. (2020). FastAl: A layered API for deep learning. Information, 11(2):108.
Jain, S., Salman, H., Wong, E.,, Zhang, P., Vineet, V., Vemprala, S., and Madry, A. (2021). Missingness bias in model debugging. In International Conference Learning Representations.
Jain, S. and Wallace, B. C. (2019). Attention is not explanation. arXiv preprint
10186.
Jethani, N., Sudarshan, M., Aphinyanaphongs. Y., and Ranganath, R. (2021a). Have we learned to explain?: How interpretability methods can learn to encode predictions their interpretations. In International Conference Artificial Intelligence and Statistics, pages 1459-1467. PMLR.
Jethani, N., Sudarshan, M., Covert, I. C., Lee, S.-I., and Ranganath, R. (2021b). Fast Real-time Shapley value estimation. In International Conference on Learning Representations.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., et al. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583-58,
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International Conference Machine Learning, pages 2668-2677 PMLR.
Kokhlikyan, N., Miglani, V., Martin, M., Wang, E.. Alsallakh, B., Reynolds, J., Melnikov, A., Kliushk- ina, N., Araya, C., Yan, S., et al. (2020). Captum: A unified and generic model interpretability library for PyTorch. arXiv preprint arXiv:2009.07896.
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang. Z., Lin, S., and Guo. B. (2021). Swin trans- former: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10012-10022.
11

---

Published as a conference paper at ICLR 2023
Loshchilov, I. and Hutter, F. (2018). Decoupled weight decay regularization. In International Conference on Learning Representations.
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., and Lee, S.-I. (2020). From local explanations to global understanding with explainable Al for trees. Nature Machine Intelligence, 2(1):2522-5839.
Lundberg, S. M. and Lee, S.-1. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems,
Naseer, M. M., Ranasinghe, K., Khan, H., Hayat, M., Shahbaz Khan, F., and Yang, M.-H. (2021). Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34.
Olah, C., Mordvintsev, A., and Schubert, L. (2017). Feature visualization. Distill, 2(1
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. (2012). Cats and dogs. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-3505, IEEE.
Petsiuk, V., Das, A., and Saenko, K. (2018). RISE: Randomized input sampling for explanation of black-box models. arXiv preprint TrXXiv:8066.044
Rajpurkar, P., Irvin, J., Bagul, A.. Ding, D., Duan, T., Mehta, H., Yang, B., Zhu, K., Laird, D.,
Ball, R. L., et al. (2017). MURA: Large dataset for abnormality detection in musculoskeletal radiographs. arXiv preprint arXiv:1712.069)
Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144.
Rogers, A., Kovaleva, O., and Rumshisky, A. (2020). A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866 866.
Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In International Conference Medical Image Computing and Computer- Assisted Intervention, pages 234-241. Springer.
Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-2155
Ruiz, L. M., Valenciano, F., and Zarzuelo, J. M. (1998). The family of least square values for transferable utility games. Games and Economic Behavior, 24(1 24(1-2):109-130
Saporta, A., Gui, X., Agrawal, A., Pareck, A., Truong, S. Q., Nguyen, C. D., Ngo, V.-D., Seckins, J., Blankenberg, F. G., Ng. A. Y., et al. (2021). Benchmarking saliency methods for chest x-ray interpretation. medRxiv, pages 2021-02.
Sclvaraju, R. R., Cogswell, M., Das, A.. Vedantam, R., Parikh, D., and Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference Computer Vision, pages 618 6266.
Serrano, and Smith, N. A. (2019). attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-2951.
Shapley, L. (1953). A value for n-person games. Contributions to the Theory of Games, 2(28):307- 317.
Shrikumar, A., Greenside, P., Shcherbina, A., and Kundaje, A. (2016). Not just black box: Learning important features through propagating activation differencee. arXiv preprint arXiv:1605.01715.
Simon, G. and Vincent, T. (2020). A projected stochastic gradient algorithm for estimating Shapley value applied in attribute importance. In International Cross- -Domain Conference for Machine Learning and Knowledge Extraction, pages 97-115. Springer.
12

---

Published as a conference paper at ICLR 2023
Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualis- ing image classification models and saliency maps. arXiv preprint 132.v21312.6034
Smilkov, D., Thorat, N., Kim, B., Viégas, F,, and Wattenberg, M. (2017). SmoothGrad: Removing noise by adding noise. arXiv preprint 17Xxiv:7000.000
Strumbelj, E. and Kononenko, I. (2010). An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 11:1-18.
Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. (2021). Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, E.., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
Vig, J., Madani, A., Varshney, L. R., Xiong. C.,, Rajani, N., et al. (2020). BERTology meets biology: Interpreting attention in protein language models. In International Conference Learning Representations.
Wang. Y., Mohamed, A., Le, D., Liu, C., Xiao, A., Mahadeokar, J., Huang, H., Tjandra, A., Zhang, X., Zhang, F., al. (2020). Transformer-based acoustic modeling for hybrid speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6874-6878. IEEE.
Waskom, M. (2021). Seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021.
Wiegreffe, S. and Pinter, Y. (2019). Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joins Conference Natural Language Processing (EMNLP-IJCNLP), pages 11-20.
Wightman, R. (2019). PyTorch image models.
https://github cww. www/rwiggtnttt
pytorch image models
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arXiv preprint arXiv:2203.0544 05482.
Xu, S., Venugopalan, S., and Sundararajan, M. (2020). Attribution in scale and space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9680-9689.
Zeiler, M. D. and Fergus, R. (2014). Visualizing and understanding convolutional networks. In European Conference on Computer Vision, pages 818-833. Springer.
Zhou, Y., Booth, S., Ribeiro, M. T., and Shah, J. (2022). Do feature attribution methods correctly attribute features? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9623-9633.
13

---

Published as a conference paper at ICLR 2023
A ATTENTION MASKING
This section describes our attention masking approach in detail. First, recall that ViTs use query-key- value self-attention (Vaswani al. 2017 Dosovitskiy et al. 2020), which accepts a of input tokens and produces weighted sum of learned token values. Given an input E RRdx and parameters U Ugton E IR 1xh'' we compute the self attention output (A) for single head as follows:
Q, K, VJ - zU
(5)
A softmax QK /VN)
(6)
SA(z) AV.
(7)
In multihead self-attention, we perform this operation in parallel over k attention heads and project the concatenated outputs. Denoting each head's output (z,(z) and the projection matrix as U Umsa E RR& the multihead self-attention output MSA(z)
MSA(z) (2) (
(8)
Multihead self-attention can operate with any number tokens, so given a subset s E 1] and an input x, evaluate VIT using only tokens for the patches x2 {x; 8; 1). However, for implementation purposes is preferable to maintain the same number of tokens within a minibatch. We therefore provide all tokens the model and achieve the same effect using attention masking. Our exact approach is described below.
Let z Rd represent the full token for an input x and let s be a subset. At each self- attention layer, we construct a mask matrix S = [s. E {0, jaxd and calculate the masked self-attention output SA (z, s) as follows:
-S) (1-S) A = (Q -(1 -1--))
oo)/vh1)
(9)
SA(z, s) = AV.
(10)
The masked multihead self-attention output is then calculated similarly to the original version:
MSA(z, s) = (z, s). SAA( Umaa
(11)
Due to the masking in eq. (9) each output token in MSA(z, s) is guaranteed not to attend to tokens from X1-x {x, s; = 03. We use masked self-attention in all layers the network, so that the tokens for Xg remain invariant to those for X1-s throughout the entire model, including after the layer norm and fully-com fully-connected layers. When the final prediction is calculated using the class token, the output is equivalent to using only the tokens for x,. If the final prediction is instead produced using global average pooling (Beyer et al. 2022), we can modify the average to account only for tokens we wish include.
B MASKED TRAINING
In this section, we provide proofs to justify training a VIT classifier with held-out tokens, either as part of the original training or part of a post-hoc fine-tuning procedure (the surrogate model training described in Section 4h. Our proofs are similar to those in prior work that discusses marginalizing out features using their conditional distribution (Covert al. 2021).
First, consider model trained directly with masking. Given a subset distribution p(s) and the data distribution p(x, y), we can train model f(xx; with cross-entropy loss and random masking by minimizing the following:
min Ep(x.y) log fy (xx: ()]:
(12)
14

---

Published as a conference paper at ICLR 2023
To understand the global optimizer for this loss function, consider the expected loss for the prediction given a fixed model input Zz:
1z),) log fy ()] Bp(ylr,)(- log fy ())
(13)
The expression in en. (13) is equal to the KL. divergence DKL(p(y x.) f(x up to constant value, the prediction that minimizes this loss is p(y F.). For any subset [0,1) 1)dd where p(s) > 0, we then have the following result for the model f(xs:t)) that minimizes eq. (12)
(u(aa))))) ply x.)) a.e. in p(x).
Intuitively, this means that training the original model with masking estimates f f(x;: = p(y x,).
In practice, we use subset distribution p(s) where p(s) 0 for all {0,1)d: we set p(s) by sampling the cardinality uniformly random and then sampling the members, which equivalent to defining p(s)
p(s)
( ..).)(+))
Alternatively, we can use model f(x; 17) trained without masking and fine-tune to better handle held-out features. In our case, this yields surrogate model (Frye et al. 2020) denoted g(xx; that we fine-tune by minimizing the following loss:
min D
Ex(x) DKL (f(x;) ll 9(xx;
(14)
To understand the global optimizer for the above loss, we can again consider the expected loss given fixed input Nz:
Epx(s Il.)) DKL. (f(x; g( B)) DKL.
D (T[[((x;))
xx)] |l g( ()) + const. (15)
The distribution that minimizes this loss is the expected output given the available features, ([f();))) x.). By the same argument presented above, we then have the following result for the optimal surrogate g(x;; (3)) that minimizes eq. (14)
E[f(x: x.] a.e. in p(x).
Notice that if the initial model is optimal, or f(x; m) ply x), then the optimal surrogate satisfies g(x,: (*)) = p(y xx).
C EXPLAINER TRAINING APPROACH
In this section, we summarize our approach for training the explainer model and describe several design choices. Recall that the explainer vision transformer ςvvr(x,, (x, ; 0) Rd that we train by minimizing the following loss:
min ep (o 0
(Uxy (s) - Uxy (0) - -T (x,
s.t. (1r y3 Usy (1) (2y(0) A (x.
15

---

Published as a conference paper at ICLR 2023
Additive efficient normalization The constraint the explainer predictions is necessary to ensure that the global optimizer outputs the exact Shapley values, and we use the same approach as prior work to enforce this constraint (Jethani et al. 2021b). We allow the model to make unconstrained predictions that we then modify using the following transformation:
Φvi 0) dv vs 0) Vay (1) Uxy (0) 1 Φvi (x, y;
(16)
This operation is known as the additive efficient normalization (Ruiz et al. 1998). and it can be interpreted as projecting the predictions onto the hyperplane where the constraint holds (Jethani et al., 2021b). We implement it as an output activation function, similar to how softmax is used to ensure valid probabilistic predictions for classification models.
Subset distribution The specific distribution Pss ) in our loss function is motivated by the Shapley value's weighted least squares characterization (Charnes et 1988 Lundberg and Lee 2017). This result states that the Shapley values for a game v {0, 1ja + I are the solution to the following optimization problem:
Σ T sea ( (1T (d - sTs) (u(s) -( - Td)
d d-1
min PERd
s.t. 1 1 v1)=v(1) v())
We obtain (s) by normalizing the weighting term in the summation, and doing so yields distribution Psa (s) x (1"s 1) 1 1)! for 0 < 1ss and Psa(1) Psa(O) To sample from (s), we calculate the probability mass each cardinality, sample a cardinality m from this multinomial distribution, and then select m indices uniformly random.
Stochastic gradient descent As is common in deep learning, we optimize our objective using stochastic gradients rather than exact gradients. To estimate our objective, we require set tuples yx s) that obtain follows. First, sample input p(x). Next, sample multiple subsets ~ (s). To reduce gradient variance, we use the paired sampling trick (Covert and Lee 202T) and pair each subset with its complement 1 Then, use explainer to output Shapley values simultaneously for all classes E {1...., K}. J. Finally, we minibatch this procedure across multiple inputs and calculate our loss across the resulting set of tuples (x. y)s)
Fine-tuning Rather than training the ViT explainer from scratch, we find that fine-tuning an existing model leads to better performance. This is consistent with recent work that finds ViTs challenging to train from scratch (Dosovitskiy et al., 2020). We have several options for initializing the explainer: we use 1) the original classifier [xx v), 2) the fine-tuned classifier g(x;; (). or 3) a ViT pre-trained on another task. We treat this choice as hyperparameter, selecting the initialization that yields the best performance. We also experiment with freezing certain layers in the model, but we find that training all the parameters leads to the best performance.
Explainer architecture We use standard ViT architectures for the explainer. These typically append class token to the set of image tokens (Dosovitskiy et al.) 2020). and we find it beneficial to preserve this token in pre-trained architectures even though unnecessary for Shapley value estimation. We require a separate output head from the pre-trained architecture, and our explainer head consists of one additional self-attention block followed by three fully-connected layers. Each image patch yields one Shapley value estimate per class, and we discard the results for the class token.
Hyperparameter tuning To select hyperparaters related to the learning rate, initialization and architecture, we use a pommcomptted set tuples (x, y. s) to calculate a validation loss. These are generated using imputs that were not used for training, an unbiased estimator of the objective function. This approach serves as an inexpensive alternative to comparing with ground truth Shapley values for large number of samples.
validation loss can be interpreted as
16

---

Published as a conference paper at ICLR 2023
Training pseudocode Algorithm 1 shows a simplified version of our training algorithm, without minibatching, sampling multiple subsets 8, or parallelizing across the classes y.
Algorithm 1: Explainer training
Input: Coalitional game Fxy (s), learning rate Output: Explainer dovir (x, y; 0)
initialize (x, y:
while not converged do sample (x, y) ~ p(x, y), 8 Psh (s)
predict Φ dvi
set ó + o d' (Uey (1) Vxy (0) 1 T calculate L + (Vay (s) Tay (0) - ②)
update 0 -- 0 - a VoL
end
C.I HYPERPARAMETER CHOICES
When training the original classifier and fine-tuned classifier models, we used a learning rate of 10-5 and trained for 25 epochs and 50 epochs. respectively. The MURA classifier was trained with an upweighted loss for negative examples to account for class imbalance. The best model was selected based on the validation criterion, where we used 0-1 accuracy for ImageNette and Oxford-IIIT Pets, and Cohen Kappa for MURA.
When training the explainer model, used the same ViT-Base architecture the original classifier and initialized using the tine-tuned classifier, as this gave the best results. We used the AdamW optimizer (Loshchilov and Hutter, 2018) with cosine learning rate schedule and maximum learning rate of 10-2, and we trained the model for 100 epochs, selecting the best model based on the validation loss. We used standard data augmentation steps: random resized crops, vertical flips, horizontal flips, and color jittering including brightness, contrast, saturation, and hue. We used minibatches of size 64 with 32 subset samples per æ sample, and we found that using tanh nonlinearity on the explainer predictions was helpful to stabilize training.
Finally, we modified the ViT architecture to output Shapley values for each token and each class: we removed the classification head and added an extra attention layer, followed by three fully-connected layers with width times the embedding dimension, and we fine-tuned the entire ViT' backbone. These choices were determined by an ablation study with different model configurations, and we also compared with training training the ViT from scratch and training a separate U-Net explainer model (Ronneberger et al. 2015) (see Table 3)
Table 3: Ablation experiments for VIT Shapley explainer architecture on the ImageNette dataset, with and without fine-tuning.
Fine-tuning config.
Extra attention block
Frozen backbone
Val loss
Test loss
A
False
True
4.332
4.351
False
False
4.331
4.351
DD
Truc
True
4.319
4.339
True
False
4.309
4.318
VIT trained from scratch
4.331
4.341
U-Net trained from scratch
4.332
4.338
We used a machine with 2 GeForce RTX 2080Ti GPUs to train the explainer model, and due to GPU memory constraints loaded the classifier and explainer to separate GPUs and trained with mixed precision using PyTorch Lightning Training the explainer model required roughly 19 hours for the ImageNette dataset and 60 hours for the MURA dataset.
17

---

Published as a conference paper at ICLR 2023
D PROOFS
Here, we re-state and prove our main results from Section 5
Lemma 1. For a single input-ouiput pair (x, v). the expected loss under eq. (4) for the predic-
tion dor y, 0) is pu-strongly convex with = Hz-1 where Ha-1 the (d 1) 1)th harmonic number.
Proof. For an input-output pair (ar, y), the expected loss for the prediction Φ = (1, O) under our objective is given by
(x) = Φ (s) J 20" Bpaatt) s(U2y(() Uzyy (0)) + Epsi(e) (Vey(() May (0))
This is quadratic function of & with its Hessian given by
Viltay = 2 Epp.(s) js8 [s
The convexity of (o) is determined by the Hessian's 's cigenvalues, and its entries can be derived from the subset distribution Psa (s): see similar results in Simon and Vincent (2020) and Covert and Lee (202I). The distribution assigns equal probability to subsets of equal cardinality, so we define k. Specifically, we have
the shorthand notation pi: Psh(s) for such that
d d-1
d-1 d-1
Pk Q
and
k() - k)
kld - k)
k=1
We can then write A = E ( (ss] J and derive its entries as follows:
d
(d-1) 1)
Ag = 4
= 1) Σ
Pk
k -
k=1
d-1 (d-1)
Q Σ
d(d k)
k=1
Σi d(d
d-1
Σ d-1 k(d-k]
d
d - 2
Ag Pr(s:(s: = Sj 1)
pk
1:
d-1 k-1
= Q Σ
d(d-k) k)
k=2
Σk=2 djd-k
d-1
k= k(d-k]
Based on this, we can see that has the structure A A A (b- + c11 for b A. - -iy and Ay. Following the argument by Simon and Vincent (2020), the minimum eigenvalue is then given by Amin (A) = - c. Deriving the specific value shows that depends on the (d - 1)th harmonic number, Ha-1:
18

---

Published as a conference paper at ICLR 2023
Amin (A) - c = Ati - -ij
j + -==
d-1
k-1
d(d-k)
d(d-k)
d-1
kmm k(d-R)
k= k(d-k)
Lk=l k
2Ha-1
The minimum eigenvalue is therefore strictly positive, and this implies that hry(0) is p-strongly convex with ja given by
= (min = H
Note that the strong convexity constant does not depend on (xr, y) and is determined solely by the number of features d.
Theorem 1. For model Φwr(x. y; 0; whose predictions satisfy the constraint inle. (4) the objective value (()) upper bounds the Shapley value estimation error as follows,
Ep(x,) [ yx ((vxy))) )12] - (2H)) -2-)
where L represents the loss achieved by the exuct Shapley values.
Proof. We first consider single input-output pair (a, y) with prediction given by Φ Φvr y 0). Rather than writing the expected loss (o), we now write the Lagrangian Laz (Ø, 7) to account for the linear constraint in our objective, see eq. (4)
(db 7) h (d) + (vay - Tay (0y - 1T
Regardless of the Lagrange multiplier value 7 R, the Lagrangian Lxy(Ø, 7) is p-stronggly convex quadratic with the same Hessian as (d);
VeLoy(0,7) = (o))
Strong convexity enables to bound ø's distance to the global minimizer via the Lagrangian's 's value. First, we denote the Lagrangian's optimizer as (Ø*, )"), where oi is given by the exact Shapley values (Charnes et al. 1988):
d(U2g))
Next, we use the first-order strong convexity condition to write the following inequality:
Cry(d.7") 2 (dy (,""""""") + (Ø (66- V Cay (0",'")) ,x))+
4 110- $11
19

---

Published as a conference paper at ICLR 2023
According to the Lagrangian's KKT conditions (Boyd et al. 2004), have the property that VoLrylo ,
= =0. The inequality therefore simplifies to
Lay(d, 2 Layl ,,'),,")0000"""))")))""" + 7 211 1'1
2
equivalently,
2 ( (Cay(0)n") - (xold' -appo""")=_(0007))) 1
11
If we constrain 6 to be a feasible solution (i.e., it satisfies our objective's ' linear constraint), the KKT primal feasibility condition implies that the inequality further simplifies to
1 - 6'11 I
(Iayy()) - Nay (
(17)
Now, we can consider this bound in expectation over p(x, y). First, we denote our full training loss as [(0), which is equal to
[()) = 1x((xy)) psa()s) (Uxy(s) - Uxy (0) - Φvv
0)) xx;
Bp(x,y) |Ixxy (dvi (x, (x;
Next, we denote L as the training loss achieved by the exact Shapley values, or
L p((r)) bxx (d(
Given a network dvvI (x, y; 0 whose predictions are constrained satisfy the linear constraint, laking the bound from eq. (17) in expectation yields the following bound on the distance between the predicted and exact Shapley values:
Ep(x,) (x) - ((vxy) (1xx)) s 2 (((0)) --))
Applying Jensen's inequality to the left side, rewrite the bound as follows:
2 V2 (c(0))-
E Ep(x.)) [ y: (( |12]] $
Substituting in the strong convexity parameter from Lemma
we arrive at the final bound:
o(xxy))l s (2H))
E Ep(x.)
We also present a corollary to Theerem 1 This result formalizes the intuition that we can iteratively optimize the explainer such that its loss approaches the optimum, Shapley value estimation error will go to zero.
Corollary 1. Given a sequence of models dor xx y; 01); divr(xx y; 02),... whose predictions satisfy the constraint in eg. (4) and where L(On) - L² the Shapley value estimation error goes to
lim Epix.) n-000
(x (x, 0; ((xx)))) =0.
20

---

Published as a conference paper at ICLR 2023
Proof. Fix c O. By assumption, there exists value n' such that [())) -L² ne²
2 for n > n'.
Following the result in Theorem we have Ppx(y)) | (x; ((luy(())) < for n'.
Finally, we also consider the role of our loss function quantifying the Shapley value estimation enror, which we define for given explainer model (x; as
SVE = Ep(x.)) [ y; of (1xx))l12
One natural approach is to use an external dataset (e.g., the test data) consisting of samples (I;, y4) for i 1,..., n, calculate their exact Shapley values (VVrra) and generate Monte Carlo estimate as follows:
SVE., 1 Σ der (zi, Ue: 0) - db (f
While standard concentration inequalities allow us to bound SVE using SVE,, generating the ground truth values can be computationally costly, particularly for large n. Instead, another approach is to use our result from Theorem I which bypasses the need for ground truth Shapley values, For this, recall that (0) represents our weighted least squares loss function, where we assume that the explainer @vrr xx 8) satisfies the constraint in (4) for all predictions. If we know L(I) exactly, then Theorem yielld the following bound with probability 1:
SVE S 2Ha ((0)
If we do not know C(0) exactly, we can instead form Monte Carlo estimate L(O),, using samples (Ii, (6, s;) for i ...... n. Then, using concentration incqualities like Chebyshev Hoeffding (the latter only applies if we assume bounded errors), we can get probabilistic bounds of the form P(I2(0)- 6) < 6. With these, we can say with probability at least 5 that L(0) S [()),, + Finally, combining this with the last steps of our Theorem T proof, we obtain the following bound with probability at least - 5:
SVE < v 221a- (2:0), L +ee
(18)
Naturally, 8 is a function of and the number of samples n used to estimate L())... with the rate of convergence to probability 1 depending on the choice of concentration inequality (Chebyshev or Hoeffding). Although this procedure yields an inexpensive upper bound on the Shapley value estimation error, the bound's looseness, well as the fact that we do not know L priori, make unappealing as an evaluation metric. The more important takeaways are 1) that training with the loss L(0) effectively minimizes upper bound the Shapley value estimation error, and 2) that comparing explainer models via their validation loss, which effectively L(()),. n. principled approach to perform model selection and hyperparameter tuning.
E DATASETS
The ImageNette dataset contains 9,469 training examples and 3,925 validation examples, and we split the validation data to obtain validation and test sets containing 1,962 examples each. The MURA dataset contains 36,808 training examples and 3,197 validation examples. We use the validation examples as a test set, and we split the training examples to obtain train and validation sets containing 33,071 and 3.737 examples, ensuring that images from the same patient belong single split. The Oxford-Ile IIIT Pets dataset contains 7,349 examples for 37 classes, and we split the data to obtain train, validation, and test sets containing 5,879, 735, and 735 examples, respectively. For all datasets,
21

---

Published as a conference paper at ICLR 2023
Table 4: Performance metrics for target-class explanations with additional baselines. Methods that fail to outperform the random baseline are shown in gray, and the best results are shown in bold (accounting for 95% confidence intervals).
ImageNNette
MURA
Ins. (f)
Del. (U)
Faith. (t)
Ins. (1)
Del. (L)
Faith. (1)
Attention last
0,962 (0.004)
0.793 (0.013)
0.694 (0.015)
0.890 (0.010)
0.592 (0.013)
0.635 (0.016)
Attention rollout
0.938 (0.005)
0.880 (0.010)
0.704 (0.015)
0.845 (0.011)
0.692 (0.014)
0.618 (0.016)
GradCAM (LN)
0,914 (0.006)
0.937 (0.008)
0.680 (0.015)
0.899 (0.009)
0.681 (0.015)
0.631 (0.016)
GradCAM (Attn)
0,938 (0.006)
0.948 (0.006)
0.656 (0.014)
0.843 (0.012)
0.835 (0.011)
0.580 (0.016)
IntGrad (Pixel)
0.967 (0.004)
0,930 (0.008)
0,403 (0.024)
0.897 (0.010)
0.796 (0.015)
0.201 (0.022)
IntGrad (Embed.)
0.967 (0.004)
0,930 (0.008)
0.403 (0.024)
0.897 (0.010)
0.796 (0.015)
0.201 (0.022)
Vanilla (Pixel)
0.938 (0.005)
0.860 (0.011)
0.700 (0.015)
0.890 (0.010)
0,561 (0.014)
0.627 (0.016)
Vanilla (Fmmbed.)
0,950 (0.004)
0.808 (0.013)
0.703 (0.015)
0.890 (0.010)
0,537 (0.014)
0.629 (0.016)
SmoothGrad (Pixel)
0.960 (0.005)
0.779 (0.013)
0.706 (0.015)
0.873 (0.010)
0.634 (0.014)
0.618 (0.016)
SmoothGrad (Embed.)
0,947 (0.005)
0.942 (0.006)
0.703 (0.015)
0.870 (0.010)
0.813 (0.011)
0.617 (0.016)
VarGrad (Pixel)
0.958 (0.005)
0.796 (0.013)
0.682 (0.015)
0.871 (0.010)
0.660 (0.013)
0.577 (0.015)
VarGrad (Embed.)
0.949 (0.005)
0,946 (0.005)
0.700 (0.015)
0.857 (0.011)
0.823 (0.011)
0.615 (0.016)
LRP
0.967 (0.004)
0.779 (0.014)
0.705 (0.015)
0.900 (0.009)
0.551 (0.013)
0.646 (0.016)
Leave-one-oul
0,969 (0.002)
0.917 (0.010)
0,140 (0.040)
0.926 (0.008)
0.694 (0.017)
0,308 (0.032)
RISE
0.977 (0.001)
0.860 (0.014)
0.704 (0.015)
0.957 (0.004)
0.573 (0.018)
0.618 (0.016)
VIT Shapley
0.985 (0.002)
0.691 (0.014)
0.711 (0.015)
0.971 (0.002)
0.307 (0.013)
0.707 (0.013)
Random
0.951 (0.005)
0.951 (0.005)
0.849 (0.010) 0.847 (0.010)
the training and validation data were used to train the original classifiers, fine-tuned classifiers and explainer models, and the test data was used only when calculating performance metries.
F BASELINE METHODS
This section provides implementation details for the baseline explanation methods. We used variety of attention-, gradient- and removal-based methods as comparisons for ViT Shapley, and we modified several approaches to arrive at patch-level feature attribution scores.
Attention last This approach calculates the attention directed from each image token into the class token in the final self-attention layer, summed across attention heads (Abnar and Zuidema 2020 Chefer et al.] (022)). The results are provided at the patch-level, but they are not generated separately for each output class.
This approach accounts for the flow of attention between tokens by summing across attention heads and multiplying the resulting attention matrices each layer (Abnar and Zuidema 2020). Like the previous method, results are not generated separately for each output class. We used an implementation provided by prior work (Chefer al. 2021).
Attention rollout
Common gradient- based methods Several methods that operate via input gradients are Vanilla gradients (Simonyan et al., 2013), SmoothGrad (Smilkov et al. 2017), VarGrad (Hooker et al. 2019), and IntGrad (Sundaraaaajan et al. (2077). These methods were run using the Captum package (Kokhlikyan et al. 2020), and we used 10 samples per image for SmoothGrad, VarGrad and IntGrad. We tried applying these at the level of pixels and patch embeddings, and in both cases we created class-specific, patch-level attributions by summing across the unnecessary dimensions. We calculated the absolute value before summing for Vanilla and SmoothGrad, VarGrad automatically produces non-negative values, and we preserved the sign for IntGrad because it should be meaningful.
GradCAM Originally designed for intermediate convolutional layers (Selvaraju et al.) 201 7 GradCAM has since been generalized to the ViT context. The main operations remain the same, only the representation being analyzed is the layer-normed input to the final self-attention layer, and the aggregation is across the embedding dimension rather than convolutional channels (GradCAM LN) (Gildenblat and contributors 2021). We also experimented with using different internal layer for
22

---

Published as a conference paper at ICLR 2023
generating explanations (the attention weights computed in the final self-attention layer, denoted as GradCAM Attn. (Chefer et al. 2021)).
Layer-wise relevance propagation (LRP) Originally described set of constraints for a modi- fied backpropagation routine (Bach et al. 2015), LRP has since been implemented for variety of network layers and architectures, and it was recently adapted VITs (Chefer al. 2021). We used an implementation provided by prior work (Chefer et al. 2021).
Leave-one-out
The importance scores in this approach are the difference in prediction probability for the full-image and the iamge with a single patch removed. We removed patches by setting pixels to zero, similar to the original version for CNNs (Zeiler and Fergus 2014).
This approach involves sampling many occlusion masks and reporting the mean prediction when each patch is included. The original version for CNNs (Petsiuk et al., (018)) used complex approach to generate masks, but we simply sampled subsets of patches. As in the original work, we sample from all subsets with equal probability, and we use 2,000 mask samples per sample to be explained. We occlude patches by setting pixel values to zero, similar to the original work.
RISE
Random Finally, we included a random baseline as a comparison for the insertion, deletion and ROAR metrics. These metrics only require a ranking of important patches, so we generated ten random orderings and averaged the results across these orderings.
Table 4 shows the same metrics Table I with additional results for alternative implementations of several baselines. For the methods based on input gradients, we experimented with generating explanations at both the pixel level and embedding level; the preferred approach depends the method and metric, but both versions tend to underperform ViT Shapley, with the exception of faithfulness ImageNette where the 95% confidence intervals overlap for many methods. We also experimented with two versions of GradCAM (described above) and find that the GradCAM LN implementation generally performs slightly better. In the main text, we present results only for GradCAM LN and the remaining gradient-base -based methods generated at the embedding level.
G METRICS DETAILS
This section provides additional details about the performance metrics used in the main text experi- ments (Section 6).
These metries involve repeatedly making predictions while either inserting or deleting features in order of most to least important (Petsiuk et al., 2018). While the original work removed features by setting them zero, we use the fine-tuned classifier that was trained to handle partial information. We calculated the area under the curve for individual predictions and then averaged the results across 1,000 test set examples; we used random examples for ImageNette, and only examples that were predicted to be abnormal for MURA. Table presents results for the true class only, and Table 2 presents results averaged across all the remaining classes.
Insertion/delltion
Sensitivity-n This metric samples feature subsets random and calculates the correlation between the prediction with each subset and the sum of the corresponding features' attribution scores (Ancona al. 2018). It typically considers subsets of a fixed size n, which means sampling from the following subset distribution (s):
(x) 1(1 = -n) (
Mathematicallyy, the metric is defined for model fxx), individual sample and label y, feature attributions & E IRd and subset size n as follows:
Sens(f, x, U, Φ. n)) = Corr, (s fz(x) - (21
Similar to insertion/deletion, we use the fine-tuned classifier to handle held-out patches and calculate the metric across 1,000 test set images. We use subset sizes ranging from 14 to 182 patches with step size 14, and we estimate the correlation for each example and subset size using 1,000 subset samples.
23

---

Published as a conference paper at ICLR 2023
Faithfulness This metric is nearly identical to sensitivity-n. only it calculates the correlation across subsets of all sizes (Bhatt et al. 2021). Mathematically, is defined as
Faith x, U, 6) = Corr f,)) f1(11=)))
and we sample from a distribution with equal probability mass on all cardinalities,
p(s)
( (d+1)
We use the fine-tuned classifier to handle held-out patches, and compute faithfulness across 1,000 test set images and with 1,000 subset samples per image.
ROAR Finally, ROAR evaluates the model's accuracy after removing features in order from most to least important (Hooker et al. 2019). We also experimented with inserting features in order of most least important. Crucially, the ROAR authors propose handling held-out features by retraining the model with masked inputs. We performed masked retraining by performing test-time augmentations for training. validation and test set images, generating explanations to identify the most important patches for the true class, and setting the corresponding pixels to zero.
Because masked retraining leaks information through the masking, we also replicated this metric using the fine-tuned classifier model, and with separate evaluator model trained directly with random masking; the evaluator model trained with random masking has been used in prior work (Jethani et al. 2021a b). We generated results for each number of inserted/deleted patches (1, 3, 7, 14, 28, 56, 84, 112, 140, 168, and 182) with the final accuracy computed across the entire test set.
Ground truth metrics Previous work has considered evaluations involving comparison with ground truth importance, where the ground truth is either identified by humans (Chefer et al.) (022)) or introduced via synthetic dataset modifications (Zhou et al., 2022). An important issue with such methods is that they test explanations against what model should depend rather than what it does depend on, so the results do not reflect the explanation ' accuracy for the specific model (Petsiuk) et al. 2018). We thus decided against including such metries.
H ADDITIONAL RESULTS
This section provides additional experimental results. We first show results involving similar base- lines and metrics as in the main text, and we then show results comparing VIT Shapley to Ker- nelSHAP.
H.1
MAIN BASELINES AND METRICS
Figure 5 shows evaluation of attention masking for handling held-out image patches using two separate metrics: 1) KL divergence relative to the full-image predictions (also shown in the main text), and top-1 accuracy relative to the true labels. The former can be understood a divergence measure between the predictions with masked inputs and the predictions with patches marginalized out using their conditional distribution (see (ece[Appddixxx B}. The latter more intuitive measure of how much the performance degrades given partial inputs. The results are similar between the two metrics, showing that the predictions diverge more quickly if the model is not fine-tuned with random masking.
Table 5]shows insertion, deletion and faithfulness results for the MURA dataset with examples that were predicted to be normal, but while evaluating explanations for the abnormal class. ViT Shapley outperforms the baseline methods, reflecting that our explanations correctly identify patches that influence the prediction towards and against the abnormal class even for normal examples.
Table 6 shows insertion, deletion and faithfulness results for the Pets dataset. We observe that ViT Shapley outperforms other methods for all metrics with the exception of faithfulness for target classes, where 95% confidence intervals overlap for many methods (similar to the other datasets).
Table 7. Table 8 and Table 9 show insertion, deletion, and faithfulness metrics for ImageNette when using other ViT architectures (i.c., ViT-Tiny, -Small, and -Large, respectively) (Wightman 2019 Dosovitskiy et al. 2020) for the classifier and explainer. They show results for target-class
24

---

Published as a conference paper at ICLR 2023
explanations and non-target-class explanations, respectively. The results are consistent with those obtained for ViT-Base, and ViT Shapley outperforms the baseline methods across all three metrics. This shows that explainer model can be trained successfully with architectures of different sizes, including when using a relatively small number of parameters.
Table shoshows insertion, deletion and faithfulness results for a ViT classifier trained directly with random masking. Whereas our [Section 6 exxperiments utilize a fine-tuned classifier to handle missing patches, classifier trained with random masking allows us bypass the fine-tuning stage and train the explainer directly. The results are similar to Table and we find that ViT Shapley consistently achieves the best performance.
Figure 6. Figure 7 and Figure 8jshow the average curves used to generate the insertion/delction AUC results. All sets of plots reflect that explanations from ViT Shapley identify relevant patches that quickly move the prediction towards or away from given class. In the case of ImageNette and Pets, we observe that this holds for both target and non-target classes.
Figure 9 shows the sensitivity-n metric evaluated for non non-target classes on the ImageNette dataset. Similarly, these results show that ViT Shapley generates attribution scores that represent the impact of withholding features from a model, even for non-target classes. In this case, RISE and leave-one-out are more competitive with ViT Shapley, but their performance is less competitive when the correlation calculated for subsets of all sizes (see faithfulness in Table 2).
Next, Figure 10 shows ROAR results generated in both the insertion and deletion directions, using the four patch removal approaches: 1) the fine-tuned classifier, 2) the separate evaluator model trained directly with random masking, 3) masked retraining, and 4) masked retraining without positional embeddings. The results show that in addition strong performance in the deletion direction, ViT Shapley consistently achieves the best results in the insertion direction, even in the case of masked retraining with positional embeddings.
Figure 11 shows ROAR results for the same settings, but when using the ViT-Small architecture. We observe the same results obtained with ViT-Base. Except for the deletion direction with masked retraining and positional embeddings enabled, ViT Shapley achieves the best performance among all methods.
shows the time required to generate explanations using each approach. Because ViT Shapley requires a single forward pass through the explainer model, is among the fastest approaches and is paralleled only by the attention-basee and backward passes for all classes, and sometimes for many altered inputs (e.g., with noise injected for SmoothGrad). RISE is the slowest of all the approaches tested because it requires making several thousand predictions to explain each sample. Our evaluation was conducted on a GeForce RTX 2080 Ti GPU, with minibatches of 16 samples for attention last, attention rollout and ViT Shapley; batch size of for Vanilla Gradients, GradCAM, LRP, leave-one-out and RISE; and internal minibatching for SmoothGrad, IntGrad and VarGrad (implemented via Captum (okhlikkann et al. 2020)).
Finally, Table
Table 11
methods. The gradient-bases methods require forward
ViT Shapley the only method considered here to require training time, and as described in Appendix C training the explainer models required roughly 0.8 days for ImageNette and 2.5 days for MURA. The training time not insignificant, but investing time in training the explainer is worthwhile if 1) high-quality explanations required, 2) there many examples to be explained (e.g., an entire dataset), or 3) fast explanations are required during a model's 's deployment.
25

---

Published as a conference paper at ICLR 2023
ImageNette
MURA
3.0
3.0
divergence
83 divergence
2.0
2.0
1.0
0.5
28
56
84
112
140
168
196
28
56
84
112
140
168
196
ImageNette
MURA
0.8
0.8
Accuraty
Accurary
0.6
0.6
0.2
0.2
0.0
0.0
Patches
Patches
Masking
Zeros input
Masking (Post:soltmax)
Zeros at embeading
Random imputation
Masking (Hine tuned]
Mask token (Fine tuned)
Figure 5: ViT predictions given partial information. We delete patches at random using several removal mechanisms, and we then measure the quality of the resulting predictions via two metries: the KL divergence relative to the original, full-image predictions (top), and the top-1 accuracy relative the true labels (bottom).
Table 5: MURA non-target metrics for images that were predicted to be normal. Methods that fail to outperform the random baseline shown in gray, and the best results are shown in bold (accounting for 95% confidence intervals).
MURA
Ins. (1)
Del. (4)
Faith. (t)
Attention last
0.157 (0.01
0,210 (0.009)
-0.555 (0.022)
Attention rollout
0.199 (0.01
0.169 (0.010)
-0.480 (0.023)
GradCAM
0.197 (0.012)
0.152 (0.010)
-0.449 (0.022)
IntGrad
0.209 (0.014)
0.151 (0.010)
0.103 (0.023)
Vanilla
0.174 (0.01
0.197 (0.009)
-0.477 (0.023)
SmoothGrad
0.169 (0.01
0.176 (0.011)
-0.480 (0.023)
VarGrad
0.166 (0.011)
0.175 (0.011)
-0.788 (0.023)
LRP
0.169 (0.012)
0.200 (0.009)
-0,461 (0.023)
Leuve-one-out
0.326 (0.016)
0.093 (0.007)
0.272 (0.029)
RISE
0.406 (0.018)
0.075 (0.006)
-0,479 (0.023)
ViT Shapley
0.580 (0.016)
0.039 (0.003)
0.642 (0.014)
Random
0.169 (0.01
0.170 (0.011)
26

---

Published as a conference paper at ICLR 2023
Table 6: Performance metrics for ViT-Base on Pets. Methods that fail to outperform the random
baseline are shown in gray, and the best results are shown in bold (accounting for 95% confidence intervals).
Target
Non-Target
Ins. (t)
Del. (1)
Faith. (1)
Ins. (t)
Del. (4)
Faith. (t)
Attention last
0.876 (0.011)
0.494 (0.016)
0.545 (0.017)
Attention rollout
0.837 (0.012)
0.672 (0.017)
0.559 (0.017)
GradCAM (Attn)
0.837 (0.013)
0.737 (0.018)
0.527 (0.016)
0.005 (0.000)
0.008 (0.000)
-0.499 (0.013)
GradCAM (LN)
0.837 (0.011)
0,717 (0.019)
0.553 (0.017)
0.012 (0.001)
0.004 (0.000)
-0.511 (0.014)
IntGrad (Pixel)
0.899 (0.010)
0.802 (0.016)
0.437 (0.017)
0.006 (0.001)
0.003 (0.000)
0.126 (0.008)
IntGrad (Embed.)
0.899 (0.010)
0,802 (0.016)
0.437 (0.017)
0.006 (0.001)
0,003 (0.000)
0.126 (0.008)
Vanilla (Pixel)
0.862 (0.011)
0.596 (0.018)
0.559 (0.017)
0.004 (0.000)
0.011 (0.000)
-0.526 (0.014)
Vanilla (Embed.)
0.876 (0.011)
0,519 (0.017)
0.561 (0.017)
0.004 (0.000)
0.013 (0.000)
-0.529 (0.014)
SmoothGrad (Pixel)
0.889 (0.011)
0.488 (0.016)
0.564 (0.017)
0.003 (0.000)
(0.000)
-0.532 (0.014)
SmoothGrad (Embed.)
0,840 (0.013)
0.834 (0.013)
0.558 (0.017)
0.004 (0.000)
0,005 (0.000)
-0.526 (0.014)
VarGrad (Pixel)
0.891 (0.011)
0.513 (0.016)
0.557 (0.017)
0.003 (0.000)
0.013 (0.000)
-0.522 (0.014)
VarGrad (Embed.)
0.847 (0.012)
0.835 (0.013)
0.555 (0.017)
0.004 (0.000)
0.005 (0.000)
-0.523 (0.014)
LRP
0.890 (0.01
0.466 (0.016)
0.567 (0.017)
0.004 (0.000)
0.013 (0.000)
-0.529 (0.014)
Leave-one-out
0.936 (0.006)
0.653 (0.022)
0.215 (0.036)
0.018 (0.001)
0.001 (0.000)
0.138 (0.023)
RISE
0.946 (0.005)
0.505 (0.022)
0.559 (0.017)
0.032 (0.002)
0.001 (0.000)
-0.523 (0.014)
VIT Shapley
0.945 (0.006)
0.388 (0.017)
0.590 (0.016)
0.052 (0.002)
0.001 (0.000)
0.529 (0.012)
Random
0.848 (0.012)
0.845 (0.012)
0.004 (0.000)
0.004 (0.000)
Table 7: Performance metrics for ViT-Tiny ImageNette. Methods that fail to outperform the random baseline are shown in gray, and the best results are shown in bold (accounting for 95% confidence intervals).
Target
Non-Target
Ins. (f)
Del. (1)
Faith. (1)
Ins. (f)
Del. (1)
Faith. (t)
Attention
0.917 (0.008)
0.702 (0.015)
0.649 (0.014)
Attention rollout
0.901 (0.009)
0.698 (0.015)
0.658 (0.015)
GradCAM (LN)
0.918 (0.007)
0.809 (0.014)
0.659 (0.014)
0.034 (0.002)
0.006 (0.000)
-0.616 (0.013)
GradCAM (Attn)
0.898 (0.009)
0,814 (0.013)
0.636 (0.014)
0.011 (0:001)
0.020 (0.001)
-0.607 (0.012)
IntGrad (Pixel)
0.928 (0.006)
0.852 (0.013)
0.293 (0.025)
0.018 (0.002)
0.008 (0.001)
0.094 (0.014)
IntGrad (Embed.)
0.928 (0.006)
0,852 (0.013)
0.293 (0.025)
0.018 (0.002)
0.008 (0.001)
0.094 (0.014)
Vanilla (Pixel)
0.884 (0.009)
0.801 (0.013)
0.656 (0.015)
0.015 (0.001)
0.020 (0.001)
-0.629 (0.013)
Vanilla (Fimbed.)
0.902 (0.008)
0.752 (0.015)
0.659 (0.015)
0.012 (0.001)
0.025 (0.001)
+0.632 (0.013)
SmoothGrad (Pixel)
0.908 (0.009)
0.733 (0.015)
0.659 (0.015)
0.011 (0.001)
0.029 (0.002)
-0.632 (0.013)
SmoothGrad (Embed.)
0.917 (0.008)
0.756 (0.014)
(0.015)
0.010 (0.001)
0.025 (0.002)
-0.632 (0.013)
VarGrad (Pixel)
0.914 (0.008)
0.754 (0.014)
(0.014)
0.011 (0.001)
0.027 (0.002)
-0.620 (0.012)
VarGrad (Embed.)
0.921 (0.008)
0.777 (0.014)
0.648 (0.014)
0.010 (0:001)
0.023 (0.001)
-0.618 (0.012)
LRP
0.938 (0.007)
0.648 (0.016)
0.673 (0.014)
0.014 (0.001)
0.024 (0.001)
-0.623 (0.013)
Leuve-one-out
0.956 (0.004)
0.737 (0.019)
0.221 (0.039)
0.052 (0.004)
0.003 (0.000)
0.145 (0.025)
RISE
0.970 (0.003)
0.602 (0.019)
0.658 (0.015)
0.092 (0.005)
0.002 (0.000)
-0.628 (0.013)
VIT Shapley
0.981 (0.002)
0.457 (0.015)
0.694 (0.014)
0.198 (0.006)
0.001 (0.000)
0.641 (0.011)
Random
0.908 (0.008)
0.907 (0.008)
0.010 (0.001)
0.010 (0.001)
27

---

Published as a conference paper at ICLR 2023
Table 8: Performance metrics for ViT-Small on ImageNette. Methods that fail to outperform the random baseline are shown in gray, and the best results are shown in bold (accounting for 95% confidence intervals).
Target
Non-Target
Ins. (t)
Del. (1)
Faith. (1)
Ins. (1)
Del. (4)
Faith. (?)
Attention last
0.949 (0.006)
0,706 (0.014)
0.781 (0.011)
Attention rollout
0.914 (0.007)
0.814 (0.013)
0.787 (0.011)
GraCCMM (LN)
0.908 (0.006)
0.898 (0.011)
0.770 (0.010)
0.027 (0.002)
0.006 (0.000)
-0.740 (0.009)
GradCAM (Attn.)
0.904 (0.009)
0,911 (0.009)
0.724 (0.010)
0.008 (0.001)
0.016 (0.001)
-0.717 (0.009)
IntGrad (Pixel)
0,951 (0.006)
0.908 (0.010)
0.541 (0.022)
0.011 (0.001)
0.005 (0.001)
0.384 (0.018)
IntGrad (Embed.)
0,951 (0.006)
0,908 (0.010)
0.541 (0.022)
0.011 (0.001)
0.005 (0.001)
0.384 (0.018)
Vanilla (Pixel)
0.905 (0.007)
0.830
0.784 (0.010)
0.011 (0.001)
0.018 (0.001)
-0.754 (0.009)
Vanilla (Embed.)
0.921 (0.007)
0.793 (0.013)
0.786 (0.011)
0.009 (0.001)
0.022 (0.001)
-0.756 (0.009)
SmoothGrad (Pixel)
0.943 (0.006)
0.744 (0.014)
0.786 (0.011)
0.007 (0.001)
0.027 (0.001)
-0.756 (0.009)
SmoothGrad (Embed.)
0,946 (0.006)
0.753 (0.014)
0.787 (0.011)
0.006 (0.001)
0.026 (0.001)
-0.757 (0.009)
VarGrad (Pixel)
0.946 (0.006)
0.766 (0.013)
0.742 (0.010)
0.007 (0.001)
0.025 (0.001)
-0.709 (0.009)
VarGrad (Embed.)
0,947 (0.006)
0.774 (0.013)
0.758 (0.010)
0.007 (0.001)
0.024 (0.001)
-0.723 (0.009)
LRP
0.957 (0.005)
0.695 (0.015)
0.793 (0.010)
0.007 (0:001)
0,027 (0.001)
-0.753 (0.009)
Leave-one-out
0,967 (0.002)
0.855 (0.015)
0.044 (0.042)
0.024 (0.002)
0.003 (0.000)
0.079 (0.030)
RISE
0.976 (0.002)
0.752 (0.018)
0.787 (0.010)
0.049 (0.004)
0.002 (0.000)
-0.755 (0.009)
VIT Shapley
0.982 (0.002)
0.591 (0.015)
0.801 (0.010)
0.130 (0.005)
0.001 (0.000)
0.747 (0.009)
Random
0.933 (0.006)
0.934 (0.006)
0.007 (0.001)
0.007 (0.001)
Table 9; Performance metrics for ViT-Large on ImageNette. Methods that fail to outperform the random baseline are shown in gray, and the best results are shown in bold (accounting for 95% confidence intervals).
Target
Non-Target
Ins. (t)
Del. (1)
Faith. (f)
Ins. (f)
Del.
Faith. (t)
Attention last
0.915 (0.005)
0.896 (0.008)
0.401 (0.022)
Attention rollout
0.929 (0.005)
0.928 (0.007)
0.423 (0.023)
GradCAM (LN)
0.955 (0.004)
0.924 (0.009)
0.424 (0.023)
0.012 (0.001)
0.005 (0.000)
-0.372 (0.020)
GradCAM (Attm)
0.935 (0.005)
0,930 (0.007)
0.395 (0.022)
0.007 (0.001)
0.006 (0.001)
-0.351 (0.018)
IntGrad (Pixel)
0.965 (0.003)
0.947 (0.006)
0.195 (0.020)
0.006 (0.001)
0.004 (0.000)
0.114 (0.013)
IntGrad (Embed.)
0.965 (0.003)
0.948 (0.006)
0.198 (0.020)
0.006 (0.001)
0.004 (0.000)
0.117 (0.013)
Vanilla (Pixel)
0.898 (0.007)
0.921 (0.007)
0.411 (0.022)
0.012 (0.001)
0.008 (0.001)
-0.370 (0.020)
Vanilla (Fmmbe.))
0.911 (0.006)
0.905 (0.009)
0.415 (0.023)
0.010 (0.001)
0.010
-0.373 (0.020)
SmoothGrad (Pixel)
0.949 (0.004)
0.861 (0.011)
0.424 (0.023)
0.006 (0.001)
(0.001)
-0.822 (0.020)
SmoothGrra (Embed.)
0.956 (0.004)
0.857 (0.011)
0.423 (0.023)
0.005 (0.000)
0.015 (0.001)
-0.380 (0.020)
VarGrad (Pixel)
0.942 (0.005)
0.878 (0.010)
0.387 (0.021)
0.007 (0.001)
0.013 (0.001)
-0.350 (0.018)
VarGrad (Embed.)
0.944 (0.005)
0.875 (0.010)
0.383 (0.020)
0.007 (0.001)
0.013 (0.001)
-0.343 (0.017)
LRP
0.946 (0.004)
0.869 (0.010)
0.424 (0.023)
0.006 (0.001)
0.013 (0.001)
-0.381 (0.020)
Leave-one-out
0.970 (0.003)
0.938 (0.007)
0.123 (0.031)
0.009 (0.001)
0.003 (0.000)
0.118 (0.020)
RISE
0.978 (0.002)
0.882 (0.013)
0.425 (0.023)
0.019 (0.003)
0.002 (0.000)
-0.381 (0.020)
VIT Shapley
0.986 (0.002)
0.742 (0.013)
0.439 (0.023)
0.077 (0.003)
0.001 (0.000)
0.391 (0.019)
Rundom
0.957 (0.004)
0.956 (0.004)
0.005 (0.000)
0.005 (0.000)
28

---

Published as a conference paper at ICLR 2023
Table 10: Performance metrics for target classes explanations for ViT classifier trained with random masking. Methods that fail to outperform the random baseline are shown in gray, and the best results shown bold (accounting for 95% confidence intervals).
ImageNette
Del. (
Faith. (f)
Attention last
0,963 (0.003)
0.839 (0.011)
0.593 (0.017)
Atteution rollout
0.945 (0.004)
0.903 (0.008)
0.597 (0.017)
GraCCAM (LN)
0.955 (0.003)
0.898 (0.011)
0.585 (0.016)
GradCAM (Att))
0.945 (0.005)
0.942 (0.006)
0.564 (0.016)
IntGrad (Pixel)
0.977 (0.002)
0.933 (0.008)
0.474 (0.017)
IntGrad (Embed.)
0.977 (0.002)
0.933 (0.008)
0.474 (0.017)
Vanilla (Pixcl)
0.934 (0.005)
0.901 (0.009)
0.593 (0.017)
Vanilla (Embod.)
0.948 (0.004)
0.864 (0.011)
0.595 (0.017)
SmoothGrad (Pixel)
0.966 (0.004)
0.809 (0.012)
0.599 (0.017)
SmoothGrad (Embed.)
0.954 (0.004)
0.950 (0.005)
0,597 (0.017)
VarGrad (Pixel)
0,964 (0.004)
0.826 (0.011)
0.573 (0.016)
VarGrad (Fmbee.)
0.953 (0.00S)
0.954 (0.004)
0.594 (0.017)
LRP
0.972 (0.002)
0.806 (0.013)
0.603 (0.017)
Leave-one-out
0.977 (0.001)
0.881 (0.014)
0.134 (0.034)
RISE
0.982 (0.001)
0.793 (0.017)
0.599 (0.017)
VIT Shapley
0.988 (0.001)
0.736 (0.013)
0.605 (0.017)
Random
0.957 (0.004)
0.957 (0.004)
Insertion
Deletion
Probabilty (Targe))
0.8
0.8
0.6
0.6
0.4
0.0
0.0
168
168
Probabiity (or-target)
0.08
0.3
0.04
0.0
0.00
# of Inserted Patches
# of Deleted Patches
Attention last
GradCAM
SmaothGrad
Leava-one-out
Random
Attention
IntGrad
VarCrad
Vanilla
LRP
VIT Shapley
Figure 6: ImageNette average insertion/dellttion curves. Top:
the mean prediction probability for the
target class as features are inserted or deleted in order of most to least important. Bottom: the mean prediction probability, averaged across all non-target classes.
29

---

Published as a conference paper at ICLR 2023
Insertion
Deletion
Probability (Abnamall)
0.6
0.6
0.0
0.0
28
56
140
168
196
140
168
196
Inserted Patches
Deleted Patches
Attention
GradCAM
SmoothGrad
Leave one
Attention rollout
IntGrad
VarGrad
RISE
Vanilla
URP
Shapley
Figure 7: MURA average insertion/deletion curves. The mean prediction probability for the target class plotted features are inserted deleted in order of most least important.
Insertion
Deletion
1.0
Probabiiity [arget
0.8
0.8
0.6
0.6
0.2
0.2
0.0
0.0
56
196
0.175
Probabiity (Norstarged
0.150
0.125
0.100
0.015
0.075
0.010
0.050
0.005
0.025
0.000
0.000
Inserted Patches
Patches
Attontion last
GradCAM
SmaothGrad
Leave one out
Random
Attention rollout
IntGrad
VarGrad
Vanilla
LRP
VIT Shapley
Figure 8: Pets average insertion/deletion curves. Top: the mean prediction probability for the target class as features are inserted or deleted in order of most to least important. Bottom: the mean prediction probability, averaged across all non-target classes.
30

---

Published as a conference paper at ICLR 2023
ImageNette
Correlation (Nom target unly)
0.5
0.4
0.2
0.1
-0.1
-02
-0.3
112
168
Patches
Attention last
GradCAM
SmoothGrad
Leave-one-out
Attention rollout
IntGrad
VarGrad
RISE
Shapley
Figure 9: Sensitivity-n evaluation for non-target classes. The results are generated separately for each subset size and averaged across all non-target classes.
Table 11: Time to generate explanations for single sample (in milliseconds), For class-specific explanations, the running time involves generating explanations for all classes.
ImageNette
MURA
Pets
Attention last
6.8
6,2
Attention rollout
10.6
10.0
11.1
GradCAM
275.9
26.2
986.4
IntGrad
1236.8
123.3
5004.9
Vanilla
230.1
23.0
810.7
SmoothGrad
1218.0
121.2
4854.8
VarGrad
1218.9
121.3
4882.5
LRP
1551.4
155.7
5583.5
Leave-one-out
810.0
815.9
922.5
RISE
8213.4
8171.1
8919,8
VIT Shapley
10.1
10.2
10.8
31

---

Published as a conference paper at ICLR 2023
Fine-tuned Classifier (Insertion)
Fine-tuned Classifier (Deletion)
1.0
1.0
0.8
0,8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
56
84
112
140
168
196
28
56
84
140
168
196
of Inserted Patches
of Delcted Patches
Separate Evaluator (Insertion)
Separate Evaluator (Deletion)
1.0
1.0
(ccur
Accuracy
0.2
0.2
0.0
0.0
56
84
112
140
168
196
56
34
112
140
168
196
Masked Retraining w/ Positional Embed. (Insertion)
Masked Retraining w/ Positional Embed. (Deletion)
1.0
1.0
0.8
0.8
Accuracy
Acouracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
160
168
Inserted Patches
Deleted Patches
Masked Retraining Positional Embed. (Insertion)
Masked Retraining Positional Embed. (Deletion)
1.0
1.0
0.8
0.8
Accurad
0.6
Accuracy
0.6
0.4
0.4
0.2
28
56
84
112
140
168
196
28
56
84
112
1400
166
196
Inserted Patches
Deleted Patches
Attention last
GradCAM
SmoothGrad
Leave-one-out
Random
Vanilla
LAP
VIT Shapley
Figure 10: ImageNette accuracy when removing features in order of their importance, run with four evaluation strategies (fine-tuned classifier, separate evaluator, masked retraining and masked retraining without positional embeddings) on the VIT-Base architecture. Left: inserting features from most to least important. Right: removing features from most to least important.
32

---

Published as a conference paper at ICLR 2023
Fine-tuned Classifier (Insertion)
Fine-tuned Classifier (Deletion)
1.0
1.0
0.8
0.8
Accurasy
0.6
Accuracy
0.6
0.4
0.4
0.2
0.2
0.0
0.0
56
140
168
196
56
168
196
of Inserted Patches
Deleted Patches
Separate Evaluator (Insertion)
Separate Evaluator (Deletion)
1.0
1.0
0.8
0.8
Accuracy
Accuracy
28
56
84
112
140
168
196
28
56
84
112
140
168
196
Patches
Masked Retraining w/ Positional Embed. (Insertion)
Masked Retraining w/ Positional Embed. (Deletion)
1.0
1.0
0.8
0.8
Accurecy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
56
84
168
168
Inserted Patches
Deleted Patches
Masked Retraining Positional Embed. (Insertion)
Masked Retraining Positional Embed. (Deletion)
1.0
1.0
0.8
0.8
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
28
56
84
112
140
168
196
28
56
84
112
140
168
196
Inserted Patchee
of Deleted Patches
..... Attention last Attention
GradCAM
SmoothGrad
Leave-one-out
Random
Vanilla
LAP
VIT Shapley
Figure 11: ImageNette accuracy when removing features in order of their importance, run with four evaluation strategies (fine-tuned tuned classifier, separate evaluator, masked retraining and masked retraining without positional embeddings) on the ViT-Small architecture. Left: inserting features from most to least important. Right: removing features from most to least important.
33

---

Published as a conference paper at ICLR 2023
Table 12: Comparing the quality of Shapley value estimates obtained using ViT Shapley and
KemelSHAA via insertion/deletion scores.
Target
Non-target
Ins. (1)
Del. (1)
Ins. (1)
Del. (L)
VIT Shapley
0.985 (0.002)
0.691 (0.014)
0.093 (0.004)
0.001 (0.000)
KernelSHAP
0.990 (0.002)
0.589 (0.053)
0.158 (0.021)
0.001 (0.000)
H.2 KERNELSHAP COMPARISONS
Here, we provide two results comparing ViT Shapley with KerneISHAP.
First, Figure 12 compares the approximation quality Shapley value estimates produced by ViT Shapley and KemelSHAP. The estimates are evaluated in terms of L2 distance, Pearson correlation and Spearman (rank) correlation, and our ground truth generated by running Keme1SHAP for the convergence detection approach described by
large number of iterations. Specifically, Covert and Lee (021) with a threshold of = 0.1. The results are computed using just 100 randomly selected ImageNette images due to the significant computational cost.
we observe that the original version of KerneISHAP takes roughly 120,000 model evaluations reach the accuracy that ViT Shapley reaches with single model evaluation. KerneISHAP with paired sampling (Covert and Lec, (201)) converges faster, and it requires roughly 40,000 model evaluations on average. VIT Shapley's 's estimates are not perfect, but they reach nearly 0.8 correlation with the ground truth for the target class, and nearly 0.7 correlation average across non-target classes.
Based on Figure 12
Next. Table 12 compares the ViT Shapley estimates and the fully converged estimates from Ker- neISHAP via the insertion and deletion metries. The fully converged KemelSHAP estimates per- formed better than ViT Shapley both metries, and the gap is largest for deletion with the target class. The results were also computed using only 100 ImageNette examples due to the computational cost. These results reflect that there is room for further improvement if ViT Shapley's estimates can be made more accurate. KernelSHAP itself not a viable option in practice, as we found that its estimates took between 30 minutes and 2 hours converge when using paired sampling (Covert and Lee. 202I) (equivalent to roughly 300k and 1,200k model evaluations), but it represents an upper bound on how well ViT Shapley could perform with near-perfect estimation quality.
34

---

Published as a conference paper at ICLR 2023
Target
Non-targes
109
10a
1.2 distance
distance
10-1
10-4
10-2
10
10-1
100,000 Evals
150,000
200,000
50,000
100.
150,000
200,000
Evals
1.0
1.0
Pearson correlation
Pecasson commlaaiio
0.4
0.2
0.2
0.0
0.0
200,000
100.
200,000
Evals
Evals
1.0
1.0
Speermen correlation
0.8
Spaarman correlation
0.2
200,000
Evals
Evals
Kemel5HAP Paired Sampling
SHAP
VIT Shapley
Figure 12: Comparing the quality of Shapley value estimates obtained by VIT Shapley and Ker- SeP The shaded areas represent 95% confidence intervals.
35

---

Published as a conference paper at ICLR 2023
I QUALITATIVE EXAMPLES
This section provides qualitative examples for VIT Shapley and the baseline methods. When vi- sualizing explanations from each method, used the icefire color palette, a diverging colormap implemented in the Python Seaborn package (Waskom 2021). Negative influence is an important feature for ViT Shapley, and diverging colormap allows us to highlight both positive and negative contributions. To generate the plots shown in this paper, we first calculated the maximum absolute value of an explanation and then rescaled the values each end of the color map; next, we plotted the original image with an alpha of 0.85, and finally we performed bilinear upsampling on the explanation and overlaid the volor-mapper result with an alpha of 0.9. The alpha values can be tuned to control the visibility of the original image.
Figure 13
shows comparison between VIT Shapley and KerneISHAP explanations for several examples from the ImageNette dataset. The results are nearly identical, but VIT Shapley produces explanations in a single forward pass while KernelSHAP is considerably slower. Determining the number of samples required for KerneISHAP to converge challenging, and used the approach proposed by prior work with a convergence threshold of / = 0.2 (Covert and Leel (022]) ). With this setting and with acceleration using paired sampling, the KernelSHÄP explanations required between 30 minutes to hour to generate per image, versus a single forward pass for ViT Shapley.
Figure 15 and BadaanFFiagg Figure 16 show comparisons between VIT Shapley and the baselines on ImageNette samples. We only show results for attention last, attention rollout, Vanilla Gradients, Integrated Gradients, SmoothGrad, LRP, leave-one-on and VIT Shapley; we excluded VarGrad, GradCAM and RISE because their results were less visually appealing. The explanations are shown only for the target class, and we observe that ViT Shapley often highlights the main object of interest. We also find that the model is prone to confounders, as VIT Shapley often highlights parts of the background that are correlated with the true class (e.g., the face of a man holding a tench, the clothes of man holding chainsaw, the sky in parachute image).
Figure 14
Similarly, Figure 17 Figure 18 and Figure compare VIT Shapley to the baselines on example images from the MURA dataset. The ViT Shapley explanations almost always highlight clear signs of abnormality, which are only sometimes highlighted by the baselines. Among those shown here, LRP is most similar to VIT Shapley, but they disagree in several cases.
compare VIT Shapley to the baselines on example images from the Pets dataset. We showed one randomly sampled image per each class (i.e., breed). We observe that ViT Shapley highlights distinctive features of breeds (e.g., the mouth of Boxer or the fur pattern of Egyptian Mau), and rarely puts significant importance on background patches.
Next, Figure 20
Figure 21 Figure 22, and Figure 23
Finally, Fiiuure 24 shows several examples of non-target class explanations. These results show that ViT Shapley highlights patches that can push the model's 's prediction towards certain non-target classes, which not the case for other methods. These results corroborate those in Table 2 and Table 5 which show that VIT Shapley offers the most accurate class-specifee explanations.
36

---

Published as a conference paper at ICLR 2023
stayet
Dna shaet raya
Ourch
Parechule
sump
Cartage truck
VF 2uee
Ennch
Vir thep
tamanteg
spreger
VIr Shepayy
VI Shaltt
Gas pump
VTT chipiyy
Figure 13: ViT Shapley vs. KemnelSHAP comparison (ImageNette).
37

---

Published as a conference paper at ICLR 2023
Caccette player
lastAttention
Shapley
Garbage truck
Attention last Attention rellout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
Shapley
Tonch
Attention Attenttin rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VTT Shapley
Engllich springor
Altention Atentton rellout
Vanilla
IntGrad
SmoothGred
LRP
Leaverone-out
Shapley
Attention
Vanilla
SmoothGGaad
LRP
Shapley
Parachute
Attention last rollout
Vanilla
IntGrad
SmoothGradd
LRP
Leave-one-out
VTT She
French horn
Attention lastAttention rellout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Attention last Attention rollout
Vanilla
IntGrad
Smooth
LRP
Leave-one-out
Shapley
lastAttention
Shapley
Gas pump
Attention last Attention rollout
Vanilla
IntGrad
SmoothhGrad
LRP
Leave-one-out
VTT Shapley
Figure 14: ViT Shapley vs. baselines comparison (ImageNette).
38

---

Published as a conference paper at ICLR 2023
Cascatte player
lastAttention
SmoothGrad
Shapley
Garbage truck
Attention last Attention rellout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
Shapley
Tonch
Attention Attenttio rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VTT Shapley
Englich springor
Altention last Attenton rellout
Vanilla
IntGrad
SmoothGred
LRP
Leaverone-out
Shapley
Church
Attention
Vanilla
SmoothGGaad
LRP
Shapley
Parachute Attention last Attention rollout
Vanilla
IntGrad
SmoothGradd
LRP
Leave-one-out
VTT She
French horn
Attention lastAttention rellout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-ou
ViT Shapley
Attention last Attention rollout
Vanilla
IntGrad
Smooth
LRP
Leave-one-out
Shapley
lastAttention
SmoothGrad
Shapley
Gas pump
Attention last Attention rollout
Vanilla
IntGrad
SmaothhGrad
LRP
Leave-one-out
VTT Shapley
Figure 15; ViT Shapley vs. baselines comparison (ImageNette).
39

---

Published as a conference paper at ICLR 2023
Cassette player
lastAttention
Shapley
Garbage truck
Attention last Attention rellout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one:out
Shapley
Tonch
Attention sttteettoo rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VTT Shapley
Engliich springer
Altention Attention rellout
Vanilla
IntGrad
SmoothGrad
LRP
Leaverone-out
Shapley
Church
Attention
Vanilla
SmoothGGaad
LRP
Shapley
Parachute
Attention Attentnnn rollout
Vanilla
IntGrad
SmoothGGrad
LRP
Leave-one-out
VTT She
French horn
Attention lastAttention rellout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
ViT Shapley
saw
Attention last Attention rollout
Vanilla
IntGrad
Smooth
LRP
Leave-one-out
Shapley
lastAttention
SmoothGrad
Shapley
Gas pump
Attention last Attention rollout
Vanilla
IntGrad
SmeothhGrad
LRP
Leave-one-out
VT Shapley
Figure 16: ViT Shapley vs. baselines comparison (ImageNette).
40

---

Published as a conference paper at ICLR 2023
Abnormal
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Abnormal
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Abnormal
Attention last Attention rollout
Vanilla
IntGrad
SmeothGrad
LRP
Leave. one out
VIT Shapley
Leave-one-out
Shapley
Abnormal
Attention
SmoothGrad
Shapley
Abnormal
Attention Altention rollout
Vanillla
IntGrad
SmeothGrad
LRP
Leave-one-out
VTT Shapley
Abnormal
Attention Attention rollout
Vanillla
IntGrad
SmeothGrad
LRP
Leave-one-out
VT Shapley
Abnormal
Attention last Attention rollout
Vanillla
IntGrad
SmoothGGrad
LRP
Leave-one-out
VIT Shapley
Attention Attentiooo rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
Shapley
Abnormal
SmeothGrad
Shapley
Figure 17: ViT Shapley vs. baselines comparison (MURA).
41

---

Published as a conference paper at ICLR 2023
Abnormal
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Abnormal
Attention last Attentiion rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VTT Shapley
Abnormal
Attention last Attention rollout
Vanilla
IntGrad
SmeothGrad
LRP
Leave one out
VIT Shapley
SmoathGrad
Leave-one-out
Shapley
Abnormal
Attention
SmoothGrad
Shapley
Abnormal
Attention last Attention rollout
Vanillla
IntGrad
SmeothGrad
LRP
Leave-one-out
VTT She
Abnormal
Attention Attentino rollout
Vanillla
IntGrad
SmeothGrad
LRP
Leave-one-out
VT Shapley
Abnormal
Attention last Attention rollout
Vanillla
IntGrad
SmoothGGad
LRP
Leave-one-out
VIT Shapley
Attention Attentioon rollout
Vanillaa
IntGrad
SmeothGrad
LRP
Leave-one-out
Shapley
Abnormal
SmeothGrad
Shapley
Figure 18: ViT Shapley vs. baselines comparison (MURA).
42

---

Published as a conference paper at ICLR 2023
Abnormal
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Abnormal
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Abnormal
Attention last Attention rollout
Vanillaa
IntGrad
SmeothGrad
LRP
Leave one out
VIT Shapley
SmoathGrad
Leave-one-out
Shapley
Abnormal
Attention
IntGrad
SmoothGrad
Shapley
Abnormal
Attention last Attentioo rollout
Vanillla
IntGrad
SmeothGrad
LRP
Leave-one-out
VTT Shapley
Abnormal
Attention Attentioo rollout
Vanillla
IntGrad
SmeothGrad
LRP
Leave-one-out
VIT Shapley
Abnormal
Attention Attention rollout
Vanillaa
IntGrad
SmoothGGad
LRP
Leave-one-out
VIT Shapley
Attention Attention rollout
Vanilla
IntGrad
SmeothGrad
LRP
Leave-one-out
Shapley
Abnormal
SmeothGrad
Shapley
Figure 19: ViT Shapley vs. baselines comparison (MURA).
43

---

Published as a conference paper at ICLR 2023
Alyssinian
Attentio n last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out out
VIT Shapley
American Bulldog
Attention last Attention rollout
Vanilia
IntGrad
SmosthGrad
IRP
Leave-one-out
VIT Shapley
American Pit Bull Terrier
Attention last Attontion rollout
IntGirad
SmoothGrnd
LRP
Shapley
Beagle
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LP
Leav-onee-out
VIT Shapley
Bengal
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Birman
Attention
Vanilla
IntGrad
SmoothGrad
Shapley
Bombay
Vanillaa
LRP
Leave-one-out
Shapley
bower Attention lest Altention rollout
Vanillaa
IntGrad
SmoothGred
LRP
Leave-one-out
VIT Shapley
British Shorthair
Attention last Altention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Figure 20: VIT Shapley vs. baselines comparison (Pets).
44

---

Published as a conference paper at ICLR 2023
Chihuahua
IntGrad
SmoothhGrad
Egyptian Mau
Attention st Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave one one-out.
VIT Shapley
English Cocker Spaniel
Attentian la Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-our
VIT Shapley
Engilsh
Attention Allention
IntGrad
SmoothGGadd
Shapley
German Shorthaired
Attention st Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave one out
VIT Shapleyy
Great Pyrenees
Altention last Allention rollout
Venilla
IntGrad
SmoothGradd
LRP
Leave-one-out
VIT Shapley
Shapley
Japanese Chin
Attention last Attontion rollout
Vanilla
IntGrad
SmoothGrad
LRP
Laave-one-o4
VIT Shapley
Keeshond
Attention last, Allention rollout
Vanilla
IntGrad
Smoothhrrad
LRP
Leave-one-out
VIT Shapley
Leonberger
Figure 21: ViT Shapley vs. baselines comparison (Pets).
45

---

Published as a conference paper at ICLR 2023
Maine Coon
Altention last Attention rollout
Vanilla
IntGrad
SmeothGrad
LRP
Leave-one-ouR
Shapley
Ministure Pinscher
Attention last Attention rallout
Vonilla
IntGrad
SmaothGrad
LRP
Leave-Bn onn-out
VIT Shapicy
eewfoundland
Attention
IntGrad
SmaothGrad
Shapley
Persian
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
VIT Shapley
Attention last Attention
IntGrad
SmaothGrad
Leave-one-out
Shapley
Pug
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
Shapley
Ragdoll
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one oneee----out
Shapley
Russian Blue
Attention last Attention rallout
Vanilia
IntGrad
SmaathGrad
Shapley
Saint Bernard
Attention last Attention rollout
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-out
ViT Shapley
Samoyed
Shaplcy
Figure 22: ViT Shapley vs. baselines comparison (Pets).
46

---

Published as a conference paper at ICLR 2023
Scottish Terrier
Attention la st Attention roll out
Vanilla
IntGrad
SmoothCrad
LRP
Leave-one-oout
VIT Shapley
SmoothGrad
Leave-one-out
Shapley
Siamese
Attention la Attention roll
Vanilla
IntGGadd
SmoothGrad
LRP
Leave-one-oul
VIT Shapley
Sphynx
Vanilla
Shapley
Staffordshire Bull Terrier Altention last Altention roll out
Vanillaa
IntGrad
Smoothrrad
LRP
Leave-one-out
VIT Shapley
Wh Wheaten Terrier
Attanttion Attention
Vanilla
IntGrad
SmoothGrad
Shaplay
Yorkshire Terrier
Attention la Attentian roll out
Vanilla
IntGrad
SmoothGrad
LRP
Leave-one-oul
VIT Shapley
Figure 23: ViT Shapley vs. baselines comparison (Pets).
47

---

Published as a conference paper at ICLR 2023
The label English springer
The whel:
The lnbel pumpp
English springer
Golfball
Golf ball
Parachutt
Eas pump
French hom
Acter Eaam last
Attention Riterten
Artention
Attantion
Vamilta
intGrad
Intrra
SmozthGrod
SmoothGradd
UPP
Shepley
VIT Shopley
Stap
Insertion
Insertion
Insertion
Insertion
Insertion
Insertion
Probasilty
Potasilty
Pobauiity
Deletion
Del etion
Deletion
Deletion
Deletion
Deletion
Probabiity
Protabity
Shapley
Shapley
VIT Shapley baselines for non-target classes (ImageNette). Left: VIT Shapley shows that the grass in the background provides evidence for the golf ball class. Middle: ViT Shapley shows that the sky and its reflection the water provide evidence for the parachute class. Right: ViT Shapley shows that the metallic gas pump handle provides evidence for the French horn class.
Figure 24:
48


Reviews:
[{"review_id": "xD1qk3g_ToY", "reviewer": "Reviewer_QpHf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper builds on the FastSHAP approach (Jethani et al., 2021) and applies it to the ViT attention-based architecture used in image analysis. The goal of FastSHAP is to learn a dedicated predictor to estimate the Shapley values expressing input feature attribution for explainability purposes. Another contribution is a mathematical justification of the FastSHAP loss used to estimate the Shapley value predictor. The approach is evaluated on two small-size datasets using insertion, deletion and faithfulness metrics.\n", "summary_of_the_review": "The main contribution of the paper is to apply the FastSHAP approach (Jethani et al., 2021) to ViT architecture, an attention-based model, and to propose a mathematical justification that the loss used to estimate the Shapley value predictor.\n\nThe application to ViT architecture is rather straightforward: deactivate the feature/tokens using the attentional weights. The mathematical justification of the loss seems well funded but is independent from the attention-based architecture. The approach is evaluated on small size databases using standard metrics but is not compared, as it seems to me, to another feature attribution study that explicitly exploits attention to build the explanation (Chefer et al., 2021).\n\nThe paper is well written and clear, but the content, for me, is rather at the level of a specialized workshop given the nature of its novelty (application of a known approach to another neural architecture and small theoretical contribution).\n\n", "strength_and_weaknesses": "== Strengths ==\n\n\n- Well funded mathematical result, at least seems to be (I didn’t check the proofs in the appendix).\n\n- Reasonable number of experiments to justify the approach, although on small size image databases (ImageNette & MURA).\n\n\n== Weaknesses ==\n\n- Limited novelty: rather straightforward application of FastSHAP to a single attention-based architecture, ViT (see the “novelty” section).\n\n- No explicit exploitation of attention information as done in  (Chefer et al., 2021), for instance. Attention is only exploited for the optimization of the Shapley value estimator.\n\n- Small size of the datasets used for evaluation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EsfXLtPc1I0", "reviewer": "Reviewer_qY28", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel approach to compute Shapley value explanations for vision transformers. To evaluate shapley values for any model we need model's predictions on masked feature inputs. Authors propose a novel way to approximate these by training a ViT model using a loss function designed specifically for Shapley values. The method is evaluated using explainability metrics that do not require ground truth information. ", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "Overall the approach of leveraging ideas from transformer training to compute shapley is interesting and building on the work of Jethani et al. is novel. Authors also present analytical error bounds on shapley approximation which can be useful. The experimental results are convincing. \n\nOne possible explanation evaluation approach to consider is the following, where explanations are compared with ground truth: \nhttps://arxiv.org/pdf/2104.14403.pdf\n\nIt would be good to discuss how ideas presented in this work could be utilised/extended for text data. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Bkw1Njop8um", "reviewer": "Reviewer_eEy2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new way to estimate Shapely values for Vision Transformers. It is required to evaluate the model on partial inputs for estimating Shapely values. For CNNs, this is difficult (e.g. masking areas with gray produces out-of-distribution samples). However, the inputs of vision transformer can be more easily removed, as the attention itself can be masked. \n\n\n\nAs the estimation of Shapely values is notoriously difficult, the paper proposes to learn them. For this, an existing loss function is used from Jethani et al. (2021). Furthermore, the authors show this loss function to bound the Shapely value estimation error.\n\n\n\nThe evaluation is done against several baselines on the ImageNette and MURA datasets.\n\n## ", "summary_of_the_review": "This is a good paper. It has both interesting applied contributions (using ViT for masking) and theoretical contributions. Also, the presentation is well done. ", "strength_and_weaknesses": "**Strengths:**\n\n- This work provides new theoretical insight into how the optimized objective bounds Shapley values.\n- The evaluation is done against a large selection of baselines\n- The method outperforms the baselines consistently\n\n**Weakness**:\n\n- The proposed approach could have also been tested on NLP transformer models.\n- Training an additional network adds complexity.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "5VrQ5OGYITl", "reviewer": "Reviewer_trn3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary.\n\nThis paper is dedicated to developing efficient Shapley value approximation for vision transformers. The authors first leverage an attention masking approach to evaluate ViTs with partial information, and then they propose a procedure for generating Shapley value explanations via a separate, learned explainer model. Extensive experiments with diverse comparisons are conducted.", "summary_of_the_review": "Good idea, but not more discussions and experimental analyses", "strength_and_weaknesses": "Pros.\n\n1. Confidence intervals of experimental results are provided.\n\n2. Multiple metrics and baseline approaches are considered.\n\n3. The paper is well-written and easy to follow.\n\nCons.\n\n1. It is very hard to say whether model-agnostic can perform better. More discussions between model-dependent and model-agnostic approaches are needed. In my opinion, involving more ViT-specific information can produce a reliable result.\n\n2. The authors first consider the ViT evaluation with partial information and then leverage a separate, learned explainer model to generate the Shapley value. It seems these two parts are loosely connected.\n\n3. Only one natural image dataset and one medical image dataset are used for experiments. More datasets are needed to support the effectiveness of the paper's proposals.\n\n4. Related works are outdated. More references from 2021 and 2022 should be included.\n\n5. More ViT backbones should be considered.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%