Class-Discriminative Attention Maps for Vision Transformers
Authors: Lennart Brocki
Jakub Binda
Neo Christopher Chung
Venue: Accepted by TMLR ()

Abstract: Importance estimators are explainability methods that quantify feature importance for deep neural networks (DNN). In vision transformers (ViT), the self-attention mechanism naturally leads to attention maps, which are sometimes interpreted as importance scores that indicate which input features ViT models are focusing on. However, attention maps do not account for signals from downstream tasks. To generate explanations that are sensitive to downstream tasks, we have developed class-discriminative attention maps (CDAM), a gradient-based extension that estimates feature importance with respect to a known class or a latent concept. CDAM scales attention scores by how relevant the corresponding tokens are for the predictions of a classifier head. In addition to targeting the supervised classifier, CDAM can explain an arbitrary concept shared by selected samples by measuring similarity in the latent space of ViT. Additionally, we introduce Smooth CDAM and Integrated CDAM, which average a series of CDAMs with slightly altered tokens. Our quantitative benchmarks include correctness, compactness, and class sensitivity, in comparison to 7 other importance estimators. Vanilla, Smooth, and Integrated CDAM excel across all three benchmarks. In particular, our results suggest that existing importance estimators may not provide sufficient class-sensitivity. We demonstrate the utility of CDAM in medical images by training and explaining malignancy and biomarker prediction models based on lung Computed Tomography (CT) scans. Overall, CDAM is shown to be highly class-discriminative and semantically relevant, while providing compact explanations.

Published in Transactions on Machine Learning Research (11/2024)
Class-Discriiinative Attention Maps for Vision Transformers
Lennart Brocki
L.brocki @ war. edu...du.yy pl j. b.binda @student. ww. edu.pl n. n.chung &wns edu.
Jakub Binda
Neo Christopher Chung
Institute of Informatics, University of Warsaw
Reviewed on OpenReview: https: // openreview. net/ farmm? id- MWT7
Abstract
Importance estimators are explainability methods that quantify feature importance for deep neural networks (DNN). In vision transformers (ViT), the self-attention mechanism natu- rally leads to attention maps, which are sometimes interpreted as importance scores that indicate which input features VIT models are focusing on. However, attention maps do not account for signals from downstream tasks. To gencrate explanations that are sensitive to downstream lasks, have developed class-dssciiminative allention maps (CDAM), gradient-based extension that estimates feature importance with respect to known class or latent concept. CDAM scales attention scores by how relevant the corresponding to- kens are for the predictions of classifier head. In addition to largeting the supervised classifier, CDAM can explain an arbitrary concept shared by selected samples by measur- ing similarity in the latent space of ViT. Additionally, we introduce Smooth CDAM and Integrated CDAM, which average series of CDAMs with slightly altered tokens. Our quan- titative benchmarks include correctness, compactness, and class sensitivity, in comparison to 7 other importance estimators. Vanilla, Smooth, and Integrated CDAM excel across all three benchmarks. In particular, results suggest that existing importance estimators may not provide sufficient class-sensitivity. We demonstrate the utility of CDAM in med- ical images by training and explaining malignancy and biomarker prediction models based on lung Computed Tomography (CT) scans. Overall, CDAM is shown to be highly class- discriminative and semantically relevant, while providing compact explanations.
Code available:
Attention Map
CDAM
Attention Map
CDAM
Gas Pump
Tench
Golf Ball
French Horn
Figure 1: Extending attention maps (AM), the proposed class- discrimiiiieeee attention maps (CDAM) quan- tily and visualize input features that are relevant for the target class in Vision Transformers (VIT). We visualize importance scores obtained with CDAM for a linear classifier with VIT-S/8, trained with DINO Caron et al. 2021). Orange and blue colors correspond to positive and negative values, respectively.

---

Published in Transactions on Machine Learning Research (11/2024)
1 Introduction
Vision transformers (ViT) are a family of computer vision models based the transformer architecture
Vaswani et al. 2017 that demonstrate comparable or even better performance than convolutional neural
networks (CNN) on many object detection and classification tasks (Dosovitskiy el. al. 2020: Wu el al. 2020).
A powerful feature of the transformers is that the attention mechanism provides a built-in explanation of the model's inner workings, if attention is understood as a weighing of the relevance of individual tokens Muulenbach et al. 2018; Bahdanna et al. 2014 Serrano & Smith 2019: Thorne et al. 2019). While atlention maps provide intuitive and inherent interpretability, they fail to be discriminative with respect to a target class. Recognizing the potential of attention maps for explaining a ViT's inner workings, we have developed clas-diieriminatiie attention maps (CDAM), specific signals based on classifier or a latent concept.
madient-bbssed extension that makes them sensitive to class-
Attention maps are the visualization of the self-attention of the class token query [CLS] in the final attention layer. Attention heads can be visualized separately, and in certain instances, may be related to semantically meaningful concepts (e.g., in language (Clark et al.] 2019)). Recently, ViT trained with self-supervise learning (SSL) methods |Caron et al. [021] Touvron et al.j 2021 Oquab et al., 2024) have shown excellent performance leveraging a greater amount of unlabeled data. Attention maps from such self-supervised ViT models provide high-quality semantic segmentations of input images. These attention maps clearly distinguish regions of interest (ROT) from the background with very little noise, with different attention heads often focusing on semantically distinct regions (Caron et al. 2021 b. There have been many improvements to ViT including the introduction of convolution (Wu et al.] 2021 Yuan et al. 2021) and spatial sparsity (Li itt al. 2021). ). Attention maps, as well as our proposed CDAMs, are applicable to all VIT architectures with self-attention.
The major shortcoming of attention maps, when considering them as explainable AT (XAT) method, is that they purely operate the level of the last attention layer and therefore do not take into account any information coming from downstream task, such as a classifier head. The proposed CDAM overcomes this shortcoming by calculating the gradients of the class score, i.e. the logit. value of a target class, with respect to the token activations in the final transformer block (visualized in Fig. ((a)), Essentially, CDAM scales the attention scores by how relevant the corresponding tokens are for the model's decision. CDAM therefore inherits the desirable properties of attention maps, but additionally indicates the evidence or counter-evident for target class that the VIT relies on for its predictions (examples in Fig. D. We demonstrate that it is class-discimmnative in the sense that the heat maps clearly distinguish the object representing the target class from the background and, importantly, from objects that belong to other classes. Therefore, CDAM is a useful tool for explaining the decision-maliee process of classifiers with a ViT backbone. Note that the goal of CDAM is to explain the classificr head and the final transformer block.
In addition to the classifier-based explanations, we present a concept-based approach that replaces the class score with the similarity measure in the latent space of the ViT, where a concept is defined through selected images. Averaging the latent representations of the example images yiclds a concept embedding, which is analogous to a concept vector in variational autoencoders (VAE) and related generative models [Brocki& & Chung, 2019). This allows obtaining a heat map for the target concept the model has never been explicitly trained on, which is of particular interest for self-supervised models. We further demonstrate when computing gradients of a class (Section 3.1 or concept similarity (Section 3.2h score, with respect to the token activations the final transformer block, smoothing or averaging could be applied to improve performance. Specifically, we introduce two additional variants of CDAM; (1) Smooth CDAM averages over multiple gradients that are computed with additional noise to tokens (inspired by Smoothrrad (Smilkov et al. 2017)) and (2) Integrated CDAM takes the integral of the gradients along the path from the baseline to the token in the final transformer block (inspired by Grad (Sundararajan et al. 2017))
We conduct several quantitative evaluations focusing on correctness, class sensitivity, and compactness. Correctness is measured by perturbing an increasing amount of input features and measuring the model output (Brocki & Chung. 2023c b). By using the ImageNet samples (Deng et al. (209) with multiple objects (Beyer et al.] 2020) and applying importance estimators for different classes, we quantify the level of dlass-iisrimminatinn Sparsity and shrinkage of explanations are measured by the number of near-zero
2

---

Published in Transactions on Machine Learning Research (11/2024)
importance scores. Besides the proposed vanilla, Smooth, and Integrated CDAM, we also employed 7 different explainability methods in comparison. Three variants of CDAM out outperfom other methods on quantitative benchmarks.
Lastly, we have applied CDAM on a ViT fine-tuned on the Lung Image Database Consortium image col- lection (LIDC) (Armato ITI et al. 2011). Two ViT-based models are built for predicting malignancy and biomarkers. CDAM is shown to be highly class-diseriminative and semantically relevant, while providing implicit regularization of importance scores.
2
Related Works
Interpretability of VIT, and more broadly explainable artificial intelligence (XAI), is an active area of re- search, since understanding their decision-making process would help improve the model, identify weaknesses and biases, and ultimately increase human trust in them. The proposed dlass-disrimminative attention map (CDAM) related to both attention and gradient-based explanation methods for deep learning models.
Gradient-based methods operate by backpropagating gradients from the prediction score to the input features to produce feature attribution maps, which is also called saliency maps. Many variants have been proposed, see (Simonyan ct al.
Sundararajan et al. 2017 Selvaraju et al.) 2017 Smilkov et al. 2017 Shrikumar let al. 2016) to just name few. In particular, CDAM is similar to and inspired by the input x gradient method
2013
(Shrikumar et al., 2016 because it involves calculating the gradients with respect to tokens multiplied by their activation (Eqs. (2 and (4)). The main difference to most gradient based methods is that we don't 't backpropagate the gradients all the way to the input tokens. CDAM stops backpropagation at the tokens that enter the final attention layer.
In that regard, our method is related to GradCam (Selvaraju et al., 2017) which backpropagates the gradients to the final convolutional feature map in a CNN. GradCam and CDAM therefore share the property that they operate on high-level features which is presumably the reason why they are both sensitive to the targeted class, a feature that many explanation methods do not have [Rudin) 2019). Several methods have attempted to apply gradient-based methods or Layer-Wing Relevance Propagation (LRP) (Bach et al.] 22015)
to Transformer architectures (Atanasova et al. 2020 Ali et al. 2022: Voita et al., 2019 A Abnar& & Zuidema
2020 Chefer et al. 20 mostly in the context of NL. tasks. GradCam for ViT backpropages the gradients
to the outputs of the final attention layer (Gildenblat
2021
Attention has often been used to gain insight into model's inner workings (Mullenbach et al. 2018 Bah-
danau et al. 2014 Serrano & Smith 2019 Thorne et al. 2019) since it seemss intuitive that this weighing of individual tokens correlates with how important they are for the performed task. There is also an ongoing discussion about whether allention provides meaningful explanations (Serrano & Smith 2019 Wiegrefle & Pinter 2019 Jain & Wallace 2019). Furthermore, methods such as "attention rollout" (Abnar & Zuidema 2020 have been proposed that aim to quantify how the attention flows through the model to the input tokens. Chefer et al.) (021aa) defines the relevancy matrices, as the Hadamard product of the attention map and the gradient of fo w.r.t. to the attention layer, accumulated at each layer. This relevance propagation (RP) method (Chefer et al. 2021a), to the best of our knowledge, shows the highest degree of class-discrimination in ViT models. This is used in our study as one of the bascline importance estimators.
Many papers have been published on the problem of evaluating post-hoc explanation methods, leading even to the proposal of meta-evaluations (Hedström al., 2023). In a recent overview of quantitative evaluation methods (Nauta et al. 2023b) 12 desirable properties have been identified. We primarily focus three of them, namely correctness, contrastivity, and compactness. In particular, carefully designed occlusion catusiion/petuuraateei pprtuurbatoonnn studies can help us calculate the fidelity statistics (Brocki & Chung) 2023c). which is also called the symmetric relevance gain (SRG) measure (Bluecher el al. 2024). Adebayo el al. (2018) first demonstrated that many importance estimators, which purport to be sensitive to the internals of classifier, produce similar importance scores regardless of the relation of inputs and classes in the training data, which they called "the data randomization test". Generally, if two importance scores with respect to two different classes produce very similar heat maps, the importance estimator is not class-sensitive. We quantify this
3

---

Published in Transactions on Machine Learning Research (11/2024)
concept in class sensitivity evaluation benchmark to measure whether explanation is sensitive to the classes.
3 Methods
Adaptation of the transformer architecture to computer vision hinges creating tokens from small patches of the input image (Dosovitskiy et al., 2020). In addition to those image patch tokens, the classification token [CLS] is added, which is utilized for downstream tasks such as classification. Attention maps are the two-dimensional visualization of the self-attention in the final attention layer with [CLS] as query token, where the attention scores are averaged over all heads. In our notation, [CLS] refers to the classification token before and [CLS]' after the final transformer block.
(a) Overview of CDAM
(b) Final Transformer Block
f.
Classifier
7
(CLS)
((),
Similarity
Gradients
[oue]
Norm
Transformer
Multi Head Attention
Inference
1)x
Transformer Block
Norm
Input Image
CLSS
Figure 2: Graphical scheme of mhaa-discrimmnative attention maps (CDAM). (a) CDAMs are obtained by first propagating image through the transformer blocks and using its latent representation | = [CLS] to infer class score fe or similarity score g(l,tc) (see Eq. or Eq. 4 transformer block, adapted from (Dosovitskiy et al. 2020
(b) Detailed view of the final
Before fine-tuning, attention maps are not class-discrmminative since they do not take into account any signal coming from a downstream task. Therefore, have developed class-discrmminative attention maps (CDAM) that be computed based known class latent. concept. In order to estimate token's 's relevance. CDAM computes gradients of class score (Section 3.1 or concept similarity score (Section 3.2 obtained downstream using the [CLS]' token, with respect to the token activations in the final transformer block (Fig. 2 Note that CDAM may not explain the full ViT architecture with multiple transformer blocks. Instead, CDAM focuses on the downstream task and the last transformer block.
In the first scenario, the class score obtained from the corresponding activation in the prediction vector of a classifier trained top of the VIT in supervised fashion. In the second scenario, define concept through small set of examples (e.g. ten images of dog) and obtain concept vector by averaging their latent representations. Then, without training classifier, similarity measure between the concept vector and the latent representation of a targeted sample image is computed and used as the target for the gradients. We call the resulting maps of feature relevance "class-discriminatie attention maps" because they scale the attention scores
(QUass K. Vdl
A; = softmax
(1)
by how relevant the corresponding tokens are for the class similarity score. Here Q[CLS] T(CLs) Wº with WQ E Rdmoddal xdę and K; K with WK Bdmoad xde In this notation Q and K are the query
4

---

Published in Transactions on Machine Learning Research (11/2024)
and key matrices which pack together all the query and key vectors (Vaswani et al. 2017 and the indices, e-g. K2, select single query or key vectors.
Self-supervised learning (SSL) methods such as DINO can train VIT models whose attention maps are high- quality semantic segmentations of the input, without using labels (Caron et al.) 2021). Therefore, we utilize ViT model (Dosovitskiy et al.) 2020) pre-trained with DINO as backbone for our main experiments and applications. Furthermore, we demonstrate general applicability of CDAM by using other VIT models, namely pre-trained with Supervised Weakly through Hashtags (SWAG) (Singh et al.) 2022), Data-efficient image Transformers (DelT) (Touvron et al. 2021) and DINOv2 (Oquab et al. 2024). We denote details about the ViT architecture, training schemes, and model performance in the Section and Appendix A.5. Note that the heat maps shown in this article are scaled independently.
3.1 CDAM for known class
Consider a linear classifier f Rddmodel + RC on top of the VIT that takes the [CLS]' token as input and maps it to a prediction vector with C classes, where dmodel is the embedding dimension of the ViT. The importance score She with respect to the token T3 is defined as
Ofc Σ Tis
St,c
(2)
OT.
where Tl is the j-th component of the i-th token that is fed to the last attention layer (Fig. 2 our experiments that calculating the gradients with respect to the layer-normalized tokens yields far better results than using the not normalized ones.
We found in
This definition is first-order estimation of the contribution of T; to fo in the sense that it would be exact if fa depended on Ti only linearly. It is inspired by gradient-based explanation methods with the difference that we don't backpropagate the gradients all the way to the input layer but only to the input tokens of the final attention layer. This allows us to gain insight into the decision-maakng process of the combination of ViT and classifier based on high-level features. This approach leads to attention maps that are class- discriminative and well-aligned with the semantic relationships of the class and different parts of an image (Tig. 1.
3.2 CDAM for latent concept
Another way of probing high-level features learned by the VIT is through the use of similarity measure g in the latent space of the VIT. We compare the latent representation 1(X) = Tiass (X) of a sample image X and a concept vector le (Brocki & Chung 2019). i.e. the averaged latent representation of images that are chosen to represent the concept of interest:
1 (X)).
Le This approach is directly inspired by the concept saliency map Brocki & Chung. 2019). The definition of the importance score in this setting is analogous to Eq.
(3)
X
2
Sr.c = ET 0g(1.L)
(4)
ØT,
Possible choices for g are for example the L distance, cosine similarity or dot product, where we use the latter in this article. The definition Eq. 4 cstimates the contribution of T; to g, such that it allows us gauge how much each token is driving the similarity between and le-
3.3 Smooth and Integrated CDAM
SmoothGrad (SG) Smmlkov et al. 2017) and Integrated Gradients (IG) (Sundararajan al. 2017 extended
saliency maps (Simonyan et al. 2013), which were introduced in the context of CNNs visualizing gradients
5

---

Published in Transactions on Machine Learning Research (11/2024)
of the class with respect to the imput features. Briefly, (Smilkov et al. 2017 adds small Gaussian noise to sample image times, resulting in n vanilla saliency maps. SG is an average of those saliency maps. In IG, instead of adding random noise, (Sundararajan et al. 2017) uses multiple saliency maps obtained from interpolating between a given sample and a basclinc (c.g., a black image).
We have adapted the analogous modifications to CDAM, resulting in two additional variants which we call Smooth and Integrated CDAM. Importantly, instead of modifying the input images, we vary the tokens that enter the final transformer block. In Integrated CDAM, we interpolate between baseline and the tokens themselves, resulting in n maps. Notice that, since we apply the same method, the axioms of IG still hold with the only difference that they apply to the tokens that enter the final transformer block instead of the input pixels. In Smooth CDAM, we repeatedly add a small Gaussian noise to the tokens and again obtain n maps.
Specifically, Smooth and Integrated CDAM are defined as follows:
1
(sc(T))
T
(5)
0T
n Σ
Of k/n (T T))
S = (Tay-Ta
(6)
OTis
where TV = T; N T "²) with N (0, representing Gaussian noise with standard deviation ø. For experiments, we used n 50. For the baseline 7; we choose the null vector of 0. We implement these two variants by first propagating an image through the ViT up to the final transformer block, where the activations of the tokens are obtained. Next, the activations are manipulated by either applying the noise interpolation step and the manipulated tokens are then further propagated to obtain the final VIT output.
3.4 Quantitative Evaluation
We perform quantitative evaluations of the correctness, class sensitivity, and compactness of importance estimators. In addition to proposed methods (vanilla, Smooth, and Integrated CDAM), we perform evaluations on attention maps, Input Gradients, relevance propagation (RP) (Chefer et al. 2021a). partial
Layer-wise Relevance Propagation (LRP) (Voita et al.] 2019). SmoothGrad (Smilkov et al. 2017 , IntGrad
(Sundararajan et al. 2017), and GradCam for VIT
Gildenblat 2021 Note that basod on classic LRP
[Bach et al. 2015), partial LRP (Voita et al.) 2019) provides improvement for multi-head self-attention mechanisms in modern transformer architecture (Voita et al. 2019). Relevance propagation is state-of- the-art importance estimator for explaining a prediction (e.g., class discriminatory) based on VIT (Chefer et al. 2021a). For IntGrad, SmoothGrad, Integrated CDAM, and Smooth CDAM, 50 is used. The noise (a) used in SmoothGrad and Smooth CDAM is selected based on the best performance validation set of 200 images.
To evaluate the correctness (also known as faithfulness) of importance scores, we employ two versions of input perturbation-basen approaches (Samek et al., 2016 Petsiuk et al., 2018 Kindermans et al. 2017 Generally, an input feature (c.g., pixel) masked and a resulting model performance obscrved. In all cases, input features are perturbed by being replaced with their blurred versions. A Gaussian blur with 14 turns pixels uninformative while minimizing perturbation artifacts (Brocki & Chung 2023c).
Fidelity. The first evaluation method uses accuracy curves obtained from the Most Important First (MIF) and the Least Important First (LIF) perturbations, which are abbreviated as MIF MIF/LIF perturbation (Brocki & Chung 2023c b). Pixels are ranked by importance scores from a specific explainability method (e.g.. CDAM) and are perturbed the MIF or LIF order. In the MIF perturbation, an accurate importance score should lead to a relatively rapid decrease of the model output (logit) for the target class. For the LIF perturbation, we expect inverse relationship. Note that some importance scores are unsigned (i.e., only
6

---

Published in Transactions on Machine Learning Research (11/2024)
non-negative values) and others are signed. For signed importance, with negative scores faithfully indicating counter-evidence, there may be an increase in the logits if pixels with negalive scores are masked. For unsigned and correct importance scores, one expects initially constant logits in the LIF perturbation.
The model output (x) or (L)r()) as a function of the fraction of perturbed pixels p gives the perturbation curve (Brocki & Chung 2023c b). We average these curves over all samples, providing an overview of fidelity for given importance estimator. We further summarize the MIF and LIF perturbation evaluation by using the area MIF (called the fidelity statistics) under the difference of perturbation curves fLJF fMIP to quantify the fidelity of the explanation methods (Brocki & Chung, 2023b). Generally speaking, higher value of fidelity statistics ALIF-MIP indicates better explanation (Brocki & Chung. 2023c). Similar ideas are developed into the symmetric relevance gain (SRG) measure (Bluecher et al. 2024
Box sensitivity. For the second evaluation method, inspired by Sensitivity-n (Ancona et al. 2017), we mcasure how the model output (logit) changes as a group of pixels is perturbed. Pixels are selected by a randomly placed square box (of size x 8); then, the sum of their importance scores and the change in the model output are measured. For accurate importance estimators, we expect to high correlation between the two measures.
For a given image, we sample these values 100 times and calculate the Pearson correlation between the sum of importance scores and the model output change. This is performed for a range of differently sized boxes to obtain (s), the correlation statistics as a function of the box size s. These sensitivity curves are averaged over all samples. Performance of importance estimators varies with s, and therefore we use the area Abox under the box sensitivity curve to quantify correctness. Generally, larger box sensitivity statistics Abox implies greater correctncss.
In contrast to Sensitivity-n which randomly selects and perturbs pixels (Ancona et al. 2017 we use a random square of pixels. A box removes a certain region which mimics how image patches are processed by VIT. Box sensitivity may therefore remove larger parts or complete objects. In contrast, removing random pixels may leave enough neighboring pixels with sufficient semantic information such that perturbation may not have any impact of the model output. Indeed, we have observed that for Sensitivity-n, the correlation is very low (< 0.1) for all methods.
Class discrimination. We have designed evaluation benchmark to measure class discrimination, which can be applied on images containing multiple objects (see Xmulti dataset below). We obtain explanations for two distinct classes corresponding to the present objects. When performing the aforementioned evaluation benchmarks (fidelity and box sensitivity), we measure the change in the model output with respect to one of the classes. Evaluation metrics are thus obtained using importance scores based on either the correct target class (denoted by MIF/LIP or box) or wrong class (denoted by MIFI/LIF" or box').
censitiveedissrrinnatte For important estimators that are not sensitive discriminative to the target class, the importance scores are similar (or the same) regardless of which class is targeted. The change of the model output during perturbation would remain the same, irrespective of the class that was targeted to obtaín the importance ranking of pixels. For highly class-diserininative importance estimators, importance scores would be dif- ferent depending on which class is targeted. Furthermore, for accurate disrrniiiciiiiaavvee methods, the performances on fidelity and box sensitivity benchmarks can be expected to deteriorate substantially when perturbing according to the importance scores largeting the wrong class.
The areas under MIF/LIF perturbation box sensitivity curves, when using wrong class, denoted by MIF and Abox' respectively. The areas AALIFM and AAbo under the respective perturbation curves
AfLFEMI = (fLI - fMIR) (JUI - fMIF)), Afbox = foox - fbox'.
(7)
(8)
measure the degroe of class-disciimnation. In particular, if an importance estimator not sensitive to the target class at all, AfLIF-MIF Afbox 0. An importance estimator that performs poorly in the data randomization test of [Adebayo et al.) 2018) would have low AALIFMIP and A
7

---

Published in Transactions on Machine Learning Research (11/2024)
Compactness. We evaluate sparsity and shrinkage by counting the number of input pixels with importance scores smaller than 5% of the maximum value. A threshold is defined by x S with / = 0.05, where S is the maximum importance score for a given image and an importance estimator. Generally, a compact explanation is considered desirable, where sparsity and shrinkage are quantitative approaches (Nauta et al. 2023a).
3.5 Data
We have used two subsets of the ImageNet (denoted as Xval and Xmuiti) (Deng al. 2009 as well as lung computed tomography (CT) scans from the Lung Image Database Consortium image collection (LIDC) (Armato III et al., 22111. The first one, Xval, is a randomly selected subset of the ImageNet validation set consisting of 1000 images (Deng et al. 2009). The second one, Xmulti, selects 1000 images from the validation set that contain multiple objects (Beyer et al. 2020). We filtered out instances in which the class labels provided by (Beyer et al.; 2020) actually refer to the same object (c.g. notcbook and laptop) by calculating and thresholding the cosine similarity of the embedded class names. Embeddings were obtained using OpenAl's nezt-embedding---ooe
model.
The Xmmtti dataset is used to evaluate the degrec of dlas-dissrimination exhibited by the explanations. To this end, we obtain explanations w.r.t to two annolated classes, perform the /IIF perturbation and box sensitivity, and record the model output for the chosen target class. For example, for the top right sample in Fig. we would obtain the explanations for the classes Tench and Golden Retriever (not shown) and record the model output for the class Golden Retriever. For the right target (Tench), we obtain a fidelity statistics ALIF-MF- For the wrong target (Golden Retriever), AII'' the worst case scenario, where explanations for those two distinet classes are the same, ALIF' MIF' ALIF-MIP and the resulting AALIF-MIF is zero. Equivalent for AAbox.
The Lung Image Database Consortium image collection (LIDC) contains 1018 records of lung CT scans, that were collected from and validated by seven academic centers and eight medical imaging companies (Armato III el al. 2011). Presence of nodules (nodule < 3 mm;; nodule < 3 mm; non-nodule 2 3 mm) is annotated by up to 4 human annotators. Segmentation of nodules (i.e., ROI) is the average contour given by annotators. Eight biomarkers that are informative in clinical practices are also collected in LIDC: subtloty, caleification, sphericity, margin, lobulation, spiculation, texture, and diameter. For clinical descriptions of biomarkers, refer to (Opulencia et al. 2011). After preprocessing, 854 nodule crops of size, corresponding biomarkers, and labels (benign malignant) were saved and used for downstream procedures. Overall, the LIDC dataset used in this study contains 443 benign examples and 411 malignant examples.
4 Results
For our experiments using the ImageNett (Deng et al. 2009 we use the ViT-S/8 architecture (with a patch size of 8 5x8) 8) trained with DINO as backbone (Caron et al. 2021). Instead of using a ViT model that pre-
trained in self-supervised manner, we demonstrate CDAMs directly from an alternative VIT architecture trained with supervised weakly through hashtags (SWAG) (Singh et al. 2022) in Appendix A.5 We further
demonstrate applicability of CDAM on computed tomography (CT) scans, using DINO CCaron al. 2021
in Section 5 as well as using Data-efficient image Transformers (DelT) (Touvron et al. 2021 and DINOv2
(Qqab et al. 2024 in Appendix A.5.2
4.1
Qualitative Evaluation
4.1.1
CDAM for a known class
To demonstrate our method, we have trained a linear classifier on the ImageNet (Deng et al. 2009) that achieved an accuracy of 77.0%. We use random resized cropping and horizontal flipping with Torch default arguments as augmentation, Adam optimizer with learning rate 3 x 10-4, batch size of 128, and train for 10 epochs. The parameters of the ViT backbone frozen during training, so only the classifier head is
8

---

Published in Transactions on Machine Learning Research (11/2024)
CDAM
Partial LRP
Rel. Prop.
Input x Grad IntGrad
GradCam
Burger
Hotdog
Figure 3: Visual comparison of different importance estimators which obtain explanations w.r.t. different output classes. Similar heat maps regardless of target classes imply that importance estimators are not class discriminative. Attention maps (AM) are, by design, not class discriminative. Orange and blue colors correspond to positive and negative values, respectively.
trainable.
The predictions of this classifier were then used to create the heat maps shown in Fig. via
Eq. (2)
CDAM clearly distinguishes the region of interest (ROI) semantically related to the targeted class with positive values (yellow) from the rest of the image. Most backgrounds and other objects are assigned either zero (irrelevant) or negative scores (counter-evidence). Note that CDAM does not assign non-zero St, to tokens that have zero attention in attention maps, see also the scatter plot between CDAM and attention map Fig. ST CDAM therefore appears to compute Si,e by multiplying the attention with the relevance of the corresponding token for the targeted class. We discuss the proportionality of CDAM to the attention scores in Appendix A.I It is a nice feature since CDAM inherits the high-quality object segmentations present in attention maps. This qualitative evaluation of shrinkage and sparsity of CDAM is reflected in the quantitative measures for compactness (Section 4.2.1)
One of the most useful features of CDAM is that it is highly sensitive to the chosen class. In Fig. 5 (left), when either the zuccini or bell pepper class is chosen, CDAM clearly highlights that vegetable. Most of the other vegetables are assigned either near-zero or negative values (indicated in blue). Many other importance estimators do not distinguish the target class or only to a lesser degree. In Fig. 3 targeting the burger or the hotdog does not seem to change importance scores from partial LRP, Input Gradd, and InaG. Relevance propagation demonstrates some selective focus the RO1, but is not as discriminative as CDAM (Fig. 3). This visual impression of class-scnsitivity is supported by the quantitative evaluation in Section 1.2.2 Additional examples of CDAM and other interpretability methods are shown in Appendix A.3
4.1.2 CDAM for a latent concept
Concept embeddings le have been obtained by averaging latent reprosentations of 30 images. Those images were randomly selected from class in the ImageNet validation set (Deng et al. 2009 Then, CDAMs in Fig. 4 were obtained using Eq. 4 We observe that CDAM reveals that the model clearly distinguishes the parts of the images that are relevant for the latent concepts shared by selected images from the rest of the image. For example, when having selected images containing hammer (bottom right), CDAM primarily highlights the hammer. While the annotated classes were used for this experiment, the concepts shared by those images are likely more nuanced, specific, and complex than the class itsclf. For example, CDAM also provides positive importance scores for the wooden handles of the screwdrivers and the socket attached the ratchet, which imply relevance for the concepts shared by 30 images selected from the class hammer. Semantically, 30 images that are selected from a class hammer have wooden handle (i.c., concept), which is shared by multiple tools. Similarly, the socket at the top of the ratchet shares similarities with typical metal heads at the top of the hammer. Unrelated tools or parts appear in blue indicating that the model deems them to be unrelated to the concept hammer.
CDAMs are discriminative with respect to the target concepts shared by selected samples (Fig. 5 right). This makes the proposed method particularly valuable for understanding the representations the VIT has learned,
9

---

Published in Transactions on Machine Learning Research (11/2024)
Attention Map
CDAM
Attention Map
CDAM Analog Clock
Bottle
Balloon
Hammer
Figure 4: Class-Dscriminatie Attention Maps (CDAM) for user- definee concepts (Section 3.2 The concept embedding le has been obtained by averaging the latent representations of 30 images that include the common concept. In each instance, from left to right: Sample image, AM, and CDAM, Orange and blue correspond to positive and negative values, respectively.
CDAM from classifiers
CDAM from similarity measures
Zuochini
Bell Pepper
Zebra
African Elephant
Digitel Clock
Analog
Tabby Cat
221
Figure 5: When targeting different classes (left) concepts (right), the resulting CDAMs are distinct and align with the corresponding objects. Note that green parts of vegetables are shown to activate the zuccini class (left). The wrinkles in the elephant's 's trunk appear to be related to the zebra concept, semantically mistaken by the model as zebra stripes.
since the explanations for different concepts give complementary view. For example, by selecting 30 images that contain african elephant, CDAM highlights areas that are important for concepts which coincide largely with the body of an elephant. It also shows that parts of the elephant's 's trunk are counter-evidence for the concepts shared by images with african elephant (Fig. 5 right). For the same sample image, we can also select 30 images with zebra, which suggests that the model deems the part of the elephant's trunk to be similar to concepts shared by those images.
Although we selected group of images based on annotated classes, which are denoted in Fig. 5 (right), to illustrate concept-based CDAMs, generally the concepts shared by selected images not identical to the classes themselves. One could manually provide set of images with shared concept that is not originally annotated in the dataset. For example, 10 images that contain zebras may share semantic concepts (c.g.. "stripes" and "lail"), that make up the class zebru. We demonstrate this by selecting a set of 10 images, outside the ImageNet, that exclusively show stripes and another set 10 images that contain diverse animal tails. Shown Fig. 0 CDAM for stripes highlights the body of the zebra, but not the body of the cheetah. On the other hand, CDAM for tails tends to focus on the tail of the chectah. Note that 10 images of diverse animal tails include body parts of animals, often ends and legs.
10

---

Published in Transactions on Machine Learning Research (11/2024)
Stripes
Tail
Figure Concept-based CDAM based on 10 images that show stripes or contain animal tails.
4.2
Quantitative evaluation
We have performed quantitative evaluations of the correctness and compactness of our proposed methods, in comparison multiple well-known importance estimators. We further investigated class sensitivity by using importance scores from multiple classes.
4.2.1
Evaluation of correctness
Fidelity ALIF-MIF
Box Sensitivity
AMIF
ALIFT
Abox
CDAM
404
1038
632
12.2
Integrated CDAM
375
1045
669
13.0
Smooth CDAM
282
1110
827
12.4
Relevance Propagation
315
957
643
10.0
Partial LRP
489
837
348
4.5
Attention map
452
846
393
4.8
SmoothGrad
534
529
-4
0.4
IntGrad
226
802
574
7.6
GradCAM
300
944
643
8.6
Input x Grad
273
713
439
3.5
Table 1: Evaluation of correctness by fidelity and box sensitivity. In Xmulus importance scores for the target class used to perturb pixels. The changes in the model output w.r.t. the perturbation percentage (fuFFMIF; Fig. $2 the box size (fboxi Fig. S3 are quantified by the area under the curves.
CDAM performs favorably in terms of the correctness of estimated feature importance compared to other methods. For the dataset containing multiple objects (Table 3 the Smooth CDAM outperforms all other methods on the fidelity evaluation using MIF/ perturbation (Fig. 52), with ALIF-MIF = 827 (Table 3 On the box sensitivity benchmark, Integrated CDAM (Abox = 13.0) performs the best, closely followed by Smooth CDAM (Abox 12.4) (Fig. S3 Some methods, such as IntGrad, perform very well on the MTF benchmark but poorly for LIF (Table 3 We consider the difference (LIF-MIF) be more meaningful and intuitive ideally, one would want large area under the LIF perturbation curve and a small area under the MIF perturbation curve. Thus, larger ALIF-MI indicates higher correctness. Particularly, good score in MIF could stem from simply triggering perturbation artifacts of the model (Brocki & Chung 2023b Hooker
et al.
2019
For the random subset ImageNet, the Integrated CDAM demonstrates the best performance (Abox 12.9), closely followed by Relevance Propagation (Abox 12.7) on the box sensitivity benchmark (Fig. S6 Table S1). Relevance Propagation outperforms all other methods on the MIF/LIF perturbation, followed by the Smooth CDAM (Fig. S7
11

---

Published in Transactions on Machine Learning Research (11/2024)
Fidelity A AI(LII-MIF)
Box Sensitivity AAbux 1
CDAM
739
14.6
Integrated CDAM
708
14.8
Smooth CDAM
823
13.4
Relevance Propagation
293
6.6
Partial LRP
0.0
Attention map
0
-0.2
SmoothGrad
-11
-0.1
IntGrad
540
7.6
GradCAM
681
9.4
Input x Grad
433
3.4
Table 2: Evaluation of class discrimination by differences in fidelity and box sensitivity metrics. By using the mumlt datasct, we measure how sensitive importance scores are changing their target class (Details
in Section 3.5 Area under the AfI.F.MII (Fig- 54 and Afhox curves (Fig. S5
4.2.2 Evaluation of class discrimination
Class discrimination (or sensitivity) is measured by the difference in fidelity statistics AACLIF-MI)) when perturbing according to importance scores obtained for the correct wrong target class. Larger AA(LIF-MIF) is considered to indicate better performance.
CDAM (AA(Li-- MIP) 71r)= Smooth CDAM (AA(LIP-MI))8233)) and Integrated CDAM (AA(LIF-MI))- clearly outperform other methods in terms of class discrimination (Table 3). IntGrad (540), Input x Grad (433), and Relevance Propagation (293) follow. Surprisingly, Partial LRP and SmoothGrad perform defi- ciently. Whereas SmoothGrad has low class-discrmmination because its importance scores have low accuracy to begin with, Partial LRP's scores have mediocre accuracy but it appears be insensitive to the choice of the target class (Fig. 3 By design, attention maps do not consider the target class and therefore, AA(LIF-MIP)=D
The importance scores for CDAM become anti-correlated with the model output when the wrong importance scores are used for the perturbation (Fig. $5 This anti-correlation probably results from the fact that pixels that are evidence for the correct target class are counter-evidence for the wrong class. This result corroborates the visual impression that CDAM correctly assigns importance scores with opposite signs to the objects corresponding to the targeted class and other objects that are present (Fig. 3). This is also reflected in the negative values of fur faIr for CDAM (Fig. S4] indicating that the ranking from least to most important has been, at least partially, reversed.
4.2.3 Evaluation of compactness
Sparsity and shrinkage are evaluated by our compactness evaluation, which one of the major desired properties in the explainability of deep learning (Nauta et al. 2023a). While sparsity may not always imply the most accurate estimation at a single data point, bias-variance tradc-off's is well-known machine lcarning [Lahlou Kitane| 2022). Ideally, compact explanations (e.g,, sparse and shrunken importance estimators) are preferred, when other properties such as correctness and class sensitivity constant.
CDAM and Integrated CDAM resulted in the highest degree of compactness, together with Input x Grad. For those Uhree importance estimators, on average, 88% of importance scores were less than 5% of the maximum score Table 3 IntGrad and SmoothGrad show relatively high sparsity 0.84 and 80%, respectively. But both are lacking in correctness (Section 4 and class-discimmnatiion (Soction 4.2.2). Relevance Propagation shows the lowest sparsity of all considered methods 48%, but high level of correctness. These results suggest. that these correctness and compactness metrics orthogonal.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Compactnesst
CDAM
0.88
Integrated CDAM
0.88
Smooth CDAM
0.75
Relevance Propagation
0.48
Partial LRP
0.74
Atlention map
0.56
SmoothGrad
0.8
IntGrad
0.84
GradCAM
0.54
Input Grad
0.88
Table 3: Compactness, measuring sparsity and shrinkage, is defined by the fraction of pixels with an impor- tance score lower or equal to 5% of the maximum score, evaluated on the Xmulti dataset.
5 Application to Medical Images
To demonstrate CDAM for another use case we apply it to nodule malignancy and biomarker prediction using
the Lung CT scans the LIDC dataset (Armato III al. (011). Here, we show CDAMs from DINO-basod models, we also showcase use of DelT (Dosovitskiy et al. 2020; Touvron al. 2021] and DINOv2 (Oquab)
al. 2024 Darcet et al. 2024
backbones that fine-tuned the LIDC dataset in Appendix A.5.2
5.1
Malignancy Prediction
After preprocessing, the LIDC data consists of 443 benign and 411 malignant lung CT scans. Training, validation, and test sets (in the ratios of 0.7225, 0.1275, 0.15) were stratified by and balanced according to these labels, c.g.; benign and malignant. We finc-tuned ViT model, with DINO backbone pro-trained the ImageNet, for 50 epochs. In parameler sweep, we varied the number of trainable layers (10 - 50) and dropout rates (0.0 0.09), where the learning rate was exponentially decaying (a = 0.0003 and 3 = 0.95). The best accuracy on the test set 0.85 was obtained with 50 trainable layers and dropout rate of 0.00: Atlention maps and CDAM are obtained based on this model.
The obtained attention maps suggest that the model focuses on patches with nodule fragments (Fig. S11 b)).
However, note that attention maps almost always focus pulmonary nodules without taking into account the downstream classification into benign and malignant samples. CDAM provides detailed structures where positive and negative values are indicated by orange and blue, respectively. In this binary classification where 0 is for benign and 1 for malignancy, input pixels with positive values in CDAMs are driving the classification towards malignancy.
5.2 Biomarker prediction
We turn our attention to clinical biomarkers in LIDC that are routinely used by medical practitioners, Inspired by the concept bottleneck model (CBM) (Koh et al. 2020). we fine-tuned a ViT model pre-trained with DINO on 8 biomarkers (subtlety, calcification, sphericity, margin, lobulation, spiculation, texture, and diameter). Essentially, a regression model was built on pre-trained ViT model with the mean squared error (MSE) as Joss function. For investigation of LIDC and incorporation of CBM, see (Brocki Chung 2023a
CDAMs were obtained for 8 biomarkers (Fig. 7. In the context of the margin, patches including the edge of nodule get positive CDAM scores, whereas patches corresponding to the nodule body gel negative CDAM scores. For the diameter, patches corresponding to the nodule body get a positive score, whereas the background pixels exhibit neutral (near zcro) or slightly negative scores. When it comes to sphericity, patches including fragments of nodule curvature get a positive CDAM score. Spiculation refers to the presence of
It. might be possible increase the model performance further investigation. Here, straightforward model and training scheme demonstrate the application CDAM medical images.
13

---

Published in Transactions on Machine Learning Research (11/2024)
needle and spike-like structures, the margin. High spiculation often is indication of malignancy. In majority of cases, CDAM highlights spicules with positive values, whereas the inner bodies of nodules (i.e.. not spicules) tend to get low importance scores.
CDAM score blomarkers
Original Image
map
bubdett
calcificat
margin
100
302
300-
150
150
150
250
230
100
200
100
200
100
202
100
200
100
200
CDAM: loburattonn
03M: spiculation
CDAM: diameter
CO::: texture
CDAM: sphencitty
30
92
150
150
150
350
350
200
200
200
702
700
200
100
202
100
200
Figure 7: Explanations for the biomarker prediction model. Eight biomarkers available in the LIDC datasct are used to fine-tune ViT regression model with DINO backbone. CDAMs provide distinet and relevant explanations for different biomarkers. For more examples, see Appendix A.4
6 Discussion
Daass-iscriminetive Attention Map (CDAM) is a gradient-based extension of attention maps, that pro- vides strong discrimination with respect to a chosen class or concept, while exhibiting excellent fidelity and compactness. Our proposed methods retain appealing characteristics of attention maps, namely their high- quality semantic segmentation. The proposed importance estimation for CDAM scales the attention scores by how relevant they are for target class or concept (Appendix A.II. Therefore, zero attention scores remain zero in CDAM and it produces compact explanations with high sparsity and shrinkage. Besides explaining predictions of the classifier on the top of ViT, CDAM can also provide importance scores specific to the user-defined concept. In the context of self-supervised models, where class labels are absent, this makes it valuable technique to investigate the latent representations learned by the model.
Post-hoc explanation methods for DNNs aim to make the full decision-making process of the model more transparent by providing an approximation with certain desirable properties, such as correctuess, sensitivity, and compactness. Particularly, many explanation methods that estimate feature importance are not very useful because they give almost identical results when targeting different classes in the model output (Rudin) 2019 Adebayo et al. 2018). Even randomization in the model weights is often shown to have minimal impact on explanations (i.e., saliency maps (Adebayo et al. 2018)). We find, both qualitatively and quanti- tatively, that CDAM is highly sensitive to the targeted class, assigning positive importance scores to objects corresponding to the target class and negative ones to other (semantically distinct) objects in the image.
We introduce Smooth and Integrated CDAM, which essentially average a series of (vanilla) CDAMs. Notably, instcad of adding noise to or modifying the input images as in SmoothGrad (Smilkov et al. 2017) or IntGrad (Sundararajan al., 2017), respectively, our methods act on tokens in the final transformer block. While conducted number of quantitative and qualitative evaluations, there is no single XAI method that triumph in all scenarios. However, considering that Smooth and Integrated CDAM require computing n vanilla CDAMs (e.g, n 50), we recommend first trying vanilla CDAMs. If high fidelity is singularly desirable, we suggest utilizing Smooth CDAM.
Our application on modical images exemplifies the need for fine-grained class-sensitive explanations offered by CDAM. Other explainability methods including attention maps often highlight whole nodules and tumors. However, they are uninformative about the inner workings of the ViT model predicting malignancy
14

---

Published in Transactions on Machine Learning Research (11/2024)
As the methods of explainable AI (XAI) are considered for practical implementations, the investigation of human interactions with different explainability methods represents promising direction for future studies.
biomarkers.
Broader Impact Statement
The introduction of our importance estimators would enhance the transparency, trustworthiness and account- ability of vision transformer models. By providing clearer insights into model decision-making processes, accurate explainability methods help foster broader adoption of AI technologies. However, the explanations could be manipulated by adversarial input perturbation or data poisoning. In turn, enhanced explanalions may inadvertently expose vulnerabilities, making models more susceptible to adversarial attacks or malicious exploitation. Therefore, it imperative to continuously monitor and ensure the robustness of explanation methods while implementing safeguards mitigale the risks associated with adversarial attacks.
Acknowledgments
This work was partially funded by the SONATA BIS grant P/23//5///SST////// from the National Science Centre of Poland (Narodowe Centrum Nauki). This work was partially funded by the ERA-Net long term challenges in ICT project INFORM (ID: 93603), by
CHIST-ERA grant (CHIST-ERA-19-XALL the National Science Centre (NCN) of Poland /2020/02/Y/STO /0000/ This research carried out with the support of the Interdisciplinary Centre for Mathematical and Computational Modelling University of Warsaw (ICM UW) under computational allocation no GDM-3540; the IUUB program (Excellence Initiative- Research University), the NVIDIA Corporation's 's Academic Hardware Grant; and the Google Cloud Research Innovators program.
References
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Anaual Meeling of the Association for Compulational Linguislics, pp. 4190 4197. Association for Computational Linguistics, 2020.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanily checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus- Robert Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. In International Conference on Machine Learning, pp. 435-451. PMLR, 2022.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- based attribution methods for deep neural networks. arXiv preprint arXi::1711.06104 2017.
Samuel G Armato 111, Geoffrey McLennan, Lue Bidaut, Michael F McNitt-Gray, Charles R Meyer, Authony P Reeves, Binsheng Zhao, Denise R Aberle, Claudia I Henschke, Eric A Hoffman, et al. The lung image database consortium (lide) and image database resource initiative (idri): a completed reference database of lung nodules on et scans. Medical physics, 38()):915-98) 931, 2011.
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. A diagnostic study of explainability techniques for text classification. arXiv preprint arXiv:2009.132: 2020.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7)::0130140, 2015.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXio preprinl arXio:1409.044 2014.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? Xio preprint or Vin:2006.0715) 2020.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Stefan Bluecher, Johanna Vielhaben, and Nils Strodthoff. Decoupling pixel flipping and occlusion strategy for consistent XAT benchmarks. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
https://openreviier http:///ppaapoeeeeeeeerrrrrrrttr net/fforummIitttILXXEtt
Lennart Brocki and Neo Christopher Chung. Concept saliency maps visualize relevant features in deep generative models. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 1771-1778. IEEE, 2019.
Lennart Brocki and Neo Christopher Chung. Integration of radiomics and tumor biomarkers in interpretable machine learning models. Cancers, 15(9), 2023a. TSSN 2072-6694. doi: 13330/canccer1555 https://www http:/pwwwwpppppo/ttttttttt.wt.t /ttp:////002220000/11/////399990
URL
Lennart Brocki and Neo Christopher Chung, Feature perturbation augmentation for reliable evaluation of importance estimators in neural networks. Pallern Recognition Letlers, 176:131 139, 2023b.
Lennart Brocki and Neo Christopher Chung, Fidelity of interpretability methods and perturbation artifacts in neural networks. In Krystal Maughan, Rosanne Liu, and Thomas F. Burns (eds.), The First Tiny Papers Track at ICLR 2023, Tiny Papers @ TCL.R 2023, Kigali, Rwanda, May 5, 2023. OpenReview.net, 2023c. URL pf
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9650-9660, 2021.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-moder explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 397 406, 2021a.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 782 791, 2021b.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? analysis of BERT's attention. In Tal Linzen, Grzegorz Chrupala, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACI. Workshop BlackbozNI.P: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, Florence, Italy, August 2019. Association for Computational Linguistics. URL
doi: 10.18653/v 10155/3v1//199/00 4824888...
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. International Conference on Leorning Representations (ICLR), 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 2rXiv:2010.11922 2020.
Jacob Gildenblat. Pytorch library for cam methods. httpe///ghhthb.ccc//oooooolllaos 2021.
Anna Hedström, Philine Bommer, Kristoffer K Wickstrøm, Wojciech Samek, Sebastian Lapuschkin, and Marina M-C Höhnc. The meta-ovaluation problem in explainable ai: Identifying reliable estimators with metaquantus. Xix preprinl 2rXio:2302.07265, 2023.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint arXi::1902.10133 2019.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternattribation. arXiv preprint ar Xie: 705.0 705.05598, 2017.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338-52 5348. PMLR, 2020.
Driss Lahlon Kitanc. Sparsity in Machine Learning: Theory and Applications. PhD thesis, Massachusotts Institute of Technology, 2022.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timolle, and Lue Van Gool. LocalViT: Bringing Locality arXiv:
Vision Transformers. CoRR, abs/2104.055077, 2021. URL /tcpp:///arxxv.org/aa/////////// 2104.05707.
James Mullenbach, Sarah Wiegrefse, Jon Duke, Jimeng Sun, and Jacob Eisenstein. Explainable prediction of medical codes from clinical text. arXic preprint 18x.:180..0050 2018.
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Compuling Surveys, 55(13)):1144 2023a.
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Compuling Surveys, 55(13s):1 42, 2023b.
Pia Opulencia, David Channin, Daniela S Raicu, and Jacob D Furst. Mapping lide, radlex and lung nodule image features. Journal of digital imaging, 24:256 270, 2011.
Maxime Oquab, Timothée Darcet, Thco Moutakanni, Huy V. Vo, Marc Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Syn- naeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2024.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:8066 07421, 2018
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use inter- pretable models instead. Nature machine intelligence, 1(5):206-2)) 215, 2019.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robers Müller. Evaluating the visualization of what a deep noural network has learnod. JEEE transactions on neural networks and learning systems, 28(11):2660- 2673, 2016.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Sofia Serrano and Noah Smith. Is attention interpretable? arXiv preprint arXiv:1906.0373: 2019.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.011 017113 2016.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprinl Xrxi::1312 211.312200000 6034,, 2013.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Mannat Singh, Laura Gustafson, Aaron Adeock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateck Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. URL https: /tte::///xxvvv http://avvvv...raa...//a wtgg/bb//20..0000/333
Danicl Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint N06:0706.03825, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319 3328. PMLR, 2017.
James Thorne, Andreas Vlachos, Christos Christodooulppollss and Arpit Mittal. Generating token-level explanations for natural language inference. arXie preprint XXXx:1994..1011 2019.
Hugo Touvron, Malthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. International Conference on Machine Learning, 2021. URL https://arxix: http:///aaxiv.oor/bbbbeeeee wwwp/bg////.......////// 12877
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informalion processing systems, 30, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi head self- attention: Specialized heads do the heavy lifting, the rest be pruned. arXiv preprint arXiv:1005 09.5009418 18, 2019.
Sarah Wiegreffe and Yuval Pinter. Attention not not explanation. arviv preprint
04666, 2019.
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual Transformers: Token-based Image Representation and Processing for Computer Vision. orXiv e-prints, June 2020. doi: 10.48550 20.44550/aaxiv... aavaa wtt.20006.06777.
URL
https://ui adsabs harvard. edu/ aactuu/bbs/2000 /ttp/aaas/20000/020000006667
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, La Yuan, and Lei Zhang, CvT: Introducing Convolutions to Vision Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22-31. IEEE Computer Society, October 2021. ISBN 9781665448825,, doi:
10.1109/JCCV44992.220 100009. URL https::/ http://www........ooooo.o/ootaaa htw_2222220000222222X00200
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 579 588, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022. URL
18

---

Published in Transactions on Machine Learning Research (11/2024)
A Appendix
A.1 Proportionality of CDAM to attention maps
The proposed class-disriminattie attention scores scale the attention scores. Empirically, in all of our evaluation lasks and applications (e.g,, Fig. ST we observe that features with atlention scores A; = softmax Craa K_ of O have clas--discrimmmative attention scores 0: Vdx
A; = 0 St. 0.
(9)
This operating characteristic is highly favorable, as besides introducing class discrimination, CDAM provides implicit regularization resulting in sparser heat maps. Based on a combination of empirical observations and mathematical derivations, we argue that the attention score A; softmax (Qras) K. is scaled by the vidy
relevance of the corresponding token T, for concept or class.
To better understand this observation, we first consider a concept-based CDAM with single attention head. Eq. is expanded:
Σ Tij 1g(1,
Si, =
(10)
ÖTij
Σ Tij iT;
Σ
lahe,k
(11)
k
θ
Σ
Tis 0T
Σ hi
Em
Am Vn + [CLS]
look
(12)
Σ Tij Σ
hh x
(AAm
Vnn + Am HTis
OVmn
) lakk
(13)
"Tig
k.m.n
0Am TTi
Σ
hig
V Vinle,k
A.Vin Σ
V huluko
(14)
j.k.m.n
where we assume the architecture of Dosovitskiy et al. 2020 and use the projection of the tokens onto the key vector VÚ = T.W. with WV E B. Rmmddlll Xd. We have also used the fact that the latent representation (i.c. [CLS]') is the sum of value voctors weighted by the corresponding attention, = Σ, A,V, with A; = softmax ( (Qtaak) plus [CLS] due to the residual conncction, see Fig, bb). The function h Rd. - Rdmdiial
consists of the layer normalization, MLP, and residual connection and performs the final processing of the [CLS] token before it enters the classifier (Fig. bb)) V is the n-th component of the gradient and Q (CLS), Vi and K; are the rows of matrices Q, V and K that correspond to the [CLS] and the i-th tokens.
Eq. (9) implies that in Eq. 10), all terms that are not proportional to A; are zero or cancelled out and we are left with
S,c = A.V. V.)
(15)
We thus find that St, is obtained by multiplying A, with the directional derivative of h - 1, in the direction of V. In other words, CDAM scales A; by the rate at which the dot product (similarity) between h and changes in the direction of Vi.
In the special case that h the identity function, we obtain an even simpler interpretation of CDAM due to V,lua (ʃ) fr dnk- Assuming is scaled by the similarity of V2 and le-
24m. 32. 0, Eq. 110 simplifies to S.e A,V; le, which means that A
We note that introducing h attention heads does not change the above derivations qualitatively. Instead of the sum 7 Emm Am Vmn there will be a term Concat Zt 722 ,.,, Z*)Wo wo E Rdd Xdmadal so that in the
19

---

Published in Transactions on Machine Learning Research (11/2024)
case of multi-head attention the equivalent of Eq.
10
reads
dAm
C = Concat (Y', y² .. )n) v (o). Y = T. Σ
V + A,V,"
(16)
0T,
and since A7 > 0, = En A implics that A = = - = A = 0, Vn we can again conclude that in order for Eq. 1 to hold all terms that are not proportional to A, either vanish or cancel out. For multi-head attention, we thus arrive at.
h
0 = Concat (r",," y",,,. )") w 10), Y = V,","
(17)
which does not have such a straightforward interpretation as the single-headed case duc to the mixing of the atlention heads.
If we think of w acting first on V(h- creating a column vector of dimension hdv, we can understand Eq. [17 the sum of directional derivatives of the de-long segments of this columns vector in the directions of y to y Each of the summands is proportional to the corresponding A"; and the interpretation of CDAM from the single-head case therefore extends to the multi-head one.
Although our observation A, = 0 + Sie = implics that all terms that are not proportional to A₁ in Eq. (10) should vanish, we don't have a theoretical argument why that would be the case. Tt seems unlikely that the various terms in the first sum cancel out, which leaves dAm. 0 as an explanation. However, in general, A, a7. depends on T; through K2 and one would therefore expect the derivative to be non-zero.
We can also show that CDAM scales the attention by the relevance of the corresponding token for a chosen class (instead of concept), in the case that a classifier with weights W/C has been trained on top of the VIT. Analogously to the above single-head head discussion we have
Ofe Tis OT;
Sic =
Σ
(18)
Tis OT:
Σ
We
(19)
k
dAm
Σ
V3 h4
Σ An V"hy V mm W.C. kc
(20)
Tii
j.k.m.n
and the only difference is that the directional derivative now acts on Σx W ke: Le. the contribution of h to the activation of class e in the prediction vector, instead of Σx lgle,k- All the intuitions we have gained from the sancept-based CDAM therefore extend to the class-bas based one.
20

---

Published in Transactions on Machine Learning Research (11/2024)
A.2 Quantitative Evaluations
0.010
0.025
Score
0.015
0.010
0.005
0.000
-0.5
0.0
0.5 CDAM
1.0
1.5
0.2 Smooth CDAM
0.0
0.4
0.00
0.06
0.10
0.15
0.0
0.1
0.2
0.3
Integrated CDAM
Rel. Prop.
Partial LRP
Figure S1: Scalter plot of raw attention scores and token importance scores obtained from 200 randomly selected samples of the Net validation set X
for fa
10
20
60
6D
DC
100
20
40
60
D0
100
0
20
40
65
100
perturbation pertentage
pertur bation percentage
perturbation percentage
CDAM Integrated COAN
Atbention Map
Parbill LMP IntGred
Gradient Smooth CONN
SmoothGrred GradCMM
Rai.
Figure S2: Fidelity of importance estimators based on MIF /LIF perturbation curves obtained from the Xmmlti dataset. The change in the model output is measured as a proportion of input features are perturbed according to the Least Important First (LIF) the Most Important First (MIF) orders. Importance estimators are evaluated by the area between LIF and MIF perturbation curves, which is equivalent to the area underneath fur--xar (right). See Table 3
21

---

Published in Transactions on Machine Learning Research (11/2024)
0.3
0.4
0.3
30
bea
CDAM
Attention
LBP
SmeothCrad
Woogisted CoaM
Pas.
InGrod
Smooth COAM
GraCCAM
Figure S3: Correctness of importance estimators measured by box sensitivity evaluation. The change in box are perturbed. The sum of their importance
the model output (logit) is measured pixels within scores would correlate with the change in the model output, if importance estimators are accurate. This is repeated 100 times per sample and the Pearson correlation between the sum of importance scores and the change in the model output is calculated, based on the Xmulti dataset.
Fir far
fur
fer
Mur
10
10
10
2
50
100
50
100
50
100
100
50
100
perturbetion
perturbation
perturbation
perturbation
perturbation
COMN Integrated
Attention Pertial Prop.
Gradient
SmoothGred
Figure S4: MIF/LIF perturbation curves for the evaluation of class-discrimination. Different classes are targeted for obtaining importance scores and measuring model output (Section 3.5 large A therefore, indicates importance scores that are clas-ddscriminative. See Table 2
22

---

Published in Transactions on Machine Learning Research (11/2024)
foo
Mbox
0.6
0.25
0.20
0.5
0.10
correlaton
0.05
0.00
-0.5
0.1
0.10
0.0
30
box size
box size
CDAM
Attention Map
Partial LRP IntGrad
Input x Gradient
SmoothGrad GradCAM
Integrated CDAM
Rel. Prop.
Smooth CDAM
Figure S5: Box sensitivity curves for the evaluation of class-sensitivity
foox
0.5
0.4
0.2
0.1
0.0
5
10
20 box size
25
30
CDAM
Attention Map
Partial LRP IntGrad
Input Gradient Smooth CDAM
SmoothGrad GradCAM
Integrated CDAM
Rel. Prop.
Figure S6: Box sensitivity curves obtained from the random ImageNet subset Xval. Refer to Section 3.5 for data description.
23

---

Published in Transactions on Machine Learning Research (11/2024)
fgr
12
14
12
10
150
20
100
100
perts rhatton percentage
perturbation percentage
pertuttation percestnge
CDAM
Attention Rel. Prop
Part Intfled
Inpur Gradient Smooth CDAM
Smoothhrred EOedCAM
Intecrated CDAN
Figure S7: /IFF perturbation curves obtained from the random ImageNet subset Nyal- Refer Sec- tion 3.5 for data description.
AMIF
ALIFt
ALIF-MIFT
Abox
CDAM
801
1423
623
11.3
Integrated CDAM
696
1459
753
12.9
Smooth CDAM
591
1482
890
11.9
Rel. Prop.
475
1432
957
12.7
Partial LRP
654
1329
674
8.4
Att. map
580
1352
771
9.1
SmoothGrad
796
786
-10
0.2
IntGrad
411
1055
644
8.3
GradCAM
515
1249
733
9.0
Input x Grad
470
935
465
3.1
Table S1: Area under the curve for MIF, LIF and LIF-MIF curves (Fig. FFig. S6 obtained from the random ImageNet subset.
S7
and box sensitivity curves
24

---

Published in Transactions on Machine Learning Research (11/2024)
A.3 Examples of CDAMs from the ImageNet
Partial LRP
Rel. Prop.
Input x Grad
SmoothGrad
IntGrad
Int: CDAM
Smooth CDAM
CDAM
AM
tractor
terrapin
cuirass
washer
Figure S8: Additional examples for CDAM and other explanation methods on the ImageNet. The indicated class the one predicted by the model and is the target for the explanation methods.
25

---

Published in Transactions on Machine Learning Research (11/2024)
Partial LRP
Rel. Prop.
Input x Grad
SmoothGrad
IntGrad
Int. CDAM
Smooth CDAM
CDAM
AM
wine bottle
park bench
cardigan
chain
Figure S9: Additional examples for CDAM and other explanation methods the ImageNet. The indicated class is the one predicted by the model and is the target for the explanation methods.
26

---

Published in Transactions on Machine Learning Research (11/2024)
Partial LRP
Rel. Prop.
Input x Grad
SmoothGrad
IntGrad
Int. CDAM
Smooth CDAM
CDAM
AM
nortarboard
lab coat
pajarma
Windsor
academic
Figure S10: Additional examples for CDAM and other explanation methods on the ImageNet. The indicated class is the one predicted by the model and is the target for the explanation methods.
27

---

Published in Transactions on Machine Learning Research (11/2024)
A.4 Examples for CDAMs from lung CT scans
(a) Lung CT Scans
(b) Attention Maps
(c) CDAM
Figure S11: Explanations for the malignancy prediction model. ViT-based classifier fine-tuned on the lung CT scans (LIDC) was used to predict benign vs. malignant lung nodules, followed by obtaining their attention maps and CDAMs. Orange and blue correspond to positive and negative values, respectively.
28

---

Published in Transactions on Machine Learning Research (11/2024)
CDAM score across biomarkers
Criginal Image
Altention map
CDAM: subtiety
CDAM: calcilication
CDAN: me riin
230
100
200
13C
200
100
200
CDAM: lobu ation
CDAM: spiculation
CDAM: diameter
CDIA: texture
COAM: sphericity
50
200
200
200
200
10C
200
100
CDAM score across biomarkers
Criginal Image
Attention
CDAM: subtlaty
CDAM: calcification
CDAM: margin
50
100
200
CLAAM: tobu atton
COAM: spiculation
CDAM: diameter
CDAM: texture
CDAM: sphencity
50-
50
200
200
Figure S12: Examples for the Attention Map and CDAM based ViT biomarker model trained the lung nodule data from LIDC.
29

---

Published in Transactions on Machine Learning Research (11/2024)
CDAM score across biomarkers
Criginal Imege
Altention map
CDAM: subtiety
CDAM: calcilication
CDAN: me
230
100
200
10C
200
300
200
CDAM: lobu ation
CDAM: spiculation
CDAM: diameter
CDIM: texture
COAM: sphericity
50
200
200
200
200
10C
200
100
CDAM score across biomarkers
Criginal Image
Attention
CDAM: subtlaty
COAM: calcification
CDAM: margin
50
100
200
CLAMM: tobu atton
COAM: spiculation
CDAM: diameter
CDAM: texture
CDAM: sphencity
50
50
200
200
Figure S13 Examples for the Attention Map and CDAM based ViT biomarker model trained the lung nodule data from LIDC.
30

---

Published in Transactions on Machine Learning Research (11/2024)
A.5 Additional ViT architectures and training strategies
While the main manuscript has used ViT models pre-trained using DINO (Caron et al. 2021 other archi- tectures and pre-trained weights can be used for CDAM. In this section, we show examples of alternative ViT models that are trained in alternative manners. As expected, pre-trained VIT models that produce high-quality attention maps result in high-quality CDAMs. In particular, note that the patch size of DINO CCron et al. 2021 is 8 compared to much larger patch sizes used in other backbones.
Table S2:
Pre-trained VIT models and training strategies.
Name
Model sizes
Patch sizes
References
DelT (original ViT)
86M
16
Dosovitskiy ct al.
(020)):
Touvron et al. 2021
SWAG
86M
16
Singh et al.
2022
DINOv2 (with registers)
21M, 86M
14
Oquab et al.
2024
Darcet et al. 2024
DINO (used in the main text)
21M, 85M
Caron et al.
[2021]
A.5.1 Alternative ViT models for the ImageNet
We use a VIT model pre-trained on the ImageNet-1K dataset using supervised weakly through hashtags (SWAG) (Sighh et al. 2022). Unlike DINO and other sell-supervised ViT models, the SWAG model has been trained end-to-end. Therefore, without pre-training, we can obtain attention maps and CDAMs directly.
Three examples are shown Fig. S14
Original Image
Attention Map
Original Map
CDAM) ninitture poodle
Original Image
Map
Figure S14: Attention maps and CDAMs from the ImageNet-IK dataset, based on the pre-trained VIT model backbone using SWAG Singh et al. 2022).
In general, VIT models require a massive amount of training data and therefore often trained in self- supervised manner. As shown in DINO (Caron et al. 2021). self-supervised learning can not only improve performance of fine-tuned models, but also improve attention maps. CDAM relics on attention maps, there- fore the quality of attention maps strongly influence that of CDAM. Note that the SWAG model has patch size of 16 16, which results in lower-resolution attention maps and CDAMs.
31

---

Published in Transactions on Machine Learning Research (11/2024)
A.5.2 Alternative ViT models for the LIDC
We demonstrate the use of CDAM with VIT models trained using Data-efficient image Transformers (DelT) (Tourron et al. 2021) and DINOv2 (Qquab el al. 2024). Note that despite the use of the same acronym, DINOv2 is substantially different from DINO (Caron et al. 2021] due to using a different training strategy much larger scale (Oquab et al.] 2024h. The original DINO pre-trained the ImageNet, whereas DINOv2 (Oquab et al. 2024 used a newly curated dataset LVD-142M. Additional tokens called "registers" were introduced to improve attention maps of DINOv2 (Darcet et al. 2024).
introduced in iBOT (Zhou et al.) 2022)
(Caron el al. 2021
The LIDC dataset was split into 5 folds and stratified according to malignancy status. The best-performing models were chosen based on means of Accuracy (ACC) and Mean Squared Error (MSE) over folds, the validation set. In parameter sweep, we varied the number of trainable layers (10-all) dropout rates in the backbone (0.0 0.12), batch size (8 - 32) and learning rate scheduler parameters. The learning rate was scheduled with CyclicLR scheduler. Binary Cross Entropy Loss and Huber Loss were used to train classification and regression models, respectively. Weights optimization was performed with Adam optimizer and random rotation was applied as data augmentation. For malignancy classification, the best-performing models based on DefT and DINOv2 achieved a mean ACC of 0.896 and 0.904, respectively. For biomarker regression, the best- pess-perforing models based on DeIT and DINOv2 archived the mean MSE of 0.409 and 0.35, respectively.
DelT
DINOv2 ViT
Lung Nodule CT Scon
Attention Map
Attention Map
Figure S15: Attention Maps and CDAMs for malignancy models using VIT backbones pre-trained with DelT and DINOv2.
32

---

Published in Transactions on Machine Learning Research (11/2024)
Original
CoMMopiculatien
CDAM]
(16M))ppericky
Original
Attention Map
coMMisubdetyy
caMMi
O
CDAMiDemmter
COMM]Tecture
CoMMiSphenccyy
Figure S16: Attention Maps and CDAMs for biomarkers models using ViT backbones pre-trained with DelT
33

---

Published in Transactions on Machine Learning Research (11/2024)
Original
Attantion Nap
Subiiay
Margin
CDAMiSchericty
Attantion
(DAMiSubitaty
Clcffcciiiitntionn
CoAMiaargin
Spicuacioiatiinn
CDAMiDlamater
CDAM Texture
CDAMiSphericky
Figure S17: Atlention Maps and CDAMs for biomarkers models using ViT backbones pre-trained with DINOv2
34


Reviews:
[{"review_id": "90itwnWYS1", "reviewer": "Reviewer_D8Vi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is well written and easy to follow.\n\n2.\tThe proposed method successfully combines the good semantic segmentation-like heatmap provided by attention maps and the discriminative information provided by the gradient of the logit of a certain class.\n\n3.\tThe visualizations in Figure 1, 3, 4, 5 is clear and show that the proposed attribution map is highly discriminative for different target classes.\n\nWeaknesses:\n\n1.\tMy main concern is about the generality of experiments. The experiments are mainly conducted on ViT trained with the DINO method. It is not clear whether CDAM still works well on ViTs with other training objectives.\n\n2.\tThe paper does not compare the proposed method with the Shapley value, which is a popular attribution method due to its good axiomatic properties. Could you also conduct evaluation on the Shapley value method?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new attribution method for vision transformers. The proposed method reweights the attention scores by the gradient of the logit of a certain class w.r.t. to token embeddings. Experiments demonstrate that this attribution method shows different heatmaps for different target classes, which means it is highly discriminative. The proposed method is also evaluated on several interpretability metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tIt is highly encouraged to put figures and tables in the main text rather than in the appendix. Doing so can greatly enhance the readability of the paper.\n\n2.\tCould you explain why CDAM usually produces negative attribution scores on the object other than the target object? For example, in Figure 1, when the target class is “Tench”, the attribution scores on the “Dog” pixels are negative, while the attribution scores on the background pixels are near zero.\n\n3.\tMinor. The paragraph below Equation (6), $T_i = T_i + N(0,\\sigma^2) \\rightarrow T’_i = T_i + N(0,\\sigma^2)$. Besides, what is a “null vector”?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uh58TahzYH", "reviewer": "Reviewer_8PW2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\n-\tUsed multiple metrics to analyze the explanations.\n-\tAddresses transformers networks.\n-\tProvides class- discriminative explanations\n\n**Weaknesses**\n\n-\tThe method only explains the linear classification layer and the last transformer block not the full ViT (i.e., has similar limitations as the original GradCam method for CNNs). Unclear how much of the conclusions are mainly about the classification layer than the full transformer architecture. It gives the impression that the explanation is just the weighting of the tokens for the classification, i.e., explaining the downstream application/linear layer and not the full ViT. The main goal of the explanation should be clarified. \n-\tGradCam and integrated gradient have already been applied to ViTs in other papers (e.g., [a,b]). If the implementation here differs this should be better explained.\n-\tNo clear conclusion from the experiments. From the tables it seems like the smooth version or integrated version perform better. [b] also provides an evaluation framework with ground truth data which could provide further insights.\n- Only one architecture has been tested, however the paper claims to be more broadly applicable.\n-\tPresentation of paper can be improved. Tables and Figures of the main paper are all after the references but not yet in the appendix (making the main paper theoretically quite long as well). It would be easier for the reader to have the relevant data and plots closer to the text describing it. Also the descriptions are partly very lengthy (e.g., conclusion mentions twice the introduction of the  additional variations)\n\n\n[a] Chefer et al., \"Transformer interpretability beyond attention visualization.\", CVPR, 2021\n[b] Hesse et al., \"FunnyBirds: A Synthetic Vision Dataset for a Part-Based Analysis of Explainable AI Methods\", ICCV, 2023", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes a method for post-hoc explaining transformer architectures focusing on providing class-discriminative explanations. The method is an adaption of the gradient-based method GradCam from CNNs to ViTs. Additionally, other straightforward extensions of common  methods for CNNs adapted to ViTs have been described and analyzed. The methods have been analyzed on a single backbone and wrt. to multiple evaluation metrics.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide clearer conclusion why CDAM is preferred compared to the other versions (smooth and integrated). Or which method is recommend when. In the current version the take away message from the presented experiments is not very clear.\n- Discuss how the proposed method differs from GradCam for ViT, e.g, as used in Chefer et al.,  \"Transformer interpretability beyond attention visualization.\", CVPR, 2021\n- Rearranging the important figures and tables within the main paper.", "broader_impact_concerns": "none present, importance and challenges, e.g., explanations used for attacks could be discussed", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oT4QWsLqBv", "reviewer": "Reviewer_UH7P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the submission is well-motivated\n- the choice of evaluation metrics covers known overlooked pitfalls of explanation methods (e.g., \"data perturbation\")\n\nWeaknesses:\n- the organization of the submission could be improved\n- certain claims need to be rephrased\n\nI will expand on my points of confusion below and I am looking forward to discussing with the authors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission studies interpretability of Vision Transformers by computing importance as the gradients of the final predictor with respects to the tokens in the last attention-layer of the model. Experiments are performed to evaluate correctness, class-sensitivity, and sparsity of the produced saliency maps on subsets of the ImageNet dataset and the LIDC dataset.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "> Averaging the latent representations of the example images yields a concept embedding, which is analogous to a concept vector in variational autoencoders (VAE) and related generative models (Brocki & Chung, 2019; Kim et al., 2018)\n\nCould the authors expand on this claim and clarify in which way this is analogous to concept activation vectors (CAVs) as in Kim et al.? CAVs are the directions of separating hyperplanes in the representation space of a deep neural network. Here, the concept representation is defined as the centroid of images with a certain concept of interest. In the general case, these two concept representations may be different.\n\n---\n\n> In that regard, our method is related to GradCam, which backpropagates the gradients to the final convolutional feature map in a CNN.\n\nIt is true that GradCAM was introduced in the context of CNNs, but it can be readily applied to ViT architectures (see, for example `pytorch-grad-cam/blob/master/tutorials/vision_transformers.md`). Similarly, other class-discriminative notions of importance have been applied to ViTs (see, Covert et al. \"Learning to Estimate Shapley Values with Vision Transformers\" [2022]). Could the authors expand on their choice not to compare with these methods? Comparing with these methods may help readers place the proposed methods in the context of existing literature.\n\n---\n\n> By design, attention maps are not class-discriminative since they do not take into account any signal coming from a downstream task.\n\nThis claim may be too strong in its current formulation. It is true that a self-supervised ViT is independent of the downstream task. But, after finetuning, the weights of the model may be updated according to a loss function that does depend on the downstream task. So, since the attention weights depend on the model parameters, and the model parameters depend on the downstream task, the attention weights do depend on the downstream task.\n\n---\n\n**Eq. (1)**\n\n* $T_i$ is mentioned in the text but it is not defined in the equation.\n* Similarly, $Q$ is mentioned in the text but the equation only contains $Q_{CLS}$.\n* Appendix A.1 contains several elements that have not been defined at this point in the main text. I would suggest moving the cross-ref.\n\n---\n\n**Appendix A.1**\n\nCould the authors expand on why the discussion focuses on Eq. (4) only (i.e., concept activation), rather than the general case of any linear classifier? Also, $g$ is introduced in the text as a general function, but here, discussion is carried out for the particular case of dot product. What are the implications of using a different similarity function $g$?\n\nPlease number all equations before Eq. (10).\n\n---\n\n**Results**\n\nFigures and Tables in support of the main claims of the paper should appear in the main body of the manuscript. Currently, Figures and Tables are placed after the bibliography, as if they were an Appendix. No figures or tables appear in the main body of the submission to corroborate claims.\n\n---\n\n**CDAM for a latent concept**\n\n> by averaging 30 latent representations of images depicting the concept of interest. These 30 images were randomly selected from the corresponding classes in the ImageNet validation set\n\nCould the authors expand on this choice? If concepts are equivalent to classes, then what is the difference between classification experiments and concept experiments? Usually, when talking about semantic concepts, people refer to sub-concepts that make up a class (e.g., class \"zebra\" and concepts \"stripes\", \"hooves\", \"tail\", ...).\n\n> when targeting the concept zebra, the model deems this part of the trunk to be similar to a zebra. Speculatively, the trunk may be too similar to the zebra’s muzzle or the trunk’s wrinkles may be mistaken for the zebra’s stripes.\n\nI agree with the authors that this claim is highly speculative, and I would suggest removing it because it does not add evidence in support of the main findings of the submission.\n\n---\n\n> Last but not least, while this paper focuses on a ViT model, CDAM is applicable more broadly to all transformer models with self-attention, including large language models (LLMs).\n\nThis is a very broad claim that needs refining or removal. The submission places itself on linear classification for bi-directional representations of images, which is a very different task from autoregressive language modeling. If there is evidence in support of the use of CDAM for certain tasks that may be performed with LLMs, then it should be included in the submission.\n\n---\n\n**Minor comments**\n\n* Inconsistent notation for [CLS] token, sometimes [CLS], sometimes [CLS]' (with a prime '). I am not sure I understand the difference.\n* Eq. (10), $\\nabla_n$ as the $n$-th coordinate of the gradient may be misinterpreted because $\\nabla_x$ commonly refers to the gradient with respects to $x$.\n* Typo in $T_i = T_i + \\mathcal{N}(0, \\sigma^2)$, the LHS needs to have a different symbol.\n* Please rephrase \"We perform quantitative evaluations on two datasets\". This paragraph introduces \"two datasets\" as two subsets of ImageNet, but the the LIDC dataset is also used.\n* Please rephrase \"For application in medical images [...] of CT scans\". The first sentence introduces general \"CT scans\". And then later, the paragraph talks about the LIDC dataset. Please state from the beginning of the paragraph that the LIDC dataset is used for this experiment.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors presented a method for post-hoc explanation of vision Transformers. Specifically, a gradient-based extension called class-discriminative attention maps (CDAM) was proposed, along with a smooth CDAM and an integrated CDAM. Experimental analysis show the effectiveness and the discriminative ability of the proposed method wrt several evaluation metrics. The paper is well-written and easy to follow in general.

Three expert reviewers were invited to comment on the submitted paper, and both strengths and weaknesses were raised i nthe reviewing process. With back-and-forth discussions between the reviewers and authors, along with revisions, all major concerns were well addressed by the authors in the revised version. All three reviewers are satisfied with the revision, and in the end, a consistent positive score (one *Accept* and two *Leaning Accept*) was recommended. Both the AE and reviewers agreed on the interesting approach and contributions made in this paper, which could be of interest to a large potential audience of TMLR. As a result, the AE is please to inform the authors that the paper has been accepted to be published in TMLR.
Decision: Accept (poster)