ATMAN: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation
Authors: Björn Deiseroth
Mayukh Deb
Samuel Weinbach
Manuel Brack
Patrick Schramowski
Kristian Kersting
Venue: NeurIPS 2023 poster (2023)
Keywords: explainability
attention manipulation
perturbation
large language model
multi-modality
generative decoder
efficiency
transformer
TL;DR: We show how attention manipulation can be used as a perturbation method; to steer or explain large language models outputs w.r.t. the input relevance; on text and text/visual models.

Abstract: Generative transformer models have become increasingly complex, with large numbers of parameters and the ability to process multiple input modalities. Current methods for explaining their predictions are resource-intensive. Most crucially, they require prohibitively large amounts of additional memory, since they rely on backpropagation which allocates almost twice as much GPU memory as the forward pass. 
This makes it difficult, if not impossible, to use explanations in production. 
We present AtMan that provides explanations of generative transformer models at almost no extra cost. Specifically, AtMan is a modality-agnostic perturbation method that manipulates the attention mechanisms of transformers to produce relevance maps for the input with respect to the output prediction. Instead of using backpropagation, AtMan applies a parallelizable token-based search method relying on cosine similarity neighborhood in the embedding space. 
Our exhaustive experiments on text and image-text benchmarks demonstrate that AtMan outperforms current state-of-the-art gradient-based methods on several metrics while being computationally efficient. As such, AtMan is suitable for use in large model inference deployments.

ATMAN: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation
Björn Deiseroth 1,23+
Mayukh Deb Samuel Weinbach
Manuel Brack
Patrick Schramowski Kristian Kersting
Aleph Alpha "Technical University Darmstadt
2XX1v:2301.081100 [cs.LG] 5 Nov 2023
3 Hessian Center for Artificial Intelligence (hessian.Al)
*German Center for Artificial Intelligence (DFKI) SLAION
(bjoern.cm deiseroth, mayukh. deb, samuel weinbach) Baapaaabaaa@aaahpaaaaaaaaaa wawpphaalphppa....meeaaaaaaaaaaa
bhanuel.brache patrick, cchramowskki@@ffki.. (kersting) @cs. tu- uu-darmstadt..e
Abstract
Generative transformer models have become increasingly complex, with large numbers of parameters and the ability to process multiple input modalities. Current methods for explaining their predictions they require prohibitively large amounts of additional memory since they rely on backpropagation which allocates almost twice as much GPU memory as the forward pass. This renders it difficult, if not impossible, to use explanations in production. We present ATMAN that provides explanations of generative trans- former models at almost no extra cost. Specifically, ATMAN is modality-agnostic perturbation method that manipulates the attention mechanisms of transformers to produce relevance maps for the input with respect to the output prediction. Instead of using backpropagation, ATMAN applies a parallelizable coken-based search method relying on cosine similarity neighborhood in the embedding space. Our exhaustive experiments on text and image-text benchmarks demonstrate that ATMAN outperforms current state-of- the-art art gradient-buse methods on several metrics and models while being computationally efficient. As such, ATMAN is suitable for use in large model inference deployments.
resource-intensive. Most crucially,
Explainability through attention maps
Generalizing beyond single-task solutions using large-scale transformer- language models has gained increasing attention from the community. In particular, the switch open-vocabulary predictions promises Al systems capable of generalizing to new tasks. Arguably, transformers are the state-of-the-ard method in Natural Language Processing (NLP) and Computer Vision. They have shown exceptional performance in multi-modal scenarios, e.g., bridging Computer Vision (CV) capabilities with text understanding to solve Visual Question Answering (VQA) scenarios (10) 18
31 30 However, the increasing adoption of transformers also raises the need to understand their otherwise black-box predictions better. Unfortunately, the "scale is you need" assumption of recent transformer models results in severely large and complex architectures. This renders tasks such as their training, inference deployment, and explaining of their outputs resource -ntuuree-nttnsveee. requiring multiple enterprise- interprise-grade GPUs or even entire computing nodes, along with prolonged runtimes.
These authors contributed equally this work Source code; Inttps:/ /taps://gittaan /htbbtt....t com/ Aleph- Allpha/AtMan Atman (Sanskrit) "essence"
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Multimodal prompt
Completion and AtMan Expl.
Didpud Logis
Jonelyy cabin
the edge of iske
Wodiied Cauaal
To Transformee Blocks
Food Forward
Layer Norm
with a geese
Scores
Massed Mlui Set Atsetion
Soll
MalMi
This is painting of
(a) "What am I looking at?"
(b) ATMAN in the transformer architecture.
Figure I: (a) The proposed explainability method ATMAN visualizes the most important aspects of given image while completing the sequence, displayed above the relevance maps. The generative multi-modal model MAGMA is prompted to describe the shown image with: "<Image> This is painting of (b) The integration of ATMAN into the transformer architecture. We multiply the modifier factors and the attention scores before applying the diagonal causal attention mask as depicted on the nigt-hand side. Red hollow boxes (C) indicate one-values, and green ones ()) -infinity, (Best viewed in color.)
Explainable AI (XAI) methods target elucidate the decision-making processes and internal work- ings of AI models to humans. State-of-the-art methods for transformers focus on propagating gradients back through the model. This backpropagation allows for the accumulation of influence of each input feature on the output by utilizing stored activations during the forward pass [6, (7). Unfortunately, this caching also leads to significant overhead in memory consumption, which renders their productive deployment to be uneconomical, if not impossible. Often half of the available memory of the GPU has to remain unused at inference, requires an entirely separate deployment of the XAI pipeline.
much more memory-efficient. However, it has not yet been proven beneficial for explaining the predictions of transformers, as the immense number of necessary forward trials to derive the explanation accumulate impractical computation time.
Fortunately, another popular XAI idea, namely perturbation 19 24
To tackle these issues and in turn, scale explanations with the size of transformers, propose to bridge relevance propagation and perturbations. contrast to existing perturbation methods- executing perturbations directly in the input space-ATMAN pushes the perturbation steps via attention manipulations throughout the latent layers of the transformer during the forward pass. will show-to produce state interpolations, token-based similarity measures and to accurately steer the model predictions. Our explanation method leverages these predictions compute relevancy values for transformer networks. Our experimental results demonstrate that ATMAN significantly reduces the number of required perturbations, making them applicable at deployment time, and does not require additional memory compared backpropagation methods. In short, ATMAN can scale with transformers. Our exhaustive experiments on the text and image-text benchmarks and models demonstrate that ATMAN outperforms current sate-of-the-arr gradient-bas based methods while being more computationally efficient. For the first time, ATMAN allows one to study generative model predictions as visualized Fig. la without extra deployment costs. During the sequence generation with large multi-moda models, ATMAN is able to additionally highlight relevant features wrt. the input providing novel insights on the generation process.
This enables
Contributions. In summary, our contributions are: (i) An examination of the effects of token-based attention score manipulation on generative transformer models. (ii) The introduction of novel and memory-efficient XAI perturbation method for large-scale transformer models, called ATMAN, which reduces the number of required iterations to a computable amount by correlating tokens in the embedding space. (iii) Exhaustive multi-modal evaluations of XAI methods several text and image-text benchmarks and autoregressive (AR) transformers.
We proceed as follows. We start by discussing related work in Sec. In Sec. we derive ATMAN and explain attention manipulation as a perturbation technique. Before concluding and discussing the benefits well as limitations in Sec. we touch upon experimental evaluation Sec.

---

showing that ATMAN not only nullifies memory overhead and proves robust to other models but also outperforms competitors on several visual and textual reasoning benchmarks.
2 Related Work
Explainability in CV and NLP. The explainability of Al systems is still ambiguously defined g XAI methods are expected to show some level of relevance to the input with respect to the computed result of an algorithm. This task is usually tackled by constructing an input relevance map given the model's prediction. The nature of relevance can be class-specific, e.g. depending on specific target instances of a task and showing a "local solution" [26 27]. or class-agnostic, i.e., depending on the overall "global behavior" of the model only 4 The level of fine granularity in the achieved explanation thus depends on the selected method, model, and evaluation benchmark. Explainability in CV is usually achieved by mapping the relevance maps to pixel level and treating the evaluation as a weak segmentation task [25] 20 27]. I. On the other hand, NLP explanations are much more vaguely defined and usually mixed with more complex philosophical interpretations, such as labeling a given text to a certain sentiment category [8
The majority of XAI methods can be divided into the classes of perturbation and gradient analysis. Perturbations treat the model as a black box and attempt to derive knowledge of the model's behavior by studying changes in input-output pairs only. Gradient-based methods, on the other hand, execute a backpropagation step towards a target and aggregate the model's parameter adoptions to derive insights. Most of these XAI methods not motivated by specific discipline, e.g., neither by NLP nor CV. They are so generic that they can be applied to both disciplines to some extent. However, architecture-specine XAI methods exist well, such as GradCAM 25 leveraging convolutional neural networks' spatial input aggregation in the deepest layers to increase efficiency.
Explainability in transformers. Through their increasing size, transformers are particularly chal- lenging for explainability methods, especially for rccitecture-agnostt ones. Transformers' core components include an embedding layer followed by multiple layers of alternating attention and feed-forward blocks Fig. 1b The attention blocks map the input into separate "query", "key", and "value" matrices and are split into an array of "heads". As with convolutions in CNN networks, separated heads are believed to relate to specific learned features tasks [13.. Further, the attention mechanism correlates tokens wrt. the input sequence dimension. Care needs to be taken here as GPT-style generative models, in contrast to BERT-style embedding models, apply a causal mask- ing of activations as we will highlight further below. This in particular might obscure results for gradient-based methods.
Consequently, most explainability adoptions to transformers focus on the attention mechanism. Rollout ( assumes that activations in attention layers are combined linearly and consider paths along the pairwise attention graph. However, while being efficient, it often emphasizes irrelevant tokens, in particular, due its class-agnostic nature. The authors also propose attention flow, which, however, is unfeasible to compute as it constructs exhaustive graphs. More recently, Chefer et al. (2021) proposed to aggregate backward gradients and LRP [20] throughout all layers and heads of the attention modules in order to derive explanation relevancy. Their introduced method outperforms previous transformer-specific and unspecific XAI methods on several benchmarks and transformer models. is further extended to multimodal transformers [7] by studying other variations of attention. However, they evaluated only on classification tasks, despite autoregressive transformers' remarkable generative performance, e.g., utilizing InstructGPT [2] or multimodal transformers such as MAGMA [10].
BLIP [8] and OFA 3T
Multimodal transformers. Contrarily to these explainability studies evaluated on object-detcction models like DETR and VIT 5.. we study explainability open-vocabulary tasks, i.p., generated text tokens of a language model, and not specifically trained classifiers. Due to the multimodality, the XAI method should produce output relevancy either on the input text or the input image as depicted Fig. Ia Specifically, to obtain image modality in MAGMA [00,, the authors propose to fine-tune a frozen pre-trained language model by adding sequential adapters to the layers, leaving the attention mechanism untouched. It uses CLIP [2 vision encoder to produce image embeddings. These embeddings are afterward treated as equal during model execution, particularly regarding
3

---

other modality imput tokens. This multi-modal open-vocabulary methodology has shown competitive performance compared single-task solutions.
3 ATMAN: Attention Manipulation
We formulate finding the best explainability estimator of a model as solving the following question: What is the most important part of the input, annotated by the explanator, produce the model's output? In the following, we derive our perturbation probe mathematically through studies of
influence functions and embedding layer updates on autoregressive (AR) models [4 21.
Then we show how attention manipulation on single tokens can be used in NLP tasks to steer a model's prediction in directions found within the prompt. Finally, we derive multi-modal XAI method ATMAN by extending this concept to the cosine neighborhood in the embedding space.
3.1 Influence functions as explainability estimators
Transformer-based language models are probability distribution estimators. They map from some input space X, e.g., text or image embeddings, to an output space y e.g., language token probabilities. Let & be the space of all explanations, i.e., binary labels, over X. An explanator function then be defined as e (X + }) x (X' x y) + E, in words, given a model, an input, and target, derive a label on the input.
Given sequence of words w [wi,..., wy ] X an AR language model assigns a probability to that sequence pl (() by applying factorization p(w) = II. Plyp(w)) The loss optimization during training can then be formalized as solving:
max log ( (W)targer targer
Σ
log po wwe (wee tearrer targest Σ log softmax(ho(w:<))) Durger
log
iarger
Large (Wer, 0) turyet (w,0)
(2)
Here he denotes the model, Wo the learned embedding matrix, and targer' the vocabulary index of the t th target token, with length Itarge] = N. Eq. D derived by integrating the cross-entropy loss, commonly used during language model training with targer w. Finally, L denotes our loss function.
Perturbation methods study the influence of the model's predictions by adding small noise to the input and measuring the prediction change. We follow the results of the studies 04 2 to approximate the perturbation effect directly through the model's parameters when executing Leaving- One-Onne---u experiments the input. The influence function estimating the perturbation of input is then derived as:
dL dageet (2, 0c) c-n Tarret (z, 0=22))) - T.rrret (z,
Tlarget 2)
de
Here 8-z, denotes the set of model parameters which & would not have been seen during training. In the following, further show how to approximate 0-z.-
3.2 Single token attention manipulation
The core idea of ATMAN the shift of the perturbation space from the raw input to the embedded token space. This allows us to reduce the dimensionality of possible perturbations down to a single scaling factor per input token. Moreover, we do not manipulate the value matrix of attention blocks and therewith do not introduce the otherwise inherent input-distribution shift of obfuscation methods. By manipulating are able to interpolate the focus of the prediction distribution of the mooell-applffiiin concepts of the prompt resulting in an XAI method as described in the following.
attention scores at the positions of the corresponding input sequence tokens, we
suppressing
Attention was introduced in (28) as: O = softmax (HaaaxxxH) (H) V, where denotes matrix multiplication. The pre-softmax query-key attention scores are defined as
H Q K" dh..
In the case of autoregression, a lower left triangular unit mask M applied to these scores as H H M, with the Hadamard product. The output of the self-attention module is O

---

Prompt
target token
fries
Original completion
Hell, name
(1)
Benn
likes
to
eat
burgers
and
1.72
tewppors Fun been working on
susber and malls Brow the
combuse edincce
Ben
likes
to
eat burgers
and
1.06
entropy
completions
Ben
eat burgers and likes
likes
to
amplyy
the soccer fleld for the
and pist 2 yearx
(2)
Ben
eat burgers and
2.29
Suppress 500GOr
Ben
likes
to
eat burgers and
2.2
amplyy
Benn
likes
to
eat
burgers
and
14.2
trath and scikinee
Benn
likes
to
eat
burgers
and
0.89
SUgjANGSO (OAM)
my can gaene and Fro boon slaying while
(a) Illustration of the perturbation method.
(b) Steering through ATMAN.
Figure 2: (a) Illustration of the proposed explainability method. First, we collect original cross-entron score of the target tokens (1). Then we iterate and suppress one token at a time, indicated by the red box (O), and track changes in the cross-entropy score of the target token (2). (b) Manipulation of the attention scores, highlighted in bluc, steers the model's prediction into different contextual direction. Note that we found mcasuring of such proper generative directions to perform better than radical liken-masking. as we show later (c.f. Fig. ITI. (Best viewed color.)
R Rixxxdd the query matrix is Q hxsxda and K, V e Rixaxdd the keys and values matrices. Finally H. H.M.HM hxxass The number heads is denoted as h, and d, is the embedding dimension of the model. Finally, is the length seputesequence.
The perturbation approximation 0-,, required by Sec. c.n score manipulation as follows: Let w be input token sequence of length |w| n. Let i be token index within this sequence to perturbated by a factor f. For all layers and heads H, we modify the pre-softmax attention scores as:
can now approximated through attention
H H.,... (1-f),
where 1 [1]a* denotes the matrix containing only ones and f the suppression factor matrix for token é. In this section we set fr. = f. for A: = and f RR and 0 elsewhere. As depicted in Fig. lb we thus only amplify i- column of the attention scores of H by a factor (1 (1- f- however, equally for all heads. We denote this modification to the model as 0_, and assume fixed factor f Note that this position in the architecture is somewhat predestined for this kind of modification, it already applies the previously mentioned causal token-masking.
We define the explanation for a class label target as the vector of the positional influence functions: Jwr,eerge) := (I (Wi,, (W1, w),..., (Wn, w))
(5)
with Iirrget derived by Eq. 23 Itrret (w;, w) == Ilarget(ww (w, - (w, 0) In words, average the cross-entropy of the AR input sequence wrt. all target tokens and measure the change when suppressing token index i to the unmodified one. The explanation becomes this difference vector of all possible sequence position perturbations and thus requires n forward passes.
Fig. 2a illustrates this algorithm. The original input prompt is the text "Ben likes to eat burgers and" for which we want to extract the most valuable token for the completion of the target token "fries". Initially, the model predicts the target token with cross-entropy score of 1.72. Iterated suppression over the input tokens, as described, leads to "burgers" being the cost-inffuential input token with the highest score of 14.2 for the completion.
Token attention suppression steers the model's prediction. Intuitively, for factors 0 << < 1, we call the modifications "suppression", as we find the model's 's output now relatively less influenced by the token at the position of the respective manipulated attention scores. Contrarily, f <0 "amplifies" the influence of the manipulated input token on the output.
An example of the varying continuations, when single token manipulation is applied, can be seen in Fig. (bb We provide the model a prompt in which the focus of continuation largely depends on two tokens, namely "soccer" and "math". We show how suppressing and amplifying them alters the prediction distributions away from or towards those concepts. It is precisely this distribution shift we measure and visualize as our explainability.
We ran parameter sweep (c.f. App. A.4 and fixed this parameter to 0.9 for this work.
5

---

diniieet Fresadding
Context with AlNan Explenation
Normnnds
in the 15th and 11h oontrr thair name Normant, region France. They were desconded bcmn Herae "Nommar" corees from "Norsmman' radies and peikks
Denmank, iocland Norway who, under isodor Rollo agueed socar fosily
Input Imagge
Withuut CS
crten entropy
From which countrie the Norse onginate?
Dermerk, insland
Nonway
A
Wieen wore the Nermars is Normanty??
10th and 11th contarics
In shat countyy is Normandy located?
Fisnas
(a) Correlated token suppression on images.
(b) SQuAD example with ATMAN Explanations.
Figure 3: (a) Correlated token suppression of ATMAN enhances explainability in the image domain. i) Shows an input image along with three perturbation examples (A, B, C). In A, we only suppress single image token (blue). In B, the same token with its relative cosine neighborhood (yellow), and in C, a non-related token with its neighborhood. Below depicted are the changes in the cross-entropy loss. The original score for the target token "panda" is denoted by and the loss change by A. ii) Shows the resulting explanation without Cosine Similarity (CS) and with CS. We evaluated the influence of the CS quantitatively Fig. 2a
(b) An example of
the SQuAD D dataset with ATMAN explanations. The instance contains three questions a given context,
each with a labeled answer pointing to fragment of the context. ATMAN is used to highlight the corresponding
fragments of the text responsible for the answer. It can be observed that the green example is full, the blue in part, and the yellow is not at all recovered according to the given labels. However, the yellow highlight seems least related to the label. (Best viewed in color.)
3.3 Correlated token attention manipulation
Suppressing single tokens works well when the entire entropy responsible for producing the target token occurs only once. However, for inputs with redundant information, this approach would often fail. This issue is especially prominent in Computer Vision (CV). where information like objects in an image is often spread across multiple tokens as the image is usually divided into patches, each of which is separately embedded. It is a common finding that applied cosine similarity in the embedding space, i.p., for CLIP-like encodings used in MAGMA, gives good correlation estimator [17 This is in particular due to the training by a contrastive loss. We integrate this finding into ATMAN in order to suppress all redundant information corresponding to a particular input token at once, which we refer to as correlated token suppression.
Fig. 3a summarizes the correlated token suppression visually. For input tokens and embedding dimension d, the embedded tokens result in a matrix T =t+, e RRddd The cosine similarity is computed from the normalized embeddings T = (7+), with T, ts/ltall. for 1,...n, n, as S = (s:) = T. T e [,1, 1|xxm Note that the index i denotes a column corresponding to the respective input token index. Intuitively, the vector s; then contains similarity scores to all input tokens. In order to suppress the correlated neighborhood of a specific token with the index i, we, therefore, adjust the suppression factor matrix for Eq. 4 as
Hu,.. = H e ((1 f)+ f(1 )). with Fr.- Si,ks if S $, otherwise.
(6)
In words, we interpolate linear between the given suppression factor 1-1 and 1, by factor of the thresholded cosine similarity fi... As we only want suppress tokens, restrict the factor value range to greater than 0 and smaller than 1. The parameter x O is to ensure lower bound on the similarity, and in particular, prevents flip of the sign. Overall, we found this to be the most intuitive and robust equation. In Fig. II we ran sweep over the hyper-parametee and recall values. Note in particular, that our modifications are rather subtle compared to entire token-masking. The latter did not lead to satisfying results, even given the context. We compute the similarity scores once on the initial embedding layer, and uniformly apply those modifications throughout all layers.
to obtain the best precision
Note that variations of Eq. and could also lead to promising results. In particular in the active research field of vision transtormers, hierarchical token transformations could be applicable and speed
We empirically fixed 0.7 through parameter sweep (c.f. Appendix

---

AIMan
Chefer
AtMan
Chefer al.
AIMan
Chefer et
orangutan
couch
quadbike
green plants
birthday
Figure 4: ATMAN produces less noisy and more focused explanations when prompted with multi-class weak segmentation compared Chefer. The three shown figures prompted explain the target classes above and below separately. (Best viewed in color.)
up computations (c.f. 33 Fig T6.. Furthermore, preliminary experiments using the similarity between the query and key values per layer, a different similarity score such as other embedding models or directly sigmoid values, or even non-ume non-uniform application of the manipulations yield promising, yet degenerated performance (c.f. App. [A.15. As we could not conclude a final conclusion yet, we leave these investigations future research.
With that, we arrived at our final version of ATMAN. Note that this form explanation & is "local", target refers to our target class. We can, however, straightforwardly derive "global explanation" by setting target = y, for y, a model completion to input w of a certain length. could then be interpreted rather abstract as the model's general focus [4]
4 Empirical Evaluation
We ran empirical evaluations on text and image corpora to address the following questions: (Q1) Does ATMAN achieve competitive results compared previous XAI methods for transformers the language as well as vision domain? (Q2) Is ATMAN applicable to various transformer models, and does ATMAN scale efficiently, and can therefore be applied to current large-scale AR models?
To answer these questions, we conducted empirical studies on textual and visual XAI benchmarks and compared ATMAN standard approaches such as IxG (26), IG (27). GradCAM (5 and the transformer-specifie XAI method of [6) called Chefer in the following. Note that all these methods utilize gradients and, therefore, categorize as propagation methods leading memory inefficiency.
We also tried to apply existing perturbation methods such as LIME [24] and SHAP 19 and Attention Rollout [T] However, they failed due extremely large trials and, in turn, prohibitive computation time, or qualitatively (c.f. Appendix A.9). We adopt common metrics, namely mean average precision (mAP) and recall (mAR) (c.f. Appendix I.II). and state their interquartile statistics in all experiments. To provide a comparison between the XAI methods, we benchmarked all methods on GPT-J [29] for language and MAGMA-6B [10] for vision-language tasks. Through memory efficiency, ATMAN can be deployed on really large models. We further evaluate ATMAN MAGMA-13B and 30B to show the scaling behavior, and, on BI.IP [T8) show the architecture independence. To obtain the most balanced comparison between the methods, we selected datasets focusing on pure information recall, separating the XAI evaluation from the underlying model interpretations.
4.1 ATMAN can do language reasoning
Protocol. All experiments were performed the open-source model GPT-J 29 Since with ATMAN we aim to study large-scale generative models, we formulate XAI on generative tasks as described in Sec. 3.3 For evaluation, (SQuAD) 23 The QA dataset is structured as follows: Given a single paragraph of information, there are multiple questions, each with corresponding answer referring paragraph. visualization an instance of this dataset can be found in Fig.Bb) SQuAD contains 536 unique paragraphs and 107,785 question//epllantion pairs. The average context sequence length 152.7 tokens and the average label length 3.4.
used the Stanford Question Answering (QA) Dataset
position in the given
The model was prompted with the template: "(Context) Q: (Question) A:", and the explainability methods evaluated on the generations to match the tokens inside the given context, c.f. Fig. Sec. (3.3] If there multiple tokens in the target label, computed the mean of the generated scores.
7

---

Table 1: ATMAN (AM) outperforms other XAI methods (a) the text QA benchmark SQuAD and (b) the image-text VQA task of OpenImages. Shown are the (interquartile) mean average precision and recall (the higher, the better). The best and second best values are highlighted with e and o. Evaluations are on 6B model. (c) ATMAN evaluated on other model types: MAGMA variants (indicated by their parameter size) and BLIP.
(a) SQUAD.
(b) OpenImages (OI).
(c) Other models on OL
IG
IxG Chef. AM
GC XxG IG Chef. AM
AM
AMBLIP
mAP
49.5
51.7 o72.7 73.77
55.2 59.1 c60.5 58.3 65.5
64.8
63.6
64.6
mAP Q
49.5
61.4 o77.5 81.8
54.3 59.0 c60.6 57.8 70.2
65,6
65.1
66.1
mAR
87.1
91.8 99..6 o93.4
4.2 10.8 10.7 o11.7 15.7
14.6
13.6
26.4
mARIQ
98.6 . 100 . 100 100
3.8 10.2 10.0 o11. 19.7
16.7
15.6
28.4
Similar to weak segmentation tasks in computer vision, we regarded the annotated explanations as binary labels and determined precision and recall over all these target tokens.
Results. The results are shown in Tab. It can be observed that the proposed ATMAN method
thoroughly outperforms all previous approaches on the mean average precision. This statement holds as well for the interquartile mean of the recall. However the entire recall, Chefer slightly outperforms ATMAN. Note that the high values for recall scores throughout all methods are partly due to the small average explanation length, such depicted in Fig. 3b] Further details and some qualitative examples can be found in Appendix(A.2 (A.2
Paragraph chunking. ATMAN can naturally be lifted to the explanation of paragraphs. We ran experiments for ATMAN dividing the input text into several paragraphs by using common delimiters as separation points and evaluating the resulting chunks at once, in contrast to token-wise evaluations. This significantly decreases the total number of required forward passes and, on top, produces "more human" text explanations of the otherwise still heterogeneously highlighted word parts. Results are shown in Appendix A.8
4.2 ATM can do visual reasoning
Protocol. Similar to language reasoning, we again perform XAI on generative models. We evaluated the OpenImages [16] dataset as VQA task and generated open-vocabulary prediction with the autoregressive model. Specifically, the model is prompted with the template: "[Image) This is a picture of and the explainability methods executed to derive scores for the pixels of the image on the generations with respect to the target tokens. If there were multiple tokens in the target label, we computed the mean of the generated scores. For evaluation, we considered the segmentation annotations of the dataset as ground truth explanations. The segmentation subset contains 2, 7.7M annotated images for 350 different classes. order to ensure good performance of the large-scale model at hand and, in turn, adequately evaluate only the XAI methods, we filtered the images for minimum dimension of 200 pixels and maximal proportional deviation between width and height of 20%. We randomly sample 200 images per class on the filtered set to further reduce evaluation costs. In total, this leads to a dataset of 27.871 samples. The average context sequence length 144 tokens, and the average label coverage is 56% of the input image. To provide a fair comparison between all XAI methods, we compute token-based attributions for both ATMAN and Chefer, as well as IG, IxG, and GC. Note that the latter three allow pixel- attribution granularity, however, these resulted in lower performance, as further discussed in Appendix A.2
The results are shown in Tab. lb. It can be observed that ATMAN thoroughly outperforms all other XAI approaches on the visual reasoning task for all metrics. Note how explicit transformer XAI methods (ATMAN, Chefer) in particular outperform generic methods (GradCAM, IG, IxG) in recall. Moreover, while being accurate explanations compared to Chefer. Through the memory efficiency of ATMAN, we were able to evaluate a 13B and 30B upscaled variant of MAGMA (c.f. Tab. K.). Interestingly, the general explanation performance slightly decreases, when upscaled, compared to the 6B model variant. However, throughout all variants ATMAN scored comparable high. Note that the overall benchmark performances of the models slightly increase (c.f. App. A.14). This behavior could be attributed to
Results.
efficieet (c.f. next section), ATMAN also generates more
8

---

T
AtMan
400
Chefer et
128
GPU memory usage (GB
256
512
60
300
60
1.5
768
1024
200
40
1.0
Baseline
Runtime
20
100
0.5
6.0
8.9
13.1
17.3
8.9
Model size (billion parameters)
Figure 5: ATMAN scales efficiently. Performance comparison the XAI methods ATMAN and Chefer et al., on various model sizes (x-axis) executed on single 80GB memory GPU, Current gradient- based approaches do not scale; only ATMAN can be utilized large-scale models. Solid lines refer to the GPU memory consumption in GB (left y-axis). Dashed lines refer to the runtime seconds (right y-axis). Colors indicate experiments on varying input sequence lengths. baseline (green) a plain forward pass with a sequence length of 1024 measured. The lightning symbol emphasizes the non non-deployability when memory resource is capped to 80GB. (Best viewed in color.)
the increased complexity of the model and, subsequently, the complexity of the explanation at hand. Hence, the "human alignment" with the model's explanations is not expected to scale with their size.
Model agnosticism. Note that the underlying MAGMA model introduces adapters a pre-trained decoder text model to obtain the image multi-modality. is yet unclear to what extent adapters change the flow of information in the underlying model. This may be reason for the degenerated performance of gradient-base based models. As ATMAN however outperforms such methods, we may conclude that information processing still remains bound to the specific input-sequence position. This finding generalizes to encoder architectures, as experiments with BLIP suggest in Tab. [y. Precisely, BLIP's multi-modality is achieved by combining decoder model. These are responsible for encoding the image, the question, and finally for generating response separately. Information between the models is passed through cross-attention. We only applied ATMAN to the image-encoder model of this architecture (c.f. App. A0). The results demonstrate the robustness of the suggested perturbation method over gradient-based approaches.
image-encoder, text-encoder, and final text-
Qualitative illustration. Fig. 4 shows several generated image explanations of ATMAN and Chefer for different concepts. More examples of all methods can be found Appendix A.7 and eval- uation on the more complex GQA dataset [12] is discussed in Appendix A.12 We generally observe more noise in gradient- based methods, in particular around the edges. Note that ATMAN evaluates the change in its perturbated generations. These are independent of the target tokens. Compared to other methods like Chefer, we, therefore, do not need to process the prompt more than once for VQA on different object classes with ATMAN. Note again that the aforementioned cosine similarity is crucial to obtain these scores. Quantitative results can be found in App. A.4
In general, the results clearly provide an affirmative answer to (QI): ATMAN is competitive with previous XAI methods, and even outperforms transformer-specife ones on the evaluated language and vision benchmarks for precision and recall. Next, we will analyze the computational efficiency of ATMAN.
4.3 ATMAN can do large scale
While ATMAN shows competitive performance, it computes, unlike previous approaches, expla- nations at almost no extra memory cost. Fig. illlustraas the runtime and memory consumption on single NVIDIA A100 80GB GPU. We evaluated the gradient-based transformer XAI method 6 and ATMAN. The statistics vary in sequence lengths (colors) from 128 to 1024 tokens, and all experiments executed sequentially with batch size 1 for best comparison.
One can observe that the memory consumption of ATMAN around that plain forward pass (Baseline; green) and increases only marginally over the sequence lengths. In comparison, the method of 6 and other gradient-based methods- exceeds the memory limit with more than double in memory consumption. Therefore, they fail on larger sequence lengths models.

---

Runtime (5)
AtMan Chefer
Memory (GB)
75
AtMan
50
25
6b
13b 17b
34b
6b
13b 17b
34b
Model size (billion parameters)
Model size (billion parameters)
Figure 6: Examplified average runtime and total memory consumption for the task of explaining image. comparing AtMan with Chefer et al.. We applied memory improvements on Chefer et al., appe-paralll 4 on both and additionally batch-size 16 on ATMAN. Chefer et al. still is not applicable to 34b models when the total memory consumption is capped to 80GB. AtMan's 's increase in total computational time lies around 1 magnitude. Note that AtMan could be computed entirely parailel, and as such, this total clock time can still be reduced to subseconds. Measured are 144 explanation tokens (full given image on MAGMA), for average total prompt length of 165 tokens.
Whereas the memory consumption of ATMAN stays is almost constant, the execution time signifi- cantly increases over sequence length when no further token aggregation is applied upfront. Note, that Fig. 5 shows complete and sequential execution. The exhaustive search loop of ATMAN can be run partially and in parallel to decrease its runtime. In particular, the parallelism can be achieved by increasing the batch size and naturally by a pipeline-paralle
execution. For instance, since large
models beyond 100B are scattered among nodes and thus many GPUs, the effective runtime is reduced by magnitudes to a proximate scale of the forward pass. Furthermore, tokens can aggregated in chunks and evaluated on a more coarse level, which further reduces the runtime drastically, as demonstrated in Fig. 16
In Fig./owe provide performance results realistic scenario. It compares a practical query (average 165 tokens) on the task of explaining a complete image (144) tokens. Runtimes are measured with pipe-parallel 4, and ATMAN additionally with batch size 16. Specifically, ATMAN requires between I and 3 total compute seconds, around magnitude longer compared to Chefer. Note that this can still further be divided by the number of available idling workers, to reduce the absolute clock time to subseconds. Each batch of ATMAN can be processed entirely parallel. Moreover Chefer, even with memory optimizations, fails to scale to 34b with the given memory limit (80GB).
Overall, these results clearly provide an affirmative answer to (Q2): Through the memory efficiency of ATMAN, it can be applied to large-scale transformer- baasddd models.
5 Conclusion
We proposed ATMAN, modality and architecture agnostic perturbation-based XAI method for generative transformer networks. In particular, ATMAN reduces the complex issue of finding proper input perturbations ATMAN outperforms current approaches relying on gradient computation. It moreover performs well decoder as well encoder architectures, even with further adjustments like adapters or crossattention in place. Most importantly, through ATMAN's memory efficiency, it provides the first XAI tool for deployed large-scale AR transformers. We successfully evaluated 30B multi-modal model on a large text-image corpus.
single scaling factor per token. As our experiments demonstrate,
Consequently, ATMAN paves the way to evaluate whether models' explanatory capabilities scale with their size, which should be studied further. Besides this, our paper provides several avenues for future work. including explanatory studies of current generative models impacting our society. Furthermore, it could lay the foundation for not only instructing and, in turn, improving the predictive outcome of autoregressive models based on human feedback (1 but also their explanations [11]
Acknowledgments
We gratefully acknowledge support by the German Center for Artificial Intelligence (DFKI) project "SAINT", the Federal Ministry of Education and Research (BMBF) project "AISC" (GA No. 011S22091), and the Hessian Ministry for Digital Strategy and Development (HMinD) project "Al Innovationlab" (GA No. S- S-DIW04/0013/0033, This work also benefited from the ICT-48 Network
10

---

of AI Research Excellence Center "TAILOR" (EU Horizon 2020, GA No 952215), the Hessian Ministry of Higher Education, and the Research and the Arts (HMWK) cluster projects "The Adaptive Mind" and "The Third Wave of AI", and the HMWK and BMBF ATHENE, project "AVSV". Further, we thank Felix Friedrich, Marco Bellagente and Constantin Fichenberg for their valuable feedback.
References
[1] Samira Abnar and Willem H. Zuidema. Quantifying attention flow transformers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, (ACL), pages 4190-448 4197. Association for Computational Linguistics, 2020.
[2] Daniel Bis, Maksim Podkorytov, and Xiuwen Liu. Too much in common: Shifting of embeddings in transformer language models and its implications. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL- HHT),, pages 51 17-3100 Association for Computational Linguistics, 2021.
[3] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations, 2023, URL BtRLltttts://opereevieee http::.ee////oorrreooeeo www.orowwwdddddoddoooor.e
[4] Nadia Burkart and Marco F. Huber. A survey on the explainability of supervised machine learning. J. Artif. Intell. Res., 70:245-317, 2021.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference of Computer Vision (ECCV), Lecture Notes in Computer Science, pages 213-229. Springer, 2020.
[6] Hila Chefer, Shir Gur, Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 782-791, 2021.
[7] Hila Chefer, Shir Gur, and Lior Wolf. Generie attention model explainability for interpreting bi-modal and secoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 397-406, 2021.
[8] Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state explainable Al for natural language processing. In Proceedings of the Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, (AACL/JJNLL)), pages 447-459. Association for Computational Linguistics. 2020.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations (ICLR), 2021.
[10] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. Magma multimodal augmentation of generative models through adapter-based finetuning. In Findings of EMNLP, Association for Computational Linguistics, 2022. /BLL/tttps///claantollogg org/2022
Lindings-emn/ 179
[11] Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting, A typology explore and guide explanatory interactive machine learning. Nature Machine Intelligence, 2023.
[12] Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world vi- sual reasoning and compositional question answering. In IEFE Conference on Computer Vision and Puttern Recognition. CVPR 2019. Long Beach. CA, USA. June 16-20. 2019. pages 6700-6709. Computer Vision Foundation IEEE, 2019. doi: 10.1109/CVPR.2019 00686. URL http:///peenaccess thecvf wwweevvw.ooo//oooeeeeeeeeeeeeeeo Setoontett_CR_______e__e Stt___CVV/RP_CP_PE_e 20119/ttm/Huuseee Dataset_ for_ heal wetdd_iivvaaaaaaaaaaaeee www.eeaaassnnee......... aacggannd__oae wtt.aaddoooossstiionnal.l Btt___CVV______P____ 2019_paper htmll
_atpdsoo_ooo______..____a___ GQA___Neww
[13] Jae-young Jo and Hyyn Myaeng. Roles and utilization of attention heads in transformer-basee neural language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, (ACL), pages 3404-3417. Association for Computational Linguistics, 2020.
[14] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the International Conference on Machine Learning, (ICML), volume 70 of Proceedings of Machine Learning Research, pages 1885 1894. PMLR, 2017.
11

---

[15] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds. Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblizz-ichaardson. Captum: A unified and generic model interpretability library for pytorch, 2020.
[16] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Cupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng. Dhyanesh Narayanan, and Kevin Murphy, Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from 2017.
https://gihubh.comoppenee
[17] Ray Larson. Introduction to information retrieval. J. Assoc. Inf. Sci. Technol. 61:852 853, 2010.
[18] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. Proceedings of the International Conference on Machine Learning (ICML), 2022.
[19] Scott M. Lundberg and Su-In Lec. A unified approach to interpreting model predictions. In Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurlPS), pages 4765-4774, 2017.
[20] Grégoire Montavon, Alexander Binder, Sehastian Lapuschkin, Wojeiech Samek, and Klaus-Robert Müller. Layer-wise relevance propagation: An overview. In Explainable Al: Interpreting, Explaining and Visu- alizing Deep Learning, volume 11700 of Lecture Notes in Computer Science. pages 193-209. Springer. 2019.
[21] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurlPS), 2022.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Confer- ence on Machine Learning (ICML), Proceedings of Machine Learning Research, pages 8748-8765. PMLR, 2021.
[23] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2383-2392. The Association for Computational Linguistics, 2016.
[24] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pages 1135-1144 144, ACM, 2016.
Explaining the
[25] Ramprasaath Rs, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM, Visual explanations from deep networks via gradient- based localization. International Journal of Computer Vision, 128, 02 2020. doi: 10.1007// 1223--19-12-
[26] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation differences. In Proceedings of the International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pages 3145-3153. PMLR, 2017.
[27] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Proceedings of the International Conference on Machine Learning, (ICML), page 3319-3328. JMLR.org, 2017.
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), pages 5998-6008. 2017.
[29] Ben Wang and Aran Komatsuzaki, Gpt-j-6b: A 6 billion parameter autorogressive language model, 2021.
[30] Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative srage-to-texx transformer for vision and language. Transactions on Machine Learning Research, 2022.
12

---

[31] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unitying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
13

---

A Appendix
A.1 Remarks on executed benchmarks
We executed all benchmarks faithfully and to the best of our knowledge. The selection of compared methods was made to be rather diverse and obtain good overview in this field of research. In particular, with regards to the multi-modal transformer scaling behavior, as there are in fact no such studies for AR models yet to compare to. It is possible, for all methods, that there are still improvements we missed in quality as well as performance. However, we see the optimizations of other methods to multi-modal AR transformer models as a research direction on its own.
Chefer. The integration of Chefer was straightforward. As it can be derived by the visualizations, there are noticeable artifacts, particularly the edges of images. In this work the underlying transformer model was MAGMA, which is finetuned using sequential adapters. It is possible that this, or the multi-modal AR nature itself, the cause for these artifacts. We did not further investigate to what extent the adapters are to be particularly integrated in the attribute accumulation of Chefer. Also notice that ATMAN often has similar, but not as severe, artifacts.
IxG, IG and GradCAM. The methods IxG, IG, and (guided) GradCAM failed completely from the quality perspective in recall when applied to pixel granularity (c.f. Tab. 2 Those are the only ones directly applicable the image domain, and thus also included the vision encoder in the backward pass. We did not further investigate or fine-tune evaluations to any method. For better comparison therefore only present the akkenbbaedd comparisons. All methods are evaluated with the same metrics and therewith give us a reasonable performance comparison without additional customization configuration.
Details on Results. For a fair comparison, all experiments were executed on single GPU, as scaling naturally extends all methods. We also want to highlight that we did not optimize the methods for performance further but rather adopted the repositories as they were. The memory inefficiency of gradient- -based methods arises from the backward pass. maximal memory performant representative is the Single-Layer-Attrine Attributtint method IxG, which only computes partial derivatives on the input with respect to the loss. Even this approach increases the memory requirement beyond an additional 50% and fails for the scaling experiments up to 34B.
In Fig. 5 we ran Chefer with full backward pass. We adopted this the minimum amount of gradients (we saw) possible and plot the full scaling benchmark below in Fig The key message remains the same. With the given IxG argument, we do not see much potential in improving memory consumption further.
The methods IxG, IG and GradCam are integrated using the library Captum [15]. We expect them to be implemented as performant as possible. IntegratedGradients is a perturbation method on the input, integrating changes over the gradients. The implementation at hand vastly runs OOM. Finally GradCam is a method specialized on CNN networks and therefore does not work for text only (or varying sequence lengths). It requires the least amount of resources but also produces poor results, without further investigations.
ATMAN Parallelizability. As a final remark, we want to recall again that the runtime measured in the sequential execution can be drastically reduced due to its parallelizability, i.p., as it only requires forward passes. For sequence length 1024, we measured 1024 iterations in order to explain each token. However note that ATMAN can also be applied to only parts chunks of the sequence (c.f. Sec. A. in contrast to gradient methods. Moreover, all tokens to explain can be computed entirely in parallel. In a cluster deployment, these can be distributed amongst all available workers. On top, can be divided by the available batch size and true
(c.f. Fig. 3)
"Setting sanuires_gra===tt False to every but the attention tensors.
14

---

Solids CPP) memoryy usagee (G8)
(CB)
Chefer (riemory optimizel)
110
(seconds
1.8
10t
1.2
129
Dashed: Runtimm
100
12
Solid GPU
250
Solid GPU
R2
17.
341
121
26.
600
31.
Model size (billion parameters)
Model size (billion param eters)
Model size (billion parameters)
Solid: GPU memory usage (GB)
Dached: Ru mimtnn (sezonids)
Solid: GPU memory ussge (GB)
GradiCam
Dotch Runtime (sacands
Model sDe (billion parameters)
Model site (billion parameters)
Figure 7: Performance comparison of the explainability methods over various model sizes (x-axis) executed on a single 80GB memory GPU, with fixed batch size 1. Solid lines refer to the GPU memory consumption in GB (left y-axis). Dashed lines refer to the runtime in seconds (right y-axis). Colors indicate experiments input sequence lengths. As baseline (green) plain forward pass with sequence length of 1024 is measured. Note that GradCAM can only applied the vision domain, is therefore fixed to 144 tokens. Note that already consumes as much memory as a forward pass of 1024 tokens. (Best viewed in color.)
varying
0.9
Number of instances
16000
AlMlan
Chifter
14000
Grad
12000
0.7
10000
6000
0.5
4000
2000
12 14
Number of tokens in explanation
Maximum number of questions
(a) Histogram of explanation token length.
(b) mAP all methods, grouped by number ques- tion/ answer pairs in the dataset.
Figure 8: Further evaluations the SQuAD dataset. (Best viewed in color.)
A.2 Detailed SQuAD Evaluations
This sections gives more detailed statistics the scores presented in Tab. First Fig. Sal is the histogram of the token lengths of all explanations. Bb the mAP score for all methods on the entire dataset, grouped by the number of questions occuring per instance.
A.3 Detailed OpenImages Evaluations
This section gives more detailed statistics on the scores presented in Tab. Fig. iis the histogram of the fraction label coverage on images. Fig. 10a and TOb are boxplots for methods on the entire dataset, for mean average precision as well as recall. Note that the methods IG, IxG and GradCam can be evaluated on image-pixel level, contrary to ATMAN and Chefer. However their performances (i.p. wrt. recall) significantly degenerate, c.f. Tab. Fig 15
15

---

Number Samples
2002
0.0
0.2
9.4
0.6
0.8
1.5
Fraction image covered by Label
Figure 9: Histogram of percentage of label coverage of the images.
0.8
0.6
0.2
0.0
0.3
ADMan
6C
Chefer
(a) mAP Boxplot for all methods of all images.
(b) mAR Boxplot for all methods of all images.
Figure 10: Further evaluations on the Openlmages dataset. (Best viewed color)
A.4 Discussion of Cosine Embedding Similarity and Parameter values
We fixed the parameter x = 0.7 of Eq. and f = 0.9 Eq. throughout this work. They were empirically concluded by running line sweep randomly sampled subset of the Openlmages dataset once, c.f. Fig. 11 Note that pure token-masking ("mask") always leads to worse performance compared to more subtle conpept-suppreesion operations. We the same parameters throughout this work, however, parameters may be optimizable depending on the use case and dataset.
In Fig. 12a and 12b we compare the mean average precision and recall scores for OpenImages for both variants, with and without correlated token suppression (to threshold k). Clearly the latter outperforms single token suppression.
The following Fig. 1 shows visually the effect on weak image segmentation when correlated suppression of tokens is activated, or when using single token suppression only. Notice how single token only occasionally hits the label, and often marks token the edge. This gives us reason to believe that entropy is accumulated around such edges during layer wise processing. This effect (on these images) completely vanishes with correlated suppression of tokens.
A.5
Variation Discussion of the method
Note that the results of Eq. 4 are directly passed to softmax operation. The softmax of a vector z is defined as
softmax(z), = ea IE,e*
In particular, the entries 0 and z will yield to the results softmax(z): = 1, e and softmax (z): one might argue as follows: If we intent to suppress the entropy of token i, we do not want to multiply it by factor 0, but rather subtract -oo of it. I.c propose the modification
(H),, (H) + log(f).
The only problem with this Eq. 7 is, that skews the cosine neighborhood factors. While we experienced this working more naturally in principle, for hand-crafted factors, did not get best
16

---

precision
recall
6/0
02
02
6.00
0.
0.06
615
0.35
610
05-
0.-
0,50
Figure 11: Parameter sweep for Eq. 6 It can be observed that indeed cosine similarity (y-axis) along with more subtle modification (X-axis) outperforms other configurations, i.p. now "mask" that entirely masks single tokens (i.e. multiplies 1 f). This result indicates that indeed conceptual entropy is scattered across tokens, that need be suppressed once.
AlMan
AtMan
22003
Numbbe of samagle
5000
Number of samples
1.5000
4000
3000
12000
2000
2000
1000
50
0.2
0.4
1.5
0.8
1.0
5.0
0.4
0.5
2.8
1.0
(a)
(b)
Figure 12: Histogram metric evaluation OpenImages for ATMAN with (green) and without (orange) correlated suppression of tokens. (Best viewed in color.)
performance in combination with Eq. 6 In the following Fig. 14a and 14b we show analogous evaluations Fig. [2a and 12b It is in particular interesting that the mode without correlated tokens slightly improves, while the one with slightly decreases in scores, for both metrics.
A.6 Artifacts and failure modes
Merged into A.12 (kept here for reference).
A.7
Qualitative comparison weak image segmentation
In the following Fig. 15 we give several examples for better comparison between the methods on the task of weak image segmentation. To generate the explanations, we prompt the model with "<Image> This is a picture of and extract the scores towards the next target tokens as described with Eq. 5 for ATMAN. For multiple target tokens, these results are averaged. the same fashion, but with an additional backpropagation towards the next target token, we derive the explanations for Chefer and the other gradient methods.
A.8 Application to document q/a
In Fig. 16 Le apply ATMAN on a larger context of around 500 tokens paragraph wise. The Context is first split into chunks by the delimiter tokens n. "'n" and " and". Then iteratively each chunk is evaluated by prompting in the fashion " "(Conlert) Q: D=(Question) A: and the cross entropy extracted towards the target tokens, suppressing the entire chunk at once, as described in Sec. 3. can be observed that the correct paragraphs are highlighted for the given questions and expected targets.
17

---

Table 2: OpenImages (OI) evaluations pixel granularity.
IxG
IG
GC
mAP
38.0 46.1
56.7
mAPro TQ
34.1
45.2
60.4
mAR
0.2
0.3
0.1
mAR
0.1
0.1
0.1
Input image target
Label
Single
Frog
Shoop
Fish
Figure 13: Example images showing the effect of correlated suppression as described in Correlated Token
Attention Manipulation Sec. 3.3 and Single Token Attention Manipulation Sec. 3.2. (Best viewed in color.)
In particular, one observe the models interpretation, like the mapping of formats or of states to countries. Note particular that it is not fooled by questions not answered by the text (last row).
A.9 Attention Rollout, Flow, other Perturbation method
Attention Flow constructs graph representing the flow of attention through the model. Unfortunately this approach is computationally too intense for evaluation on large language models; Chefer et al. (2021) already ran into issues smaller BERT models. LIME (24] and SHAP [19] are perturbation methods closely related to ATMAN. However they result in several 100.000 forward trials which is inpractical for with large language models.
Finally, we ran additional experiments for Attention Rollout 1. Note that the authors stress its design for encoder classification architectures, while we explore the applicability to generative decoder models in this work. Further, Rollout is by default class-agnostic, in contrast to ATMAN or the studied gradient methods. We adopted Attention Rollout by appending the target to the prompt, and only aggregating attentions over those last target sequence positions. Note that in contrast to the other studied methods, Rollout disregards the output distribution.
We observed large artifacts on the explanations the positions the last 2 image patches; apparently most attention aggregation take place at those positions. which may be due to the causal mask. On OpenImages our adoption of Rollout achieves a mAP score of 43.6 6 and a mAR of 1.3 (c.f. AtMan with 65.5 and 13.7). Scores do increase when removing the last image patch, however, this also entirely erases explanation on this patch. On Squad we achieved mAP AP 23.1 and m A of 53.4 (AtMan: 73.7, 93.4).
18

---

AlMan without
AtNon
12000
AtMan
5000
Number of instances
Number instances
20000
4000
8000
3000
6000
2000
2000
(a)
(b)
Figure 14: Histogram of the evaluation metrics OpenImages for ATMAN with and without correlated token suppression. (Best viewed color.)
Table 3: Evaluation of XAI on BLIP model.
Chefer
ATMAN
mAP 59.7
64.6
MAPıQ
60.3
66.1
MAR
20.2
26.4
mQR
19.4
28.4
A.10 Demarcation to other architectures, BLIP experiments
To answer the question of "whhy we think that AtMan could be generalized other (generative) multimodal Transformers", consider the following arguments.
We conduct for the first time experiments generative decoder models. The underlying MAGMA model extends the standard decoder transformer architecture, having causal attention masks in place, by additional sequential adapters that could potentially shift distributions. However, the proposed attention manipulation still leads to robust perturbation method - in text-only as well as image-text modality. We thus conjecture that the skip connections lead to somewhat consistent entropy flow throughout the network. OFA (< 1B) and BLIP rely on standard encoder-decoder architectures even without adapters- and the same skip connections. Furthermore, both models process the input image in a similar fashion to obtain "image tokens" forwarded throughout the transformer at the positions.
BLIP, in contrast to OFA, separates the processing of the image, the prompt, and the answer generation into 3 different models standard encoders, only the answer generation model being decoder. The different outputs are gross-attended. We apply ATMAN only on the iision-encoder model.
For comparison, we furthermore evaluate Chefer on BLIP, its attention aggregation being only applied the vision encoder as well. Tab. 3 compares these two methods, showing that ATMAN still outperforms Chefer. Note, however, that Chefer achieves better scores compared to being applied to the MAGMA model.
On language models, we found that "chat" fina--fffnnnedeeessoooo are more applicable to this method, most likely they are specifically trained to respect the given input. In general, prompt-tuning instructions may improve the results, such as "Answer with respect to the given context"..
A.11 Metrics
All recall and precision metrics in this work are based on the normalized continuous values as follows: For relevancy scores x and binary segmentation label y, let = a/ max, max; We compute AP (E: /. Σ, z; for precision and for recall AR (.r)) Σ. y4- The final mean scores
19

---

Target
AtMan
Chefer
GradCAM
Input Image
Monviy
1pod
Camera
Coflee
STABUU
STARBUCK
Bay
Dog
Figure 15: Weak image segmentation comparison of several images for all methods studied in this work. Note that IG and IxG are presented in token (left) and pixel-granularity (right). (Best viewed in color.)
are the average of all samples. This yields a comparable untuned metric also catching the robustness of the methods.
A.12 Interpretability of "complex facts" and failure modes
All experiments were executed in the generative setting, c.f. Eq. 1,2 and the remark to "global explanations" the end of Sec. 3 Fig. Ta shows "real text generation" and the accumulation of explanations on the input image during the model's generation process. For comparison benchmarks,
20

---

Question (Q) Target (T)
Context
This contract, dated the 2nd Nevember in corsitutes an employ agreement between these two partes and is govemed by the lave of the state Mach WHE AS the Employer desime reaan the services of the Emplaynn, and the Emproyes cesree to nensier such servose, these ferme condinan partics
1943,
betwoon Tomatiuuiiers Corp. and Ronaid Smith. document
set forth IN he she will faintuly and the best of their abilly to carry out the dudis and respons responsblillee biiles communcaated to them by the Eyployer. Employan shall comply with company polices, rules and procedures at limes. Position As jary derk, the dity of Empkyye
CONSIDERATION
following
Q: What is the pay per hour?
oyer. The
Employes's work Compensation As componsation for the services provided, lhee Emipllyce shall be paid wade S10 (por hou and will be subtect to a quarterly performance revlow. All payments shal be subtect to mancalory employment coductions (State & Federa Taxes, Socal right participale benefins will only be possible wher the
T: $10.00
employer currentyy Employer perod has
Access pessed Probationary Porlod is understodd that the frrt 2 months employment constitules probasonary poriod. During this lime, the
offers Uniimtted PTO. Medical and Dentil
Emoloyee eligill paid time other benofis.. During time, Employer also exer diseesss the right to torminate omp mmpllymmnnttoommm any time without advanced notice
This contraci, (attdd on the 2nd day November year 1883 is made botwoon Tomatouuicers Corp. and Ronaid Smith. This document constlurees an perpkymen men sgrenmmen leeen Frose two parties and governee by lave the state Michiann WHE AS Employer CONSIDERATION this mult nutual indersaanding the partiss agree to the following tarns and condltons Employment The Employes apreas that he or she will famuully and the bost of ther abilly to carry cut the dudes and respons
Q: What is the date on the contract? Please answer dd-mm- yy format
communccaion to them by the Employer. The duty also and other duties within the mavorable Employen of the
company dules, From Employas's s work Compens compensatinn As compensation for the services provided, the Employee shall be paid a wage 510 (por hou? and will be Sedurily Menican)) Benefit The Employon has the right particppate
jary
perform esseetial job fundions
the Employe
beneiii plans offered by the Employer. The cemplnyee currenty offers Unimitte PTO. Med cal and Dental insuranie. Access to these benoltis will only be possible after the proballonay poriod flss dersttod
T: 2-11-43
Probationary Pertod Employen olgible paid time other benefios. time without advanced notion.
2 months emoloyment consilutes probationary poriod
time.
time, Employer also acer the serminate mininall emplyymet any
This contract dated on the 2nd day of November in the year 1943, is made betweon Licatou Corp. and Ronaid Smith. This document consiluies an employment agree ment between these two parties and is govemedl by the laws the state at Michiann WHEREAS the Employrr desimes retain serviais of Employes, and the Empinyan cesires to render such servoce, these lerms condition understanding partiis agres to follawing
IN them Empooyea agrees times. Postion jury derk, the duty of Emplyyee
CONSIDERATION this Employee shall comply serform all essantial job functions and dules. From fime fmo, the Employer may also add other dulles within the reasonable scope of the Employee's a work Compenestion As compensttin for the services provided, the Emplyyee shall be paid a wage S10 (per hour) and will be subject to quarterly performance revlow. All payments shall be subrect to mancatory employment deductions (State Federal Taxes, Social amplsser curmently
and conditios: Emplyymmn respona
Q: Which country contract based upon ?
faitullly
best policies,
carry
Swcurtyy Macican)) Benniit The Emplyyen has the right to participate any benefis plans offered by Employer: probailonary period During time,
T: United States
Untim
Medical Dental Insurance. Access these beneths will only possible constiuute probationrry
the months Employeo elgible time other benefits. During timo, the Employer exeraiees the terminate employment
Probalionary
line without advanced notios.
This contract, dated on the 2nd day of November in the year 1943, is made betwaon Tomatouuicers Corp, and Ronaid Smith. This document consiitutes an amploymen spreament belweion these two partis and govemed by laws the Michigan WHEREAS Emplooyrr these
Employee, CONSIDERATION ION of thes my dual undensaaning ng. the partiss agmee to the follwing terms and condioon:: Emplymeen The Employee agrees that Employaa shall comply perform all essential job fundionn and dules. From time imm, the Employer may also add other dulles within the reasonable soope of the Emplooye s work Compensstion compencttion for services provded. the Employee shall paid wage S10 (oer hour) and will be subject
the
condrions
forth
Q: How many wheels does a bike have?
and the bast
sbilly carry policies,
dudios and respa billis commin unmmuuroated to them by Empoyer The Posliion
amployment deductions (Stats Securily Medlccano Benofts The Employes has the right to participate in any benefts plans offered by the Employer. The amployer currently offers Unlimted Plo, Medical and Dental insuranee Access to these benefiis will only be possible after the probationayy penao has empkoyment constiutte proberionary Employea is aligito for paid time off other benefis. During this sime, Emplyyer also oxer ciseess the right to sonminale ampleyment at time without advanced notice.
Two
Probationary
2
Figure 16: Showing ATMAN capabilities to highlight information in a document q/a setting. The model is prompted with "( ([Context) Q: ('Questton) A: " and asked to extract the answer (target) of the given Explanation. Here, ATMAN is run paragraph wise, described in text, and correctly highlights the ones containing the information. All Explanations were split into 50 paragraphs (thus requiring only 50 ATMAN forwad- passes). In particular is shown row that the model can interpret, i.c. convert date-time formats. Row 3 shows that it can derive from world knowledge that Michigian is in the US. Row 4 shows that the method ATMAN is robust against questions with non- includicli information. (Best viewed in color.)
21

---

Table 4: Performance of scaled MAGMA models.
6b
13b
30b
VQA
60.0
62.6
64.2
OKVQA
37.6
38.2
43.3
GQA
47.4
43.7
45.6
target was separate generated explanations from the underlying model's interpretation, by retrieving "obvious facts". Fig. demonstrates qualitatively target-class awareness.
More complex results are shown in Fig. 17 Fig. 17 gives an example where the model is VQA prompted and ATMAN explains both modalities at the same time for the completion token "white". highlights the words "tub" and "color" of the text, and an edge of the tub in the picture as significant to produce the completion. Note that we find these "joint-modal explanations" to work best when the target token is not just a yes/no answer, c.f. Fig. a'). Further note how these artifacts remain amongst explainability methods such as Chefer, which may be due to the underlying models architecture. We leave these investigations for further research.
In Fig. ITp. ATMAN highlights multiple heterogenous concepts to answer the counting task. Note that here the explanation for the completion "two", which actually highlights visually the animals, is found at the token position "animals", and not at the position of the predicted token. ATMAN returns at the same time explanations to all previous context positions, without additional forward passes (in contrast to tradient-based methods).
Finally, we ran evaluations on visual reasoning using the GQA dataset 17 Fig. 17 shows an instance of GQA, in which the model is prompted answer "Does the man ride horse?", while the man in the picture actually rides a bike. ATMAN highlights at two different sequence positions of the prompt the respective concepts in the image required to answer the question, once the man, and once the bike. Fig. 1 and d.2 are the resulting IG explanations when applied to explain the same token positions; somewhat it captures parts of the concept but focuses on noise the edge. Note that many samples of this dataset are rather difficult to answer, it is often not clear on what token position an explanation is supposed to be found. In particular, often ambiguous concepts occur multiple times in the image and are further restricted to the correct instances much later in the text description. This is, particular, difficult to grasp for generative decoder models due to the causal masking. As the dataset contains annotations for areas as in example Fig. 17c, we ran preliminary quantitative results on 10k randomly chosen instances. ATMAN matches bounding boxes with m.P 52.4 and mAR 24.3. For IG we obtained mAP 33.3 and mAR 27.0.
A.13 Partial attention manipulation
We apply the same manipulation in each layer. We hypothesised -and first experiments indicate- that one needs "critical mass", remove the entire concept entropy from the generation process. This is i.p. due to the skip connections. Only applying it to single (arbitrary) layer did not yield comparable results. However we did not finalize conclusion yet and leave this to future work.
A.I5 Fig. [8
A.14
MAGMA scaling
MAGMA Model performances on standard text-image evaluation benchmarks are shown in Tab.
A.15
Variants of ATMAN and final remarks
There are many promising variants of ATMAN not yet fully understood. In particular finding the relevant layers and other formulae to manipulate reliably the activations, or to normalize the output scores, could yield further insights into the transformer architecture. We showed how conceptual suppression improves overall performance. For pure NLP tasks, utilizing an embedding model such as BERT to compute similarities could be beneficial. Moreover, should be possible to aggregate tokens and increase the level of details as needed, to save additional computational time.
22

---

a) Whet the of the tub?
White
b) Q: How many animals are this picture? A: two.
d.1)
c.1) Q: Does the
c.2) Q: Does the ride horse? A: No.
d.2)
a!1)
a!22) Chefer
AtMan
Is tub
yes
Is the bath tub white in color?
A: yes
white color?
Figure 17: More complex examples showing generative modal explanations (c.f. Sec. A.12 for detailed description). italic: model completion; underline: token position for explanation. (Best viewed in color.)
TD
mup extan
Dayer
Figure 18: Measured mAP and mAR when applying AtMan only single layer, with similarity of the first embedding layer. Evaluated subset of openimages. Apparently, layers contribute more the explanation than others. "Full AtMan" as proposed yet outperforms on both metrics with scores mAP 0.66 mAR = 0.24.
Fig. 18 shows scores when applying ATMAN only to a single layer. Lp. the second half of layers seem more influential to the ATMAN scores, which is in accordance with recent research studying that higher-level concepts form later throughout the network.
In Fig. 19 we applied softmax on the embeddings (still only of the first layer) instead of the proposed thresholded cosine similarity Eq. 6 The proposed cosine similarity clearly outperforms its softmax variant in quality, however visible correlations can be found in the softmax-image as well.
As final remark, we want to highlight again the rather undefined question of "what an explanation is" Afterall, we show what input leads diverging generation of the model faithfully, even if would not match the expected outcome.
23

---

Figure 19: Qualitative example for an explanation of AtMan as proposed (middle) and AtMan when applying the softmax the embedding instead of cosine similarity (right), as suggested by reviewer. While some correlation can be found, it much more noisy.
24


Reviews:
[{"review_id": "hHEjdCxET9", "reviewer": "Reviewer_HHf8", "summary": "This paper proposes atman, a method for understanding the predictions of transformers by masking out the tokenized inputs. The paper evaluates atman on explainability tasks in text pipelines and vision pipelines. The authors argue that atman is superior to gradient-based explanation methods since it uses less memory.", "questions": "Suggested list of questions to improve the work:\n1. What other \"surprisingly simple\" interventions can be used to explain predictions?\n2. Can you look at embeddings of tokens in the final output?\n3. What are the differences (in NLP) between generative-models (BERT) and encoder-only models (GPT)?\n4. How are the explanations for vision affected when the correlations are not taken into account?\n5. What happens if this intervention is done partway through the backbone instead of at the beginning?", "limitations": "N/a", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is well-written and the contributions are easily understood. The method is simple to implement and understand, and does take less memory footprint than gradient-based methods. The evaluation on explainability benchmarks seem to show that it behaves at par with gradient-based methods. The figures make the method very easy to understand.", "weaknesses": "The novelty of the method seems questionable, and it doesn't seem necessary to create an entire \"method\" with a name to describe the observation that masking out input tokens can be useful in tracing which parts of the input sequence are important. This type of analysis is standard in analyzing the outputs of Transformers, e.g. it is a standard technique in Olsson et al 2022 (https://arxiv.org/abs/2209.11895). To claim that it is a novel method is a bit much in terms of framing.\n\nThere may be some novelty in the vision domain in looking at the correlated outputs instead of masking individual patches of the input image, but as far as I can tell this methodological change is not ablated.\n\nThese novelty questions point to a weakness in the evaluation and the discussion of related work. I don't have any problem with a simple method (in fact I like it) - but if the method is a first standard trick that most ML engineers would use to understand what is happening when a Transformer makes predictions, then it suggests a different framing and evaluation.\n\nFor example: What other \"surprisingly simple\" interventions can be used to explain predictions? Can you look at embeddings of tokens in the final output? What are the differences (in NLP) between generative-models (BERT) and encoder-only models (GPT)? How are the explanations for vision affected when the correlations are not taken into account? What happens if this intervention is done partway through the backbone instead of at the beginning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nthank you for the feedback, however, we kindly disagree with your novelty assessment of our proposed XAI method. First and foremost, our work studies, and compares to, state-of-the-art interpretability research in the field of transformers and neural networks, correlating input token importance to the generated output tokens of the otherwise black-box transformer model. While it is true that masking out input tokens has been explored in the context of transformer analysis to some extent, we do not merely mask tokens. We study ---for the first time--- the adjusted generative behavior of decoder models when amplifying or attenuating (ref Fig 2b) the query and key scores consequently throughout all transformer layers (not only in a single layer). We moreover present large benchmark evaluations for explanations on two different modalities, outperforming current gradient-based state-of-the-art methods. Our contribution lies in providing a structured and systematic approach to leverage this observation for solving a specific problem (XAI), which we believe is novel. Specifically, we introduce attention manipulation as a perturbation technique for XAI and derive it as a form of influence functions and carefully devise its efficient application to autoregressive transformer models. \n\nFig 10 of the appendix section and the attached Fig.17 (‘mask’ row) demonstrate quantitatively, that “simple” interventions would lead to degenerated performance when comparing changes in the logit distribution as proposed in the submission. Note that we already gave qualitative and quantitive evidence to this with Fig. 3a of the main paper and Fig.10 and 11 in the appendix. We have not come across any previous work that specifically addresses the exact problem and approach described in our submission. Therefore, we would appreciate references to relevant publications, even to ‘best ML engineer practices’. Please also consider the global response to this rebuttal.\n\nWe addressed the issue rendering state-of-the-art perturbation methods impractical, perturbing in a large discrete token-space, by shifting this sto a continuous factor value space. \n\nWith regard to the reference of Olsson et al, the induction-heads proposal indeed is an interesting line of work (even though it seems unpublished to this date). Note that we do not claim to have an explanation for the behavior or responsibilities of generic architecture components, in particular, their development during training. We empirically show and evaluate rigorous consistent behavior when altering attention scores amongst all layers. Therefrom we derive an explainability method that reveals insights into the black-box correlation of input to output tokens. It also gives some insights into the processing behavior of input tokens throughout the layers of a transformer network. Recent work such as [0], highlight the relevance and novelty of investigating influence functions for transformer interpretability. \n\n[0] Grosse et al. 2023, ‘Studying Large Language Model Generalization with Influence Functions’\n\nQ1\nWe again would kindly disagree that AtMan relies on “simple” interventions as we argue above.\n\nQ2\nThat is indeed an interesting question that unfortunately is out of scope for this work. In the visual domain, it is natural to ‘look at’ the cosine similarity directly after the embedding layer, in particular as we use CLIP components (c.f. line 199) that are trained with a contrastive loss. We empirically show how it can be utilized to ‘sufficiently suppress’ the input entropy of some concepts, and that measuring those remains stable throughout a large benchmark set.\nIn particular, in decoder models, the embeddings of the last layer are not as simple to compare. We assume, as your next question suggests, that you refer to running an embedding model (like BERT) and use those embeds to compute the cosine similarity on the inputs. This indeed would work, and tailor the application to a specific focus (the one BERT is trained with). However one would require an additional inference on a different model to produce those embeddings on the input chunks. In the scope of our work we introduce a XAI method for autoregressive models such as GPT which don’t provide a final output embedding (cf Q3).\n\nQ3\nBERT is an encoder by design, without the causal mask (cf line 165), thus tokens can attend to all other tokens in the attention mechanics. It is usually not applied in practice as a text-generator, but trained with a contrastive loss to generate comparable embeddings. GPT, in contrast, is generative, applying the causal mask, thus tokens can only attend to preceding tokens. This architecture has shown outstanding text-generation capabilities, but due to the masking, the altered meanings of the inner workings remain widely unclear. The resulting embeddings are not as easy to digest.\n\nQ4\nWe addressed this quantitatively with attached Figure 17 (‘mask’ row). In the main paper we gave qualitative and quantitative evidence with Fig 3a, Fig 10 and 11. Precision and Recall both severely degenerate.\n\nQ5\nOur intervention receives cosine similarity measures from the first (embedding) layer, and rigorously applies manipulations throughout the network. It is not possible to apply cosine similarity on intermediate layer embeddings out of the box, most likely due to the causal mask. However, we used the initial similarity values and applied the manipulations only in one layer at a time. In the attached Figure 18 we observed an interesting variance, but overall degenerative performance.\n\nWe hope we clarified our contribution and resolved your concerns. If you have further questions, please let us know.\n\n\nThe authors.\n"}, {"review_id": "WzzwPp1SY8", "reviewer": "Reviewer_YuQG", "summary": "This paper proposed a new permutation method for Transformer explanation, named ATMAN, which directly manipulates attention scores in the hidden layers of the Transformer model rather than permuting the model inputs. Importantly, the authors established a bridge between ATMAN and the relevance between inputs and output predictions. Experimental results demonstrate the effectiveness and efficiency of ATMAN on interpreting Transformer model on different types of data and different model scales.", "questions": "NA.", "limitations": "NA.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The proposed ATMAN method is novel, and the experimental results shows that ATMAN significantly improve the effectiveness and efficiency over existing perturbation and propagation methods on interpreting Transformer model on different types of data and different model scales.", "weaknesses": "NA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nwe want to express our gratitude for taking the time to review our paper and the appreciation of our work. \nIf you have further questions let us know.\n\nThank you,\nthe authors.\n"}, {"review_id": "4cVOUHqdgq", "reviewer": "Reviewer_sfjw", "summary": "This paper proposed ATMAN, a modality and architecture-agnostic perturbation-based XAI method for generative transformer models. Specifically, ATMAN evaluates the sensitivity of model prediction w.r.t. the perturbation of attention scores on a set of cosine-distance neighboring tokens. Empirical results suggest that ATMAN can effectively identify the relevant input tokens for target prediction, and outperforms the baseline methods in terms of precision and recall. ", "questions": "Can you provide more model details about Section 3.1? Which black-box model do you evaluate? ", "limitations": "The limitation is the high runtime cost when evaluating every input token/pixel. It would be great if the authors can extend the analysis into higher granularity. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "As far as I know, this paper is the first work that identifies the crucial tokens/pixels by attention perturbation. The idea is novel and looks interesting to me. The authors further apply the cosine similarity to collect the related neighboring tokens and reduce the searching redundancy. It helps us to understand how transformer models process input signals and identify crucial ones. ", "weaknesses": "1. As perturbing every token to evaluate their sensitivity, ATMAN can incur high time complexity to generate the explanation for transformer models. As suggested by Figure 5, the runtime is much higher than the baseline. Therefore, to reduce the time complexity, would there be any solution to evaluate the token/pixel sensitivity in a higher granularity, like dividing the tokens/pixels into clusters? \n2. The paper mentioned steering model prediction by applying the attention scores of identified tokens but I did not find any empirical results to claim the advantages of this approach. Could you provide more details and results or correct me if I am wrong? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nthank you for your valuable feedback and the invested time reviewing our work. \n\nW1 (inference time)\n\nAs discussed in the global review, we want to highlight once again, that this is the first XAI method deployable at all for large transformer networks, without additional allocation on deployment resources. For instance, models exceeding 9B parameters are not deployable on a single 80GB GPU using current methods such as Chefer et al. To emphasize this more throughout our paper and prevent further misunderstanding we updated the main text and Figure 5. Please see attached Fig 5 in the supplemented pdf. Furthermore, we now illustrate the computation time using AtMan and previous methods on a simple deployment layout, see Fig 16. To explain an image, it can be observed that AtMan takes about 1 magnitude longer in total processing time, but can still be processed in seconds of total compute with pipe-parallel 4 and batch-size 16. Note that the forward passes are entirely independent, and thus the processing time can further be pushed to subseconds, with more idle workers. When chunking input to paragraphs as described with Appendix section A8, Figure 14, moderately larger prompts with around 500 tokens and 50 ‘explanation passes’ remain around 10 computation seconds (that can be executed in parallel).\nFurther note that our perturbation is executed in the transformers hidden space. In generative models such as the here studied MAGMA or BLIP, is already clustered into ‘tokens’ (c.f. line 107). In case of MAGMA, 144 ‘image tokens’ are perturbed, in contrast to the large pixel space. Further aggregation such as adopting ‘token merging’ or exploiting the symmetry of cosine similarity should be possible. We will clarify this further in the main paper.\n\nW2 (steering)\n\nIt is unclear what comparison in advantages you’d like to see. Throughout the paper, we always apply the attention score modification to ‘steer’ the generative process and derive the XAI method from the distribution shift in token generations. Table 1 empirically shows how we outperform other state-of-the-art methods in performance.\n\nIf you meant a comparison to other available perturbation methods such as SHAP, perturbing directly on the input space is resulting in an uncomputable number of trials (c.f. line 540). \nIf you meant a comparison to entirely masking out tokens with AtMan (setting $f=1$), we’d like to point to Fig 10 and the attached Fig 17. The top row ‘mask’ refers to ‘entirely masking’ single tokens, throughout the network, not applying the cosine similarity. It can clearly be observed that simply masking single tokens leads to degenerative performance. We assume this arises from the destructive influence on the trained input structure. Our method on the other hand applies a more subtle manipulation that aims to downgrade ‘a concept entirely’ found in the input. We will include this discussion in the paper. We hope this addressed your concern. Let us know if open questions remain.\n\nQuestion\n\nWe do not clearly understand what you are referring to with ‘black-box’ in section 3.1. \n\nIf you meant section 4.1, we indeed missed to clearly describe the model under investigation. Thank you for pointing this out. That would be plain gpt-j, the underlying opensource model MAGMA was trained upon. We will update the manuscript accordingly.\n\nIf however you really meant section 3.1, assumably you refer to the title of the cited paper of ‘understanding black-box predictions via influence functions’. For us, there is no black-box (other than the garbled hard-to-trace computations executed). The model (MAGMA) is, in fact, white-box, which we exploit as follows:\nWe use Influence functions to give a  measurement based on robust statistics for our derived XAI method, which in fact is the visual heatmap of 'a point’s influence'. We regard transformers to be distribution estimators and aligning with the work of [13], define an influence function (Eq 3), that approximates the input-perturbations influence into changes of the model parameter space, ‘as if the model would not remember the to-probe concept’, and trace the distribution loss change. Here we raised the somewhat natural hypothesis, that the query and key scores are the relevant parameters in observing or altering those input changes of ‘point masses’. Our exhaustive empirical evaluation indeed reinforces this argument. We will clarify this in the main paper.\n\n\n\nThank you for your very valuable feedback that contributed to improving this submission.\nThe authors.\n"}, {"review_id": "Fp31uRo7ca", "reviewer": "Reviewer_FKKY", "summary": "The authors proposed a new resource-friendly modality-agnostic perturbation method named AtMan to explain the predictions of generative transformers. AtMan suppresses or amplifies the attention entry relying on the cosine similarity neighborhood to evaluate the influence of each token, and it saves many memory resources compared with current gradient-based methods, which means that it's more suitable for large model inference developments. Empirical results show that AtMan can do language/visual reasoning better than the current state-of-the-art gradient-based methods.", "questions": "Similar to Weakness-1 and Weakness-2.\n", "limitations": "The author mentions that their work provides reference methods for future work, including explanatory studies of current generative models impacting our society. The author doesn't mention other potential negative societal impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. AtMan is simple but useful, and the idea that changing perturbation space from the raw input to the embedding space is important. It can perform on decoder as well as encoder architectures and the empirical results are supportive.\n2. The core contribution of AtMan is that it reduce the memory cost and can be applied to the large models, which will be important for the future works of explainability of large models. Now it can be applied to 30B multi-modal model but previous methods fail.\n3. The paper is well written and easy to follow.", "weaknesses": "1. The paper mentions that the $\\kappa$ parameter in AtMan is set to be $0.7$, but it may be important for the performance of AtMan, so it is better to do additional experiments to show the effect of different kappa on the experimental results on different datasets. Furthermore, the choice of (6) is not unique, there are some other ways like using the softmax value of $s_{i,k}$ to be $f_{k,*}^i$ and using temperature as a hyper-parameter. It's better to compare these ways.\n2. The increase in the time cost of AtMan is significant compared with gradient-based methods, so it's better to show the time-cost figure after pipeline-parallel execution.\n3. The results of the locality are important, maybe the authors can try to add them (like figure 10 in Appendix) into the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nthank you for your valuable and very inspiring feedback. \n\n\nW1 (parameters $f$ and $\\kappa$) \n\nWe apologize for not showing that figure in the appendix already. Please find attached Fig. 17 for the influence of $f$ and $\\kappa$ on the openimages benchmark. Note that indeed utilizing the cosine similarity improves upon ‘plain masking of single tokens’, as shown in the ‘mask’ row. We can further run and attach this ablation on the GQA and Squad dataset and BLIP outside the rebuttal period. \nThank you for the suggestion of iterating on Eq 6. It is indeed a very interesting idea you propose, however, it is not a subtle change. In Fig 2b. we show how the generation process is not degenerated, but rather ‘steered’. The range of the softmax values is usually entirely off and rather incomparable among different samples, as it is normalized to sum up to one. It, therefore, is hard to fix a single temperature to a large benchmark dataset. We tried some samples that however did not work as well (attached Fig 19). Normalizing the softmax by dividing by its max value is almost like returning to the original cosine similarity, so we are not sure how to proceed here. \nWe want to foster exactly those discussions by making the source code publicly available and inviting collaborations.\n\nW2 (inference speed) \n\nThank you for this remark, in the main paper we tried to do the fairest comparison possible, as this plot highly depends on the actually available resources. Nevertheless, we run through reasonable assumptions on those and added exactly this point with attached Fig 16; showing that indeed this algorithm is very well computable and not that far off in magnitudes.\n\nW3 (locality) \n\nThank you for this suggestion. We agree and will add it to the main paper.\n\n\nLimitation) \n\nWe derive an XAI method correlating input to output tokens for the otherwise black-box generations. These first levels of explanations pave the way for exploring e.g. underlying trained, potentially harming, biases when using machine-learned processes.\n\nThe underlying process of AtMan includes suppression (or facilitation) of concepts found in the input, to steer the models generation. Indeed the steerable nature may as well be used to also correct those findings, or be abused to increase them. We agree that this should be further discussed as potential negative societal impact and will add this concern to the discussion.\n\nAgain thank you for your valuable feedback,\nthe authors\n"}, {"review_id": "lbQ2p6YiI8", "reviewer": "Reviewer_wnDz", "summary": "The paper introduces AtMan, a method that manipulates the attention mechanism of generative transformer models, to provide explanations. More specifically, it perturbs raw attention scores instead of the token inputs. The authors provide extensive experiments for different modalities to back their claims regarding better results to established baselines.", "questions": "1. Line 62 states that the source code is released although I could not find it in the supplementary.\n2. Line 33-34 states that memory consumption leads to uneconomical productive deployment. This is a overstatement, especially given the computational requirements of the new proposed method. \n3. A lot of literature has been exploring the effect of attention scores in the context of Stable Diffusion, e.g. [2] and follow-ups.\n4. Is there a minus missing before $L^{\\text{target}}$ in Eq. (2)?\n5. Line 245, it is not clear to me how the mean of generated scores affects the final predictions. Are some tokens in the target label more important than others as expected? \n6. Is there a way to prioritize checking some tokens first instead of exhaustively trying out all tokens to find the ones that explain better the target? This might help to reduce the inference speed. You can think about having a method to rank tokens to check, and then create a plot of computational budget vs performance, where you check iteratively more tokens. You could also think of doing this in a more hierarchical way, similar to what is proposed in Appendix A8. \n\n[2] Hertz, Amir, et al. \"Prompt-to-prompt image editing with cross attention control.\" arXiv preprint arXiv:2208.01626 (2022).", "limitations": "The paper proposes a way to manipulate attention scores to explain prediction of Transformer models. Manipulating these scores is done a token at a time, leading to inference speed that scales with the sequence length, and is orders of magnitude bigger than previous approaches. Although a lot of experiments are presented, some of the design choices are not adequately justified. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper introduces a new approach towards Explainable AI, that does not require calculation of gradients through the network, thus enabling evaluation of larger models with the same hardware.\n2. The idea of manipulating attention scores is novel and interesting.\n3. The authors embark on an extensive evaluation of their method on different modalities. The universality of the Transformer architecture, enabled them to seamlessly switch between modalities. The methods performs well on a range of architectures (encoder, decoder). ", "weaknesses": "1. One of the main motivation of this work is to enable more widespread adoption of explainable AI methods, via reducing required computational resources. The issue of memory is discussed extensively. Inference speed is only mildly explored, in Figure 5. For large sequence lengths, the method requires upwards of 2 orders of magnitude more time to make the final predictions.\n2. The method requires one to set new hyperpatameres, namely $f$ and $\\kappa$. The effect of those with respect to e.g. sequence length or model architecture, are not discussed. For example, different sequence lengths may lead to different softmax entropy and thus different $f$ values might be required. \n3. According to Eq. (4), tokens are masked by setting the corresponding $H$ values to 0. This does by no means lead to masking of the tokens, as briefly explained in Appendix A5. Setting the values to the value $0$ constitutes another hyperparameter, that needs to be set for the proposed method. In principle any real value could have been chosen (In Appendix A5 the authors explore the values $0$ and $-\\infty$). The range of the possible values was not explored at all. \n4. In general a lot of approximations are made for the derivations of the last method, with possible alternatives not discussed. Why was Eq. (4) chosen to mask tokens? Why is the cosine similarity of the embeddings a good measure to suppress common tokens? In [1] for example, the cosine similarity between keys is used as a more suitable measure. The cosine similarity is calculated based on the embeddings of what layer? Is there a benefit from calculating this similarity on different layers? There is a lot of work on what type of information token embeddings carry for different layers in a Vision Transformer. \n\n[1] Bolya, Daniel, et al. \"Token merging: Your vit but faster.\" arXiv preprint arXiv:2210.09461 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, \n\nthank you for the invested time reviewing our work and your valuable feedback.\n\nW1 (inference speed)\n\nAs discussed in the global review, we want to highlight once again, that this is the first XAI method deployable at all for large transformer networks, without additional allocation on deployment resources. For instance, models exceeding 9B parameters are not deployable on a single 80GB GPU using current methods such as Chefer et al. To emphasize this more throughout our paper and prevent further misunderstanding we updated the main text and Figure 5. Please see attached Fig 5 in the supplemented pdf. Furthermore, we now illustrate the computation time using AtMan and previous methods on a simple deployment layout, see Fig 16. To explain an image, it can be observed that AtMan takes about 1 magnitude longer in total processing time, but can still be processed in seconds of total compute with pipe-parallel 4 and batch-size 16. Note that the forward passes are entirely independent, and thus the processing time can further be pushed to subseconds, with more idle workers. When chunking input to paragraphs as described in Appendix section A8, Figure 14, moderately larger prompts with around 500 tokens and 50 ‘explanation passes’ remain around 10 computation seconds (that can be executed in parallel).\n\nW2 (factors $f$ and $\\kappa$)\n\nThank you for the good question. Indeed $f$ and $\\kappa$ depend on model architecture and the type of dataset. To determine these factors we ran the following search on openimages, cf attached Fig. 17. As it can be observed, $f=0.9$ and $\\kappa=0.7$ leads to the best values in mean precision and recall. We used the same factors for the tasks VQA and GQA (A12 Fig15), even though the style of prompts largely varies. We will add a discussion on this limitation and how to find fitting parameters to the main text.\n\nRegarding sequence length, computing the precision and recall as described in A11 is invariant against a shift in e.g. magnitude of the actual values of the influence function --- the explanation becomes the (normalized) heatmap. \nWe gave qualitative examples on large prompt lengths (500+ tokens) with A8 Fig14. As it can be observed, along with (varying sequence lengths) in the Squad benchmark, the results remain stable. We will clarify this further in the submission by extending Fig 17 to BLIP and Squad.\n\nW3 (masking tokens) \n\nThank you for this valuable feedback. However, there seems to be a misconception here. We never mask out tokens entirely by setting them to 0 (or -$\\infty$). To provide further clarifications for the readers, we additionally show that masking tokens entirely would even yield degenerated performance as shown in the top row ‘mask’ of the Fig 17 (c.f. W2).\n\nW4 (questions regarding Eq.4) \n\nThank you for this remark. We will add further discussions on approximations and alternatives to the main paper. In particular, we will be more precise regarding your following questions:\n\na) Why Eq. (4)?\n\n   It is the most generic equation possible, i.p. as we want to apply cosine similarity afterward. The function of ‘the causal mask’ in fact is ‘to apply suppression’ already, such that attention cannot attend to ‘future tokens in the sequence’. It is therefore a reasonable position to apply perturbation. \n \nb) (cosine similarity)\n\n   We compute the cosine similarity directly after the embedding layer. The vision encoder of MAGMA is a pre-trained CLIP-like model (c.f. line 133) that is trained on a contrastive loss and thus expected to already produce comparable embeddings (and our results reinforce). They are consistently applied throughout the network.\n   Improving the similarity matching we proposed is indeed an interesting line of research. We will add this to the discussion section. Note that we study decoder models contrary to encoders like ViT. The applied causal mask most likely destroys the similarity of ‘future’ tokens required, when calculating similarity in middle layers. Initial experiments did not succeed, but will be discussed.\n\nQ1 (source code) \n\nIndeed, we just sent the code to the AC, we hope they will forward them to you soon.\n\nQ2 (runtime) \n\nIn contrary to previous gradient-based methods, AtMan is indeed practically performant. Please cf. W1.\n\nQ3 (stable diffusion) \n\nThank you for the references, we will add Hertz, Amir, et al. and further references to a dedicated related work section on the effect of manipulating attention scores.\n\nQ4 (Eq 2) \n\nMathematically the sign is not necessary, however, we will include it for the sake of readability.\n\nQ5 (mean approximation)\n\nWe used the mean as we assumed for simplification standard Gaussian distribution and tried to model the expectation value. Indeed we will further investigate and discuss it. Due to the generative token-wise process, surely the first token of each ‘concept word’ carries the most entropy.\n\nQ6 (token reduction) \n\nIndeed ways to prioritize tokens could lead to performance improvements. However, we would like to emphasize again that AtMan’s inference speed is not impractical, c.f. W1. \nToken aggregation is indeed possible, c.f. appendix section A8. Exploring this further would be interesting follow-up work. In particular, exploiting symmetric cosine similarity or adopting ‘token merging’ seems promising.\n\nLimitation \n\nWhile we agree that the current implementation of AtMan leaves improvements for inference speed, we would like to highlight again that inference speed is not orders of magnitude bigger than previous approaches. In fact, AtMan is the first deployable XAI method not requiring further infrastructure costs, as it only relies on normal forward-pass workers cf. global review. \nWe will clarify the discussion in the paper and in particular add Fig 16 showing practical real-time performance of average prompts. \nThe discussed design choices and parameter justifications will be added to the paper.\n\nThank you again, \nthe authors.\n"}]

Author rebuttal:
Dear All,

thank you for taking the time to review our paper and the valuable feedback. We appreciate the insights and suggestions, which have inspired us to refine and improve our work. In response, we herewith aim to clarify some major concerns that arose among several reviewers

First, we'd like to emphasize that our proposed method is the first XAI technique specifically suited and evaluated for large generative models, that is moreover fit for productive deployment. This key message seems to be misunderstood at times. In particular, to deploy AtMan, it does not require additional workers specifically waiting for interpretability requests, as it would be the case for other methods. This is due to the fact that AtMan only relies on plain forward passes, while gradient-based methods would require a different compiled execution graph, and a different layout among GPUs to ensure enough free memory for the backward pass overhead, optimizer state, and such. Yes, we achieve this by trading memory consumption for run time. However, previous gradient-based methods such as Chefer et al. would not have been deployable at the same hardware cost. Furthermore they require additional workers specifically for XAI requests. To prevent further misunderstandings we updated Figure 5 (see attached Fig. 5).
Efficient deployable XAI methods for generative transformers are especially relevant considering recent stipulations of the EU AI Act, i.e. Article 13, stating that ‘users should be able to “interpret” the system’ and that it ‘requires human oversight measures that can facilitate the “interpretation” of its outputs’.


However, we agree that demonstrating a realistic scenario with the computational cost of AtMan would further clarify the almost negligible additional run time costs.  The attached Fig. 16 compares a practical query (average 165 tokens) explaining an image (144) tokens. Runtimes are measured with parallel execution (pipe-parallel 4, batch size 16). This demonstrates that AtMan is absolutely practical and usable. Specifically, AtMan requires between 1 and 3 total compute seconds, around 1 magnitude longer compared to Chefer. Note that this can still further be divided by the number of available idling workers to further reduce absolute computation time. Each batch of AtMan can be processed entirely parallel.
Further, note that existing perturbation methods are simply not computable at all on large transformer networks due to their large number of trials. Moreover Chefer, even with memory optimizations, fails to scale to 34b with the given memory limit (80GB).


Please further note:

1) We have anonymized the code related to our paper and submitted it to the AC, who will handle further processing and forward it to you.

2) We have incorporated your feedback into the paper to the best of our understanding, enhancing the clarity and improving the overall quality of the submission.

3) We attached Figure 17 demonstrating an exhaustive search on $f$ and $\kappa$ as described in the paper. It can indeed be observed that our proposed subtle modifications generate better explanations than e.g. entirely masking out tokens.


We believe this submission contains enough novel material to better understand the inner workings of transformers, as well as encourages further research, as the reviewers also rightly pointed out plenty-wise, in this understaffed segment of XAI methods on transformers.


Again thank you so much for this fruitful discussion, and kindly find the Figures attached to this post,

the authors of AtMan.





Meta-review: This paper proposes a perturbation method for explaining the prediction of generative Transformer model. The proposed scheme is to manipulate the attention mechanism of the Transformer, so that relevance map of the input can be obtained. The proposed method is memory friendly. Experiments on text and image-text benchmarks demonstrate that the proposed method outperforms gradient-based methods.

The paper has been reviewed by five reviewers. Overall the reviews are positive, except one reviewer HHf8, who gave a rating 3. This reviewer is mainly concerned with novelty. The authors provided a detailed reply on this issue. Reviewer wnDz, who gave a rating 5, is concerned with the inference speed and the choice of tuning parameters. The authors addressed these concerns reasonably. There are other concerns that were not very serious, and were addressed by the rebuttal and discussion.

Overall the proposed method is a useful contribution to explainable AI.
Decision: Accept (poster)