Title: ATMAN: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation
Venue: NeurIPS 2023
Accepted: yes
Keywords: explainability

attention manipulation

perturbation

large language model

multi-modality

generative decoder

efficiency

transformer
Abstract: Generative transformer models have become increasingly complex, with large numbers of parameters and the ability to process multiple input modalities. Current methods for explaining their predictions are resource-intensive. Most crucially, they require prohibitively large amounts of additional memory, since they rely on backpropagation which allocates almost twice as much GPU memory as the forward pass. 
This makes it difficult, if not impossible, to use explanations in production. 
We present AtMan that provides explanations of generative transformer models at almost no extra cost. Specifically, AtMan is a modality-agnostic perturbation method that manipulates the attention mechanisms of transformers to produce relevance maps for the input with respect to the output prediction. Instead of using backpropagation, AtMan applies a parallelizable token-based search method relying on cosine similarity neighborhood in the embedding space. 
Our exhaustive experiments on text and image-text benchmarks demonstrate that AtMan outperforms current state-of-the-art gradient-based methods on several metrics while being computationally efficient. As such, AtMan is suitable for use in large model inference deployments.

Abstract

Generative transformer models have become increasingly complex, with large numbers of parameters and the ability to process multiple input modalities. Current methods for explaining their predictions are resource-intensive. Most crucially, they require prohibitively large amounts of additional memory since they rely on backpropagation which allocates almost twice as much GPU memory as the forward pass. This renders it difficult, if not impossible, to use explanations in production. We present ATMAN that provides explanations of generative transformer models at almost no extra cost. Specifically, ATMAN is a modality-agnostic perturbation method that manipulates the attention mechanisms of transformers to produce relevance maps for the input with respect to the output prediction. Instead of using backpropagation, ATMAN applies a parallelizable token-based search method relying on cosine similarity neighborhood in the embedding space. Our exhaustive experiments on text and image-text benchmarks demonstrate that ATMAN outperforms current state-of-the-art gradient-based methods on several metrics and models while being computationally efficient. As such, ATMAN is suitable for use in large model inference deployments.

1 Explainability through attention maps

Generalizing beyond single-task solutions using large-scale transformer-based language models has gained increasing attention from the community. In particular, the switch to open-vocabulary predictions promises AI systems capable of generalizing to new tasks. Arguably, transformers are the state-of-the-art method in Natural Language Processing (NLP) and Computer Vision. They have shown exceptional performance in multi-modal scenarios, e.g., bridging Computer Vision (CV) capabilities with text understanding to solve Visual Question Answering (VQA) scenarios [10, 18, 31, 30]. However, the increasing adoption of transformers also raises the need to understand their otherwise black-box predictions better. Unfortunately, the 'scale is all you need' assumption of recent transformer models results in severely large and complex architectures. This renders tasks such as their training, inference deployment, and explaining of their outputs resource-intensive, requiring multiple enterprise-grade GPUs or even entire computing nodes, along with prolonged runtimes.

This is a painting of

a lonely cabin

on the edge of a lake

with a truck nearby

Multimodal prompt

and the geese ...

Completion and AtMan Expl.

(a) 'What am I looking at?'

Token Embeddings

Transformer Blocks

Mask

Scale

MatMul

MatMul

Q

K

V

o

Modifier

Factors

Attention

Scores

Attention Scores

Output Logits

(b) ATMAN in the transformer architecture.

Explainable AI (XAI) methods target to elucidate the decision-making processes and internal workings of AI models to humans. State-of-the-art methods for transformers focus on propagating gradients back through the model. This backpropagation allows for the accumulation of influence of each input feature on the output by utilizing stored activations during the forward pass [6, 27]. Unfortunately, this caching also leads to significant overhead in memory consumption, which renders their productive deployment to be uneconomical, if not impossible. Often half of the available memory of the GPU has to remain unused at inference, or it requires an entirely separate deployment of the XAI pipeline.

Fortunately, another popular XAI idea, namely perturbation [19, 24], is much more memory-efficient. However, it has not yet been proven beneficial for explaining the predictions of transformers, as the immense number of necessary forward trials to derive the explanation accumulate impractical computation time.

To tackle these issues and in turn, scale explanations with the size of transformers, we propose to bridge relevance propagation and perturbations. In contrast to existing perturbation methodsexecuting perturbations directly in the input space-ATMAN pushes the perturbation steps via at tention man ipulations throughout the latent layers of the transformer during the forward pass. This enables us-as we will show-to produce state interpolations, token-based similarity measures and to accurately steer the model predictions. Our explanation method leverages these predictions to compute relevancy values for transformer networks. Our experimental results demonstrate that ATMAN significantly reduces the number of required perturbations, making them applicable at deployment time, and does not require additional memory compared to backpropagation methods. In short, ATMAN can scale with transformers. Our exhaustive experiments on the text and image-text benchmarks and models demonstrate that ATMAN outperforms current state-of-the-art gradient-based methods while being more computationally efficient. For the first time, ATMAN allows one to study generative model predictions as visualized in Fig. 1a without extra deployment costs. During the sequence generation with large multi-modal models, ATMAN is able to additionally highlight relevant features wrt. the input providing novel insights on the generation process.

Contributions. In summary, our contributions are: (i) An examination of the effects of token-based attention score manipulation on generative transformer models. (ii) The introduction of a novel and memory-efficient XAI perturbation method for large-scale transformer models, called ATMAN, which reduces the number of required iterations to a computable amount by correlating tokens in the embedding space. (iii) Exhaustive multi-modal evaluations of XAI methods on several text and image-text benchmarks and autoregressive (AR) transformers.

Weproceed as follows. We start by discussing related work in Sec. 2. In Sec. 3, we derive ATMAN and explain its attention manipulation as a perturbation technique. Before concluding and discussing the benefits as well as limitations in Sec. 5, we touch upon our experimental evaluation Sec. 4,

showing that ATMAN not only nullifies memory overhead and proves robust to other models but also outperforms competitors on several visual and textual reasoning benchmarks.

2 Related Work

Explainability in CV and NLP. The explainability of AI systems is still ambiguously defined [8]. XAI methods are expected to show some level of relevance to the input with respect to the computed result of an algorithm. This task is usually tackled by constructing an input relevance map given the model's prediction. The nature of relevance can be class-specific, e.g., depending on specific target instances of a task and showing a 'local solution' [26, 27], or class-agnostic, i.e., depending on the overall 'global behavior' of the model only [1, 4]. The level of fine granularity in the achieved explanation thus depends on the selected method, model, and evaluation benchmark. Explainability in CV is usually achieved by mapping the relevance maps to a pixel level and treating the evaluation as a weak segmentation task [25, 20, 27]. On the other hand, NLP explanations are much more vaguely defined and usually mixed with more complex philosophical interpretations, such as labeling a given text to a certain sentiment category [8].

The majority of XAI methods can be divided into the classes of perturbation and gradient analysis. Perturbations treat the model as a black box and attempt to derive knowledge of the model's behavior by studying changes in input-output pairs only. Gradient-based methods, on the other hand, execute a backpropagation step towards a target and aggregate the model's parameter adoptions to derive insights. Most of these XAI methods are not motivated by a specific discipline, e.g., neither by NLP nor CV. They are so generic that they can be applied to both disciplines to some extent. However, architecture-specific XAI methods exist as well, such as GradCAM [25], leveraging convolutional neural networks' spatial input aggregation in the deepest layers to increase efficiency.

Explainability in transformers. Through their increasing size, transformers are particularly challenging for explainability methods, especially for architecture-agnostic ones. Transformers' core components include an embedding layer followed by multiple layers of alternating attention and feed-forward blocks Fig. 1b. The attention blocks map the input into separate 'query', 'key', and 'value' matrices and are split into an array of 'heads'. As with convolutions in CNN networks, separated heads are believed to relate to specific learned features or tasks [13]. Further, the attention mechanism correlates tokens wrt. the input sequence dimension. Care needs to be taken here as GPT-style generative models, in contrast to BERT-style embedding models, apply a causal masking of activations as we will highlight further below. This in particular might obscure results for gradient-based methods.

Consequently, most explainability adoptions to transformers focus on the attention mechanism. Rollout [1] assumes that activations in attention layers are combined linearly and consider paths along the pairwise attention graph. However, while being efficient, it often emphasizes irrelevant tokens, in particular, due to its class-agnostic nature. The authors also propose attention flow, which, however, is unfeasible to compute as it constructs exhaustive graphs. More recently, Chefer et al. (2021) proposed to aggregate backward gradients and LRP [20] throughout all layers and heads of the attention modules in order to derive explanation relevancy. Their introduced method outperforms previous transformer-specific and unspecific XAI methods on several benchmarks and transformer models. It is further extended to multimodal transformers [7] by studying other variations of attention. However, they evaluated only on classification tasks, despite autoregressive transformers' remarkable generative performance, e.g., utilizing InstructGPT [21] or multimodal transformers such as MAGMA [10], BLIP [18] and OFA [31].

Multimodal transformers. Contrarily to these explainability studies evaluated on object-detection models like DETR and ViT [5, 9], we study explainability on open-vocabulary tasks, i.p., generated text tokens of a language model, and not specifically trained classifiers. Due to the multimodality, the XAI method should produce output relevancy either on the input text or the input image as depicted in Fig. 1a. Specifically, to obtain image modality in MAGMA [10], the authors propose to fine-tune a frozen pre-trained language model by adding sequential adapters to the layers, leaving the attention mechanism untouched. It uses a CLIP [22] vision encoder to produce image embeddings. These embeddings are afterward treated as equal during model execution, particularly regarding

other modality input tokens. This multi-modal open-vocabulary methodology has shown competitive performance compared to single-task solutions.

3 ATMAN: Attention Manipulation

We formulate finding the best explainability estimator of a model as solving the following question: What is the most important part of the input, annotated by the explanator, to produce the model's output? In the following, we derive our perturbation probe mathematically through studies of influence functions and embedding layer updates on autoregressive (AR) models [14, 2].

Then we show how attention manipulation on single tokens can be used in NLP tasks to steer a model's prediction in directions found within the prompt. Finally, we derive our multi-modal XAI method ATMAN by extending this concept to the cosine neighborhood in the embedding space.

3.1 Influence functions as explainability estimators

Transformer-based language models are probability distribution estimators. They map from some input space X , e.g., text or image embeddings, to an output space Y , e.g., language token probabilities. Let E be the space of all explanations, i.e., binary labels, over X . An explanator function can then be defined as e ∶ (X → Y) × (X × Y) → E , in words, given a model, an input, and a target , derive a label on the input.

Given a sequence of words w = [ w 1 , . . . , w N ] ∈ X N , an AR language model assigns a probability to that sequence p ( w ) by applying factorization p ( w ) = ∏ t p ( w t ∣ w < t ) . The loss optimization during training can then be formalized as solving:

Here h θ denotes the model, W θ the learned embedding matrix, and target t the vocabulary index of the t -th target token, with length ∣ target ∣ = N . Eq. 1 is derived by integrating the cross-entropy loss, commonly used during language model training with target = w . Finally, L denotes our loss function.

Perturbation methods study the influence of the model's predictions by adding small noise ϵ to the input and measuring the prediction change. We follow the results of the studies [14, 2] to approximate the perturbation effect directly through the model's parameters when executing Leaving-One-Out experiments on the input. The influence function estimating the perturbation ϵ of an input z is then derived as:

Here θ -z ϵ denotes the set of model parameters in which z ϵ would not have been seen during training. In the following, we further show how to approximate θ -z ϵ .

3.2 Single token attention manipulation

The core idea of ATMAN is the shift of the perturbation space from the raw input to the embedded token space. This allows us to reduce the dimensionality of possible perturbations down to a single scaling factor per input token. Moreover, we do not manipulate the value matrix of attention blocks and therewith do not introduce the otherwise inherent input-distribution shift of obfuscation methods. By manipulating the attention scores at the positions of the corresponding input sequence tokens, we are able to interpolate the focus of the prediction distribution of the model-amplifying or suppressing concepts of the prompt resulting in an XAI method as described in the following.

Attention was introduced in [28] as: O = softmax ( H ) ⋅ V , where ⋅ denotes matrix multiplication. The pre-softmax query-key attention scores are defined as

In the case of autoregression, a lower left triangular unit mask M is applied to these scores as H M = H ○ M , with ○ the Hadamard product. The output of the self-attention module is O ∈

likes

burgers

and

fries

target token

likes

burgers

and

likes

burgers

and

likes

burgers

and

likes

burgers

and

1.36

2.29

14.2

2.2

(2)

(1)

1.72

to

to

to

to

to

eat

eat

eat

eat

eat

likes

burgers

and

likes

burgers

and

to

to

eat

eat

0.89

1.86

cross entropy

(a) Illustration of the perturbation method.

Hello, my name is Lucas. I like

soccer and math. Since the last

few years, I've been working on

Prompt

my degree in

computer science.

Original completion

the soccer field for the

past 2 years

my own project, and I'm

really enjoying it

math and science

my own game, and I've

been playing it for a while

amplify soccer

suppress soccer

amplify math

suppress math

New completions

(b) Steering through ATMAN.

R h × s × d h , the query matrix is Q ∈ R h × s × d h and K , V ∈ R h × s × d h the keys and values matrices. Finally H , M , H M ∈ R h × s × s . The number of heads is denoted as h , and d h is the embedding dimension of the model. Finally, s is the length input-sequence.

The perturbation approximation θ -z ϵ required by Sec. 3.1 can now be approximated through attention score manipulation as follows: Let w be an input token sequence of length ∣ w ∣ = n . Let i be a token index within this sequence to be perturbated by a factor f . For all layers and all heads H u we modify the pre-softmax attention scores as:

where 1 ∈ [ 1 ] s × s denotes the matrix containing only ones and f i the suppression factor matrix for token i . In this section we set f i k, ∗ = f , for k = i and f ∈ R and 0 elsewhere. As depicted in Fig. 1b, we thus only amplify the i -th column of the attention scores of H by a factor ( 1 -f ) -this, however, equally for all heads. We denote this modification to the model as θ -i and assume a fixed factor f . 1 Note that this position in the architecture is somewhat predestined for this kind of modification, as it already applies the previously mentioned causal token-masking.

We define the explanation for a class label target as the vector of the positional influence functions:

with I target derived by Eq. 2,3 as I target ( w i , w ) ∶= L target ( w , θ -i )-L target ( w , θ ) . In words, we average the cross-entropy of the AR input sequence wrt. all target tokens and measure the change when suppressing token index i to the unmodified one. The explanation becomes this difference vector of all possible sequence position perturbations and thus requires n forward passes.

Fig. 2a illustrates this algorithm. The original input prompt is the text 'Ben likes to eat burgers and' for which we want to extract the most valuable token for the completion of the target token 'fries'. Initially, the model predicts the target token with a cross-entropy score of 1 . 72 . Iterated suppression over the input tokens, as described, leads to 'burgers' being the most-influential input token with the highest score of 14 . 2 for the completion.

Token attention suppression steers the model's prediction. Intuitively, for factors 0 < f < 1 , we call the modifications 'suppression', as we find the model's output now relatively less influenced by the token at the position of the respective manipulated attention scores. Contrarily, f < 0 'amplifies' the influence of the manipulated input token on the output.

An example of the varying continuations, when a single token manipulation is applied, can be seen in Fig. 2b. We provide the model a prompt in which the focus of continuation largely depends on two tokens, namely 'soccer' and 'math'. We show how suppressing and amplifying them alters the prediction distributions away from or towards those concepts. It is precisely this distribution shift we measure and visualize as our explainability.

With CS

(i)

Selected patch

Cosine Similarity on Embedding Space

cross entropy

(ii)

Without CS

A

A

B

B

C

C

Label

In what country is Normandy located?

France

When were the Normans in Normandy?

From which countries did the Norse originate?

10th and 11th centuries

Denmark, Iceland and Norway

Question

Label

Context with AtMan Explanation

The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people

who in the 10th and 11th centuries gave their name to  Normandy, a region in France. They

were descended from Norse ("Norman" comes from "Norseman") raiders and pirates

from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to

Figure 3: (a) Correlated token suppression of ATMAN enhances explainability in the image domain. i) Shows an input image along with three perturbation examples ( A,B,C ). In A , we only suppress a single image token (blue). In B , the same token with its relative cosine neighborhood (yellow), and in C , a non-related token with its neighborhood. Below depicted are the changes in the cross-entropy loss. The original score for the target token 'panda' is denoted by c 0 and the loss change by ∆ . ii) Shows the resulting explanation without Cosine Similarity (CS) and with CS. We evaluated the influence of the CS quantitatively in Fig. 12a. (b) An example of the SQuAD dataset with ATMAN explanations . The instance contains three questions for a given context, each with a labeled answer pointing to a fragment of the context. ATMAN is used to highlight the corresponding fragments of the text responsible for the answer. It can be observed that the green example is full, the blue in part, and the yellow is not at all recovered according to the given labels. However, the yellow highlight seems at least related to the label. (Best viewed in color.)

3.3 Correlated token attention manipulation

Suppressing single tokens works well when the entire entropy responsible for producing the target token occurs only once. However, for inputs with redundant information, this approach would often fail. This issue is especially prominent in Computer Vision (CV), where information like objects in an image is often spread across multiple tokens-as the image is usually divided into patches, each of which is separately embedded. It is a common finding that applied cosine similarity in the embedding space, i.p., for CLIP-like encodings as used in MAGMA, gives a good correlation estimator [17, 2]. This is in particular due to the training by a contrastive loss. We integrate this finding into ATMAN in order to suppress all redundant information corresponding to a particular input token at once, which we refer to as correlated token suppression.

Fig. 3a summarizes the correlated token suppression visually. For n input tokens and embedding dimension d , the embedded tokens result in a matrix T = ( t i ) ∈ R n × d . The cosine similarity is computed from the normalized embeddings ̃ T = ( ̃ t i ) , with ̃ t i = t i /∣∣ t i ∣∣ , for i ∈ 1 , . . . n , as S = ( s i ) = ̃ T ⋅ ̃ T ⊺ ∈ [-1 , 1 ] n × n . Note that the index i denotes a column corresponding to the respective input token index. Intuitively, the vector s i then contains similarity scores to all input tokens. In order to suppress the correlated neighborhood of a specific token with the index i , we, therefore, adjust the suppression factor matrix for Eq. 4 as

In words, we interpolate linear between the given suppression factor 1 -f and 1 , by factor of the thresholded cosine similarity f i k, ∗ . As we only want to suppress tokens, we restrict the factor value range to greater than 0 and smaller than 1 . The parameter κ > 0 2 is to ensure a lower bound on the similarity, and in particular, prevents a flip of the sign. Overall, we found this to be the most intuitive and robust equation. In Fig. 11 we ran a sweep over the hyper-parameters to obtain the best precision and recall values. Note in particular, that our modifications are rather subtle compared to entire token-masking. The latter did not lead to satisfying results, even given the context. We compute the similarity scores once on the initial embedding layer, and uniformly apply those modifications throughout all layers.

Note that variations of Eq. 4, 5 and 6 could also lead to promising results. In particular in the active research field of vision transformers, hierarchical token transformations could be applicable and speed

green plants

AtMan

orangutan

quadbike

couch

bear

birthday

AtMan

AtMan

up computations (c.f. [3], Fig 16). Furthermore, preliminary experiments using the similarity between the query and key values per layer, a different similarity score such as other embedding models or directly sigmoid values, or even a non-uniform application of the manipulations yield promising, yet degenerated performance (c.f. App. A.15). As we could not conclude a final conclusion yet, we leave these investigations to future research.

With that, we arrived at our final version of ATMAN. Note that this form of explanation E is 'local', as target refers to our target class. We can, however, straightforwardly derive a 'global explanation' by setting target = y , for y , a model completion to input w of a certain length. It could then be interpreted rather abstract as the model's general focus [4].

4 Empirical Evaluation

We ran empirical evaluations on text and image corpora to address the following questions: (Q1) Does ATMAN achieve competitive results compared to previous XAI methods for transformers in the language as well as vision domain? (Q2) Is ATMAN applicable to various transformer models, and does ATMAN scale efficiently, and can therefore be applied to current large-scale AR models?

To answer these questions, we conducted empirical studies on textual and visual XAI benchmarks and compared ATMAN to standard approaches such as IxG [26], IG [27], GradCAM [25] and the transformer-specific XAI method of [6] called Chefer in the following. Note that all these methods utilize gradients and, therefore, categorize as propagation methods leading to memory inefficiency. We also tried to apply existing perturbation methods such as LIME [24] and SHAP [19] and Attention Rollout [1]. However, they failed due to extremely large trials and, in turn, prohibitive computation time, or qualitatively (c.f. Appendix A.9). We adopt common metrics, namely mean average precision (mAP) and recall (mAR) (c.f. Appendix A.11), and state their interquartile statistics in all experiments. To provide a comparison between the XAI methods, we benchmarked all methods on GPT-J [29] for language and MAGMA-6B [10] for vision-language tasks. Through its memory efficiency, ATMAN can be deployed on really large models. We further evaluate ATMAN on a MAGMA-13B and 30B to show the scaling behavior, and, on BLIP [18] to show the architecture independence. To obtain the most balanced comparison between the methods, we selected datasets focusing on pure information recall, separating the XAI evaluation from the underlying model interpretations.

4.1 ATMAN can do language reasoning

Protocol. All experiments were performed on the open-source model GPT-J [29]. Since with ATMAN we aim to study large-scale generative models, we formulate XAI on generative tasks as described in Sec. 3.3. For evaluation, we used the Stanford Question Answering (QA) Dataset (SQuAD) [23]. The QA dataset is structured as follows: Given a single paragraph of information, there are multiple questions, each with a corresponding answer referring to a position in the given paragraph. A visualization of an instance of this dataset can be found in Fig. 3b. SQuAD contains 536 unique paragraphs and 107,785 question/explanation pairs. The average context sequence length is 152 . 7 tokens and the average label length is 3 . 4 .

The model was prompted with the template: '{Context} Q: {Question} A:', and the explainability methods evaluated on the generations to match the tokens inside the given context, c.f. Fig. 3b, Sec. 3.3. If there were multiple tokens in the target label, we computed the mean of the generated scores.

Similar to weak segmentation tasks in computer vision, we regarded the annotated explanations as binary labels and determined precision and recall over all these target tokens.

Results. The results are shown in Tab. 1 a. It can be observed that the proposed ATMAN method thoroughly outperforms all previous approaches on the mean average precision. This statement holds as well for the interquartile mean of the recall. However on the entire recall, Chefer slightly outperforms ATMAN. Note that the high values for recall scores throughout all methods are partly due to the small average explanation length, such as depicted in Fig. 3b. Further details and some qualitative examples can be found in Appendix A.2.

Paragraph chunking. ATMAN can naturally be lifted to the explanation of paragraphs. We ran experiments for ATMAN dividing the input text into several paragraphs by using common delimiters as separation points and evaluating the resulting chunks at once, in contrast to token-wise evaluations. This significantly decreases the total number of required forward passes and, on top, produces 'more human' text explanations of the otherwise still heterogeneously highlighted word parts. Results are shown in Appendix A.8.

4.2 ATMAN can do visual reasoning

Protocol. Similar to language reasoning, we again perform XAI on generative models. We evaluated the OpenImages [16] dataset as a VQA task and generated open-vocabulary prediction with the autoregressive model. Specifically, the model is prompted with the template: '{Image} This is a picture of ', and the explainability methods executed to derive scores for the pixels of the image on the generations with respect to the target tokens. If there were multiple tokens in the target label, we computed the mean of the generated scores. For evaluation, we considered the segmentation annotations of the dataset as ground truth explanations. The segmentation subset contains 2,7M annotated images for 350 different classes. In order to ensure a good performance of the large-scale model at hand and, in turn, adequately evaluate only the XAI methods, we filtered the images for a minimum dimension of 200 pixels and a maximal proportional deviation between width and height of 20% . We randomly sample 200 images per class on the filtered set to further reduce evaluation costs. In total, this leads to a dataset of 27 . 871 samples. The average context sequence length is 144 tokens, and the average label coverage is 56% of the input image. To provide a fair comparison between all XAI methods, we compute token-based attributions for both ATMAN and Chefer, as well as IG, IxG, and GC. Note that the latter three allow pixel-based attribution granularity, however, these resulted in lower performance, as further discussed in Appendix A.2.

Results. The results are shown in Tab. 1 b. It can be observed that ATMAN thoroughly outperforms all other XAI approaches on the visual reasoning task for all metrics. Note how explicit transformer XAI methods (ATMAN, Chefer) in particular outperform generic methods (GradCAM, IG, IxG) in recall. Moreover, while being memory-efficient (c.f. next section), ATMAN also generates more accurate explanations compared to Chefer. Through the memory efficiency of ATMAN, we were able to evaluate a 13B and 30B upscaled variant of MAGMA (c.f. Tab. 1 c). Interestingly, the general explanation performance slightly decreases, when upscaled, compared to the 6B model variant. However, throughout all variants ATMAN scored comparable high. Note that the overall benchmark performances of the models slightly increase (c.f. App. A.14). This behavior could be attributed to

6.0

8.9

13.1

17.3

Model size (billion parameters)

0

20

40

60

80

GPU memory usage (GB,

)

0

100

200

300

400

AtMan

128

256

512

768

1024

Baseline

6.0

8.9

13.1

17.3

0

20

40

60

80

0.0

0.5

1.0

1.5

2.0

Runtime (seconds, --)

the increased complexity of the model and, subsequently, the complexity of the explanation at hand. Hence, the 'human alignment' with the model's explanations is not expected to scale with their size.

Model agnosticism. Note that the underlying MAGMA model introduces adapters on a pre-trained decoder text model to obtain the image multi-modality. It is yet unclear to what extent adapters change the flow of information in the underlying model. This may be a reason for the degenerated performance of gradient-based models. As ATMAN however outperforms such methods, we may conclude that information processing still remains bound to the specific input-sequence position. This finding generalizes to encoder architectures, as experiments with BLIP suggest in Tab. 1 c. Precisely, BLIP's multi-modality is achieved by combining an image-encoder, a text-encoder, and a final textdecoder model. These are responsible for encoding the image, the question, and finally for generating a response separately. Information between the models is passed through cross-attention. We only applied ATMAN to the image-encoder model of this architecture (c.f. App. A.10). The results demonstrate the robustness of the suggested perturbation method over gradient-based approaches.

Qualitative illustration. Fig. 4 shows several generated image explanations of ATMAN and Chefer for different concepts. More examples of all methods can be found in Appendix A.7 and evaluation on the more complex GQA dataset [12] is discussed in Appendix A.12. We generally observe more noise in gradient-based methods, in particular around the edges. Note that ATMAN evaluates the change in its perturbated generations. These are independent of the target tokens. Compared to other methods like Chefer, we, therefore, do not need to process the prompt more than once for VQA on different object classes with ATMAN. Note again that the aforementioned cosine similarity is crucial to obtain these scores. Quantitative results can be found in App. A.4.

In general, the results clearly provide an affirmative answer to (Q1) : ATMAN is competitive with previous XAI methods, and even outperforms transformer-specific ones on the evaluated language and vision benchmarks for precision and recall. Next, we will analyze the computational efficiency of ATMAN.

4.3 ATMAN can do large scale

While ATMAN shows competitive performance, it computes, unlike previous approaches, explanations at almost no extra memory cost. Fig. 5 illustrates the runtime and memory consumption on a single NVIDIA A100 80GB GPU. We evaluated the gradient-based transformer XAI method [6] and ATMAN. The statistics vary in sequence lengths (colors) from 128 to 1024 tokens, and all experiments are executed sequentially with batch size 1 for best comparison.

One can observe that the memory consumption of ATMAN is around that of a plain forward pass (Baseline; green) and increases only marginally over the sequence lengths. In comparison, the method of [6]-and other gradient-based methods-exceeds the memory limit with more than double in memory consumption. Therefore, they fail on larger sequence lengths or models.

Memory (GB)

75

50

25

0

AtMan

Chefer

6b

13b

17b

Model size (billion parameters)

6b

13b

17b

34b

Model size (billion parameters)

0

1

2

3

Runtime (s)

AtMan

Chefer

6b

13b

17b

34b

Model size (billion parameters)

0

25

50

75

Memory (GB)

AtMan

Chefer

Whereas the memory consumption of ATMAN stays is almost constant, the execution time significantly increases over sequence length when no further token aggregation is applied upfront. Note, that Fig. 5 shows complete and sequential execution. The exhaustive search loop of ATMAN can be run partially and in parallel to decrease its runtime. In particular, the parallelism can be achieved by increasing the batch size and naturally by a pipeline-parallel execution. For instance, since large models beyond 100B are scattered among nodes and thus many GPUs, the effective runtime is reduced by magnitudes to a proximate scale of the forward pass. Furthermore, tokens can be aggregated in chunks and evaluated on a more coarse level, which further reduces the runtime drastically, as demonstrated in Fig. 16.

In Fig. 6 we provide performance results on a realistic scenario. It compares a practical query (average 165 tokens) on the task of explaining a complete image (144) tokens. Runtimes are measured with pipe-parallel 4, and ATMAN additionally with batch size 16. Specifically, ATMAN requires between 1 and 3 total compute seconds, around 1 magnitude longer compared to Chefer. Note that this can still further be divided by the number of available idling workers, to reduce the absolute clock time to subseconds. Each batch of ATMAN can be processed entirely parallel. Moreover Chefer, even with memory optimizations, fails to scale to 34b with the given memory limit (80GB).

Overall, these results clearly provide an affirmative answer to (Q2) : Through the memory efficiency of ATMAN, it can be applied to large-scale transformer-based models.

5 Conclusion

We proposed ATMAN, a modality and architecture agnostic perturbation-based XAI method for generative transformer networks. In particular, ATMAN reduces the complex issue of finding proper input perturbations to a single scaling factor per token. As our experiments demonstrate, ATMAN outperforms current approaches relying on gradient computation. It moreover performs well on decoder as well as encoder architectures, even with further adjustments like adapters or crossattention in place. Most importantly, through ATMAN's memory efficiency, it provides the first XAI tool for deployed large-scale AR transformers. We successfully evaluated a 30B multi-modal model on a large text-image corpus.

Consequently, ATMAN paves the way to evaluate whether models' explanatory capabilities scale with their size, which should be studied further. Besides this, our paper provides several avenues for future work, including explanatory studies of current generative models impacting our society. Furthermore, it could lay the foundation for not only instructing and, in turn, improving the predictive outcome of autoregressive models based on human feedback [21] but also their explanations [11].

Acknowledgments

We gratefully acknowledge support by the German Center for Artificial Intelligence (DFKI) project 'SAINT', the Federal Ministry of Education and Research (BMBF) project 'AISC' (GA No. 01IS22091), and the Hessian Ministry for Digital Strategy and Development (HMinD) project 'AI Innovationlab' (GA No. S-DIW04/0013/003). This work also benefited from the ICT-48 Network

34b

of AI Research Excellence Center 'TAILOR' (EU Horizon 2020, GA No 952215), the Hessian Ministry of Higher Education, and the Research and the Arts (HMWK) cluster projects 'The Adaptive Mind' and 'The Third Wave of AI', and the HMWK and BMBF ATHENE project 'AVSV'. Further, we thank Felix Friedrich, Marco Bellagente and Constantin Eichenberg for their valuable feedback.

References

Samira Abnar and Willem H. Zuidema. Quantifying attention flow in transformers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, (ACL) , pages 4190-4197. Association for Computational Linguistics, 2020.

Daniel Bis, Maksim Podkorytov, and Xiuwen Liu. Too much in common: Shifting of embeddings in transformer language models and its implications. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , pages 5117-5130. Association for Computational Linguistics, 2021.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=JroZRaRw7Eu .

Nadia Burkart and Marco F. Huber. A survey on the explainability of supervised machine learning. J. Artif. Intell. Res. , 70:245-317, 2021.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference of Computer Vision (ECCV) , Lecture Notes in Computer Science, pages 213-229. Springer, 2020.

Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 782-791, 2021.

Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 397-406, 2021.

Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable AI for natural language processing. In Proceedings of the Conference of the AsiaPacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, (AACL/IJCNLP) , pages 447-459. Association for Computational Linguistics, 2020.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR) , 2021.

Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. Magma - multimodal augmentation of generative models through adapter-based finetuning. In Findings of EMNLP . Association for Computational Linguistics, 2022. URL https://aclanthology.org/2022. findings-emnlp.179 .

Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. A typology to explore and guide explanatory interactive machine learning. Nature Machine Intelligence , 2023.

Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pages 6700-6709. Computer Vision Foundation / IEEE, 2019. doi: 10.1109/CVPR.2019. 00686. URL http://openaccess.thecvf.com/content_CVPR_2019/html/Hudson_GQA_A_New_ Dataset_for_Real-World_Visual_Reasoning_and_Compositional_CVPR_2019_paper.html .

Jae-young Jo and Sung-Hyon Myaeng. Roles and utilization of attention heads in transformer-based neural language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, (ACL) , pages 3404-3417. Association for Computational Linguistics, 2020.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the International Conference on Machine Learning, (ICML) , volume 70 of Proceedings of Machine Learning Research , pages 1885-1894. PMLR, 2017.

Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. Captum: A unified and generic model interpretability library for pytorch, 2020.

Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github.com/openimages , 2017.

Ray R. Larson. Introduction to information retrieval. J. Assoc. Inf. Sci. Technol. , 61:852-853, 2010.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (ICML) , 2022.

Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 4765-4774, 2017.

Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller. Layer-wise relevance propagation: An overview. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning , volume 11700 of Lecture Notes in Computer Science , pages 193-209. Springer, 2019.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS) , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML) , Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 2021.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2383-2392. The Association for Computational Linguistics, 2016.

Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) , pages 1135-1144. ACM, 2016.

Ramprasaath Rs, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision , 128, 02 2020. doi: 10.1007/s11263-019-01228-7.

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the International Conference on Machine Learning (ICML) , volume 70 of Proceedings of Machine Learning Research , pages 3145-3153. PMLR, 2017.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Proceedings of the International Conference on Machine Learning, (ICML), page 3319-3328. JMLR.org, 2017.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS) , pages 5998-6008, 2017.

Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021.

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research , 2022.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-tosequence learning framework. In Proceedings of the International Conference on Machine Learning (ICML) , 2022.

A Appendix

A.1 Remarks on executed benchmarks

We executed all benchmarks faithfully and to the best of our knowledge. The selection of compared methods was made to be rather diverse and obtain a good overview in this field of research. In particular, with regards to the multi-modal transformer scaling behavior, as there are in fact no such studies for AR models yet to compare to. It is possible, for all methods, that there are still improvements we missed in quality as well as performance. However, we see the optimizations of other methods to multi-modal AR transformer models as a research direction on its own.

Chefer. The integration of Chefer was straightforward. As it can be derived by the visualizations, there are noticeable artifacts, particularly on the edges of images. In this work the underlying transformer model was MAGMA, which is finetuned using sequential adapters. It is possible that this, or the multi-modal AR nature itself, is the cause for these artifacts. We did not further investigate to what extent the adapters are to be particularly integrated in the attribute accumulation of Chefer. Also notice that ATMAN often has similar, but not as severe, artifacts.

IxG, IG and GradCAM. The methods IxG, IG, and (guided) GradCAM failed completely from the quality perspective in recall when applied to pixel granularity (c.f. Tab. 2). Those are the only ones directly applicable to the image domain, and thus also included the vision encoder in the backward pass. We did not further investigate or fine-tune evaluations to any method. For better comparison we therefore only present the token-based comparisons. All methods are evaluated with the same metrics and therewith give us a reasonable performance comparison without additional customization or configuration.

Details on Results. For a fair comparison, all experiments were executed on a single GPU, as scaling naturally extends all methods. We also want to highlight that we did not optimize the methods for performance further but rather adopted the repositories as they were. The memory inefficiency of gradient-based methods arises from the backward pass. A maximal memory performant representative is the Single-Layer-Attribution method IxG, which only computes partial derivatives on the input with respect to the loss. Even this approach increases the memory requirement beyond an additional 50% and fails for the scaling experiments up to 34B.

In Fig. 5 we ran Chefer with a full backward pass. We adopted this to the minimum amount of gradients (we saw) possible and plot the full scaling benchmark below in Fig 7 3 . The key message remains the same. With the given IxG argument, we do not see much potential in improving memory consumption further.

The methods IxG, IG and GradCam are integrated using the library Captum [15]. We expect them to be implemented as performant as possible. IntegratedGradients is a perturbation method on the input, integrating changes over the gradients. The implementation at hand vastly runs OOM. Finally GradCam is a method specialized on CNN networks and therefore does not work for text only (or varying sequence lengths). It requires the least amount of resources but also produces poor results, without further investigations.

ATMAN Parallelizability. As a final remark, we want to recall again that the runtime measured in the sequential execution can be drastically reduced due to its parallelizability, i.p., as it only requires forward passes. For sequence length 1024, we measured 1024 iterations in order to explain each token. However note that ATMAN can also be applied to only parts or chunks of the sequence (c.f. Sec. A.8), in contrast to gradient methods. Moreover, all tokens to explain can be computed entirely in parallel. In a cluster deployment, these can be distributed amongst all available workers. On top, it can be divided by the available batch size and true pipeline-parallelism (c.f. Fig. 6).

6.0

8.9

13.1

17.3

25.7

34.1

Model size (billion parameters)

0

10

20

30

40

50

60

70

80

0

100

200

300

400

500

600

Solid: GPU memory usage (GB)

AtMan

128

256

512

768

1024

Baseline

Sequ. Length

Dashed: Runtime (seconds)

6.0

8.9

13.1

17.3

25.7

34.1

Model size (billion parameters)

0

10

20

30

40

50

60

70

80

Solid: GPU memory usage (GB)

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Dashed: Runtime (seconds)

Chefer

(memory optimized)

6.0

8.9

13.1

17.3

25.7

34.1

Model size (billion parameters)

0

10

20

30

40

50

60

70

80

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Dashed: Runtime (seconds)

Solid: GPU memory usage (GB)

IxG

6.0

8.9

13.1

17.3

25.7

34.1

Model size (billion parameters)

0

10

20

30

40

50

60

70

80

0

2

4

6

8

10

Dashed: Runtime (seconds)

Solid: GPU memory usage (GB)

IG

6.0

8.9

13.1

17.3

25.7

34.1

Model size (billion parameters)

0

10

20

30

40

50

60

70

80

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Dashed: Runtime (seconds)

Solid: GPU memory usage (GB)

GradCam

2

4

6

8

10

12

14

16+

Number of tokens in explanation

0

2000

4000

6000

8000

10000

12000

14000

16000

Number of instances

2

3

4

5

6

7

Maximum number of questions

0.4

0.5

0.6

0.7

0.8

0.9

mAP

AtMan

Chefer

Input x Grad

IG

(a) Histogram of explanation token length.

(b) mAP for all methods, grouped by number of question/ answer pairs in the dataset.

A.2 Detailed SQuAD Evaluations

This sections gives more detailed statistics on the scores presented in Tab. 1. First Fig. 8a is the histogram of the token lengths of all explanations. Fig. 8b is the mAP score for all methods on the entire dataset, grouped by the number of questions occuring per instance.

A.3 Detailed OpenImages Evaluations

This section gives more detailed statistics on the scores presented in Tab. 1. Fig. 9 is the histogram of the fraction of label coverage on all images. Fig. 10a and 10b are boxplots for all methods on the entire dataset, for mean average precision as well as recall. Note that the methods IG, IxG and GradCam can be evaluated on image-pixel level, contrary to ATMAN and Chefer. However their performances (i.p. wrt. recall) significantly degenerate, c.f. Tab. 2, Fig. 15.

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of image covered by Label

0

1000

2000

3000

4000

5000

6000

Number of Samples

GC

IG

IxG

Chefer

AtMan

0.0

0.2

0.4

0.6

0.8

1.0

GC

IG

IxG

Chefer

AtMan

0.0

0.2

0.4

0.6

0.8

A.4 Discussion of Cosine Embedding Similarity and Parameter values

We fixed the parameter κ = 0 . 7 of Eq. 6 and f = 0 . 9 of Eq. 4 throughout this work. They were empirically concluded by running a line sweep on a randomly sampled subset of the OpenImages dataset once, c.f. Fig. 11. Note that pure token-masking ('mask') always leads to worse performance compared to our more subtle concept-suppression operations. We use the same parameters throughout this work, however, parameters may be optimizable depending on the use case and dataset.

In Fig. 12a and 12b we compare the mean average precision and recall scores for OpenImages for both variants, with and without correlated token suppression (to threshold κ ). Clearly the latter outperforms single token suppression.

The following Fig. 13 shows visually the effect on weak image segmentation when correlated suppression of tokens is activated, or when using single token suppression only. Notice how single token only occasionally hits the label, and often marks a token at the edge. This gives us a reason to believe that entropy is accumulated around such edges during layer wise processing. This effect (on these images) completely vanishes with correlated suppression of tokens.

A.5 Variation Discussion of the method

Note that the results of Eq. 4 are directly passed to a softmax operation. The softmax of a vector z is defined as

In particular, the entries z k = 0 and z l = -∞ will yield to the results softmax ( z ) k = 1 / ∑ j e z j and softmax ( z ) l = 0 . So one might argue as follows: If we intent to suppress the entropy of token i , we do not want to multiply it by a factor 0 , but rather subtract -∞ of it. I.e. we propose the modification

The only problem with this Eq. 7 is, that it skews the cosine neighborhood factors. While we experienced this working more naturally in principle, for hand-crafted factors, we did not get best

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

mask

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.99

precision

0.500

0.525

0.550

0.575

0.600

0.625

0.650

0.675

0.700

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

mask

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.99

recall

0.00

0.05

0.10

0.15

0.20

0.25

0.30

f

f

0.0

0.2

0.4

0.6

0.8

1.0

mAP

0

1000

2000

3000

4000

5000

6000

7000

Number of samples

AtMan without CST

AtMan

(a)

0.0

0.2

0.4

0.6

0.8

1.0

mAR

0

5000

10000

15000

20000

Number of samples

AtMan without CST

AtMan

(b)

performance in combination with Eq. 6. In the following Fig. 14a and 14b, we show analogous evaluations to Fig. 12a and 12b. It is in particular interesting that the mode without correlated tokens slightly improves, while the one with slightly decreases in scores, for both metrics.

A.6 Artifacts and failure modes

Merged into A.12 (kept here for reference).

A.7 Qualitative comparison weak image segmentation

In the following Fig. 15 we give several examples for better comparison between the methods on the task of weak image segmentation. To generate the explanations, we prompt the model with ' < Image > This is a picture of ' and extract the scores towards the next target tokens as described with Eq. 5 for ATMAN. For multiple target tokens, these results are averaged. In the same fashion, but with an additional backpropagation towards the next target token, we derive the explanations for Chefer and the other gradient methods.

A.8 Application to document q/a

In Fig. 16 we apply ATMAN on a larger context of around 500 tokens paragraph wise. The Context is first split into chunks by the delimiter tokens of '.', ',', '\n' and ' and'. Then iteratively each chunk is evaluated by prompting in the fashion ' { Context } Q: { Question } A: ' and the cross entropy extracted towards the target tokens, suppressing the entire chunk at once, as described in Sec. 3. It can be observed that the correct paragraphs are highlighted for the given questions and expected targets.

Input image

+ target

Label

Correlated Token

Single Token

Frog

Sheep

Fish

Polar Bear

In particular, one can observe the models interpretation, like the mapping of formats or of states to countries. Note in particular that it is not fooled by questions not answered by the text (last row).

A.9 Attention Rollout, Flow, other Perturbation method

Attention Flow constructs a graph representing the flow of attention through the model. Unfortunately this approach is computationally too intense for evaluation on large language models; Chefer et al. (2021) already ran into issues on smaller BERT models. LIME [24] and SHAP [19] are perturbation methods closely related to ATMAN. However they result in several 100 . 000 forward trials which is inpractical for use with large language models.

Finally, we ran additional experiments for Attention Rollout [1]. Note that the authors stress its design for encoder classification architectures, while we explore the applicability to generative decoder models in this work. Further, Rollout is by default class-agnostic, in contrast to ATMAN or the studied gradient methods. We adopted Attention Rollout by appending the target to the prompt, and only aggregating attentions over those last target sequence positions. Note that in contrast to the other studied methods, Rollout disregards the output distribution.

We observed large artifacts on the explanations at the positions of the last 2 image patches; apparently most attention aggregation take place at those positions, which may be due to the causal mask. On OpenImages our adoption of Rollout achieves a mAP score of 43 . 6 and a mAR of 1 . 3 (c.f. AtMan with 65 . 5 and 13 . 7 ). Scores do increase when removing the last image patch, however, this also entirely erases explanation on this patch. On Squad we achieved mAP of 23 . 1 and a mAR of 53 . 4 (AtMan: 73.7, 93.4).

0.0

0.2

0.4

0.6

0.8

1.0

mAP

0

1000

2000

3000

4000

5000

6000

Number of instances

AtMan w/o

CST

AtMan

(a)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

mAR

0

2000

4000

6000

8000

10000

12000

Number of instances

AtMan without CST

AtMan

(b)

A.10 Demarcation to other architectures, BLIP experiments

To answer the question of 'why we think that AtMan could be generalized to other (generative) multimodal Transformers' , consider the following arguments.

We conduct for the first time experiments on generative decoder models. The underlying MAGMA model extends the standard decoder transformer architecture, having causal attention masks in place, by additional sequential adapters that could potentially shift distributions. However, the proposed attention manipulation still leads to a robust perturbation method - in text-only as well as image-text modality. We thus conjecture that the skip connections lead to a somewhat consistent entropy flow throughout the network. OFA ( < 1 B) and BLIP rely on standard encoder-decoder architectures -even without adapters- and the same skip connections. Furthermore, both models process the input image in a similar fashion to obtain 'image tokens' forwarded throughout the transformer at the same positions.

BLIP, in contrast to OFA, separates the processing of the image, the prompt, and the answer generation into 3 different models -standard encoders, only the answer generation model being a decoder. The different outputs are cross-attended. We apply ATMAN only on the vision-encoder model.

For comparison, we furthermore evaluate Chefer on BLIP, its attention aggregation being only applied to the vision encoder as well. Tab. 3 compares these two methods, showing that ATMAN still outperforms Chefer. Note, however, that Chefer achieves better scores compared to being applied to the MAGMA model.

On language models, we found that 'chat'-finetuned-versions are more applicable to this method, most likely as they are specifically trained to respect the given input. In general, prompt-tuning instructions may improve the results, such as 'Answer with respect to the given context'.

A.11 Metrics

All recall and precision metrics in this work are based on the normalized continuous values as follows: For relevancy scores x and binary segmentation label y , let ̃ x = x / max i ( x i ) . We compute AP = (∑ i x i y i )/ ∑ i x i for precision and for recall AR = (∑ i ̃ x i y i )/ ∑ i y i . The final mean scores

Target

Monkey

Ipod

Camera

Cattle

Coffee

Bear

Boy

Corded phone

Dog

AtMan

Chefer

IG

IxG

GradCAM

RBU

RBU

are the average of all samples. This yields a comparable untuned metric also catching the robustness of the methods.

A.12 Interpretability of 'complex facts' and failure modes

All experiments were executed in the generative setting, c.f. Eq. 1,2 and the remark to 'global explanations' at the end of Sec. 3. Fig. 1a shows 'real text generation' and the accumulation of explanations on the input image during the model's generation process. For comparison benchmarks,

Layer

GradCAM

Figure 16: Showing ATMAN capabilities to highlight information in a document q/a setting. The model is prompted with ' { Context } Q: { Question } A: ' and asked to extract the answer (target) of the given Explanation. Here, ATMAN is run paragraph wise, as described in text, and correctly highlights the ones containing the information. All Explanations were split into ∼ 50 paragraphs (thus requiring only 50 ATMAN forwadpasses) . In particular it is shown in row 2 that the model can interpret, i.e. convert date-time formats. Row 3 shows that it can derive from world knowledge that Michigian is in the US. Row 4 shows that the method ATMAN is robust against questions with non-including information. (Best viewed in color.)

Table 4: Performance of scaled MAGMA models.

our target was to separate generated explanations from the underlying model's interpretation, by retrieving 'obvious facts'. Fig. 4 demonstrates qualitatively target-class awareness.

More complex results are shown in Fig. 17. Fig. 17a gives an example where the model is VQA prompted and ATMAN explains both modalities at the same time for the completion token 'white'. It highlights the words 'tub' and 'color' of the text, and an edge of the tub in the picture as significant to produce the completion. Note that we find these 'joint-modal explanations' to work best when the target token is not just a yes/no answer, c.f. Fig. a'). Further note how these artifacts remain amongst explainability methods such as Chefer, which may be due to the underlying models architecture. We leave these investigations for further research.

In Fig. 17b ATMAN highlights multiple heterogenous concepts to answer the counting task. Note that here the explanation for the completion 'two', which actually highlights visually the animals, is found at the token position 'animals', and not at the position of the predicted token. ATMAN returns at the same time explanations to all previous context positions, without additional forward passes (in contrast to gradient-based methods).

Finally, we ran evaluations on visual reasoning using the GQA dataset 17. Fig. 17c shows an instance of GQA, in which the model is prompted to answer 'Does the man ride a horse?', while the man in the picture actually rides a bike. ATMAN highlights at two different sequence positions of the prompt the respective concepts in the image required to answer the question, once the man, and once the bike. Fig. 17d.1 and d.2 are the resulting IG explanations when applied to explain the same token positions; somewhat it captures parts of the concept but focuses on noise to the edge. Note that many samples of this dataset are rather difficult to answer, it is often not clear on what token position an explanation is supposed to be found. In particular, often ambiguous concepts occur multiple times in the image and are further restricted to the correct instances much later in the text description. This is, in particular, difficult to grasp for generative decoder models due to the causal masking. As the dataset contains annotations for areas as in example Fig. 17c, we ran preliminary quantitative results on 10k randomly chosen instances. ATMAN matches bounding boxes with mAP 52 . 4 and mAR 24 . 3 . For IG we obtained mAP 33 . 3 and mAR 27 . 0 .

A.13 Partial attention manipulation

We apply the same manipulation in each layer. We hypothesised -and first experiments indicatethat one needs a 'critical mass', to remove the entire concept entropy from the generation process. This is i.p. due to the skip connections. Only applying it to a single (arbitrary) layer did not yield comparable results. However we did not finalize a conclusion yet and leave this to future work. C.f. A.15, Fig. 18.

A.14 MAGMAscaling

MAGMAModel performances on standard text-image evaluation benchmarks are shown in Tab. 4.

A.15 Variants of ATMAN and final remarks

There are many promising variants of ATMAN not yet fully understood. In particular finding the relevant layers and other formulae to manipulate reliably the activations, or to normalize the output scores, could yield further insights into the transformer architecture. We showed how conceptual suppression improves overall performance. For pure NLP tasks, utilizing an embedding model such as BERT to compute similarities could be beneficial. Moreover, it should be possible to aggregate tokens and increase the level of details as needed, to save additional computational time.

Q: Does the man

Q: Does the man ride a horse? A:

No

.

White

a)

c.1)

c.2)

d.2)

d.1)

0

1

0

1

0

1

Is the bath tub

white in color?

Is the bath tub

white in color?

yes

Chefer

AtMan

A:

yes

A:

Q:

Q:

a'.1)

a'.2)

0

5

10

15

20

25

layer

0.45

0.50

0.55

0.60

0.65

mAP (solid)

0.09

0.10

0.11

0.12

0.13

0.14

0.15

0.16

mAR (dashed)

Fig. 18 shows scores when applying ATMAN only to a single layer. I.p. the second half of layers seem more influential to the ATMAN scores, which is in accordance with recent research studying that higher-level concepts form later throughout the network.

In Fig. 19 we applied softmax on the embeddings (still only of the first layer) instead of the proposed thresholded cosine similarity Eq. 6. The proposed cosine similarity clearly outperforms its softmax variant in quality, however visible correlations can be found in the softmax-image as well.

As a final remark, we want to highlight again the rather undefined question of 'what an explanation is'. Afterall, we show what input leads to a diverging generation of the model faithfully, even if it would not match the expected outcome.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces AtMan, a method that manipulates the attention mechanism of generative transformer models, to provide explanations. More specifically, it perturbs raw attention scores instead of the token inputs. The authors provide extensive experiments for different modalities to back their claims regarding better results to established baselines.\n\nSTRENGTHS:\n1. The paper introduces a new approach towards Explainable AI, that does not require calculation of gradients through the network, thus enabling evaluation of larger models with the same hardware.\n2. The idea of manipulating attention scores is novel and interesting.\n3. The authors embark on an extensive evaluation of their method on different modalities. The universality of the Transformer architecture, enabled them to seamlessly switch between modalities. The methods performs well on a range of architectures (encoder, decoder).\n\nWEAKNESSES:\n1. One of the main motivation of this work is to enable more widespread adoption of explainable AI methods, via reducing required computational resources. The issue of memory is discussed extensively. Inference speed is only mildly explored, in Figure 5. For large sequence lengths, the method requires upwards of 2 orders of magnitude more time to make the final predictions.\n2. The method requires one to set new hyperpatameres, namely $f$ and $\\kappa$. The effect of those with respect to e.g. sequence length or model architecture, are not discussed. For example, different sequence lengths may lead to different softmax entropy and thus different $f$ values might be required. \n3. According to Eq. (4), tokens are masked by setting the corresponding $H$ values to 0. This does by no means lead to masking of the tokens, as briefly explained in Appendix A5. Setting the values to the value $0$ constitutes another hyperparameter, that needs to be set for the proposed method. In principle any real value could have been chosen (In Appendix A5 the authors explore the values $0$ and $-\\infty$). The range of the possible values was not explored at all. \n4. In general a lot of approximations are made for the derivations of the last method, with possible alternatives not discussed. Why was Eq. (4) chosen to mask tokens? Why is the cosine similarity of the embeddings a good measure to suppress common tokens? In [1] for example, the cosine similarity between keys is used as a more suitable measure. The cosine similarity is calculated based on the embeddings of what layer? Is there a benefit from calculating this similarity on different layers? There is a lot of work on what type of information token embeddings carry for different layers in a Vision Transformer. \n\n[1] Bolya, Daniel, et al. \"Token merging: Your vit but faster.\" arXiv preprint arXiv:2210.09461 (2022).\n\nQUESTIONS:\n1. Line 62 states that the source code is released although I could not find it in the supplementary.\n2. Line 33-34 states that memory consumption leads to uneconomical productive deployment. This is a overstatement, especially given the computational requirements of the new proposed method. \n3. A lot of literature has been exploring the effect of attention scores in the context of Stable Diffusion, e.g. [2] and follow-ups.\n4. Is there a minus missing before $L^{\\text{target}}$ in Eq. (2)?\n5. Line 245, it is not clear to me how the mean of generated scores affects the final predictions. Are some tokens in the target label more important than others as expected? \n6. Is there a way to prioritize checking some tokens first instead of exhaustively trying out all tokens to find the ones that explain better the target? This might help to reduce the inference speed. You can think about having a method to rank tokens to check, and then create a plot of computational budget vs performance, where you check iteratively more tokens. You could also think of doing this in a more hierarchical way, similar to what is proposed in Appendix A8. \n\n[2] Hertz, Amir, et al. \"Prompt-to-prompt image editing with cross attention control.\" arXiv preprint arXiv:2208.01626 (2022).\n\nLIMITATIONS:\nThe paper proposes a way to manipulate attention scores to explain prediction of Transformer models. Manipulating these scores is done a token at a time, leading to inference speed that scales with the sequence length, and is orders of magnitude bigger than previous approaches. Although a lot of experiments are presented, some of the design choices are not adequately justified.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a new resource-friendly modality-agnostic perturbation method named AtMan to explain the predictions of generative transformers. AtMan suppresses or amplifies the attention entry relying on the cosine similarity neighborhood to evaluate the influence of each token, and it saves many memory resources compared with current gradient-based methods, which means that it's more suitable for large model inference developments. Empirical results show that AtMan can do language/visual reasoning better than the current state-of-the-art gradient-based methods.\n\nSTRENGTHS:\n1. AtMan is simple but useful, and the idea that changing perturbation space from the raw input to the embedding space is important. It can perform on decoder as well as encoder architectures and the empirical results are supportive.\n2. The core contribution of AtMan is that it reduce the memory cost and can be applied to the large models, which will be important for the future works of explainability of large models. Now it can be applied to 30B multi-modal model but previous methods fail.\n3. The paper is well written and easy to follow.\n\nWEAKNESSES:\n1. The paper mentions that the $\\kappa$ parameter in AtMan is set to be $0.7$, but it may be important for the performance of AtMan, so it is better to do additional experiments to show the effect of different kappa on the experimental results on different datasets. Furthermore, the choice of (6) is not unique, there are some other ways like using the softmax value of $s_{i,k}$ to be $f_{k,*}^i$ and using temperature as a hyper-parameter. It's better to compare these ways.\n2. The increase in the time cost of AtMan is significant compared with gradient-based methods, so it's better to show the time-cost figure after pipeline-parallel execution.\n3. The results of the locality are important, maybe the authors can try to add them (like figure 10 in Appendix) into the main paper.\n\nQUESTIONS:\nSimilar to Weakness-1 and Weakness-2.\n\nLIMITATIONS:\nThe author mentions that their work provides reference methods for future work, including explanatory studies of current generative models impacting our society. The author doesn't mention other potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed ATMAN, a modality and architecture-agnostic perturbation-based XAI method for generative transformer models. Specifically, ATMAN evaluates the sensitivity of model prediction w.r.t. the perturbation of attention scores on a set of cosine-distance neighboring tokens. Empirical results suggest that ATMAN can effectively identify the relevant input tokens for target prediction, and outperforms the baseline methods in terms of precision and recall.\n\nSTRENGTHS:\nAs far as I know, this paper is the first work that identifies the crucial tokens/pixels by attention perturbation. The idea is novel and looks interesting to me. The authors further apply the cosine similarity to collect the related neighboring tokens and reduce the searching redundancy. It helps us to understand how transformer models process input signals and identify crucial ones.\n\nWEAKNESSES:\n1. As perturbing every token to evaluate their sensitivity, ATMAN can incur high time complexity to generate the explanation for transformer models. As suggested by Figure 5, the runtime is much higher than the baseline. Therefore, to reduce the time complexity, would there be any solution to evaluate the token/pixel sensitivity in a higher granularity, like dividing the tokens/pixels into clusters? \n2. The paper mentioned steering model prediction by applying the attention scores of identified tokens but I did not find any empirical results to claim the advantages of this approach. Could you provide more details and results or correct me if I am wrong?\n\nQUESTIONS:\nCan you provide more model details about Section 3.1? Which black-box model do you evaluate?\n\nLIMITATIONS:\nThe limitation is the high runtime cost when evaluating every input token/pixel. It would be great if the authors can extend the analysis into higher granularity.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposed a new permutation method for Transformer explanation, named ATMAN, which directly manipulates attention scores in the hidden layers of the Transformer model rather than permuting the model inputs. Importantly, the authors established a bridge between ATMAN and the relevance between inputs and output predictions. Experimental results demonstrate the effectiveness and efficiency of ATMAN on interpreting Transformer model on different types of data and different model scales.

STRENGTHS:
The proposed ATMAN method is novel, and the experimental results shows that ATMAN significantly improve the effectiveness and efficiency over existing perturbation and propagation methods on interpreting Transformer model on different types of data and different model scales.

WEAKNESSES:
NA.

QUESTIONS:
NA.

LIMITATIONS:
NA.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes atman, a method for understanding the predictions of transformers by masking out the tokenized inputs. The paper evaluates atman on explainability tasks in text pipelines and vision pipelines. The authors argue that atman is superior to gradient-based explanation methods since it uses less memory.\n\nSTRENGTHS:\nThe paper is well-written and the contributions are easily understood. The method is simple to implement and understand, and does take less memory footprint than gradient-based methods. The evaluation on explainability benchmarks seem to show that it behaves at par with gradient-based methods. The figures make the method very easy to understand.\n\nWEAKNESSES:\nThe novelty of the method seems questionable, and it doesn't seem necessary to create an entire \"method\" with a name to describe the observation that masking out input tokens can be useful in tracing which parts of the input sequence are important. This type of analysis is standard in analyzing the outputs of Transformers, e.g. it is a standard technique in Olsson et al 2022 (https://arxiv.org/abs/2209.11895). To claim that it is a novel method is a bit much in terms of framing.\n\nThere may be some novelty in the vision domain in looking at the correlated outputs instead of masking individual patches of the input image, but as far as I can tell this methodological change is not ablated.\n\nThese novelty questions point to a weakness in the evaluation and the discussion of related work. I don't have any problem with a simple method (in fact I like it) - but if the method is a first standard trick that most ML engineers would use to understand what is happening when a Transformer makes predictions, then it suggests a different framing and evaluation.\n\nFor example: What other \"surprisingly simple\" interventions can be used to explain predictions? Can you look at embeddings of tokens in the final output? What are the differences (in NLP) between generative-models (BERT) and encoder-only models (GPT)? How are the explanations for vision affected when the correlations are not taken into account? What happens if this intervention is done partway through the backbone instead of at the beginning?\n\nQUESTIONS:\nSuggested list of questions to improve the work:\n1. What other \"surprisingly simple\" interventions can be used to explain predictions?\n2. Can you look at embeddings of tokens in the final output?\n3. What are the differences (in NLP) between generative-models (BERT) and encoder-only models (GPT)?\n4. How are the explanations for vision affected when the correlations are not taken into account?\n5. What happens if this intervention is done partway through the backbone instead of at the beginning?\n\nLIMITATIONS:\nN/a", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}