SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widetilde {\mathbf {H}} _ {u, *, *} = \mathbf {H} _ {u, *, *} \circ \left(\left(\mathbf {1} - f\right) + f \left(\mathbf {1} - \mathbf {f} _ {k, *} ^ {i}\right)\right), \text {w i t h} \mathbf {f} _ {k, *} ^ {i} = \left\{ \begin{array}{l l} s _ {i, k}, & \text {i f} \kappa \leq s _ {i, k} \leq 1, \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {6}
$$
In words, we interpolate linear between the given suppression factor  $1 - f$  and 1, by factor of the thresholded cosine similarity  $\mathbf{f}_{k,\star}^{i}$ . As we only want to suppress tokens, we restrict the factor value range to greater than 0 and smaller than 1. The parameter  $\kappa > 0^2$  is to ensure a lower bound on the similarity, and in particular, prevents a flip of the sign. Overall, we found this to be the most intuitive and robust equation. In Fig. 11 we ran a sweep over the hyper-parameters to obtain the best precision and recall values. Note in particular, that our modifications are rather subtle compared to entire token-masking. The latter did not lead to satisfying results, even given the context. We compute the similarity scores once on the initial embedding layer, and uniformly apply those modifications throughout all layers.
Note that variations of Eq. 4, 5 and 6 could also lead to promising results. In particular in the active research field of vision transformers, hierarchical token transformations could be applicable and speed
2We empirically fixed  $\kappa = 0.7$  through a parameter sweep (c.f. Appendix A.4).
Figure 4: ATMAN produces less noisy and more focused explanations when prompted with multi-class weak segmentation compared to Chefer. The three shown figures are prompted to explain the target classes above and below separately. (Best viewed in color.)
up computations (c.f. [3], Fig 16). Furthermore, preliminary experiments using the similarity between the query and key values per layer, a different similarity score such as other embedding models or directly sigmoid values, or even a non-uniform application of the manipulations yield promising, yet degenerated performance (c.f. App. A.15). As we could not conclude a final conclusion yet, we leave these investigations to future research.
With that, we arrived at our final version of ATMAN. Note that this form of explanation  $\mathcal{E}$  is "local", as target refers to our target class. We can, however, straightforwardly derive a "global explanation" by setting  $target = y$ , for  $y$ , a model completion to input  $w$  of a certain length. It could then be interpreted rather abstract as the model's general focus [4].
4 Empirical Evaluation
We ran empirical evaluations on text and image corpora to address the following questions: (Q1) Does ATMAN achieve competitive results compared to previous XAI methods for transformers in the language as well as vision domain? (Q2) Is ATMAN applicable to various transformer models, and does ATMAN scale efficiently, and can therefore be applied to current large-scale AR models?
To answer these questions, we conducted empirical studies on textual and visual XAI benchmarks and compared ATMAN to standard approaches such as IxG [26], IG [27], GradCAM [25] and the transformer-specific XAI method of [6] called Chefer in the following. Note that all these methods utilize gradients and, therefore, categorize as propagation methods leading to memory inefficiency. We also tried to apply existing perturbation methods such as LIME [24] and SHAP [19] and Attention Rollout [1]. However, they failed due to extremely large trials and, in turn, prohibitive computation time, or qualitatively (c.f. Appendix A.9). We adopt common metrics, namely mean average precision (mAP) and recall (mAR) (c.f. Appendix A.11), and state their interquartile statistics in all experiments. To provide a comparison between the XAI methods, we benchmarked all methods on GPT-J [29] for language and MAGMA-6B [10] for vision-language tasks. Through its memory efficiency, ATMAN can be deployed on really large models. We further evaluate ATMAN on a MAGMA-13B and 30B to show the scaling behavior, and, on BLIP [18] to show the architecture independence. To obtain the most balanced comparison between the methods, we selected datasets focusing on pure information recall, separating the XAI evaluation from the underlying model interpretations.
4.1 ATMAN can do language reasoning
Protocol. All experiments were performed on the open-source model GPT-J [29]. Since with ATMAN we aim to study large-scale generative models, we formulate XAI on generative tasks as described in Sec. 3.3. For evaluation, we used the Stanford Question Answering (QA) Dataset (SQuAD) [23]. The QA dataset is structured as follows: Given a single paragraph of information, there are multiple questions, each with a corresponding answer referring to a position in the given paragraph. A visualization of an instance of this dataset can be found in Fig. 3b. SQuAD contains 536 unique paragraphs and 107,785 question/explanation pairs. The average context sequence length is 152.7 tokens and the average label length is 3.4.
The model was prompted with the template: "\{Context\} Q: \{Question\} A:" and the explainability methods evaluated on the generations to match the tokens inside the given context, c.f. Fig. 3b, Sec. 3.3. If there were multiple tokens in the target label, we computed the mean of the generated scores.
Table 1: ATMAN (AM) outperforms other XAI methods on (a) the text QA benchmark SQuAD and (b) the image-text VQA task of OpenImages. Shown are the (interquartile) mean average precision and recall (the higher, the better). The best and second best values are highlighted with  $\bullet$  and  $\circ$ . Evaluations are on a 6B model. (c) ATMAN evaluated on other model types: MAGMA variants (indicated by their parameter size) and BLIP.
(a) SQuAD.
IG	IxG	Chef.	AM
mAP	49.5	51.7	○72.7	●73.7
\( mAP_{IQ} \)	49.5	61.4	○77.5	●81.8
mAR	87.1	91.8	●96.6	○93.4
\( mAR_{IQ} \)	98.6	●100	●100	●100
(b) OpenImages (OI).
GC	IxG	IG	Chef.	AM
55.2	59.1	○60.5	58.3	●65.5
54.3	59.0	○60.6	57.8	●70.2
4.2	10.8	10.7	○11.7	●15.7
3.8	10.2	10.0	○11.1	●19.7
(c) Other models on OI.
AM13B	AM30B	AMBLIP
64.8	63.6	64.6
65.6	65.1	66.1
14.6	13.6	26.4
16.7	15.6	28.4
Similar to weak segmentation tasks in computer vision, we regarded the annotated explanations as binary labels and determined precision and recall over all these target tokens.
Results. The results are shown in Tab. 1a. It can be observed that the proposed ATMAN method thoroughly outperforms all previous approaches on the mean average precision. This statement holds as well for the interquartile mean of the recall. However on the entire recall, Chefer slightly outperforms ATMAN. Note that the high values for recall scores throughout all methods are partly due to the small average explanation length, such as depicted in Fig. 3b. Further details and some qualitative examples can be found in Appendix A.2.
Paragraph chunking. ATMAN can naturally be lifted to the explanation of paragraphs. We ran experiments for ATMAN dividing the input text into several paragraphs by using common delimiters as separation points and evaluating the resulting chunks at once, in contrast to token-wise evaluations. This significantly decreases the total number of required forward passes and, on top, produces "more human" text explanations of the otherwise still heterogeneously highlighted word parts. Results are shown in Appendix A.8.
4.2 ATMAN can do visual reasoning
Protocol. Similar to language reasoning, we again perform XAI on generative models. We evaluated the OpenImages [16] dataset as a VQA task and generated open-vocabulary prediction with the autoregressive model. Specifically, the model is prompted with the template: “{Image} This is a picture of”, and the explainability methods executed to derive scores for the pixels of the image on the generations with respect to the target tokens. If there were multiple tokens in the target label, we computed the mean of the generated scores. For evaluation, we considered the segmentation annotations of the dataset as ground truth explanations. The segmentation subset contains 2,7M annotated images for 350 different classes. In order to ensure a good performance of the large-scale model at hand and, in turn, adequately evaluate only the XAI methods, we filtered the images for a minimum dimension of 200 pixels and a maximal proportional deviation between width and height of  $20\%$ . We randomly sample 200 images per class on the filtered set to further reduce evaluation costs. In total, this leads to a dataset of 27,871 samples. The average context sequence length is 144 tokens, and the average label coverage is  $56\%$  of the input image. To provide a fair comparison between all XAI methods, we compute token-based attributions for both ATMAN and Chefer, as well as IG, IxG, and GC. Note that the latter three allow pixel-based attribution granularity, however, these resulted in lower performance, as further discussed in Appendix A.2.
Results. The results are shown in Tab. lb. It can be observed that ATMAN thoroughly outperforms all other XAI approaches on the visual reasoning task for all metrics. Note how explicit transformer XAI methods (ATMAN, Chefer) in particular outperform generic methods (GradCAM, IG, IxG) in recall. Moreover, while being memory-efficient (c.f. next section), ATMAN also generates more accurate explanations compared to Chefer. Through the memory efficiency of ATMAN, we were able to evaluate a 13B and 30B upscaled variant of MAGMA (c.f. Tab. lc). Interestingly, the general explanation performance slightly decreases, when upscaled, compared to the 6B model variant. However, throughout all variants ATMAN scored comparable high. Note that the overall benchmark performances of the models slightly increase (c.f. App. A.14). This behavior could be attributed to
Model size (billion parameters)
Figure 5: ATMAN scales efficiently. Performance comparison of the XAI methods ATMAN and Chefer et al., on various model sizes (x-axis) executed on a single 80GB memory GPU. Current gradient-based approaches do not scale; only ATMAN can be utilized on large-scale models. Solid lines refer to the GPU memory consumption in GB (left y-axis). Dashed lines refer to the runtime in seconds (right y-axis). Colors indicate experiments on varying input sequence lengths. As baseline (green) a plain forward pass with a sequence length of 1024 is measured. The lightning symbol emphasizes the non-deployability when memory resource is capped to 80GB. (Best viewed in color.)
the increased complexity of the model and, subsequently, the complexity of the explanation at hand. Hence, the "human alignment" with the model's explanations is not expected to scale with their size.
Model agnosticism. Note that the underlying MAGMA model introduces adapters on a pre-trained decoder text model to obtain the image multi-modality. It is yet unclear to what extent adapters change the flow of information in the underlying model. This may be a reason for the degenerated performance of gradient-based models. As ATMAN however outperforms such methods, we may conclude that information processing still remains bound to the specific input-sequence position. This finding generalizes to encoder architectures, as experiments with BLIP suggest in Tab. 1c. Precisely, BLIP's multi-modality is achieved by combining an image-encoder, a text-encoder, and a final text-decoder model. These are responsible for encoding the image, the question, and finally for generating a response separately. Information between the models is passed through cross-attention. We only applied ATMAN to the image-encoder model of this architecture (c.f. App. A.10). The results demonstrate the robustness of the suggested perturbation method over gradient-based approaches.
Qualitative illustration. Fig. 4 shows several generated image explanations of ATMAN and Chefer for different concepts. More examples of all methods can be found in Appendix A.7 and evaluation on the more complex GQA dataset [12] is discussed in Appendix A.12. We generally observe more noise in gradient-based methods, in particular around the edges. Note that ATMAN evaluates the change in its perturbed generations. These are independent of the target tokens. Compared to other methods like Chefer, we, therefore, do not need to process the prompt more than once for VQA on different object classes with ATMAN. Note again that the aforementioned cosine similarity is crucial to obtain these scores. Quantitative results can be found in App. A.4.
In general, the results clearly provide an affirmative answer to (Q1): ATMAN is competitive with previous XAI methods, and even outperforms transformer-specific ones on the evaluated language and vision benchmarks for precision and recall. Next, we will analyze the computational efficiency of ATMAN.
4.3 ATMAN can do large scale
While ATMAN shows competitive performance, it computes, unlike previous approaches, explanations at almost no extra memory cost. Fig. 5 illustrates the runtime and memory consumption on a single NVIDIA A100 80GB GPU. We evaluated the gradient-based transformer XAI method [6] and ATMAN. The statistics vary in sequence lengths (colors) from 128 to 1024 tokens, and all experiments are executed sequentially with batch size 1 for best comparison.
One can observe that the memory consumption of ATMAN is around that of a plain forward pass (Baseline; green) and increases only marginally over the sequence lengths. In comparison, the method of [6]--and other gradient-based methods--exceeds the memory limit with more than double in memory consumption. Therefore, they fail on larger sequence lengths or models.
Figure 6: Exemplified average runtime and total memory consumption for the task of explaining an image, comparing AtMan with Chefer et al.. We applied memory improvements on Chefer et al., pipe-parallel of 4 on both and additionally batch-size 16 on ATMan. Chefer et al. still is not applicable to 34b models when the total memory consumption is capped to 80GB. AtMan's increase in total computational time lies around 1 magnitude. Note that AtMan could be computed entirely parallel, and as such, this total clock time can still be reduced to subseconds. Measured are 144 explanation tokens (full given image on MAGMA), for an average total prompt length of 165 tokens.
Whereas the memory consumption of ATMAN stays is almost constant, the execution time significantly increases over sequence length when no further token aggregation is applied upfront. Note, that Fig. 5 shows complete and sequential execution. The exhaustive search loop of ATMAN can be run partially and in parallel to decrease its runtime. In particular, the parallelism can be achieved by increasing the batch size and naturally by a pipeline-parallel execution. For instance, since large models beyond 100B are scattered among nodes and thus many GPUs, the effective runtime is reduced by magnitudes to a proximate scale of the forward pass. Furthermore, tokens can be aggregated in chunks and evaluated on a more coarse level, which further reduces the runtime drastically, as demonstrated in Fig. 16.
In Fig. 6 we provide performance results on a realistic scenario. It compares a practical query (average 165 tokens) on the task of explaining a complete image (144) tokens. Runtimes are measured with pipe-parallel 4, and ATMAN additionally with batch size 16. Specifically, ATMAN requires between 1 and 3 total compute seconds, around 1 magnitude longer compared to Chefer. Note that this can still further be divided by the number of available idling workers, to reduce the absolute clock time to subseconds. Each batch of ATMAN can be processed entirely parallel. Moreover Chefer, even with memory optimizations, fails to scale to 34b with the given memory limit (80GB).
Overall, these results clearly provide an affirmative answer to (Q2): Through the memory efficiency of ATMAN, it can be applied to large-scale transformer-based models.
We proposed ATMAN, a modality and architecture agnostic perturbation-based XAI method for generative transformer networks. In particular, ATMAN reduces the complex issue of finding proper input perturbations to a single scaling factor per token. As our experiments demonstrate, ATMAN outperforms current approaches relying on gradient computation. It moreover performs well on decoder as well as encoder architectures, even with further adjustments like adapters or crossattention in place. Most importantly, through ATMAN's memory efficiency, it provides the first XAI tool for deployed large-scale AR transformers. We successfully evaluated a 30B multi-modal model on a large text-image corpus.
Consequently, ATMAN paves the way to evaluate whether models' explanatory capabilities scale with their size, which should be studied further. Besides this, our paper provides several avenues for future work, including explanatory studies of current generative models impacting our society. Furthermore, it could lay the foundation for not only instructing and, in turn, improving the predictive outcome of autoregressive models based on human feedback [21] but also their explanations [11].
We gratefully acknowledge support by the German Center for Artificial Intelligence (DFKI) project "SAINT", the Federal Ministry of Education and Research (BMBF) project "AISC" (GA No. 01IS22091), and the Hessian Ministry for Digital Strategy and Development (HMinD) project "AI Innovationlab" (GA No. S-DIW04/0013/003). This work also benefited from the ICT-48 Network
of AI Research Excellence Center “TAILOR” (EU Horizon 2020, GA No 952215), the Hessian Ministry of Higher Education, and the Research and the Arts (HMWK) cluster projects “The Adaptive Mind” and “The Third Wave of AI”, and the HMWK and BMBF ATHENE project “AVSV”. Further, we thank Felix Friedrich, Marco Bellagente and Constantin Eichenberg for their valuable feedback.
[31] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
A.1 Remarks on executed benchmarks
We executed all benchmarks faithfully and to the best of our knowledge. The selection of compared methods was made to be rather diverse and obtain a good overview in this field of research. In particular, with regards to the multi-modal transformer scaling behavior, as there are in fact no such studies for AR models yet to compare to. It is possible, for all methods, that there are still improvements we missed in quality as well as performance. However, we see the optimizations of other methods to multi-modal AR transformer models as a research direction on its own.
Chefer. The integration of Chefer was straightforward. As it can be derived by the visualizations, there are noticeable artifacts, particularly on the edges of images. In this work the underlying transformer model was MAGMA, which is finetuned using sequential adapters. It is possible that this, or the multi-modal AR nature itself, is the cause for these artifacts. We did not further investigate to what extent the adapters are to be particularly integrated in the attribute accumulation of Chefer. Also notice that ATMAN often has similar, but not as severe, artifacts.
IxG, IG and GradCAM. The methods IxG, IG, and (guided) GradCAM failed completely from the quality perspective in recall when applied to pixel granularity (c.f. Tab. 2). Those are the only ones directly applicable to the image domain, and thus also included the vision encoder in the backward pass. We did not further investigate or fine-tune evaluations to any method. For better comparison we therefore only present the token-based comparisons. All methods are evaluated with the same metrics and therewith give us a reasonable performance comparison without additional customization or configuration.
Details on Results. For a fair comparison, all experiments were executed on a single GPU, as scaling naturally extends all methods. We also want to highlight that we did not optimize the methods for performance further but rather adopted the repositories as they were. The memory inefficiency of gradient-based methods arises from the backward pass. A maximal memory performant representative is the Single-Layer-Attribution method IxG, which only computes partial derivatives on the input with respect to the loss. Even this approach increases the memory requirement beyond an additional  $50\%$  and fails for the scaling experiments up to 34B.
In Fig. 5 we ran Chefer with a full backward pass. We adopted this to the minimum amount of gradients (we saw) possible and plot the full scaling benchmark below in Fig  $7^{3}$ . The key message remains the same. With the given IxG argument, we do not see much potential in improving memory consumption further.
The methods IxG, IG and GradCam are integrated using the library Captum [15]. We expect them to be implemented as performant as possible. IntegratedGradients is a perturbation method on the input, integrating changes over the gradients. The implementation at hand vastly runs OOM. Finally GradCam is a method specialized on CNN networks and therefore does not work for text only (or varying sequence lengths). It requires the least amount of resources but also produces poor results, without further investigations.
ATMAN Parallelizability. As a final remark, we want to recall again that the runtime measured in the sequential execution can be drastically reduced due to its parallelizability, i.p., as it only requires forward passes. For sequence length 1024, we measured 1024 iterations in order to explain each token. However note that ATMAN can also be applied to only parts or chunks of the sequence (c.f. Sec. A.8), in contrast to gradient methods. Moreover, all tokens to explain can be computed entirely in parallel. In a cluster deployment, these can be distributed amongst all available workers. On top, it can be divided by the available batch size and true pipeline-parallelism (c.f. Fig. 6).
${}^{3}$  Setting requires_grad  $=$  False to every but the attention tensors.
Figure 7: Performance comparison of the explainability methods over various model sizes (x-axis) executed on a single 80GB memory GPU, with fixed batch size 1. Solid lines refer to the GPU memory consumption in GB (left y-axis). Dashed lines refer to the runtime in seconds (right y-axis). Colors indicate experiments on varying input sequence lengths. As baseline (green) a plain forward pass with a sequence length of 1024 is measured. Note that GradCAM can only be applied to the vision domain, it is therefore fixed to 144 tokens. Note that it already consumes as much memory as a forward pass of 1024 tokens. (Best viewed in color.)
(a) Histogram of explanation token length.
Figure 8: Further evaluations on the SQuAD dataset. (Best viewed in color.)
(b) mAP for all methods, grouped by number of question/ answer pairs in the dataset.
A.2 Detailed SQuAD Evaluations
This sections gives more detailed statistics on the scores presented in Tab. 1. First Fig. 8a is the histogram of the token lengths of all explanations. Fig. 8b is the mAP score for all methods on the entire dataset, grouped by the number of questions occurring per instance.
A.3 Detailed OpenImages Evaluations
This section gives more detailed statistics on the scores presented in Tab. 1. Fig. 9 is the histogram of the fraction of label coverage on all images. Fig. 10a and 10b are boxplots for all methods on the entire dataset, for mean average precision as well as recall. Note that the methods IG, IxG and GradCam can be evaluated on image-pixel level, contrary to ATMAN and Chefer. However their performances (i.p. wrt. recall) significantly degenerate, c.f. Tab. 2, Fig. 15.
Figure 9: Histogram of percentage of label coverage of the images.
(a) mAP Boxplot for all methods of all images.
Figure 10: Further evaluations on the OpenImages dataset. (Best viewed in color.)
(b) mAR Boxplot for all methods of all images.
A.4 Discussion of Cosine Embedding Similarity and Parameter values