SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We fixed the parameter  $\kappa = 0.7$  of Eq. 6 and  $f = 0.9$  of Eq. 4 throughout this work. They were empirically concluded by running a line sweep on a randomly sampled subset of the OpenImages dataset once, c.f. Fig. 11. Note that pure token-masking ("mask") always leads to worse performance compared to our more subtle concept-suppression operations. We use the same parameters throughout this work, however, parameters may be estimizable depending on the use case and dataset.
In Fig. 12a and 12b we compare the mean average precision and recall scores for OpenImages for both variants, with and without correlated token suppression (to threshold  $\kappa$ ). Clearly the latter outperforms single token suppression.
The following Fig. 13 shows visually the effect on weak image segmentation when correlated suppression of tokens is activated, or when using single token suppression only. Notice how single token only occasionally hits the label, and often marks a token at the edge. This gives us a reason to believe that entropy is accumulated around such edges during layer wise processing. This effect (on these images) completely vanishes with correlated suppression of tokens.
A.5 Variation Discussion of the method
Note that the results of Eq. 4 are directly passed to a softmax operation. The softmax of a vector  $\mathbf{z}$  is defined as
$$
\operatorname {s o f t m a x} \left(\mathbf {z}\right) _ {i} = e ^ {z _ {i}} / \sum_ {j} e ^ {z _ {j}}.
$$
In particular, the entries  $z_{k} = 0$  and  $z_{l} = -\infty$  will yield to the results  $\mathrm{softmax}(\mathbf{z})_k = 1 / \sum_j e^{z_j}$  and  $\mathrm{softmax}(\mathbf{z})_l = 0$ . So one might argue as follows: If we intend to suppress the entropy of token  $i$ , we do not want to multiply it by a factor 0, but rather subtract  $-\infty$  of it. I.e. we propose the modification
$$
(H) _ {u, *, *} = (H) _ {u, *, *} + \log (f). \tag {7}
$$
The only problem with this Eq. 7 is, that it skews the cosine neighborhood factors. While we experienced this working more naturally in principle, for hand-crafted factors, we did not get best
Figure 11: Parameter sweep for Eq. 6. It can be observed that indeed cosine similarity (y-axis) along with more subtle modification (x-axis) outperforms other configurations, i.p. row "mask" that entirely masks single tokens (i.e. multiplies  $1 - f$ ). This result indicates that indeed conceptual entropy is scattered across tokens, that need to be suppressed at once.
(b)
Figure 12: Histogram metric evaluation on OpenImages for ATMAN with (green) and without (orange) correlated suppression of tokens. (Best viewed in color.)
performance in combination with Eq. 6. In the following Fig. 14a and 14b, we show analogous evaluations to Fig. 12a and 12b. It is in particular interesting that the mode without correlated tokens slightly improves, while the one with slightly decreases in scores, for both metrics.
A.6 Artifacts and failure modes
Merged into A.12 (kept here for reference).
A.7 Qualitative comparison weak image segmentation
In the following Fig. 15 we give several examples for better comparison between the methods on the task of weak image segmentation. To generate the explanations, we prompt the model with “<Image> This is a picture of ” and extract the scores towards the next target tokens as described with Eq. 5 for ATMAN. For multiple target tokens, these results are averaged. In the same fashion, but with an additional backpropagation towards the next target token, we derive the explanations for Chefer and the other gradient methods.
A.8 Application to document q/a
In Fig. 16 we apply ATMAN on a larger context of around 500 tokens paragraph wise. The Context is first split into chunks by the delimiter tokens of “,”, “,”, “\n” and “and”. Then iteratively each chunk is evaluated by prompting in the fashion “{Context} Q:{Question} A:” and the cross entropy extracted towards the target tokens, suppressing the entire chunk at once, as described in Sec. 3. It can be observed that the correct paragraphs are highlighted for the given questions and expected targets.
Table 2: OpenImages (OI) evaluations on pixel granularity.
IxG	IG	GC
mAP	38.0	46.1	56.7
\( mAP_{IQ} \)	34.1	45.2	60.4
mAR	0.2	0.3	0.1
\( mAR_{IQ} \)	0.1	0.1	0.1
Figure 13: Example images showing the effect of correlated suppression as described in Correlated Token Attention Manipulation Sec. 3.3 and Single Token Attention Manipulation Sec. 3.2. (Best viewed in color.)
In particular, one can observe the models interpretation, like the mapping of formats or of states to countries. Note in particular that it is not fooled by questions not answered by the text (last row).
A.9 Attention Rollout, Flow, other Perturbation method
Attention Flow constructs a graph representing the flow of attention through the model. Unfortunately this approach is computationally too intense for evaluation on large language models; Chefer et al. (2021) already ran into issues on smaller BERT models. LIME [24] and SHAP [19] are perturbation methods closely related to ATMAN. However they result in several 100.000 forward trials which is impractical for use with large language models.
Finally, we ran additional experiments for Attention Rollout [1]. Note that the authors stress its design for encoder classification architectures, while we explore the applicability to generative decoder models in this work. Further, Rollout is by default class-agnostic, in contrast to ATMAN or the studied gradient methods. We adopted Attention Rollout by appending the target to the prompt, and only aggregating attentions over those last target sequence positions. Note that in contrast to the other studied methods, Rollout disregards the output distribution.
We observed large artifacts on the explanations at the positions of the last 2 image patches; apparently most attention aggregation take place at those positions, which may be due to the causal mask. On OpenImages our adoption of Rollout achieves a  $mAP$  score of 43.6 and a  $mAR$  of 1.3 (c.f. AtMan with 65.5 and 13.7). Scores do increase when removing the last image patch, however, this also entirely erases explanation on this patch. On Squad we achieved  $mAP$  of 23.1 and a  $mAR$  of 53.4 (AtMan: 73.7, 93.4).
(b)
Figure 14: Histogram of the evaluation metrics on OpenImages for ATMAN with and without correlated token suppression. (Best viewed in color.)
Table 3: Evaluation of XAI on BLIP model.
Chefer	ATMAN
mAP	59.7	64.6
\( \mathrm{mAP}_{IQ} \)	60.3	66.1
mAR	20.2	26.4
\( \mathrm{mAR}_{IQ} \)	19.4	28.4
A.10 Demarcation to other architectures, BLIP experiments
To answer the question of "why we think that AtMan could be generalized to other (generative) multimodal Transformers", consider the following arguments.
We conduct for the first time experiments on generative decoder models. The underlying MAGMA model extends the standard decoder transformer architecture, having causal attention masks in place, by additional sequential adapters that could potentially shift distributions. However, the proposed attention manipulation still leads to a robust perturbation method — in text-only as well as image-text modality. We thus conjecture that the skip connections lead to a somewhat consistent entropy flow throughout the network. OFA (< 1B) and BLIP rely on standard encoder-decoder architectures — even without adapters — and the same skip connections. Furthermore, both models process the input image in a similar fashion to obtain “image tokens” forwarded throughout the transformer at the same positions.
BLIP, in contrast to OFA, separates the processing of the image, the prompt, and the answer generation into 3 different models—standard encoders, only the answer generation model being a decoder. The different outputs are cross-attended. We apply ATMAN only on the vision-encoder model.
For comparison, we furthermore evaluate Chefer on BLIP, its attention aggregation being only applied to the vision encoder as well. Tab. 3 compares these two methods, showing that ATMAN still outperforms Chefer. Note, however, that Chefer achieves better scores compared to being applied to the MAGMA model.
On language models, we found that "chat"-finetuned-versions are more applicable to this method, most likely as they are specifically trained to respect the given input. In general, prompt-tuning instructions may improve the results, such as "Answer with respect to the given context".
A.11 Metrics
All recall and precision metrics in this work are based on the normalized continuous values as follows: For relevancy scores  $x$  and binary segmentation label  $y$ , let  $\widetilde{x} = x / \max_i(x_i)$ . We compute  $AP = (\sum_i x_i y_i) / \sum_i x_i$  for precision and for recall  $AR = (\sum_i \widetilde{x}_i y_i) / \sum_i y_i$ . The final mean scores
Figure 15: Weak image segmentation comparison of several images for all methods studied in this work. Note that IG and IxG are presented in token (left) and pixel-granularity (right). (Best viewed in color.)
are the average of all samples. This yields a comparable untuned metric also catching the robustness of the methods.
A.12 Interpretability of "complex facts" and failure modes
All experiments were executed in the generative setting, c.f. Eq. 1,2 and the remark to "global explanations" at the end of Sec. 3. Fig. 1a shows "real text generation" and the accumulation of explanations on the input image during the model's generation process. For comparison benchmarks,
Figure 16: Showing ATMAN capabilities to highlight information in a document q/a setting. The model is prompted with “{Context} Q:{Question} A:” and asked to extract the answer (target) of the given Explanation. Here, ATMAN is run paragraph wise, as described in text, and correctly highlights the ones containing the information. All Explanations were split into  $\sim 50$  paragraphs (thus requiring only 50 ATMAN forad-passes). In particular it is shown in row 2 that the model can interpret, i.e. convert date-time formats. Row 3 shows that it can derive from world knowledge that Michiganian is in the US. Row 4 shows that the method ATMAN is robust against questions with non-including information. (Best viewed in color.)
Table 4: Performance of scaled MAGMA models.
6b	13b	30b
VQA	60.0	62.6	64.2
OKVQA	37.6	38.2	43.3
GQA	47.4	43.7	45.6
our target was to separate generated explanations from the underlying model's interpretation, by retrieving "obvious facts". Fig. 4 demonstrates qualitatively target-class awareness.
More complex results are shown in Fig. 17. Fig. 17a gives an example where the model is VQA prompted and ATMAN explains both modalities at the same time for the completion token "white". It highlights the words "tub" and "color" of the text, and an edge of the tub in the picture as significant to produce the completion. Note that we find these "joint-modal explanations" to work best when the target token is not just a yes/no answer, c.f. Fig. a'). Further note how these artifacts remain amongst explainability methods such as Chefer, which may be due to the underlying models architecture. We leave these investigations for further research.
In Fig. 17b ATMAN highlights multiple heterogenous concepts to answer the counting task. Note that here the explanation for the completion "two", which actually highlights visually the animals, is found at the token position "animals", and not at the position of the predicted token. ATMAN returns at the same time explanations to all previous context positions, without additional forward passes (in contrast to gradient-based methods).
Finally, we ran evaluations on visual reasoning using the GQA dataset 17. Fig. 17c shows an instance of GQA, in which the model is prompted to answer "Does the man ride a horse?", while the man in the picture actually rides a bike. ATMAN highlights at two different sequence positions of the prompt the respective concepts in the image required to answer the question, once the man, and once the bike. Fig. 17d.1 and d.2 are the resulting IG explanations when applied to explain the same token positions; somewhat it captures parts of the concept but focuses on noise to the edge. Note that many samples of this dataset are rather difficult to answer, it is often not clear on what token position an explanation is supposed to be found. In particular, often ambiguous concepts occur multiple times in the image and are further restricted to the correct instances much later in the text description. This is, in particular, difficult to grasp for generative decoder models due to the causal masking. As the dataset contains annotations for areas as in example Fig. 17c, we ran preliminary quantitative results on 10k randomly chosen instances. ATMAN matches bounding boxes with  $mAP$  52.4 and  $mAR$  24.3. For IG we obtained  $mAP$  33.3 and  $mAR$  27.0.
A.13 Partial attention manipulation
We apply the same manipulation in each layer. We hypothesised—and first experiments indicate—that one needs a "critical mass", to remove the entire concept entropy from the generation process. This is i.p. due to the skip connections. Only applying it to a single (arbitrary) layer did not yield comparable results. However we did not finalize a conclusion yet and leave this to future work. C.f. A.15, Fig. 18.
A.14 MAGMA scaling
MAGMA Model performances on standard text-image evaluation benchmarks are shown in Tab. 4.
A.15 Variants of ATMAN and final remarks
There are many promising variants of ATMAN not yet fully understood. In particular finding the relevant layers and other formulae to manipulate reliably the activations, or to normalize the output scores, could yield further insights into the transformer architecture. We showed how conceptual suppression improves overall performance. For pure NLP tasks, utilizing an embedding model such as BERT to compute similarities could be beneficial. Moreover, it should be possible to aggregate tokens and increase the level of details as needed, to save additional computational time.
a) What is the color of the tub? White
c.1) Q: Does the man
b)Q: How many animals are in this picture? A: two.
c.2) Q: Does the man ride a horse? A: No.
d.2)
a'.1) AtMan
Q: Is the bath tub white in color?
A:yes
a'.2) Chefer
Q: Is the bath tub white in color?
A:yes
Figure 17: More complex examples showing generative multi-modal explanations (c.f. Sec. A.12 for detailed description). italic: model completion; underline: token position for explanation. (Best viewed in color.)
Figure 18: Measured mAP and mAR when applying AtMan only to a single layer, with similarity of the first embedding layer. Evaluated on a subset of openimages. Apparently, some layers contribute more to the explanation than others. "Full AtMan" as proposed yet outperforms on both metrics with scores  $\mathrm{mAP} = 0.66$  and  $\mathrm{mAR} = 0.24$ .
Fig. 18 shows scores when applying ATMAN only to a single layer. I.p. the second half of layers seem more influential to the ATMAN scores, which is in accordance with recent research studying that higher-level concepts form later throughout the network.
In Fig. 19 we applied softmax on the embeddings (still only of the first layer) instead of the proposed thresholded cosine similarity Eq. 6. The proposed cosine similarity clearly outperforms its softmax variant in quality, however visible correlations can be found in the softmax-image as well.
As a final remark, we want to highlight again the rather undefined question of "what an explanation is". Afterall, we show what input leads to a diverging generation of the model faithfully, even if it would not match the expected outcome.
Figure 19: Qualitative example for an explanation of AtMan as proposed (middle) and AtMan when applying the softmax on the embedding instead of cosine similarity (right), as suggested by a reviewer. While some correlation can be found, it is much more noisy.