Beyond Intuition: Rethinking Token Attributions inside Transformers
Authors: Jiamin Chen
Xuhong Li
Lei Yu
Dejing Dou
Haoyi Xiong
Venue: Accepted by TMLR ()

Abstract: The multi-head attention mechanism, or rather the Transformer-based models have always been under the spotlight, not only in the domain of text processing, but also for computer vision. Several works have recently been proposed around exploring the token attributions along the intrinsic decision process. However, the ambiguity of the expression formulation can lead to an accumulation of error, which makes the interpretation less trustworthy and less applicable to different variants. In this work, we propose a novel method to approximate token contributions inside Transformers. We start from the partial derivative to each token, divide the interpretation process into attention perception and reasoning feedback with the chain rule and explore each part individually with explicit mathematical derivations. In attention perception, we propose the head-wise and token-wise approximations in order to learn how the tokens interact to form the pooled vector. As for reasoning feedback, we adopt a noise-decreasing strategy by applying the integrated gradients to the last attention map. Our method is further validated qualitatively and quantitatively through the faithfulness evaluations across different settings: single modality (BERT and ViT) and bi-modality (CLIP), different model sizes (ViT-L) and different pooling strategies (ViT-MAE) to demonstrate the broad applicability and clear improvements over existing methods.

Published in Transactions on Machine Learning Research (02/2023)
Beyond Intuition: Rethinking Token Attributions inside Transformers
Jiamin Chen
jiaminchen buaa.edu.co
Bethang University & Baidu Inc.
Xuhong Li Baidu Inc.
lixuhong baidu.
Lei Yu Bethang University & Beihang Hangzhou Innovation Institute Yuhang
yulei buaa.
Dejing Dou Baidu Inc.
doudejing @ baidu.com
Haoyi Xiong
xionghaoyi @ baidu.com
Baidu Inc.
Reviewed on http:://///////ooo//////////////
Abstract
The multi-head attention mechanism, or rather the Transformer-based models have always been under the spotlight, not only the domain of text processing. but also for computer vision. Several works have recently been proposed around exploring the token attributions along the intrinsic decision process. However, the ambiguity of the expression formulation can lead an accumulation of error, which makes the interpretation less trustworthy and less applicable to different variants. In this work, we propose novel method to approximate token contributions inside Transformers. We start from the partial derivative to each token, divide the interpretation process into attention perception and reasoning feedback with the chain rule and explore each part individually with explicit mathematical derivations. In attention perception, we propose the head-wise and token-wise approximations in order learn how the tokens interact form the pooled vector. As for reasoning feedback, adopt noise-decreasing strategy by applying the integrated gradients to the last attention map. Our method is further validated qualitatively and quantitatively through the faithfulness evaluations across different settings: single modality (BERT and ViT) and bi bi-modality (CLIP), different model sizes (ViT-L) and different pooling strategies (ViT-MAE) to demonstrate the broad applicability and clear improvements over existing methods.
1 Introduction
The Transformer Vaaswani et al. 2017) and its variants (Dosovitskiy et al. 2021 Devlin et al. 2019 Brown et al. 2020) take a leading place in the domain of natural language processing (NLP), where texts are firstly tokenized into words or sub-words, identified as tokens, and then fed into the deep neural network. The main architecture of Transformer consists of several attention blocks, where the query-relevant tokens are captured and combined to make new representation. With its huge success in NLP, more and more Transformer-baat multiple tasks, such image classification (Dosovitskiy al. 2021), object detection (Carion et al. 2020) VQA (Antol ct al. 2015) and GQA (Hudson & Manning 2019). For images, tokens usually the patches which are uniformly segmented (Dosovitskiy et al.) 2021), obtained with shift window (Liu et al. 2021) or captured with an object
models have been proposed for
detector (Tan & Bansal 2019 Li et al. 2019).
Codee available https://gihab.cmmjjjanneeeee how.:m/n/oo/aaaaaaaemetttttttt /rransformerinterp and InterpretDD
2022

---

Published in Transactions on Machine Learning Research (02/2023)
Generic Attribution
Our Method
VTT-B
ViT-L
VIT-MAE
VIT-B
VIT-L
VIT-MAE
Figure I: With different ViT variants, we observe an accumulation of error for the intuitive interpretation method Generic Attribution when the model gets deeper or adopts a global pooling strategy, which makes the results contradictory with the model performances.
Explaining how tokens are mixed and used inside the Transformer to make the final prediction can help a lot in understanding, debugging and refining the model. Some classic interpretation methods can be applied with modifications considering Transformer literature, such as the input gradient based methods (Sclvaraju et al. 2017 (Sundararajan et al. 2017] (mmikov et al. 2017). Regardless of their versatility for all differentiable models, the explanation results can be noisy to certain extent, due to the saturation and vanishing of gradients caused by numerous non-linear components in the deep models. Some attention-baser explanation methods have also been proposed for the Transformers (Michel ict al. 2019 Abnar & Zuidemal 2020 Chefer et al., 202 la b Hao et al., 202I), despite the disputation about the legitimacy of attentions being an explanation (Jain & Wallace 2019 Wiegreffe & Pinter [2019] Kobayashi et al.) (2020). With more characteristies inside Transformer taken into account, these attention-based algorithms generally provide more faithful explanation results than simply adopting the raw attentions.
However, existing attention-based Transformer explanation methods are built either on too strong assumptions, or without clear theoretical frameworks. Rollout (Abnar & Zuidemal 2020) takes skip connections into consideration but ignores the existing effects of linear projections inside attention blocks (as detailed in Section 3.2 Transformer Attribution (Chefer et al. 202 and Generic Attribution (Chefer et al. 202Ib) combine the gradients with layer-wise relevance propagation (Binder et al. 2016) (LRP) or attention maps along a rolling out path, and eliminate the negative components in each attention block. Though gradients in way reflect the direct influence the final objective, involving them intuitively and masking the negatives in each attention block risk losing useful information from relevance scores or attention maps.
In this work, we intend to bring the potential risks of introducing the methods intuitively. The unidentified errors from intuitive designs can be disclosed and magnified under different settings, causing the methods with good performances in the vanilla models fail in explaining their variants. We show in Figure some visualizations with Generie Attribution to different types Vision Transformers (Dosovitskiy al. 202T He al. 2022). spite of its good performances in explaining ViT-Base the visualizations other ViT variants show some contradictions to the models' properties, where the model with better performances (ViT-L) is visualized with focus on less semantic pixels. and the explanation results do not correspond with the semantics enhancement brought by the masked pretraining technique (ViT-MAF). These contradictions decrease the trustworthiness of interpretation methods and make them less applicable to all Transformer- based models.
Contributions. In order to explicitly explain how the model predicts and theoretically formalize the token interactions inside Transformers, we introduce novel explanation method with main contributions as following:
1) Based the partial derivative of the loss function each token, propose novel interpretation framework with the chain rule, dividing the whole process into the Attention Perception and Reasoning Feedback, as shown in Figure We believe that this framework can inspire more interpretation methods with other approximation strategies.
2) attention perception, we look into the relation between the input and output in each attention block and mathemati- cally derive two recurrence formulas with our head-wise and token-wise attention maps, where the token-wise one achieves 20x more accurate on average than Attention Rollout. By recurring from the first to the last attention block, reach the expression indicating the contributions of input tokens in the pooled vector before prediction head.
2

---

Published in Transactions on Machine Learning Research (02/2023)
3) To validate the faithfulness of the proposed explanation algorithm, we follow the previous evaluation methods (Samek 2019 De Young et al.) 2020), and compare ours with existing algorithms. The evaluation settings cover different modalities, different sizes and different pooling strategies concerning Transformers, including BERT-Base (Devlin et al. 2019) for texts, ViT- -TT-Base Dosoviiskky et al. 2021) for images, CLIP (Radford et al. 2021) for bi-modality, ViT-Large ( (Dosovitskiy et al. 202T) for the large model, and ViT-MAE (He et al. 2022) for the global
jet al.) 2017: Vu et al.]
pooling strategy. Our approach outperforms other strong baselines (e.g., (Abnar & Zuidema, 2020, Chefer et al.
2021a b)) through quantitative metrics and qualitative visualizations, and shows better applicability to various settings.
More analyses and ablation studies are also provided.
2 Related work
Transformer-based Models. For the first time being proposed by Vaswani et al. (2017), the Transformer has been rapidly and widely used for almost all kinds of deep learning tasks. A variety of derived methods (Dosovitskiy et al.)
202T Devlin al. 2019 Brown et al.)
2020 Liu et al., 2019) has sprung up in recent years. These Transformer-basee
models have become the state of the art most of the NLP benchmarks and stepped in the domain of computer vision as well (Dosovitskiy et al., 2021 Liu al.
2021). Vision Transformer ( (ViT) (Dosovitskiy et al. 2021 takes sequence of image patches, regraded as tokens, into successive attention blocks and feeds the last class token [CLS] into prediction head for downstream tasks. With its broad usage covering both image and text, the Transformer architecture has been explored for lot of bi-modal models, e.g., VisualBERT (Li et al. 2019). LXMERT (Tan & Bansal 2019).
CLIP (Radford et al. (021) and VLM (Xu et al., 2021). A single or dual self att.entiioocroos-aateeteee attenntooo based encoder is used to get the representations of both text and image. Thus, being capable to explain the decision process of these Transe models is rather important for a deeper understanding and a refinement for better performance.
Explainability of Deep Models. Deep models are well known for their superb performance as well as their black-box nature. Before Transformers, many explanation methods have been proposed to explain the deep models. For instance, as a post-hoe local explanation method, LIME (Ribeiro et al. 2016) locally explains the deep model at a single data point. Many methods based on gradients have also been proposed because of the differentiability of most deep models. Smooth Gradient (SG) (Smilkov et al. 2017) and Integrated Gradient (IG) (Sundararajan et al. 2017) are two explanation algorithms to produce input gradient based saliency maps, through adding noisy inputs, or using the integral respectively. More methods focus on specific model structures, e.g., CAM/Grad-CAM (Zhou et al.) 2016 Selvaraju et al. 2017 Wang al. 2020 for convolutional networks, GNNExplainer (Ying al. 2019) for graph neural networks, and GAN dissection [Bau et al. 2019) for generative adversarial networks.
Explainability for Transformers. Some Transformer-speccffi explanation methods have also been proposed recently, especially towards the goal of better understanding the characteristic components inside. Previous studies (Bahdanau et al. 2015 Xu et al., 20115] Choi et al. (016)) show that raw attention maps can be used as explanations for attention- based models. Considering the multi-head mechanism, Michel et al. (2019) find that the contributions of heads vary lot from one to another, and therefore propose that pruning the unimportant heads has little impact the model. Attention Rollout (Abnar & Zuidema 2020) assumes that the input tokens are linearly combined based on the attention weights and takes the [CLS] token as explanations. Generic Attribution (Chefer et al.) (021b) generalizes the idea of Rollout and adds the gradient information to each attention map, while Transformer Attribution (Chefer et al. 2021a) exploits LRP (Binder et al., 2016) and gradients together for getting the explanations.
3 Method
This section introduces the explanation expression we propose for Transformers. To facilitate reading, we present in Figure a roadmap to illustrate the main idea of our proposed method.
Roadmap. We consider the token attributions inside Transformers as the derivative of the loss function to the input tokens. In Section 3.1 by proposing a novel basis B = {ECLS, X1,..., XN), where each vector represents the corresponding token, can directly study the importance of token i as ac*(x) with (X)s (to...., ... fx). The basis St;
vectors X, are identified in Section by unfolding the successive attention blocks inside Transformers. Applying the chain rule with the output of last attention block (L), we divide the derivative into: Attention Perception P(L) (2 81,
ax* and combine them have the expression of our explanation method. and Reasoning Feedback Fe az
3

---

Published in Transactions on Machine Learning Research (02/2023)
Token Attributions (3.1)
Prediction Head
0
Tas
ac
(F p(A)
p.A)ou..... (1
07
2
CLS
N
Coordinates along
token basis (from
Z
X
Reasoning Feedback
ac (xx)
2
LN&MMPPP
F ReLl
E
(ax)
DA
(T)
Attention Perception (3.2)
2 MSA (LN(Z 11)) (LN(Z(1))
Approsimation 2 (2 With 20-1
Approsimation
+1)20 13
Embedding Module Position Embedding Eps
2 @ naa 10 FIXE Will Will Welh
Figure 2: Illustration of our method with VIT Dosovitskiy al. 2021). We show in the left how information propagates inside VIT. For method in the right, start from the partial derivative, divide the whole process into Attention Perception and Reasoning Feedback with the chain rule, and analyze each sub-process to obtain the final expression.
In Section 3.2
obtain (L),, focus on deriving a recursive relation between the input zd- and output Z
in attention block I. With Approximation and Approximation 2 we derive recursive relation Z (A (!) T)z(-1) w-(t) MLP and propose our head-wise (A head and wisen-wise (A token) attention maps. This recursion eventually
(1)
leads to (LL) p()) X, which reveals both IB and P(4)
In Section 3.3 for F explaining the process of passing the pooled vector for the final prediction, we introduce the technique of integration decrease the possible noise irrelevant tokens.
Preliminaries
Before starting the introduction of our method, this subsection, we recall several important concepts prepare for the following derivation.
Transformers. We present in the left panel of Figure a Transformer-based model consisting of L blocks. The input X for the Transformer is a sequence of N tokens. After the embedding module E, the tokens are turned into vectors of dimension D, with a special [CLS] token at the top and a positional embedding Epos Epos,, Epos, Eposs Let be the (-th attention block composed layer normalisation, multi-head self-attention (MSA), skip connections and projection layers. We denote the input of the block
(1,-), and the output ( In the MSA module, the attentions A (1), defined softmax(Q) KT /dd). capture the important tokens from a certain query and combines them to make output. Here for self-attention, query (Q), key (K) and value (V) are from trainable transformations of the same input data. Layer Normalisation (LN) (Ba et al. 2016) re-scales the input for single training case. The MLP module consists of two linear layers with an activation function ReLU/GeLU in between. In brief, we have
( [XOLS: X, E; X²E; X, E Epos
E R (dx)xD Epoe R(++)xDD
(1)
0 UU-( + + (11)
1,..,
(2)
Z (Z)( )
(3)
4

---

Published in Transactions on Machine Learning Research (02/2023)
Note that the MSA module operates on the sub-spaces of dimension DA, where h refers to the head index, and have H heads so that H HD
D.
Most Transformers perform the classification task via the last [CLS] embedding, while some use global pooling instead (He et al. 2022). In this section, we introduce our method particularly with [CLS] pooling, but the same methodology can also applied for other cases.
Coordinates and Partial Derivative. Let (Vi, V,..., Vn be basis of vector space V with finite dimension Given vector Z E V, let (a1...., an) be the coordinates of Z over E; that is Z Σ a,V, We denote (Z)B (a1...., (nn)
V,..,
Setting / + IR a function. For y = f(Z) f(ai,..., an), ) the partial derivative of f with respect to the variable a; is denoted By 0f(ai .z,-' .0n))
da:
Ba;
In the derivation of our method, we use Ai, to represent the (i + 1)-th row and (j + 1)-th column of the matrix A, Ar, for the entire (i + 1)-th row of the matrix A and Ai, (). for the (i + 1)-th row and (j + 1)-th to (A + 1) columns of the matrix A. For more details, we provided a full table of notations in the appendix.
3.1 Token Attributions inside Transformers
Let X be the input and _ the loss function while predicting class The aim of our method is to obtain the importance of each input token with respect to the decision process. Using the back poopapprp gradients the input aL*(X) be an answer, but here would like study directly the entire attribution of each token, since the transformation from the obtained high-dimensional explanation results to single scalar which represents the contribution of each token can bring unexpected noise. By formulating the input-output relations in attention blocks (details in Section 3.2 we propose set of vectors,
8X could
B = (XcLs, XI,..., Xx}
(4)
where each element represents the corresponding token and will be identified later in Scction 3.2
In this way, we can rewrite the input vector as (X) (fo, ((x) where to is the coordinate over XCLS and & is the coordinate over X; for i E (1,...,N) N). Thus, we can define the attribution of the i-th token as M..
(x)
From the chain rule, we develop (2*(x) and denote Zets the [CLS] token of the output in the last L-th attention block, having
0t,
(X) Öt,
(
Oz
(L
(
a
CLS)
(5)
OZ
CLS
Ōt
a(Z(L))
Öt,
With the analyses in Section 3.2. ZeLs can rewritten under the basis BB as (Z(Ls) = P!. indicating the attributions
of each input token to the last [CLS] before the prediction head. Here we make an approximation that ac*(X)
(X)
up (P
(LT
which could be tenable for the cases where raw attentions can in way represent the attributions in last [CLS] token. So
we have (X)
(X)(X)
(4) As for the second term a(Z
can have a(Z
(O..... P(.) . 0).
3,2 (
JA
Now plugging the above formulas into Eq. (5), we can compute the attribution of i-th token, which is
(
(Z(
OL (())
(6)
Ot:
d 0,
(L) Ot,
0.4
Combining all the tokens together and using F to denote (x)
(A( we reach the expression of our explanation method:
(X)
(
(C
R
F O PP()
(7)
00
Øt
O,(1..
5

---

Published in Transactions on Machine Learning Research (02/2023)
Our Explanation Method. For the Transformer-basee based model, the attributions of partitioned tokens to the final decision T can be expressed as a Hadamard product of two parts:
T T= (P(L) F
(8)
We denote the first part P(Z), representing how the input tokens integrate in successive attention blocks and being regarded as a process of representation learning, so called attention perception. The second part, denoted as FS, implies how the last [CLS] token is used for predicting certain class c, so called reasoning feedback. Their details will be introduced in Sections 3.2 and 3.3 respectively. The subscript P E {cls, glp,... ... is determined by the pooling strategy, such as [CLS] pooling and global pooling. In [CLS] pooling, it refers to the indices {(0. .)....... (0, N)} of the matrix
Tels (P(L) @ F Do.(1...N) In global pooling, by replacing Z with (z(")... [W,,,, we obtain its expression Tglp
(P()) @ (I...N.
3.2 Attention Perception
In this subsection, in order to identify p(L), analyze how the input tokens interact to obtain the last [CLS] representation. We first unfold each attention block to derive the relation between the input and output of the block.
Approximation 1. In MSA blocks with attentions A approximate the relations between input (t-1) and output 0 as ( (A ((-)))))())) + D)) MLP
Justification. The non-linear components such layer normalisation (LN) and activation function play an important role in deep neural networks. Here we make linear assumption and neglect the effects of LN and ReLU/GeLU,, because they operate on individual tokens and have slight influence on token interaction. Thus we simplify the information flow in the attention block as linear combinations and transformations for all the tokens:
Ž linear A A (Z ((() W()))
proj proj + (1-1)
I,.,.
(9)
88sum.
(1)
linear
7
W
MLP,1 + (1) 1
MLP,1 MLP,2 + 1 1MPP. 7
(10)
assum.
where (!
m-(l)
DXD and 6 proj
RD indicate the linear transformations inside Eq. 2 and MLP,1
e
proj
IR Phinaden, 6 (1)
MLP R.P. will)
MLP,1 R
Dx Puiaaes, w-ll) MLP.2 Pusteen in Eq. (3
After putting Eq. 99) into Eq. 10), rearrange the formula and obtain the relations between the input and output of the attention block / as:
( = (A (Z + (d (2(11))W((
MLP + (
(11)
where Wr(l) (1) MLP w Wi() ILP,1 MLP,2 + W (((W proj and B(O) 1h (wil) w
MLP,1 MLP.2 MLP Wil +)+
M MLP.2 We will neglect (()) because it is shared by all tokens and does not make tokens interacted. Thus, we obtain the relations that ( (4- (1)) Z w( MLP-
Approximation 2. If we can find À satisfying A (ZZ(-)W0()) ( As (A + TyZ(-1). the rela- tions between the input tokens and the output vectors of last attention block (L) can be written (
(( XI,...,XN)) where P(L) = (A (L) + I..... (A + T) and X, (X,E + Epona) W w(1)
MLP
MLP-
Justification. With A (-11)) + 1-11 (A ( + (-1) we can obtain the recurrence relation (
(A (4) + ()Z(-) MLP. Hence, the conclusion can be reached directly from recursion. We recur from the last block I. to the first block and obtain the relation between output of block and input tokens:
(Xer. + Epos,n) W (X₁E + Epoe,1) W
XcIs X1
CLS
z
~ (L) + (U)..AA.(AA))) +1)
= p(t)
(12)
z(4
(XxE E + Epos,N) W W
XN

---

Published in Transactions on Machine Learning Research (02/2023)
WLLLD where W W ... MLP and X; (X,E+ E)os,i) W. With the normalization components in neural networks and the assumption of independence between the input tokens, (XCLS XI.., Xx) forms set of basis vectors.
Remark. The recurrence relation of Attention Rollout method (Abnar & Zuidema 2020 can be rediscovered under the assumption that W = J. With this assumption, we have A (1 + ( A (l + z(--1) so that A = (1) Hence, we obtain the expression of Attention Rollout with the conclusions from Approximation Rollout (A (L) + I) ... (A I).
However, the assumption of regrading it as an identity matrix is too strong. Although W only operates the embedding dimension, due to A (1) it also contributes to token interaction. In order to study its effect and find a more accurate (1) expression, we derive O() = (1) ((11)) by two approximations: A (4)
token and Ä head" We name the token-wise attention map using "token" since it contributes the effect of W A (1) as a norm difference between token vectors, and name the head-wise one for a difference between heads.
Token-wise Attention Map. By the definition of matrix product, we develop 0 AZW and have
Oij
Σ Aim!
Zmuhk Wk. )Zm.
token
(AAi.m
i,m
(ZW
Zm.j
(13)
Z
IZm.--
m
(!) A (1)
((()
token
A (1)
(
(14)
token
wise
where (A (A
token ) = 0i A and og II (
So with A (1)
token we obtain
(
token
Rs
token
(
+
wise
MLP- More derivation details can be found in the appendix.
Head-wise Attention Map. We redevelop O = AZW by taking different heads into consideration, and have
Zmnk Wh kJ
O₂ = ΣΣ h
A 1.
Zm
head P wise
Σ Oh(AAZ)ss
(15)
Zmj
V1
where A is the attention of head h and 02 can be regarded head weight. Recent works (Michel et al. 2019 Hao et al. (021) propose head importance concept J, by considering the sensitivity of head mask, which means that if T is small, removing this head won't it affect the model output lot. A head importance expression is also provided in their work, with the inner product the attention and its gradient: I = (AA)T ac We adopt here the concept of In with normalisation On- E,, Ia.
Jn
describe the head weight. Hence, we propose another head-wiss type, which is casier to
apply in practice, A (1)
head E 1 10 ) and (t)
head Rd
(A (1)
+ I) MLP
Σ.
head
wise
With the above defined recurrence relations and Approximation 2 attention block L:
we obtain the tokens' interactions P(L) until the last
p(L)_ (A.
(L) + p((-1) =AA (A
(1) +1)
(16)
where A can be A token from Eq. 14 or A head from Eq. 15
Our token-wise attention perception achieves 20x more accurate than Attention Rollout. order to learn
the errors between different approximation strategies and the true value, we conduct several tests for Attention
Rollout, token-wise and head-wise attention maps. In each block 1, we compare ( 1 and Z head by ((( Zapprox = (A (1) + ()Z(("))wlll WhiP MLP with the true output z(1) and calculate the square error E [ Zapprox
for each method.
Randomly selected 5000 images from ImageNet val set used for the test. Both of the averages for our token-wise and head-wiss wise methods are below Attention Rollout, especially the token-wise approximation achieves 20x more accurate than Attention Rollout. B the errors in different blocks, where we can see that our tisen-wise method has a more stable performance when the layer gets deeper.
3.3 Reasoning Feedback
7

---

Published in Transactions on Machine Learning Research (02/2023)
Attention perception, taken for the first sub- pro-procsss, is explained by P(L) introduced previously. It remains to the reasoning part FS, which refers to the stage making the final decision based on the representations before the prediction head. We (C) the pack-propagatee gradients of the final prediction with respect to the attention map the last block.
Approximation Test
attention rollout ours token-wise
explain it by 0A(4]
Error Mogg10)
Integrated Gradient for Token-level Importance. Directly calculating the gradient of the last attention map may bring some noise and irrelevant features to the saliency map, due to the lack of sensitiv- ity, which causes gradients to focus on the irrele- vant features practically (Sundararajan et al. 2017] Smilkov et al. 2017). order to reduce the gra- dient self-induced noise, we use an Integrated Gra- dient method (IG) (Sundararajan et al. 2017) get the relevant gradients of the last attention map. We set the baseline to all zero, i.e. a black image or all padding tokens, and a linear path to the in- put. With Riemann approximation, we discretize the integration, having
10
Block Number
Figure Results our approximation tests. We notice that our token-wise approximation achieves 10~100x more accurate than Attention Rollout in different blocks. Besides, the variance of errors each block also decreases in token-wise method.
1
K ( Kzze ReLU Σ
(ax Jaz OA(L)
F ReLU
da)
(17)
K OA(L)
where only the positive part of the gradient is taken into consideration and K the total steps.
Combining the results of Attention Perception and Reasoning Feedback, our explanation method Tels can be finally expressed as
K
OC
k (
(p(L) @ Fe
pp()) o ReLU
Tals
(18)
0.(1...N)
K (A
0,(1...N3
where (() (1) () + (A T) and two types of A proposed: token-wise A token in Eq. 14
and head-wise A head in Eq. 15
4 Experiments
We validate proposed explanation method by comparing the results with several strong baselines. The experiment settings are based on two aspects: different modalities and different model versions. The experimental results show the clear advantages and wide applicability of our methods over the others in explaining Transformers.
4.1 Experimental Settings
Faithfulness Evaluation. Following previous works (Abnar& & Zuidemal 2020 Chefer et al. 202 b Samek al. 2017 Vu et al. 2019 Young et al. 2020), we prepare three types of tests for the trustworthiness evaluation:
I) Perturbation Tests gradually mask out the tokens of input according to the explanation results and measure the mean accuracy of all these masked inputs, i.c., area under the curve (AUC). There are two kinds of perturbation tests, positive and negative. the positive perturbation test, tokens are masked from the highest (most relevant) to the lowest. A steep decrease (lower AUC) indicates that the relevance does reflect their importance to the classification score. And vice versa in the negative perturbation test. With the class-specificity for some methods, adopt their explanation
8

---

Published in Transactions on Machine Learning Research (02/2023)
lipstick
Rollout
GA
TA
Ours-H
Ours-1
gyromitra
Figure Localization of fine-grained regions for single class prediction of ViT-Base. Both samples are correctly predicted with high probability score.
GradCAM
GA
TA
Ours-H
Ours-T
basset
parrot +
Figure 5: Class-specifi visualization results of ViT-Base. We select the class discriminative methods at the very first place, and present the results of two different classes.
results on two classes: Predicted, which refers to the class with highest probability score, and Target, which refers to the target class.
2) Segmentation Tests equal the explanation to semantic segmentation of image and compare with the ground-truth in the dataset. We use here the predicted class to obtain the explanation results for class-specific methods. Fach explanation can be evaluated by four metries: pixel accuracy, mean Intersection over Union (mloU), mean Average Precision (mAP) and mean F1 (mF1). Both pixel accuracy and mloU are obtained after binarizing with a threshold set as the average of attribution scores. The metrics mAP and mF1 are calculated by averaging the corresponding scores at each threshold point.
3) Language Reasoning comes from a NLP benchmark ERASER (De et al., 2020) for rationales extraction, whose goal is extract the input text spans that support the classification. Similar to the segmentation test, with the provided ground-truth, measure the F1 score of the top-k tokens according to the relevance map, where consider them to be part of the "rationale".
Baselines and Ours. The following six explanation methods are used as the baselines for a comparison in our experiments: Raw Attention Map (RAM), Attention Rollout (Abnar & Zuidemal 2020) (Rollout), Generic Attribu- tion (Chefer et al. 20216) (GA), GradCAM (Selvaraju et al. 2017) (CAM), Partial LRP (Voita et al., 2019) (PLRP) and Transformer Attribution (Chefer al. 2021a) (TA). The appendix provides detailed description of each method. We choose these baselines according their explainability literature and applicability the faithfulness evaluations. Black- box methods, which are computationally expensive and intrinsically different, are not considered here. As for our methods, we have, for each experiment, the token-witg wise (Ours-T) T) and the head-wise (Ours-H) types.
4.2 Transformers on Different Modalities
In this subsection, we focus on the explanation results of all methods with different modalities. We test them on three types of Transformers: BERT (Devlín et al., 2019) for texts, ViT (Dosovitskiy al. 2021) for images and

---

Published in Transactions on Machine Learning Research (02/2023)
275
WULL 1311
517
Mind
103
Figure 6: Results for BERT: (from left to right) Negative Perturbation Test on Movie Reviews, Positive Perturbation Test on Movie Review, Language Reasoning Test on Movie Reviews, Negative Perturbation Test on 20 Newsgroups and Positive Perturbation Test on 20 Newsgroups.
Table I: Comparison of Positive (lower is better) and Negative (higher is better) perturbation AUC scores (percents) on ImageNet for ViT-Base, ViT-Large and ViT-MAE. H stands for the Head-Wise methods and T for Token-Wise.
RAM
Rollout
CAM
PLRP
GA
TA
Ours-H
Ours-T
Predicted Target
24.51
20.45
34.61
20.18
20.95
17.37
15.97
15.89
Positive
34.07
20.20
20.01
16.65
15.14
15.13
ViT-Base
Predicted
45.65
53.86
41.95
50.62
48.95
54.53
57.13
57.97
Negative
Target
42.46
50.64
51.23
55.70
58.54
59.24
Predicted
27.71
21.91
45.66
30.19
19.21
20.41
18.01
17.99
Positive
Target
45.49
30.19
18.53
19.93
17.80
17.57
Vir-Large
Predicted
40.99
53.44
47.58
37.14
54.72
52.67
55.86
56.44
Negative
Target
47.69
37.16
56.13
53.25
56.33
56.88
Predicted
38.56
38.20
56.79
26.59
33.11
34.00
20.57
20.72
Positive
Target
57.24
26.57
32.63
33.74
20.22
20.30
VIT-MAE
Predicted
40.79
52.03
24.80
55.34
57.67
56.92
65.33
64.48
Negative
Target
24.41
55.37
58.80
57.31
66.06
65.18
CLIP (Radford et al. 2 for bi-modality. And we will present the experimental results by models. In terms of a Transformer literature and model size equivalence, we select BERT-base, ViT-base-16-224 and ViT-B/32 for CLIP, where all of them are derived from the Transformer Encoder architecture. The architectures of BERT and ViT are the same, except that BERT deals with (sub-)words while ViT copes with patches of input image. CLIP adopts dual encoder, in which text and image are respectively encoded and interact by cosine similarities.
VIT. Class-discriminability and the capacity of capturing fine- fine-grained details the criteria of good visual explanation (Selvaraju et al., (2017). Figures and geNet (Deng et al.] 2009). demonstrating that our method is of such capacities. More samples can be found in appendix. As shown in the figures, Rollout captures a complete region of the target, but simply taking an average of attentions brings noises on the irrelevant pixels. TA and GA climinate all the attentions with negative gradients which may result in an ignorance of some relevant pixels. Our methods perform well in capturing very localized area without too much loss of the entire object, thanks the combination of the attention perception that captures the entire object, and the reasoning feedback that decreases the noises of irrelevant pixels.
and 5Jndviiddvdee the visualizations of some images selected from Ima-
As for the quantitative evaluation, we select randomly 5 images per class (5k in total) from the ImageNet validation set for the perturbation tests, and the dataset of tests. As shown in Tables and 2 our methods outperform the other baselines on all metrics, especially the token-wise one, which leaves a great margin in segmentation tests, because of its more accurate approximation.
(Guillaumin et al. 2014) for the segmentation
For the text modality, we first have the perturbation and language reasoning tests on the Movie Reviews Dataset YDeYo et al. 2020) containing 1999 samples, which concerns classification of positive and negative sentiments according certain movie review. Our methods outperforms all the baselines in the language reasoning test.
BERT.
10

---

Published in Transactions on Machine Learning Research (02/2023)
Table 2: Segmentation test (percent, higher is better) on the ImageNeer-Semmentttion dataset for ViT-Base, Vir-Large and ViT-MAE.
RAM
Rollout
CAM
PLRP
GA
TA
Ours-H H
Ours-1
mloU
46.37
55.42
41.30
57.95
58.34
61.92
60.74
66.32
pixel accuracy
67.87
73.54
65.91
76.31
76.30
79.68
78.04
82.15
ViT-Base
mAP
80.24
84.76
71.60
84.67
85.28
85.99
86.18
88.04
mF1
29.44
43.68
19.42
38.82
41.85
40.10
44.45
45.72
mloU 41.18
52.88
39.72
40.09
54.40
56.31
61.24
61.65
pixel accuracy
63.20
71.15
68.49
62.31
73.93
75.75
78.92
78.87
ViT-Large
mAP
74.75
83.48
63.29
73.56
81.93
83.39
85.52
86.33
mF1
25.58
42.76
10.25
23.99
38.04
38.18
42.40
43.38
mIoU 37.30
42.39
22.60
51.13
47.66
45.98
62.47
62.36
pixel accuracy
56.10
62.00
39.62
72.01
67.10
66.70
79.66
79.63
ViT-MAE
mAP
67.41
76.38
57.28
81.15
79.58
78.41
86.58
86.21
mF1
31.99
34.20
12.99
33.94
36.51
32.43
44.67
44.08
For the perturbation test on Movie Reviews, the accuracy stops at a random guess (i.e. accuracy of 50%) at the end. We can see from Figure thtt our methods outperform almost all the baselines, especially the token-wiin -wiee method on the negative perturbation and language reasoning tests, where significant improvements are observed. As for the positive perturbation test,
from the final score that TA is better than methods, whereas in the first half of positive perturbation, the decrease according to both of our methods is clearly sharper than TA. We also notice that the methods with good performance on image, such as Rollout and GradCAM, lose its advantage for text. In contrast, our methods stay competitive for both text and image.
Since both of quantitative test for BERT are based on the Movie Review Dataset and the same fine-tuned classifier, we have another text perturbation test on 20 Newsgroups Dataset (Lang) 1995) for category classification. The 20 Newsgroups Dataset is collection of approximately 20,000 newsgroup documents across 20 different newsgroups. We finetune BERT-base model on its training data, with the accuracy reaching 93% on testing set. We randomly select 3000 documents from the testing set for the perturbation. The results are consistent with the perturbation test Movie Review, as shown in Figure where GradCAM and Rollout lose their advantages and our methods still outperform the other baselines. We provide as well the visualizations for BERT in the appendix, where our methods show more unman-unddrstaddbbll explanation results without too much focus on tokens like "the" or "and".
A pretrained CLIP achieves good performances in some vision-language tasks, such as image captioning. Two visualization examples are shown in Figure 7 For images, the captured area of our method corresponds well with the text, even for multiple objects noun phrase. For texts, the captured tokens are also the key words in the noun phrase. More examples can be found in the appendix.
CLIP.
Besides, CLIP can also match the performance of original ResNet50 (He et al. 2016) on ImageNet for the zero-shot prediction task with a designed prompt, by calculating the similarities between "a photo of class name" and the corresponding image. For the perturbation test, we have a zero-shot image classification task with the same 5k images as ViT from the ImageNet validation set. As shown in Table 3 our methods outperform all the basclines as well.
4.3 Transformers of Different Versions
In this subsection, we test the applicability of all explanation algorithms to different versions of Transformers. for example, with different model depth and pooling strategies. Besides ViT-Base in Sub-section 4.2 we conduct evaluation experiments on another two variants: ViT-large-16-222 (Dosovitskiy et al. 202T), denoted ViT-Large and ViT-MAE with the global pooling strategy instead of [CLS] pooling in vanilla ViT-base- 16-224 model. ViT-Large (ViT-L/16) is a very big model, which consist of successive attention blocks and 16 attention heads with embedding size set as 1024. ViT-MAE adopts the same architecture as ViT- ViT-Base, but with different pretraining approach and pooling strategy. The self-supervised learning approach of masking random patches of the
masked autoencoder (MAF) (He al. 2022
11

---

Published in Transactions on Machine Learning Research (02/2023)
Table 3: Comparison Positive (lower better) and Negative (higher is better) perturbation AUC scores (percents) on ImageNet for CLIP.
RAM
Rollout
GradCAM
GA
Ours-H
Ours-T
Predicted Target
14.11
19.32
21.49
12.78
12.29
12.32
Positive
21.34
11.91
11.55
11.48
Predicted Target
34.02
28.83
28.78
35.38
35.74
35.69
Negative
28.88
37.09
37.21
37.10
Rollout
GradCAM
GA
Ours-H
Ours-T
"a man in white T-shirt"
l man in white shirt
man in white shirt
a man in white shirt
man in white shirt
| man in white shirt
"an elephant"
elephant
elephant
an elephant
elephant
an elephant
Figure 7: Visualization of bi-modal model: CLIP. The visualization results with our method correspond well with the
highlighted input phrases, even for that with multiple objects.
input images and reconstructing the missing pixels. By reconstruction, the encoder learns complex semanties, thus leading better performances.
ViT-Large. We provide in the first line of Figure the visualizations of ViT-Large for the class "tiger cat". It's interesting to that the explanation methods, such GradCAM, GA and TA, which capture the quite semantic tokens for ViT-base, focus very irrelevant tokens for ViT-Large, whereas the ViT-Large shows better accuracy and transferability in the prediction. We believe that the irrationality results from an intuitive formulation of expression. With more successive attention blocks, the eиror accumulates, thus leading to less trustworthy visualizations. However, our methods, derived from explicit derivations, show far more understandable results in predicting the "tiger cat"..
for quantitative tests, we show in Table and Table the segmentation results and perturbation results for ViT-Large, where methods outperform all the baselines in all the settings.
For all the baselines, their explanations of Transformer global pooling setting are not provided in the original paper. To be more fair, we modify these methods by using the average of attribution scores from the second to the last line, indicating the first the last token, instead of the first line for [CLS] token.
VIT-MAE.
The visualization of ViT-MAE is shown in the third line of Figure with the same photo and predicting class. Our methods not only capture the very semantic tokens, but also show good coherence with the masked autoencoder approach, where by masking and reconstructing, the model can learn from more semantics. We credit the applicability of our methods in global pooling setting to the theoretical analysis as well.
Table and Table T
shows the results for segmentation and perturbation tests, where our methods leave a even more significant margin than the previous settings in both the positive and negative tests. It further demonstrates that our methods can also be applied to global pooled Transformers and provide trustworthy performances. Moreover, the
12

---

Published in Transactions on Machine Learning Research (02/2023)
GradCAM
GA
TA
Ours-H
Ours-I
1st Line: ViT-Large
2nd Line: ViT-Base
3rd Line: ViT-MAE
Figure 8: Visualizations of ViT-Large (Ist line), ViT-Base (2nd line) and ViT-MAE (3rd line) on the same image for the same class tiger cat. Our methods show a more stable and less noisy capturing of semantic pixels than other baselines.
strategy of masking and reconstructing in MAE can alleviate the out-of-distriibtioo issue in the perturbation test (Hase) al. 2021 Hooker et al. 2019) and make the quantitative results more convincing.
4.4 Ablation Study
We propose two ablation studies: the first focuses on the effectiveness of our expression, while the second focuses on the benefits of introducing the token-wise and hesd-wise coefficients. We conduct our ablation analyses with the same evaluation settings for ViT-Base, mentioned in Section 4.2, From Table we notice that the results correspond well with our derivations in Section 3 Introducing token-wise and head-wise attention maps clearly enhances the quality of explanations. Moreover, the first ablation study also indicates the effectiveness of introducing the integrated version of gradients to reduce the noise.
Table 4: Results of ablation study with ViT-Base. Avg stands for simple average of attentions for all heads and A (L) for the attention map of the last block. (L) denotes the attention gradient of the last block and f G the integrated version. We report the results of perturbation test with its predicted class. The first part is designed to analyze the effectiveness our derived attribution expression T and the second part for the befits of introducing our token- wise/head wise attention maps.
Setting
Segmentation
Perturbation
Perception
Feedback
Acc.
mioU
mAP
mF1
Pos.
Neg.
f G())
67.31
48.76
77.93
38.44
18.56
52.12
Head-wise
x
71.83
53.24
83.66
42.26
20.20
52.79
Head-wise
(LL)
75.67
57.75
84.49
43.01
17.33
55.27
Hise
f (L)
78.04
60.74
86.18
44.45
15.97
$7.13
(L)
f (L))
77.55
59.16
84.65
38.82
19.05
52.45
Avg Head-wise
f G(L)
77.88
60.56
86.14
44.34
16.72
55.59
f ((L)
78.04
60.74
86.18
44.45
15.97
57.13
Token-wise
(L)
82.15
66.32
88.04
45.72
15.89
$7.97
4.5 Token-wise/Head-wise
The token-wise and head-wise manners come from the different approximation approaches we use for the recurrence relation of each attention block, as discussed in Section 3 The laken-wise one credits the effects linear projection to the differences in the norms of token vectors while the head-wise considers the head differences. As for
13

---

Published in Transactions on Machine Learning Research (02/2023)
implementation, the token-wise method may involve extra modifications to obtain the intermediates, since they are not the direct output of any component in the neural network. The head-wise one has no such issue because it only demands the attention and attention gradients for calculation.
However, both visualizations and quantitative results demonstrate that the differences between these two methods are very limited, since they are both derived within the same framework. They both outperform all the baseline methods and are applicable to different types of Transformers, which contributes the explicitness behind these two methods.
5 Conclusions
Most of the Transformer-base models give the prediction by first integrating different tokens via successive attention blocks, and then using the pooled vector of last token representations for prediction task. In this work, we propose a novel explanation method based on gradients, the change of basis and the chain rule, dividing the entire process into attention perception and reasoning processes. Moreover, by taking into account the linear projections, we derive attention maps (token- wise and head-wise) for the attention perception process, which more accurate than traditional Attention Rollout. We show in the experiments that our method achieves a better performance than all baselines in a series of evaluation settings, on specifying the class information, capturing the fine- fine-grained pixels and being applicable to different model sizes, pooling strategies and modalities.
Based on this tool, we can explore in depth those Transformer-ban based models in the fields of both computer vision and language processing. which gives a hand in debugging, training and evaluating different Transformers. In this work, we only deal with Transformer Encoder. Other model architectures, such as Encoder-Decodee (Yang et al.) 2019) Decoder types (Brown et al. 2020), will be tackled in our future work.
Broader Impact Statement
Being explainable an important criterion to implement the deep learning algorithms under real life settings. With the wide applications of Transformer-based models, our methods can be very helpful by bringing up a trustworthy explanation and increasing the transparency of model prediction. However, it may also be used for malicious purpose if the user intends to trick the model with modifications to the most or the least important tokens.
References
Samira Abnar and Willem H. Zuidema. Quantifying attention flow transformers. Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 4190-4197. Association for Computational Linguistics, 2020. doi: 1/205/3/1/22222almmmmma
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2433, 2015.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, His/1607.004551 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. GAN dissection: Visualizing and understanding generative adversarial networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In Alessandro E. Villa, Paolo Masulli, and Antonio Javier Pons Rivero (eds.), Artificial Neural Networks and Machine Learning ICANN 2016 25th
14

---

Published in Transactions on Machine Learning Research (02/2023)
International Conference on Artificial Neural Networks, Barcelona, Spain, September 6-9, 2016. Proceedings, Part II, volume 9887 of Lecture Notes in Computer Science, pp. 63-71. Springer, 2016. doi: 10. 10.1007/988-3311-44781111
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, 0oss/2005.11160 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision. ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9630-9640. IEEE, 2021. doi: 10. 10.1109//CCV49922.20020 2001000
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 782-791. Computer Vision Foundation / IEEE, 2021a.
Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder- decoder transformers. In 202/ IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 387-396. IEEE, 2021b. doi: 10. 10.1099/CCV449922220000000
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9620-9629, IEEE, 2021. doi: 10.110/0CCV4922.22210000
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. RETAIN: interpretable predictive model for healthcare using reverse time attention mechanism. In Daniel D. Lee, Masashi Sugiyama, Ulrike Luxburg, Isabelle Guyon, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 3504-3512, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR 2009), 20- 25 June 2009, Miami, Florida, USA. pp. 248-255, IEEE Computer Society, 2009, doi: 10. PAL100//CVPR.2200552666
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers). pp. 4161-4186. Association for Computational Linguistics, 2019, doi: 10.18653/v1/n19-11 1423..
Jay De Young, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NL.P models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 4443-4458 Association for Computational Linguistics, 2020. doi: 10.18653/w1/202000 1/20200.0// mailmmaii.4400
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2023)
Matthieu Guillaumin, Daniel Küttel, and Vittorio Ferrari. Imagenet auto-annotation with segmentation propagation. Int. J. Comput. Vis., 110(3)33283344 2014. doi: 10070/ 166-0740133---
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Self-attention attribution: transformer. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thirl Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 12963-12977 AAAI Press, 2021.
Interpreting information interactions inside
Peter Hase, Harry Xie, and Mohit Bansal. The cut-of-distributioo problem in explainability and search methods for feature importance explanations. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 3650-3666, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016, doi: 10.1109/CVPR.20166
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 15979-15988. IEEE, 2022. doi: 10.1109/VVP52228.22222 2022.0155.
Sara Hooker, Dumitru Erhan, Paeter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9734-9745, 2019.
Drew A Hudson and Christopher D Manning. Gqa: new dataset for compositional question answering over real-world images. arXiv preprint arXiv: 190::0990000 3(8), 2019.
Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543-3556, 2019.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 7057-7075. Association for Computational Linguistics, 2020. doi: 10. 18653/v 1/2020 122000.emmlp-main.55777
Ken Lang. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning. pp. 331-339, 1995.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. Visualbert: A simple and performant 2019.
baseline for vision and language. CoRR,
Xuhong Li, Haoyi Xiong, Xingjian Li, Xuanyu Wu, Zeyu Chen, and Dejing Dou. Interpretdl: Explaining deep models in paddlepaddle. Journal of Machine Learning Research, 23(197):1-6, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/19907 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 9992-10002. IEEE, 2021. doi: 10.1109) CCCV48922.2021 1.09866.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than onc? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence Alché-Buu Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 14, 2019, Vancouver, BC, Canada, pp. 14014-14024, 2019.
16

---

Published in Transactions on Machine Learning Research (02/2023)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Muchine Learning Reseurch, pp. 8748-8763. PMLR, 2021.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 1135-1144. ACM, 2016. doi: 10. (0.1155/9399772.937777
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE Trans. Neural Networks Learn. Syst., 28(11); 2660-2673, 2017.
doi: 10.1
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 618-626. IEEE Computer Society, 2017. doi: 10. 10.1109//CC222217..4.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, abs// 03825, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 3319-3328. PMLR, 2017.
Hao Tan and Mohit Bansal. LXMERT: learning cross-modality encoder representations from transformers. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, FMMLLP-JICILP 2019, Hong Kong, China, November 3-7, 2019, pp. 5099-5110. Association for Computational Linguistics, 2019. doi: 10. 11655/111D19-15144
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA. pp. 1998-6008 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Anna Korhonen, David R. Traum, and Lluís Marquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 5797-5808. Association for Computational Linguistics, 2019. doi: 10. (0.18653/v1/p19-111
Minh N. Vu, True D. Nguyen, NhatHai Phan, Ralucca Gera, and My Thai. Evaluating explainers via perturbation. CoRR, abs/1906.00 02032, 2019.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp. 111-119. Computer Vision Foundation / IEEE, 2020. doi: 10. 10.10090CVPN555492222222
Sarah Wiegreffe and Yuval Pinter. Attention not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP) pp. 11-20, 2019.
17

---

Published in Transactions on Machine Learning Research (02/2023)
Hu Xu, Gargi Ghosh, Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer. VL.M: task-agnostic video-language model pre-training for video understanding. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL/JJCNPP 2021, Online Event, August 1-6, 2021, volume ACL/UCNLL 2021 of Findings of ACL, pp. Association for Computational Linguistics, 2021, doi: 10. 18653/v /2021. findings-acl.370.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2048-2057. JMLR.org. 2015.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5754-5764, 2019.
Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9240-99 2019.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discrimina- tive localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. IEEE Computer Society, 2016. doi: 10.1109/CVPR:20166.199
18

---

Published in Transactions on Machine Learning Research (02/2023)
A
Table of Notations
Table 5: Tabbl of notations
Indices:
L
the number of multi-head self-attention blocks (1 E {I,..., LJ), which is used as the superscripl (1) denote the variables and parameters concerning the [-th block.
N
the number of tokens/patches.
C
the number of classes during the prediction (c (I.....C)..
H
the number of heads in the self-attention block (h E {), H)).
K
the number of steps while discretizing the integration (I E {I,...,K) K}), which is only involved in Section 3.3
p
the pooling strategy (P {cls, glp,... ...)).
i,j
the subscript indexing the matrix (i {0, 1,.,, N} and denotes the added [CLS] token). For example, Aij stands for the element at the (i + 1)-th row and (j + 1)-th column of the matrix A. We also use Ai, for the entire (à 1)-th row, (1,(1, ,N for the second to N- columns at the first row.
Our Explanation Method:
T
expression our explanation method attention perception reasoning feedback
P
F
Åtoken
the token-wise attention map the boken-wise coefficients
A head
the head-wise attention map the head importance the head-wise coefficients
Transformer
Parameters:
E
the embedding module the positional embedding the embedding size
Epos
D
Dpidden
the hidden size
W
the linear transformation matrix the bias in linear transformation
Variables:
X
the input image patches or text tokens the input/output of self-attention block
Z
Z
the output after multi-head self-attention module the set of basis vectors representing each token the basis vector the coordinate of input vector over basis vector the raw attention map
B
X
A
Q
query key value
K
V
C
the loss function
B Details of Baselines
Raw attention map (RAM) Raw attention map rather straightforward explanation method, which takes into consideration only the attention map of the last block. For the model with [CLS) pooling method, it explains the
19

---

Published in Transactions on Machine Learning Research (02/2023)
importance of different patches with the first row of the attention matrix. In experiment, the last attention layer is visualized.
Attention Rollout (Abnar & Zuidema, 2020) (Rollout) Different from raw attention, all the attention maps along a forward-pasd count for the explanation, where they assume a linear combination tokens and quantify the influence of skip connections by adding an identity matrix. Note that both raw attention and Attention Rollout are class- agnostic....
GradCAM (Selvaraju et al., 2017) GradCAM provides class-specific explanation, where they add weights gradient based feature map of the last convolution layer for ConvNet based model. Following the work Ch et al. (2021a). we use a weighted gradient map of the last attention block, which corresponds to the [CLS] token
Partial LRP (Voita ef al., 2019) (PLRP) Not only the attention map of each block is considered, the relevance propagation based methods also take the information flow inside the whole neural network into account. These methods are also class-agnostic, which means the visualization remains the same for the prediction of all classes. Here, we visualize the relevance map the last attention layer.
Transformer attribution (Chefer al. 2021a) (TA) Transformer attribution method is sfute-of-thean art class-speciff explanation method for Transformer. It combines relevancy and attention-map gradient by regarding the gradient weight to the relevance for certain prediction task.
Generic attribution (Chefer et al. b021b (GA) Generic attribution extends the usage of Transformer attribution to co-attention and self-attention based models, such as VisualBERT, LXMERT etc. and propose a more generic relevancy update rule. Meanwhile, it replaces the relevance of Transformer attribution with the attention layer in each block.
C Token-wise Attention Map
Here, we provide the mathematical derivation details of the Eq. (14) and Eq. (55) The goal is to develop the formula O AZW, where O ((N+1)xD, A E (NN+)))(N++) and W DxD
By the definition of matrix product, have:
N+1 D Ozj ΣΣ
Aim Zem Wki
(19)
Since Ai Zm,k and Wk. E R, it is possible to write when Zin F that,
N+1
D
Zm, Wk.
N+1
D
Zun WK.
O2 = Σ
Σ
Ai.
Zm.jj
Aim
7m
(20)
Zm..j
Zmo
m=
k=1
m=
1=1
N+1
Aim
N+1
Ar.
Zm, WK Zm.
(ZW) mm Imnd
(21)
Zmj
Zemi
m=1
1=1
m=1
N+1
(ZW)m-
token
N+1
(ZZW),. mm Zmmit
A A.m.
Zmni
A..m
(22)
Zmm
wise
ZZmm |l
N+1
Σ
As.m Zymuj = (Atoken Z)i.s
(23)
m==1
Here comes the definition of our token-wise attention map.
D Discussion about Self-Supervised Learning
Self-supervised learning strategy can bring new properties to the model, especially to the Transformer- based mod- els (Caron al. 2021 Chen al. 2021 He et al. 2022). DINO (Caron al. 2021 is self-supervised learning method
20

---

Published in Transactions on Machine Learning Research (02/2023)
for Vision Transformer, with specific design to reflect directly semantics via features. In such case, adopting simple explanation methods such as raw attention map might already be promising. In order to understand the differences that the self-supervised learning strategies can bring to explanation results, we add an additional segmentation test for ViT-Base-DINN on and compare them with raw attention map.
(Guillaumin et al. 2014). We apply our wreen-wise/head-WWee methods
Table 6: Segmentation test on ImageNet-Segmentaion dataset (percent) for VIT-DINO
Raw Attention Map
Ours-H
Ours-T T
IoU
65.79
67.42
66.42
ViT-Base-DINO
Ace
82.08
83.71
83.09
mAP 86.79
87.60
87.26
As shown in Table self-supervised learning can bring differences in explaining certain models, whereas our methods still perform better localizing semantics. We notice that the results of our methods for DINO is almost the same as Supervised ViT-Base in Table 2 This may indicate that there are some hidden and hard-to-get semantics in the supervised ViT-Base that are equal to ViT-DINO. And our methods succeed in revealing this information and being applicable to all Transformer- based models.
Moreover, the meggenet-Segmennation dataset only contains images with single objects. For explanation methods that are class-specific, the segmentation results can be ambiguous when it appears multiple objects on the image. We show in Figure the visualizations for multiple objects, which further validate the benefits of methods in capturing semantics.
Class 1
Ours-T
Raw Attention
Figure 9: Class-specific visualizations for ViT-Base-DINO.
21

---

Published in Transactions on Machine Learning Research (02/2023)
E
More Samples, BERT and ViTs (Supervised, MoCo-v3, DINO and MAE)
RAM
Rollout
GradCAM
PLRP
lipstick
TA
GA
Ours-H
Ours-T
RAM
Rollout
GradCAM
PLRP
TA
GA
Ours-H
Ours-T
gyromitra
RAM
Rollout
GradCAM
PLRP
TA
GA
Ours-H
Ours-T
border collie
RAM
Rollout
GradCAM
PLRP
TA
GA
Ours-H
Ours-T
barber chair
Figure 10: More samples of localization of fine-gran finengrained regions for single class prediction of ViT with full baselines.
22

---

Published in Transactions on Machine Learning Research (02/2023)
GradCAM
GA
TA
Ours-H
Ours-T
bull mastiff
tiger cat +
GradCAM
GA
TA
Ours-H
Ours-T
African elephant
zebra +
GradCAM
GA
TA
Ours-H
Ours-T
tusker +
zebra +
GradCAM
GA
TA
Ours-H
Ours-T
Persian cat
golden-> retriever
Figure 11: More samples of class-specific visualization results of ViT. We just present the results of two classes with class-specific baseline methods, while for the other methods, there are not any differences between different classes.
23

---

Published in Transactions on Machine Learning Research (02/2023)
Rollout
GradCAM
GA
Ours-H
Ours-1
"a man in white T-shirt"
man in white shirt
man in whitee shirt
man in while shirt
à man white shirt
man In white shirt
"an elephant"
an elephant
an elephant
an elephant
an ølephant
aan elephant
'nn astronaut suit"
astronsut suit
lan astronaut suit
astronaut suit
astronaut suit
astronaut suit
"a man with eyeglasses"
man with eye glesses
men with eyta glesses
man with eye Blasses
a man with eye glasses
3 man with eye glasses
"a girl climbing the floor"
girl climbing the floor
girl climbing the floor
climbing the floor
girl climbing the fleor
girl climbing the floor
"a man with lipstick"
man with lipstick
la man with lipstick
man with lipstick
man with lipstick
man with lipstick
Figure 12: More samples of CLIP visualization. Both text and image visualizations are provided here.
24

---

Published in Transactions on Machine Learning Research (02/2023)
RAM
Rollout
KCT.) ab, and 1900 gilee akore soell too the all thet the finlene distaetiinn
(UTA all and 1939 sow gyin aling willl (ion "hhe will that tow the drtiions daviantion of leitg the mance ve
fer this und frunky
lan tiie year, and quili fruddy derthe i we ancching eqpully hat. a lout hope idd a
donte: seyting wasde bod or boot boge
twe anyriiing opsally
seyttingg eqpalty
leal 11 de ' all list EM lhi story of the ano papaic gny adhod ploye ly
laad sae 'sail that idb the vtory lhe uos popelle soe in seland i paaoni lo froddte petuae jr. who accepte het trasslert the gwicent gri
tracite penss y 1] wao
and
(rccted right there, jooilou de many Tad filerralions
Sigh cooi lnto
popsice
file theee
Highs oock
the nost papiler
The Bat steestas withort the prstitutionn
sous the
withort the testicctionn of stoeylane would lave eatccupped to tay sausting uer wills this cenctial the will leves a deccce to proput the more formodt as throgh "hhe i all thet tellee compleiily au the luse and uncrzed becadds to proli it sheed there y act oue neiginal or innerwetie character the til other and that wau
lise peetty
"
atterptedd to soczellinng wath BB
coul have
teen a destin in propel the mode firoont as thoregh
couplisc us Bil game and pecrard foanala to pass B ddaad Hoae uh ongial or Inteoweting dharette
fimm
eoorgh
tless losl uood
ienturel lie sis anoii
Khere nut goad per fermuare Instuall star the uanis radad kigls cock stuple bomblee teallly be
rxpersiion Ennoeghent the liik and licks cnsiized tist ahe
hastup Entemnill Sizee
that
lerring udterable thme as
aznd and
hee mnnueeenatiio Rid iicoriieedigg ding
wastin it
bee trondherratt atiion soe monoelniing and innecnnsry. the works svere to he stying it a lettlee to be parelarr thass to be wha you see as for iroida yeaser ff to udder onffurtiy rajos he qptte hust here counte shrough the this cheess saal Lover exaidiles
better to be papraze thess wau you and to be tookae mnier actie andiaarl enjor
B Bull
through
damu. aad acnd cotablake
thameder
beotlee lin fnd
lakcu ciliis lare, to leother get
an leoogil
linooll stink
suppoeed
ber lii juat
becatan beuttag ante It tank that stagle aleneen timn sow one of the aast ollveiie things te bont a mosle it bing Gine I all thet I
because be wore beartng sice 3 con of the most ofEssivve things se seen modde in llong linee i "slee is ell that eacoi
degie ebersent the thes your
GradCAM
PLRP
KCLS ab. and 1900 was ginng along will, too abe that EM the distace distiatii of liiigg lhe noni moses mimdl ted. an bost hope story
[CLS5 az and 1939 wow going along wil too. the xil that the delitone
we for this you and quie louslly I
disiiation of letig the socil uunce ve 10M fee tiis yoar quite haakly 1 equaii had lowe hope
donbe:
epually
dente
aed
that
pogalct puy
plaee
story the mon papiila par
poased
frobdic peizar
have ans (il stocslas F cook File timee problee
cosi Iitt
begh mock leon the pepottrr that 1ROHt theme prollom A
thrse kand of
neer da and of the fias kind of
lavee soss li atapliue to oak commurnto one the and of tae fite. Mos" pestty somann ceorpe withort the posttrctionn of
The gwett sommen
alleajied to sotsellinng withort the preeitution of courien hadt tltmmaaee
Tad the titeraaloss.
with lis satterad. Bi well noes soold host been a destos propell the most forzont
elicasptol to tes semocking une with (ki usiccial Hie will soens stoeylin would lane beon diccce to provel the mone formand.
theogh Bi fus incesnol Dicesii poadi idredd
couplesty Hil june
pras
cumplilly
theee intrrotting good
Earr
good
keugh cook
stuply Eaerilie. sesally
like theoughonn fick bwit iacee corricce that
ttugle bomitn imeelly do "hee personal bas the toe Htink ineth to te crock wane sam espersccn Eanerationt HE Bod and ticks foorky
reeds to hhe axd sonk wnrr fhe
kaning sitembik a tizze to
and morte in easting in ue foolis prias theongh
and contrrent that
andd"
tet troudtenstion and
lax frazakmaatas
be papatic gume bid Kerre outki
veliiancy popritur
darnn enjer.
quile list
lusooah ali siiled
beotles
chara
enden
the bentber
audeooennble masnn, be gue bearnng mith explanationn glves and "Tey in nove tooeght up som supeord to bed
cook
cook and for indisemmald ressen he giil boscing odds caplerntion is
for tin. just
gpeee and they no never brorate up. sene rxprovedd to fre. aarry Ser lim just
lecasa le war bancia aids F' do tank fiat sinale chauese of the fikn soa
becuuse he wone bearing alts do think so that degle
the the s0e
ore the most offecaiin thing if we own mode in long thre
all thet
sas msst offozade thangy we sees a tunne in long timee I shhe aill Hast
ando |SE
TA
GA
ping donte 'Il enything
CTS ale
[CLL8] ab. 1928 was piaz along welk
that dsticra
fruaky asytting repully
quite fharkly paape who sccopta let franaion the godaes giil adace (naded. legh cack 1960 the most popelker that 188H them grondem a hov ranry thrss lacre ve soas (il stuplite to wock counestt one the and of the liee.
deale being the
bad dee all that BB story of the papelaz achool ( played who scoepe a het rroddern the grelint gir in school mochel kiali cosie isto mool popuiia (aal falt lasse prodics lawe army cams have co own sisestar
popriter
thaddee primes
frokii peize:
of flasnnkers commante sear the rnd the tim
of
lie" peetis voums casept bas uneiltation" athercited try
His' peatty vomin eocope withort the prostrationn stteepped to tey senething new
meconial, well
desine gropel fhe
companyy" on Bill
and iecessdd fieania paak thout.
compltsttyy
groh
orighell or intereliin doancter
timm sttey
original or miing chrrweer this Hhhee that soa manrigh kacre la aol cae good pefermaaze knttaod. Hil sia Hh uorie reded kiads cook- If mouste lommun ceaully do the to pereoral
daas is uoi ces good perdenaance iediand Hi sizz @ aasvic cacael hish cock smeey tornttg sseially
in promnna
atola be cook stars mas copprsaio llsoeniiion BE bastng
nooth to
coull seuas saae iapecsiou Lanveabaral MB lick Ral
las InnssRenatios
juansseess trmidte
better be
bethee
than who
ondiierly mnjo be tot B Iuat lese
though thn calles
celitecly by eajos Ie
Inll here
thoough H8 3Tt sisted
dlam,, and Brei oatablnder nod thameter keenn crien herr tos the beother
chazms Ent BONE orakiadon coal ciaeacier danes cultin hare foo ca the lootaer cook
Io pue learing side explanation ginas and day in mose lesegat up. impponte ind
mare
same fndisoematle moser 1s gei bowniig asls 10 explocatiion
for liss jus thi sou
aivea ins thay eo aeeez brorad up- pess aupposed to
aus lim juai you
bearteg
that stngle ling
becator bearing thinss
fiit degle
tlang
lime II
ands jaur
entia (S19))
Ours-H
Ours-T
KT ah, and 1000 grin, along woll too distactiin
for this thee" munaane donte: septinng metolle bod on bost hoge
JCLS 1939
delisne
the mosic
funkly i
doale thithing eupaall bai. at land, baye" isk quite frunkly fost die
thetlecttin being the moree
this
ww anything equally lea dac all dest BB the story of the and papeli any adhudd played by prane
thing equals
that bell the wtory the mon popeiler gre achool paree be
accepo het mondfrr grekint gnd
froblic peiner jr whs scogle a let to leandkens the geties gill is adaod (indad lmpp cook the ros poptier: that riiht them. groidem
Sigh cool
popsice thet fale there prodies
sons this stapyltee
liee
The"
withort the preiltution
like" peatty
atteroted to sonzellanng with lill casteral. HBl will
would hase
attrapped to tre sonething with this monelal the
owild howe
Teen a destin in propel the mode firwont as thirgg
all oite
box dessce to propcl the manie fiewerd
though six all that gais
couplesc us Bil gane gall pereand Erandd to pass B theat. Hoee nn one onghall or inteoweting dharetee
compleiely au lie gam and geatuoo iecands proli il shead thure a nut our neigiaal or interettin chazecter the
and thet soe
that yos kight
manrgh
tlass jolt seat good pestenuns iestunll Hi sits music
Kaere not good peo formuare knstuall star lhke seuple
Heseugioon
Eincopbaat
and
hastig
contrnced that
fer tronsbercatin innoeiieeigg and ousttent tast ahe andd and the work
hee mondermmtion
(iig
HE moste
westin i
stying is
'a bettrr to be poprize thess be wio you are.. Be botta pnitr scise codtuarls mnjor he B gume loin moude through thn catted dasm. gull aree ottiilaa
a bettee to be papeaarr thas to be who you wee for bruida Jr ta acter celtuoriy rajos
gatt lakna cillin
coset through the this io loother bor liin jual- supponed edersent the sits you
chameter outkin
bootles
thezus taal
gen
lif. becazas wore boattng ante 2 tamk that atogle abneere Abnn RoM one the mast oleeniig (lings i"
looeght
fnd
brougl think movie in llnng time . sile" io all tlai!
becase be
mosic king tine all thet
of HEssise
Figure 13: BERT visualizations for a sample with negative emotion in Movie Reviews Dataset. A darker color represents a greater token attribution during the prediction.
255

---

Published in Transactions on Machine Learning Research (02/2023)
RAM
Rollout
JCT 9 EM good sazund plowrit and mos gtlee missig bitt mamar cchoetrecters veziarnuna pid counre, who every sour scal prrsontaaeey awaryy .D to regort groendiigg foy the grandiing day cemtone incoon poendhogg
(CTAS thes good rotenel pleseer AMy coneedty NIll aerrac (thoadhoetees vonlerau plul oumuss cla, eatr isasst prupeitoraay to repett on pomndhtg day the gornddog day nermone tnnolve a groedding lecan, Iiodd oni u las and la doen so a saalio 2 will le us carly seciis
1394 ploe
1084; place
leisg liled of lao and doeca It sec staske,, it will be on oady syeans-
piel mally
a Jaoccerr coreatory and
produee rita taackeedd can
pail rouly luste tae cremory and ust
mial loveeer Jate cesel look prodioee ctta (msodorsl) can change
cradl for pidl
living
and be
living grondive the
lim groilllng is sel until torally mnffecttiv
acresseeiiors deadtr and danty leeally unotfoaite
bime
sell
the
writion
lest
axsanettion dimette rasst uni, daney muit
waklen funty
bust
by orettierra conedt otthongh the idea day reporting car and oore way
nerts werteneets comsly althengh the idko of a day repeating nore and curr mas aoud todaaae, thers uee custtals gcod jokes to kold the caliezeos tliekkiss ilanoualani tar stidl firs camte divection waso haie and athorghh door try any dash dlocttur tiido, lhe
soundd tediooo liace lre taocaal good paao to losk unliaaso atoculion clporadeont
the whoie filll fiu a
and atrinegs
dirveis
and the joac are to
diecteed will cusoals and jokes out set up will und ratting good eepecialiy
alsn mood expectilyy perfor rigle
pootien btll criss
gamd fnd his trumiaat the hargy
wles siis have perfect right with lif the prffecaone excoileat bill unmmy graat fin, and
alo
frims ejsacal to hapey ansoothh
and delivr his fimm his rend
il good rins atthorgh
and delives his fies lis yoasl enrry style mnlle moddiadd is good iht aittingg
socuttu dae just a litk lus too avont is parto the lims there deminity
auueiia ahz just a ltile bit too sedd is same parts of the Slss tlase cacaablty leenoes the ran laade thenitille otherntse the while thi would probabty full part dduie cliid uu the couemmmn hery aho funny allhsogh sou lasse
the too loads Esuiifilly
probuatl call part
ellon: the
kezr. B libe Exsry grine
you lave to lis
fils
pudyy atyle, otheretoe the rally going
thronghas lea ciher
tillac suppelling lst and evnn directer
stephin tabolonskty atlaroes patle schel
exploe tabsuresyy tillsmos phile oll schood mate sysraon, own diacttrr
vents popping up a neaningist is lict Chee is oot oue shul performeure ihil
canis prpplig up segitt in fae theee is aut con dhal perfictance Chis fin, and oves the sporachos arta a baay scoss anchritg a car cheae, sennil thow bn molly willing snong with gronding day at all
slis and conn the sroudling geta a fanty sone inmbbngg cae dase I sweall therr it maaly mocking wher with poonadlos day
NEE
GradCAM
PLRP
KILS) la this good astarel, planect and cay gits ouusly bill aroray silolllsssee,
JCLS Mis gool uscedd plesast tall easy pinng (imnsl 1ill munioo (alandisolton
1a84 pakee srmmy wathrmos ald comaen evrry sear is prosortannayy
1966: place granee mathermss plri conters every your lewer proconttorree
ip regiees gromddlng do the granfling cereoone insolon poonilleg It profocer rits reachonnd can chingy his minii laweeerr lite kaa crael miik for plú, tai
repect grondlon the groilllng cormany lnsilee groetting
afted af
sill
plil wally have the
pull realy lottee the
proclaee (ooddaad moall) dhange
po- living wouudllo
minl. laowerr Jaie las anrl took Se phil. and
living grouadase
day carr and ooe. and oee weel he gratially liww it. and vits tale in koo with kin I gouuilio dhy a will wellica jutals undiasis sul funs causnly the scmecentters descter tamt and donry rabi hom wettom firm,
day were and one and over selll be griftally thee and itis fulk in loee with
lim mosullln das I seil healin woall undiaaii and funt suudlff la anvernnttes diretter ramt one danny
feit
frery torms repeating,
ovclyy sesiluenaal ounol I elliongh idne die reporiiig
may
secrls
cuunly abllongh
the alaie fil more' diection lelps diciegg be doorr I theorghece
tedions, ther tar stacle Hing camts diretion beipe athoogh doon iloonggton divcctor to the this it toreted will toough aid the johns fel aet sill and the caiting thi good copectudy when a daons par of the day over aad ooer gasis carh as alwe pill triee
flads
faat
dinetie texks the film dircted will cauugh. and the are and ser again wiiiigg alio goot repecaally wheee it shons tne part the fay
well and the
pandi as whau plùl thai to lave the peifee sialt vitk cia lae perkoniness are slac mooebert marny is grott fin, and hi trmmaation from eseeal hergy ensothh. and delsers lissi ussall aury siols mulk unskonell is qued
pertect right the the
Excelieat bill namy anall aid his in
laper Braselll
elliossal
and
if poin
llane caraaistes
letwens the loads Cariifllly oclerviee whide cira aller. the communan lazry.. ale frnmy althengs hooe linn godly style. ocherwke you mally gring to hatt bim thnoughen this films and it will kone youe reposteerr of tae tits ovenil the suppecting cud sce It list citin, with stepiee tabolrwetty sifarnee ze palk ald wherl mote red reusere and corn divettr staia poptins up aesnbbggit| is fast there mit one did perferaaaee tlis sooullig mally wocking wrnng with gamdthag dop Strj
Sill part
betwens the leade thinafilly othereter the shale Bii woald probahy full part cleis eliit the easernons hary ahis funs alllongh you lavee to like his soodly style otlerrtee you are realy soing katn lem choonghost the Siil and is will lowce your exportennt the fil oveall the suporriin not ant Ted eeealass tabouoekky lillenee is plil old stand ause aamaon. and disctur fin. prrpiigg up veerokoget goouadles
arclring car perforratoo the polly notting snong with grondingg day Ista
Joscall. lheer
TA
GA
grlng cousty hill cloetlectess 1354 plepearmmyy wenlammn oid couarn, who every
KT BM good saturd plowrtt
[CLS good cantzel
eas suan bill abootisation.
seal prosedtavaay
1966:
repect grmmillon day de gommilan every lewer persoutt
ID to regort in grondhogg day the grandling day cemmone lnooke poondhng
innibea groundase
leisa iitel out of lao und dioca' lez stador, it will be in caly speaas
being Iitd nen a her and be doom i we a chodoe in well be ao early speing
plel mally haww comennyy and cradl trick for plil
produee rits
can change
phil eadls lastes far comonns and estu prodlao sin ansdloeel cus disage
los sitdd hoowoee fate low creal trok be pail and steere liitng grenddogg
guatislly scnesvettins discior and danny colar. undiorsion
lim grenndlee and
day
unil be grodelll Thes
tie fall
bise
traaly
the
writimn
lest
soveraalbes, direetse
dunry mhit
written
bat
one by orittnerca coneds otthongh the iiee day reporting and oee may
nenty weiteneete somely alllengh the idos day repeating oore and mas woud todarae, tizre are sxmmal good jokns to hold the catiezeos ttteakine tlanoghaan the shide file rurll
sound tediooo liase gre fnooial good psoo to hald the miliano atoustion tlacrabort the whace nomel
divcris [Ee liu i dineted belpe and anee
any fluds
and
andl joac tne to kell ts
discttrr tiido (e B liaetted sill
tall joker til set up sill tas le
aks Bos equectily perfos hi
rattig the Biod repectally
lave
with
jstl his fiss rend sisl sndl moddoned il good alloorgh dae just a inth lot too sweet in parts the them demuairy the two loads ciuiifllly
(ill urmmy graat fan Hidl
from
unl celivee his lices his
aomeriaa she a ltile bit too calle modliael is gool thi olllong)
leewons the leatle thenafills otheredoe parts of the tims taree carmatty
would probsall sall part
thi woild prohibly fall yart
ellen.
kazr. lhde Ersry allbenaa you lave to lis thersshent
datt clied uu the couermans hery thu funny althoou sou lasse
ralll grine
guvly style
thronghose
suppolling
cillac nith
mate syoraon, lial diesceer
tebolmewiky atlanne patte old whhol
senn directer
tabe onkky billoroe phile old
runis popping ocandegis is lict Chere oue shal performure thi dits and sonn the anondling gett a fany soes tmmooag cae dasse sweall ther i teally socling wnong with procollng day at dil SEP
nanis prppiis
arssakggie fec. dere is ant cae dhal perfictancc thi
fim. and ovn the greacho gee a bany scms anobring car cheso. seemill them
meily wolting woong with grondfng day et all (EE9)
Ours-H
Ours-T
XLL Bil good sstandl plancat msl sas giiss sausty bill anme sahooleasccss peppermayy wetteroos pii coneen every
[CLS the good norrned plnsser SEE sM0 gaing comeddy Ni merree { thoatieateee
1064; place pragg wortlrmn phil ouuns alai evary your issas pruesttoan
.P sogiel groundlng the gomallig day centans insobm spomillg
-P.a. repect gonnahie day the mrendding dae cermany trmalves groeddane and doen
lelin
will early qpling
plll wellly the
prolicer reacioonlll change
plal
musloeel) liding goondingg
his ame aowever, line cracl mnik for pld, tad
po- living gounilo
wied lonoore fate has
trik pall
day our and ooe and ove unel he gratnally lile it. and sita fate in koo with
tker and uner and
unil be
liks und din fals in love wall
kita grouilion dee B sill sattir jotall undianeee and fune cumnty the weitoms firm
lim. arosualln, das B seil wellins toakly undicaiie and fimtl cundly the anweinnttees, directe rande and donny
scnverenteer deseter nartle and donry
fest nore
firmy tores her day repeatins aad aas
oeclly au cunll sllioogh idce die reposiigg
aay
sorrl ansttnaatii cuunty altlonal
diestion lelps dicoeeni doncs thecnghete
good
the wlick: liss canch directon teipe allhoggh dosce thrmushoor choscter taddo the film dineced will
and
the files
fesl.
directee trake. the fin is diveted miil eaoeth the ard soil and the
the jokne are set siil and the
wiling aki gool repodillly when it
the day and
cilting shi good eepecially when it doons
of the day wer oove
and deisen his las goant ton fnd cenudation esuccal heryo Hi mag
mndd as wien piu. thos to lnow the perfoc stale with mrn fie
wada nn rlass pail tike laae the perfect digh with stte. the prrfermmre ses abs puenteee bill vemmy grant fin and bo
from
hupey srooth
real suay
mndk undeenell Bl goed
alllonais
uad his lises ueed sanry
mooloed is good site. iilaooad
the kods Liniifallyy
whole fimm protuall 3el part.
chete allect the Baaiffllly cilessia de
fall
aller the comernan
aac finmy atthergs
hove lioe
bury alin fine alhooggh havn like atc realyy
godly stsle oclervee
mally to late lim thnorghnt this film. and it will
hale him chooggoeee the Iis and Ls will lowse jarzz impertennt the fi oooaall Bil aupporting cue MI bad otherr soah in plils oll fact there nst one daa performancen fhi the grondd hee yeu frnny
polly style oilersise
krove sore reporteer Eae fits ovenil the sxppocting cus B8 lise ettice with stepeen iebolrwstt
en phils old slood more and rournte
seeplen ooeky asseokannt. ramt popsiia uo tin. wolling soota with goouddin
mase
diowtor
fatt. there nat one dia performance Hhi grandlog Ores finry mall wnus 9ilk poouaallae day SSE))
susia papeiar up arannbggat
their
thew
Figure 14: BERT visualizations for sample with positive emotion in Movie Reviews Dataset. A darker color represents a greater token attribution during the prediction.
26

---

Published in Transactions on Machine Learning Research (02/2023)
RAM
Rollout
GradCAM
PLRP
F
F
TA
GA
Ours-H
Ours-T
-
Polaroid: 99.0%
teddy bear: 0.1%
RAM
Rollout
GradCAM
PLRP
TA
GA
Ours-H
Ours-T
ITT-B-MoCo-v3
Polaroid: 98.9%
teddy bear: 0.4%
RAM
Rollout
GradCAM
PLRP
TA
GA
Ours-H
Ours-T
VÝT-B-DINO
Polaroid: 79.3%
teddy bear: 10.6%
RAM
Rollout
GradCAM
PLRP
TA
GA
Ours-H
Ours-T
ViT-B-MAE
Polaroid: 6.6%
teddy bear: 4.8%
Figure 15: Visualizations class: teddy, teddy bear for ViT-Base with different self-supervised learning (SSL) strategies: MoCo-v3 (Chen al. 2021), DINO (Caron et al. 202I) and MAE (He et al.) (022). Although SSL can influence explanations, our methods show stable performances and good consistency with models' properties.
27


Reviews:
[{"review_id": "K6uBBmYxIY", "reviewer": "Reviewer_Sq8c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The design of the method is methodical and mathematically motivated.\n* The experiments cover a broad range of problems in vision and natural language processing domains, with impressive performance.\n\nWeaknesses:\n* Section 3 is almost impossible to read. The introduced notation is dense and confusing; the flow of the ideas is not clear; and I was not able to understand this section despite trying to read it a few times.\n* Section 3 lacks rigor. For example, a mathematical \"proposition\" should be provable in a rigorous manner. A proposition that says $x\\approx y$ needs to at least define the sense at which $\\approx$ is defined; and prove the statement. \n* I was not able to understand where the differences between the head-wise and token-wise variants of the method come from. Understanding these two designs a bit more would have been preferred.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel method to attribute tokens in transformers. The method generates what is called attention perception and reasoning feedback combines them to get a score to interpret a transformer's decision. The design is mathematically motivated and the intuition is presented in Section 3 (which can still be improved as described later). The experimental results are impressive showing big improvements in NLP and CV domains with various tasks. Overall, I think the paper is definitely interesting to the TMLR audience and there is substantial evidence for its performance that make the paper publishable. In what follows, I will provide major points that require a revision, especially in Section 3.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Given the effectiveness of the method and the great results, I think there is merit in this work for publication. However, major revisions are needed IMO as detailed below:\n\n* Please rewrite Section 3; simplify notations, perhaps introduce a notations table; and make clear the flow/goal of each subsection so that it would be more understandable. Apart from this major requests, Section 3 suffers from minor issues, such as Proposition 3 is referenced before being presented, etc, which should be addressed in a revision.\n\n* Please either remove the use of \"proposition\" in this section (as none of such results is rigorously provable) or add rigor to this section.\n\n* Please add more discussion on the head-wise and token-wise variants and provide more guidance in what situations either would be preferable.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bhJfFv4UCE", "reviewer": "Reviewer_yn1D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- It seems that the two parts in the proposed token attribution expression, *attention perception* and *reasoning feedback*, are analogous to the attributions indicated by bottom-up and top-down signals, respectively. Therefore, it seems that the proposed method well characterizes both how a token contributes to the construction of the [CLS] representation and how a token influences the final prediction. \n- Since not all information stored in the final [CLS] representation will be used for a task-specific prediction, it makes sense that *reasoning feedback* helps reduce noises that could exist when only bottom-up attention attributions were considered.\n- The breadth of both the test types and the model types studied in this paper assures a strong faithfulness evaluation.\n\nWeaknesses\n- The mathematical notations in Section3 are a bit hard to read. Proposition3 is referenced by Section3.1 before it is introduced at a later page.\n- The effectiveness of the proposed approach is only demonstrated through explaining the [CLS] token and how [CLS] is converted to a final prediction. How the proposed approach would perform on explaining normal text tokens remains to be investigated.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an approach to compute the amount of token attributions to the final output in Transformer models. Concretely, the amount of attribution is expressed as a Hadamard product between *attention perception* and *reasoning feedback*. *Attention perception* indicates how a token contributes to the last [CLS] representation, via unfolding each attention block with the chain rule. *Reasoning feedback* indicates how a token contribute to the final prediction based on [CLS], via back-propagating the final prediction w.r.t the last layer's attention map. The paper leverages Integrated Gradient to reduce the noise from irrelevant tokens in the *reasoning feedback* process. \n\nThe effectiveness of the resulting token attributions are validated through a series of tests:\n- In the **Perturbation Test**, tokens are gradually masked based on the predicted attribution score. A sharp performance drop after masking tokens would suggest that the predicted attributions have successfully recognized important tokens.\n- In the **Segmentation Test**, a semantic segmentation map is obtained from (binarized) token attribution maps. A high segmentation metric evaluated from attribution maps would suggest that the predicted attributions have successfully recognized semantically significant tokens.\n- In the **Language Reasoning Test**, tokens with high attribution scores are extracted as a \"rationale\" supporting a classification decidion. A high F1 score between the ground-truth rationale and the rationale predicted from attribution scores would suggest the effectiveness of predicted attributions.\n\nThe paper shows that their token attributions consistently outperforms previous explanation methods on all three tests. Also, the proposed approach remains to be effective when different model sizes, modalities and [CLS]-pooling methods are used. The results hold promising for further downstream usage of token attributions for debugging, evaluating and designing Transformer models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'd like to propose a few adjustments regarding to formatting. They are not critical to securing my recommendation for acceptance because I already believe this is a strong submission.\n\nSection 4.1: Currently the format would mislead a reader into believing that \"Faithfulness Evaluation\" and \"Perturbation Tests\" are parallel to each other. I'd suggest put the three types of tests in a list, aligning the test names.\n\nSection 4.2: I might be missing something, but the caption of Table1&3 did not explain what \"Predicted\" and \"Target\" correspond to. \n\nSection 4.4: In Table4, I suppose \"&check;\" in the feedback column represents the proposed approach while \"$G^{(L)}$\" represents the old way of doing attention gradients without integration. To make the table more readable, it would be clearer to add a sentence clarifying this difference.\n\n", "broader_impact_concerns": "There might be a dual use issue when the proposed explanation approach is applied to tasks that involve predicting human attributes. On the positive side, the explanation approach can help figure out tokens having large impacts on the model prediction, thereby increasing transparency. On the flip side, this might provide a malicious user with access to how to trick a model into making certain predictions via altering the input tokens.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fT2ywjPHLK", "reviewer": "Reviewer_PzQa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. This paper proposes a novel method for inspecting transformers. Extensive experiments are conducted to demonstrate its efficiency. \n\n2. It provides a detailed derivation process to show how to calculate the inspection and provides the code. I appreciate the authors’ efforts.\n\n3. The visualization is fantastic for visually demonstrating the benefits of the proposed method compared with the previous inspection methods.\n\nWeakness:\n\n1. To evaluate the proposed methods’ performance, 5,000 images from the ImageNet val set are selected. Is there any principle behind the selection? For example, are there any special designs to make the selection balance regards the species?\n\n2. It will be better to provide the inspection results on BERT, regarding which word is emphasized. Current visualization only focuses on the visual domain.\n\n3. As shown in the results, the token-wise and head-wise inspection methods have a different emphases. Can the authors comment on the reasons for the difference between these two manners? It will be better to provide guidance on the selection of the two manners.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel method to analyze the transformers by inspecting the token contributions inside transformers. It contains a head-wise and token-wise process to inspect the transformers. Extensive experiments on vision transformers, language transformers, and multi-modality transformers. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness", "broader_impact_concerns": "A broader impact statement should be presented. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CJQ3PMp9ix", "reviewer": "Reviewer_Ut92", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- (S1) The authors proposed attention perception $P$ by directly expanding input-output relations of transformers using the change of basis, which is a novel way of getting token attributions not being investigated in previous studies.\n- (Q1) In equation 10, the authors used $(\\cdot)^{+}$ to reflect the ReLU/GeLU’s function of eliminating negative feature maps. But the implementation of the transformer’s MLP block inserts the activation between $W_{MLP, 1}$ and $W_{MLP, 2}$. Is it OK to apply a positive clamp only at the end?\n- (S2) The method is evaluated not only on fully-supervised ViT/BERT but also on contrastive language-image pre-trained (CLIP) ViT and self-supervised ViT (MAE).\n  - (Q2) I understand the authors focused more on the pooling (GAP) of MAE. Though I think self-supervised learning itself brings some differences in the explanation result. Examining other SSL training methods for ViT, including MoCo-v3 (contrastive) [4] and DINO (EMA distillation) [5], would clear this question. Especially, I am curious about DINO’s result as it already showed promising localization properties.\n- (Q3) Which $\\mathcal{L}$ is used for the visualization of CLIP? I failed to find it in the manuscript.\n- (Q4) Why is CIFAR-100 used for the perturbation test of CLIP? As other image perturbation tests are done on ImageNet, it feels natural to do so for CLIP as well.\n- (Typo, on page 12) Table 8 → Figure 8\n\n[4] Chen, Xinlei, Saining Xie, and Kaiming He. \"An empirical study of training self-supervised vision transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.  \n[5] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Explanation methods for transformers have been developed rapidly. Directly applying the methods used for convolutional neural networks (i.e., gradient-based methods) to transformers would yield noisier interpretations due to its complex global receptive field (multi-head self-attention) and vanishing/saturating gradients. Attention-based methods attract the community as it is intuitively linked with the attention blocks in transformers. This paper provides a more grounded exploitation of attention blocks than previous methods [1,2,3] do, each of which relies on some intuitions for summarizing the explanation. The proposed method factors the explanation into attention perception ($P$) and reasoning feedback ($F$). $P$ resembles [1]’s attention rollout but with a weaker assumption on the attention block’s output projection $W$. Index gymnastics is used to weaken the assumption of $W = I$, resulting in token-wise and head-wise attention maps. $F$ uses the integrated gradient method to focus on relevant positions in attention maps, making the method class-specific. Trustworthiness evaluations, including perturbation, segmentation, and language reasoning tests, are done and show the proposed method’s efficacy to its baselines.\n\n[1] Abnar, Samira, and Willem Zuidema. \"Quantifying Attention Flow in Transformers.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.  \n[2] Chefer, Hila, Shir Gur, and Lior Wolf. \"Transformer interpretability beyond attention visualization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.  \n[3] Chefer, Hila, Shir Gur, and Lior Wolf. \"Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address questions in the Strengths And Weaknesses.", "broader_impact_concerns": "I don’t have any concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your time and all the revisions that you have done to this paper, the reviewers feel they have substantially strengthened the work and hopefully its final impact on the community.  Given my discussions with them, we would like to accept the paper with minor revisions.

Specifically, they have provided the following feedback:

> Section 3 would benefit from a close read by a colleague unfamiliar with the research to help with clarity.  The following are a couple of example pointers from the first paragraph of Section 3 (page 4), where statements are unclear which would likely be caught by a new reader.  There are similar small points thought-out the rest of Section 3 they will be able to help with
> - there is still a pointer to (12) that is hard to understand,
> - $\tilde{x}_i$ needs to be better defined,
> - it is not clear what $(Z_{CLS})_B$ is,
> - Approximation 2 is referenced before it is given,
> - $P_0^{(L)}$ not defined.
> 
> The new roadmap picture is great! but it is not discussed in the main text. Please add a paragraph in the beginning of Section 3 that at a high level connects all of the components of the subsequent subsections that are used to obtain the final approximation, and ground that description both in Section numbers, equation numbers, and the roadmap figure for the reader to better understand where Section 3 is headed.

All the reviewers are positive on your work and results, but worry that the impact will be diminished if you lose readers during the critical methods section.

Decision: Accept (poster)