AttCAT: Explaining Transformers via Attentive Class Activation Tokens
Authors: Yao Qiang
Deng Pan
Chengyin Li
Xin Li
Rhongho Jang
Dongxiao Zhu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Transformer
Explanation
Attribution

Abstract: Transformers have improved the state-of-the-art in various natural language processing and computer vision tasks. However, the success of the Transformer model has not yet been duly explained. Current explanation techniques, which dissect either the self-attention mechanism or gradient-based attribution, do not necessarily provide a faithful explanation of the inner workings of Transformers due to the following reasons: first, attention weights alone without considering the magnitudes of feature values are not adequate to reveal the self-attention mechanism; second, whereas most Transformer explanation techniques utilize self-attention module, the skip-connection module, contributing a significant portion of information flows in Transformers, has not yet been sufficiently exploited in explanation; third, the gradient-based attribution of individual feature does not incorporate interaction among features in explaining the model's output. In order to tackle the above problems, we propose a novel Transformer explanation technique via attentive class activation tokens, aka, AttCAT, leveraging encoded features, their gradients, and their attention weights to generate a faithful and confident explanation for Transformer's output. Extensive experiments are conducted to demonstrate the superior performance of AttCAT, which generalizes well to different Transformer architectures, evaluation metrics, datasets, and tasks, to the baseline methods. Our code is available at: https://github.com/qiangyao1988/AttCAT.

AttCAT: Explaining Transformers via Attentive Class Activation Tokens
Yao Qiang
Chengyin Li Department of Computer Science, Wayne State University {yao, pan. , xinlee, r. jang, dzhu) @wanne. edu
Deng Pan
Xin Li Rhongho Jang Dongxiao Zhu
Abstract
Transformers have improved the state-of-the-art in various natural language pro- cessing and computer vision tasks. However, the success of the Transformer model has not yet been duly explained. Current explanation techniques, which dissect either the self-attention mechanism or gradient- attribution, do not necessarily provide faithful explanation of the inner workings of Transformers due to the fol- lowing reasons: first, attention weights alone without considering the magnitudes of feature values not adequate reveal the self-attention mechanism; second, whereas most Transformer explanation techniques utilize self-attention module, the skip-connection module, contributing significant portion of information flows in Transformers, has not yet been sufficiently exploited in explanation; third, the gradient-based attribution of individual feature does not incorporate interaction among features in explaining the model's output. In order to tackle the above problems, we propose novel Transformer explanation technique via attentive class activation tokens, aka, AtCCAT, leveraging encoded features, their gradients, and their attention weights to generate faithful and confident explanation for Transformer's 's output. Extensive experiments are conducted to demonstrate the superior performance of AuCAT, which generalizes well to different Transformer architectures, evaluation metrics, datasets, and tasks, to the baseline methods. Our http//aggyyo19////CCAAA. 1988/AuCAT.
code is available at:
Introduction
Transformers have advanced the state-of-the-art 2 and see increasing popularity in the field of computer vision [ 4. The main innovation behind
variety of natural language processing tasks
the Transformer models is the stacking of multi-head self-attention layers extract global features from sequential tokenized inputs. However, the lack of understanding of their mechanism increases the risk of deploying them in real-world applications (5]
g 9. This has motivated new research
on explaining Transformers output to assist trustworthy human decision-making [10] 12 3 14
15
16
The self-attention mechanism 18 in Transformers assigns a pairwise score capturing the relative importance between every two tokens image patches attention weights. Thus, common practice is to use these attention weights to explain the Transformer model's output by exhibiting the importance distribution over the input tokens (6) The baseline method, shown RawAtt in Figure 2 utilizes the raw attention weights from a single layer or a combination of multiple layers [10]. 33 question whether highly attentive inputs significantly impact the model outputs. Serrano et al. [ demonstrate that erasing the representations accorded high attention weights do not necessarily lead performance decrease. Jain al. [12] suggest that "attention is not explanation" by observing that attention scores are frequently inconsistent with other feature importance indicators like gradient-based measures. Abnar al. 13] argue that the contextual information from tokens gets more similar as going deeper into the model, leading to unreliable
However, recent studies [11
12
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Bedk-poppgglion
Forward pass
Eo. 12) A
[CLS]
20
A/
Ag
y/o
21
Ag
A
3/1
"like"
42
Ag
hg
1/2
Weighted Vectors
"this"
Ag
1/3
of
"movie"
44
A+
1/4
Attention Weights
Connection
45
Ng
y5
Atteriion
1. Seff
Transformerd Victorr
[SEP]
46 Ag A/ Ag
A
A + R
AAb.
Figure 1: An illustration of Transformer architecture. The left panel shows a simple three-layer Transformer model. Each layer consists of self-attention module and skip connection module (shown in the right panel). The input is sequence of tokens with two added special tokens, i.e., [CLS] and [SEP]. The third token, 'like' (x22) contributes mostly to the positive sentiment prediction since its attention weighted output is the largest. Size of the colored circles illustrate the value of the scalar or the norm of the corresponding vector. Arrows within the circles demonstrate the directions of the vectors.
explanations using the attention weights. The authors propose two methods to combine the attention weights across multiple layers to cope with this issue. Their attention rollout method, shown Rollout in Figure reassigns the important scores to the tokens through the linear combination of attention weights across the layers tracing the information flow in Transformer. However, the rollout operation canceled out the accumulated important scores uniformly distributed attention weights. The attention flow method is formulated as max- max-flow flow problem by dissecting the graph pairwise attentions. While somewhat outperforms the rollout method in specific scenarios, it is not ready to support large-scale evaluations [5].
deeper layers have almost
Recently, Bastings et al. 19] advocate using saliency method opposed to attention explanations. Although some gradient- based methods [20
22 23 have been proposed to leverage salience for explaining Transformer's output, most of them still focus on the gradients of attention weights, i.e. Grads and AttGrads as shown in Figure 2 They suffer from a similar limitation to the above- mentioned iitention-based methods. Layer-wise Relevance Propagation (LRP) method 24 25), which is also considered as a type of saliency method, propagates relevance scores from the output layer to the input. There has been a growing body of work on using LRP to explain Transformers [5]. [5). Voita et al. [14) use LRP to capture the relative importance of the attention heads within each Transformer layer (shown as PartialLRP in Figure However, this approach is limited by only providing partial information on each self-attention head's relevance; no relevance score is propagated back the input. To address this problem, Chefer al. [1] provide a comprehensive treatment of the information propagation within all components of the Transformer model, which back- propaaaass the information through all layers from the output back to the input. This method further integrates gradients from the attention weights, shown as TransAtt in Figure 2 However, TransAtt relies on the specific LRP rules that is not applicable for other attention modules, e.g., co-attention. Thus it can not provide explanations for all transformer architectures 26
21
As such, the existing Transformer explanation techniques are completely satisfactory due to three major issues. First, most attention- methods disregard the magnitudes of the features. The summation operation (Eq, 2 shown in Figure demonstrates both attention weights (the green circles) and the feature (the blue circles) contribute to the weighted outputs (the red circles). In other words, since the self-attention mechanism involves the computation of queries, keys, and values, reducing only to the derived attention weights (inner products of queries and keys) is not ideal. Second, besides the self-attention mechanism, skip connection as another major component in Transformer is not even considered in current techniques. The latter enables the delivery and integration of information by adding an identity mapping from inputs to outputs, trying to solve the model optimization problem from the perspective of information transfer [27]. Moreover, Lu et al. [28) find that a significant

---

Va
Transformer Layer (1) a
Transforeer Layer (1) a
R
Transformer Layer (L) a
T
h
Vh
Alll + (a)(
Kn (Val oa') Biess Bu(Va!")
A Vu(Val R
A = Ba @A*))
1/ Eg l @ (VA oba))
Eretsaar Ba(
BOxt 4, 1.0 A
Ensien A 21.
Basscaus 1 4 4... A14)
Bremnan A .14)... (..)(4))
ExmaA A.
Figure 2: A summary of the existing explanation methods and our methods: CAT and ArCAT. The Transformer consists of several layers denoted as Layer (1), (!).... (L). Va and Vh represent the gradients of attention weights and outputs h, respectively. R is calculated based on layer-wise relevance propagation (LRP). E denotes the explanation method. EH means averaging among multi-head attentions in each layer.
portion of information flow in BERT goes through the skip connection instead of the attention heads (i.e., three times more often than attention on average). Thus, attention alone, without considering the skip connection, is not sufficient to characterize the inner working mechanism of Transformers.
Third, the individual feature attribution-Bas based approaches [1] 29 30 cannot capture the pairwise interactions of feature since gradients relevance scores calculated independently for each individual feature. For example, the gradients directly go through the Transformer layers from the output to the specific input (the token 'like'), shown in Figure
We propose Attentive Class Activation Tokens (AtCAA)) to generate token-level explanations leverag- ing features, their gradients, and their self-attention weights. Inspired by GradCAM [31], which uses gradient information flowing into the last convolutional layer of the Convolutional Neural Network (CNN) to understand the importance of each neuron for the decision of interest, our approach quan- tifies the impact of each token to the class-specific output via its gradient information. We further leverage the self-attention weights to capture the global contextual information of each token since it determines the relative importance of a single token concerning all other tokens the input sequence. By disentangling the information flow across the Transformer layers for specific token into the information from itself via skip connection and the interaction information among all the tokens via self-attention mechanism, we integrate the impact scores, which are generated using AICAT,, from multiple layers to give the final explanation.
A summary of the baseline methods and our AtCAT method is shown in Figure 2 demonstrating their main similarities and differences. The Raw and Rollout [3] methods simply use the attention weights (a)). The Grads method leverages the gradients of attention weights (Va") from the last Transformer layer, while the AutGrads method 22 integrates the attention weights (a) and their gradients (Vo) from all Transformer layers. The PartialLRP method [14 applies LRP only on the last Transformer layer (I24). Differently, the TransAtt method [26) integrates the relevance scores (R) from LRP and the gradients of attention weights (Va). We use CAT, new gradient-baa -based attribution method leveraging the features (h) and their gradients (Vh), as our in-house baseline method. We further integrate attention weights (a) with CAT the proposed AttCAT method.
We state our contributions as follows:
We propose a novel Transformer explanation technique, AtCAT,, leveraging the features, their gradients together with attention weights to generate the so-called impact scores to quantify the influence of inputs the model's outputs.
Our ArСAT exploits both the self-attention mechanism and skip connection to explain the inner working mechanism of Transformers disentangling information flows between intermediate layers.

---

Furthermore, our class activation based method is capable of discriminating positive and negative impacts toward the model's output using the directional information of the gradients.
Finally, conduct extensive experiments different Transformer architectures, datasets, and Natural Language Processing (NLP) tasks, demonstrating a more faithful and confident explanation than the baseline methods using several quantitative metrics and qualitative visualizations.
Preliminaries
2.1
Self-Attention Mechanism
The encoders in Transformer model [T] typically stack L identical layers. Each contains two sub- layers: (a) multi-head self-attention module and (b) feed-forward network module, coupled with layer normalization and skip connection. As illustrated Figure [ each encoder computes the output
h Rd of the i-th token combining the previous encoder's corresponding output h (1-1) from the
skip connection and sequence output (i-11) = (h) (111) h self-attention mechanism:
(1-1 .o h C Rd through
Q(h ((hh-11) Q(n K(h, 15
a == softmax
R,
Vd
Oi ai V(hj (01) + h (n(-1))
W
(2)
where af. is the attention weight assigned to the j-th token for computing h,", d denotes the dimension of the vectors. Here, Q(-), K().. and (-) are the query, key, and value transformations:
Q(h) w ((h) == w h, V(h) = w'h, (WQ, w WV) Rddd
respectively. We drop the bias parameters in these equations for simplicity. For multi-head attentions. we concatenate the output from each head.
2.2 Class Activation Map
GradCAM 31 is the most successful PAM-baseds methods using the gradient information flowing into the last convolutional layer of CNN to understand the importance each neuron for the decision of interest. In order to obtain the class discriminative localization map for the explanation, Grad-CAM first computes the gradient of the score for class i.e., yf before the softmax, dy concerning feature maps A of convolutional layer as øл.. Then, these flowing back gradients are plobal-average-pooeed obtain the neuron importance weight wf:
Oy 0A4K
W
E
where denotes the Alobal-vverage-pollle The weight wc reflects partial linearization of the CNN downstream from A and captures the importance of feature map & for target class Then weighted combination of forward activation maps obtained by:
GradCAM = ReLU
Σ A:
ww AR
where ReLUO is applied filter out the negative values since we are only interested in the features that positively influence the class of interest.
3 Problem Formulation
The objective token-level explanation method for Transformer is to generate separate score for each input token in order to answer the question: Given an input text and trained Transformer

---

model, which tokens mostly inffuence the model's output? There is no standard definition of influence in literature [32] Some works use the term 'importance', whereas others use the term 'relevance' depending on the explanation methods being used. Here we note that the token influence should reflect not only the magnitude of impact but also directionality. As such, we define a new concept, Impact Score, to measure both Magnitude of Impact and Directionality. The former addresses the question "Which input tokens contribute mostly to the output?". And the latter addresses the question "Given an input token, have positive negative contributions been made to the output?" Formally, we define the Impact Score generated by our AtCAT method as follows:
Definition 1 (Impact Score) Given pre-trained T Transformer T(.), an input token and our explanation method FAUCAT(()) Impact Score define as:
[EAACAAT (T())) (igni (FxHC)T(T()))),
Magnitude of Impact,
Impact Score(EAncAr(((EE))
(6)
Directionality.
Remark 1 (Magnitude of Impact) The magnitude of impact indicates how much contribution has been made by each token. A sort function can be applied to the array of scores for the input tokens to retrieve the most impactful tokens on the output.
Remark 2 (Directionality) The sign reveals whether each token makes positive or negative impact on the output.
Attentive Class Activation Tokens
4.1 Disentangling Information Flows in Transformer
To interpret the inner working mechanism of Transformers, essential to understand how the information of each input token flows through each intermediate layer and finally reaches the output. Some previous works [3] use heuristics treat high attention weights and/or their gradients as indicators of important information flows across layers. Others [15] [4] apply LRP aiming to dissect the information flows via layer-wise back- pack-propagation. However, these approaches either rely on the simple-but-ung unrellllli assumption of linear combination of the intermediate layers ignore the major components of Transformer, i.e., the magnitudes of the features and the skip connection.
From Figure I we observe that the output sequence of the Transformer model has one-to-one correspondence to its input sequence. The skip connection is a shortcut that bridges the input and output of the self-attention operation. We note that the Transformer encoder intuitively is operator that adds the representation of token interactions (via self-attention mechanism) onto the original representation of the token (via skip connection). Therefore, from perspective of information flow, we can specify the i-th token's 's information at the (/)-th layer as:
Information
Information(x (x!-1) + Interaction(x 1-1 xx/1))
where Information(x; 11) represents the information contained in the i-th token at the (I-1)-th layer, and Interaction (x| x reflects the summation of all pairwise interaction between the i-th token and all other tokens (n/i).
This observation motivates us to interpret the inner working mechanism of Transformers via dis- entangling the information flow Transformer. Thus, considering Eq. 7 as recurrence relation, the final representation of the i-th token then consists of the original information (the input) plus token interactions between the i-th token and all other tokens at different layers. Since the CNN's last convolutional layer also encodes both high-level semantics and detailed spatial information, corresponding the original information and the interactions herein, the way GradCAM used for explaining CNN model's 's output inspired us to design Attentive Class Activation Tokens (AtCAAT) understand the impact of each token on Transformer model's output.
4.2 Class Activation Tokens
For a pre-trained Transformer, we can always find its output h' at [-th layer. Assume h' has n columns, each column corresponds to input token (including the paddings, i.c.. [CLS] and [SEP]).
5

---

We write its columns separately as hi..... h...... last Transformer layer L, to interpret the impact of i-th token to the final output y for class c, would be straightforward if we have linear relationship between ye and h as follows:
h As h is the output of i-th token from the
h
(8)
where wf is the linear coefficient vector for h.. Inspired by GradCAM 31 we obtain the token important weights as:
Oy
w Vh
ah!
where wf illustrates partial linearization from h and captures the importance of i-th token to target class c. Class Activation Tokens (CAT) is then obtained through a weighted combination:
(10) where @ is the Hadamard product. CAT. denotes the impact score of the i-th token at L-th layer towards class c. Note that we do not apply ReLUO to filter out the negative scores here since we also care about the directionality of the impact score.
CAT Vh h/
4.3 Attentive CAT
While CAT explains the model's output according to the attribution of each individual token's encoder output (Eq. [), it does not consider the interaction among tokens, which is revealed via the self- attention mechanism. The self-attention mechanism [8] assigns a pairwise similarity score between every two tokens as the attention weight, encoding the important interaction information of these tokens. Therefore, we integrate self-attention weights with CAT to further incorporate the token interaction information for better quantifying the impact of each token on the Transformer model's output. Our Attentive CAT (AtCAT)) at I-tth layer for i-th token is then formulated as:
AttCAT En (a CAT/),
(11)
where a denotes the attention weights of the i-th token at L-th layer. E+)() means averaging over multiple heads.
Recall that Eq. represenns recurrence relation,
always find the output of I-th layer and assign it as v/. We use Eq. 9 10 and 11 formulate AttCAT, denoting the impact score for i-th token at 1-th layer.
Finally, different from the Rollout and TransAtt methods that apply the rollout operation, we sum AUCAT, over all Transformer layers as the final impact score of i-th token as follows:
AutAT,
ALCATT
(12)
We empirically demonstrate that the summation is more effective way than Rollout in Figure
Experiments
5.1 Desirable Properties of an Explanation Technique
We first introduce two desirable properties of an explanation method: faithfulness and confidence. along with metrics to systematically evaluate the performance of various explanation techniques.
Faithfulness quantifies the fidelity of explanation technique by measuring if the tokens identified indeed impact the output. We adopt two metrics from prior work to evaluate the faithfulness of word-level explanations: the over the perturbation curve (AOPC) [33] 34] and the Log-odds scores [35] 34]. These two metrics measure local fidelity by deleting or masking the top k% scored words and comparing the probability change on the predicted label.
Confidence token can receive several saliency scores, indicating contribution to the prediction of each class. The tokens with higher impact scores of the predicted class should also have lower

---

impact scores for the remaining classes. In other words, the explanation techniques should be highly confident in recognizing the most impact tokens of the desired class (usually the predicted class). On the other hand, these tokens should have the most negligible impact on other classes. We use Kendall-7 correlation, the statistic measuring the strength of association between the ranked scores of different classes, evaluate the confidence of an explanation method.
5.2 Experiment Seftings
Transformer models: BERT [2 is one the most representative Transformer models with impres- sive performance across a variety of NLP tasks, e.g., sentiment analysis and question answering. We use the BERT base model and some variants (i.e., DistillBERT [66] and RoBERTa (37] in our experiments. Our method can be generally applied to other Transformer architectures with minor modifications. The pre-trained models from Huggingface are used for validating our explanation method and comparing it to others. More details of these Transformer models and their prediction performance are presented in Appendix A
Datasets: We evaluate the performance using the following exemplar tasks: sentiment analysis on SST2 [3 Amazon Polarity, Yelp Polarity [39], and IMDB 40 data sets; natural language
inference on MNLI 41 data set, paraphrase detection on QQP 42 data set; and question answering
on SQuADvl 43 and SQuADv2 44 data sets. More details of these data sets are described in
Appendix B
Several baseline explanation methods for Transformer have been compared through our experiments, including the attention-based methods (i.e., RawAtt and Rollout [3)), the attention gradient-based methods (i.e., Grads and AttGrads (22]), the LRP-based methods (i.e., Par [14] and TransAtt (15). CAT without incorporating attention weights is an ablation
Baseline methods:
version of AtCAT. Figure summarizes and compares these methods with formulations.
5.3 Evaluation Metrics
AOPC: By deleting top 1% words, AOPC calculates the average change of the prediction probability on the predicted class over all test examples as follows:
1 N Σ p()));)))))))))))))))())))))pp p(O)x*),
AOPC(k)
(13)
where N is the number of examples, ÿ is the predicted label, p(Ol-) is the probability the predicted class, and x is constructed by removing the 1% top-scored words from x; To avoid choosing an arbitrary k; we remove 0, 10, 20, ... 100% of the tokens in order decreasing saliency, thus _x000 Higher values of AOPC are better, which means the deleted words are more impactful on the model's 's output.
arriving x,, x,
LOdds: Log-odds score is calculated by averaging the difference of negative logarithmic probabilities on the predicted class over all test examples before and after masking 1% top-scored words with zero paddings,
N
1 N log
LOdds(k) =
(14)
p())
The notations are the same as in Eq. 13 with the only difference that x is constructed by replacing the top 6% word with the special token [PAD] in x; Lower LOdds scores are better.
Kendal correlation: We use the Kendal-7 to evaluate confidence of explanation method, formally:
N N
Kendal correlation
( S(x:)c/e)
(15)
where S(x;) denotes an array of the token index in order of the decreasing saliency (or attribution, or relevance, or impact) scores for a test example. A lower Kendal correlation demonstrates the explanation method is more confident in generating the saliency scores for predicting the class c.
httpp:://nugiigffaeec./

---

0.5
NISAT
0.0
CAT
TransAtt
0.4
Perli
Mttrods
0.3
LOd6s
-1.5
0.2
2.0
Grad5
2.5
Rellout
RaweM
-3.0
20
30
40
50
80
90
20
30
40
50
70
80
90
Figure 3: AOPC and LOdds scores of different methods in explaining BERT against the corruption rate & on Amazon data set. Higher AOPC and lower LOdds scores are better. The x-axis demonstrates removing or masking the 1% of the tokens order decreasing saliency.
Method
SST2
QQP
MNLI
Amazon
Ydl
IMDB
AOPC1
LOdds]
AOPC
LOdds
AOPC
LOdds
AOPC
LOdds
AOPC
LOdds
AOPC
LOdds
RawAu
0.331
-0.885
0.143
0.149
0.138
0.235
0.384
-1.729
0.394
-2.017
0.298
-1.245
Rollout
0.286
-0.641
0.139
0.262
0.151
0.321
0.324
-1.303
0.277
-1.055
0.331
-1.323
Grads
AttGrads
0,351
-0.603
0.143
0.113
0.159
0.114
0.346
-1.941
0.439
-2.054
0.310
-1.267
PartialL RP
0.341
0.922
0.142
0.137
0.138
0.231
0.418
2.019
0.424
-2.199
0.312
-1.321
TransArt
0.354
-1,038
0.145
0.114
0.130
0,214
0.415
-1 1.889
0.434
-2,508
0.421
-2.137
CAT 0.352
-1.115
0.134
0.121
0.157
0.121
0.409
2.157
0.421
2.587
0.406
.3005
-1.319
0.139
0.073
0.164
-2.332
-3.671
Table 1: AOPC and LOdds scores of different methods in explaining BERT on different data sets. Higher AOPC and lower LOdds scores are better. Best results are in bold face.
Precision @ K: Inspired by the original Precision @ K used in recommender system 45 novel Precision @ K evaluate the explanation performance SQuAD) data sets. For each test example, we count the number tokens the answer that appear in the K top-scored tokens as Precision @ K. Therefore, higher Precision @ K scores better.
we design
Results and Discussions
6.1 Quantitative Evaluations
The quantitative evaluations this Section demonstrate our ArCAT method outperforms the baseline methods on the vast majority of different data sets and tasks. Table I depicts the results of various explanation methods and data sets. We report the average AOPC and LOdds scores over & values. Due to computation costs, experiment the Amazon, Yelp, and IMDB data sets. Entire test sets are used for other data sets. AtСAT achieves the highest AOPC and lowest LOdds scores in most settings, demonstrating that the most impactful tokens for model prediction have been deleted or replaced. Among all the compared methods, the attention-baa based methods (i.e.. RawAtt and Rollout) perform worst since attention weights alone without considering the magnitudes of feature values are not adequate to analyze the inner working mechanism of Transformers. Remarkably, AtCAT also outperforms Trans recent work representing a strong baseline method. The performance CAT, shown here as an ablation study, drops markedly, corroborating the effectiveness of using self-attention weights in ArCAT.
subset with 2,000 randomly selected samples for
We also report the AOPC and LOdds scores of different methods in explaining BERT by deleting masking bottom 6% words different data sets in Appendix Table Our AtCAA achieves the lowest AOPC and highest LOdds, demonstrating that ArCAT efficiently captures the most impactful tokens for model predictions.
Figure illustrates how the evaluation metrics, namely AOPC and LOdds, change over the varying corruption rates (via removing masking the 4% top-scored words). Our ArCAT method achieves the highest AOPC and the lowest LOdds scores within a corruption rate & of 50% less, further demonstrating that AtСAT has detected the most impactful words for model predictions.

---

ADv1
SQuADrE
0.00
00.00
00.05
0.14
-0.08
.0.04
40.31 0.00
Rolbul
0.03
AMGAT
1.87
1.00
10.00
00.05
00.09
0.04
0.09
0.00
0.03
10.00
0:10
-0.09
00.02
0.00
0.00
0.00
044
1,40
1.55
00.00
00.01
0.03
0.05
-0.05
-0.03
.0.02
0.00
+0.00
00.04
0.02
0.02
-0.03
+0.00
0.00
0.00
1:30
+0.05
-0.00
0.11
-0.03
0.00
0.00
L8
10.00
+0.01
0.00
0.00
+0.06
0.00
0.00
0.00
-0.0
6.47
0.40
L30
0.00
0.00
0.01
0.01
-0.00
-0.03
10.00
0.00
Rollout
+0.00
10.00
-0.00
-0.00
-0.00
-0.00
00.00
0.00
RaBERTa
RoBERTa
movie[SEP]
Figure 4: Precision @ 20 scores of the selected ex- Figure 5: Heatmap of the normalized impact planation methods for different Transformer models scores from different BERT layers. Rollout on SQuAD data sets. Higher scores are better. The and Sum denote the rollout and summation max scores of SQuADvl and SQuADv2 are 3.72 operations (ours), respectively. Best viewed in color.
and 3.84, respectively.
Method
STT2
QQP
MNLI
Amazon
Yelp
IMDB
Grads
0.150
0.236
0.169
0.146
0.174
0.098
AttGrads
0.116
0.198
0.156
0.148
0.132
0.064
PPrtialLRP
0.955
0.949
0.935
0.965
0.952
0.858
TransAu
0.336
0.222
0.339
0.152
0.121
0.043
CAT
0.101
0.373
0.339
0.095
0.107
0.056
AuCAT
0.018
0.349
0.017
0.015
0.008
0.023
Table 2: Kendal-T correlation of different explanation methods in explaining BERT on varying data sets. Lower scores are better. Only class-specific methods are selected. Best results are in bold face.
Table shows the Kendal-T based confidence score of the different explanation techniques for BERT tested using various data sets. We do not report the confidence scores of the attention based methods since they are class agnostic. AtCAT achieves the best performance on most data sets; different classes observe distinctively sorted tokens, leading much lower Kendal correlations. In other words, our ArCAT is highly confident in recognizing the most impactful tokens for predicting the class of interest.
We show the Precision @K scores for the SQuAD data sets in Figure Here A set to 20. Our results clearly demonstrate that AUCAT is superior to other methods and generalizes well to various BERT architectures on SQuAD data sets. The higher score means that AuCAT can capture more impactful answer tokens in the TOP-20 sorted tokens, proving its capability to generate more faithful explanations. The results of varying values are shown in Appendix Figure 9 10
6.2 Qualitative Visualizations
Lastly, we show a heatmap of the normalized impact scores generated by ArCAT in Figure The first 12 rows (L.0-L11) show the impact scores of each token from different BERT layers. The darker shaded token represents a higher score, as shown in the legend. The signs of scores indicate their directionalitics. This heatmap also justifies the effectiveness of the summation operation we used in Eq. 12] As shown in the figure, the impact scores become uniform and less impactful as the layer goes deeper, which is consistent with the observation from [13] where the authors argue that the embeddings are more contextualized and tend to carry similar information in the deeper layers. Thus, the rollout operation used [3] [5] will attenuate the impact scores at shallower layers (i.e.. LO-L9) since they are multiplied by scores at the deeper layers (i.e., L10-L11). 1)). As shown in the row of 'Rollout' in the figure, the rollout operation only gives minimal impact scores of the tokens, indicating essentially no information has been captured for the explanation. While the summation operation (ours), shown as the row of 'Sum', generates a faithful explanation incorporating the impact scores from each layer. In term of Impact Score, the token 'not" with the highest positive impact

---

(CL5) whtch team ropresented the super
[SEP] supor bowl was american football game to determine
champion of the national football league nn for the 2015 seison american football conference jafe chhmmponn Seroel broncos
delecsed the notional footboll conference nilo champion carolinss ponthers 24 10 to earn their third super buxl Uile the yome was dara caltorii
played february 2016
In the francisco
50th super
bowl the league amphasized the golden anriversary with various gold thamad inttiativss as wall as tamporarty suspend maing the under which the game
tradttin naming
super howl game
sttmeral
have been known super
that the logo cound prominently feature the ambbic nu wimerol ees 50 [SEP]
(a) A visualization of the impact scores generated by AttCAT on a showcase example in SQuAD.
(a) AttCAT [CLS] really didn
1K0
this
movie
some of actors were
good
but
overall the
movie
boring
[SEP]
(b) TransAtt [CLS]
really didn
movie
some
actors were
good
but
overall movic boring
[SEP]
(c) RawAtt [CLS]
really didn
Ike
this
movie
some of the actors were
good
but overall the movie
was
boring
[SEP]
(d) Rollout [CLS]
[SEP]
(h) Visualizations of the impact scores generated by the selected methods showcase example in SST2.
Figure 6: Visualization examples. The green shade indicates an important positive impact whereas the read shade means otherwise. Darker colors represent higher impact scores. Best viewed in color. More examples are shown in Appendix.
score (0.72) contributes mostly to the negative sentiment of this sentence, whereas the token 'like' with the highest negative impact score (-0.37) contributes inversely.
The ground truth answer of the question answering example shown in Figure ba "denver brconcos". AuCAT successfully captures these two tokens with the darkest green shades, corresponding to highest impact scores. The example from SST2 shown in Figure Toarebbbiia has negative sentiment. Both AtСAA and TransAtt capture the most impactful tokens, such as 'borin',, 'didn', and 't', which contribute mostly the negative sentiment prediction. Besides the tokens explaining the negative sentiment, our ÁttСA method also identified some other tokens that contribute inversely to the negative sentiment, e.g., 'like' and 'really' (shown in dark shade of red), whereas TransAtt is not capable of differentiating positive and negative contributions. RawAtt gives more attention on some irrelevant tokens, i.e., 'overall", 'but', and the punctuations. Rollout only generates some uniformly distributed important scores for the tokens.
7 Conclusion
This work addresses the major issues in generating faithful and confident explanations for Trans- formers via novel attentive class activation tokens approach. AtCAT leverages the features, their gradients, and corresponded attention weights to define the so-called impact scores, which quantify the impact of inputs on the model's outputs. The impact score can give both magnitude and direction- ality of the input tokens' impact. We conduct extensive experiments on different Transformer models and data sets and demonstrate that our AtСAA achieves the best performance among strong baseline methods using quantitative metrics and qualitative visualizations.
Even though current AtСAT approach is mainly designed for BERT architectures NLP tasks, it can be naturally extended to Vision Transformer architectures on computer vision tasks as the future work. Since there are various versions of Transformer architectures, c.g., ViT 3 and Swin Transformer [4) which are much different from Transformers used on NLP tasks, it opens up new avenues to extend our AtСAA explain these models prediction.
Acknowledgments
This work is supported by the National Science Foundation under grant IIS-221 1897.
10

---

References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in newral information processing systems, 30, 2017.
[2] Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1XXi::810004805 2018.
[3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192 2020.
[4] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[5] Deng Pan, Xin Li, and Dongxiao Zhu. Explaining deep neural network models with adversarial gradient integration. In IJCAI, pages 2876-2888 2883, 2021.
[6] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04344 2019.
[7] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What know about how bert works. Transactions of the Association for Computational Linguistics, 8:842- 866. 2020.
[8] Yao Qiang, Xin Li, and Dongxiao Zhu. Toward tag-free aspect based sentiment analysis: A multiple attention network approach. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2020.
[9] Yao Qiang, Supriya Tumkur Suresh Kumar, Marco Brocanelli, and Dongxiao Zhu. Tiny rnn model with certified robustness for text classification. In 2022 International Joint Conference on Neural Networks (IJCNN, pages 1-8. IEEE, 2022.
[10] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXiv preprint arXiv: 1908. 08593, 2019.
[11] Sofia Serrano and Noah A Smith. attention interpretable? arXiv preprint urXiv:1906.0377 2019.
[12] Sarthak Jain and Byron C Wallace. (tt:::9902.11880 2019.
Attention is not explanation.
arXiv preprint
[13] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. arXiv preprint vrXiv:2005.00928 2020.
[14] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-heat self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:/905.00.0 2019.
[15] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 782-791, 2021.
[16] Xin Li, Xiangrui Li, Deng Pan, Yao Qiang, and Dongxiao Zhu. Learning compact features via in-training representation alignment. arXiv preprint arXiv:2211.13322 2022.
[17] Yao Qiang, Chengyin Li, Marco Brocanelli, and Dongxiao Zhu. Counterfactual interpolation augmentation (cia): A unified approach to enhance faimess and explainability of dnn. In Proceedings of the Thirty-Firs International Joint Conference on Artificial Intelligence IJCAI, pages 732-739, 2022.

---

[18] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint wrXiv:149.04772 2014.
[19] Jasmijn Bastings and Katja Filippova. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? arXiv preprint arXiv:2010.05607, 2020.
[20] Xin Li, Yao Qiang, Chengyin Li, Sijia Liu, and Dongxiao Zhu. Saliency guided adversarial training for learning generalizable features with applications to medical imaging classification system. arXiv preprint (rXiv:2209.4433) 2022.
[21] Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Self-attention attribution: Interpreting information interactions inside transformer. arXiv preprint arXiv:2004. /007, 2, 2020.
[22] Oren Barkan, Edan Hauon, Avi Caciularu, Ori Katz, Itzik Malkiel, Omri Armstrong, and Noam Koenigstein. Grad-sam: Explaining transformers via gradient self-attention maps. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 2882 2882-2887, 2021.
[23] George Chrysostomou and Nikolaos Aletras. Enjoy the salience: Towards better transformer- based faithful explanations with word salience. arXiv preprint arXiv:2108. 13759, 2021.
[24] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Rober- Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, ((7))e0130140, 2015.
[25] Grégoire Montavon, Sehastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus- Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern recognition, 65:211 1222, 2017.
[26] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397-406 2021.
[27] Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu Sun, and Yuexian Zou. Rethinking skip connection with layer normalization. In Proceedings of the 28th International Conference on Computational Linguistics, pages
2020.
[28] Kaiji Lu, Zifan Wang, Piotr Mardziel, and Anupam Datta. Influence patterns for explaining information flow in bert. Advances Neural Information Processing Systems, 34, 2021.
[29] Eric Wallace, Jens Tuyls, Junlin Wang, Sanjay Subramanian, Matt Gardner, and Sameer Singh. Allennlp interpret: A framework for explaining predictions of nlp models. arXiv preprint 1rXi::1909.099 2019.
[30] Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. diagnostic study of explainability techniques for text classification. arXiv preprint arXiv:2009.13295. 2020.
[31] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[32] Brent Mittelstadt, Chris Russell, and Sandra Wachter. Explaining explanations in ai. In Proceedings of the conference on fairness, accountability, and transparency, pages 279-288. 2019.
[33] Dong Nguyen. Comparing automatic and human evaluation of local explanations for text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1069-1078, 2018
12

---

[34] Hanjie Chen, Guangtao Zheng, and Yangfeng Ji. Generating hierarchical explanations on text classification via feature interaction detection. arXiv preprint arXiv:2004.02011 2020.
[35] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference machine learning, pages PMLR, 2017.
[36] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 2019.
[37] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:190..11992 2019.
[38] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pages 1631-1642, 2013.
[39] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[40] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150,
2011.
[41] Adina Williams, Nikita Nangia, and Samuel R Bowman. coveraceverage challenge corpus for sentence understanding through inference. arXiv preprint 1704v:/704. 2017.
[42] Zihang Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs. 2017.
[43] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension text. arXiv preprint 1606.05500 2016.
[44] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv: 1 18001800.0.800 03822, 2018.
[45] Deng Pan, Xiangrui Li, Xin Li, and Dongxiao Zhu. Explainable recommendation via in- terpretable feature mapping and evaluation of explainability. In ICCAI, pages 2690-2696, 2020.
13


Reviews:
[{"review_id": "VHTq9w1v9Gv", "reviewer": "Reviewer_HxzL", "summary": "This paper proposed two methods to understand how transformers work: CAT and AttCAT. They are motivated to take care of the magnitudes of the features, the gradients, and the skip connection to examine (1) which tokens mostly influence the model's output and (2) whether a token pose a positive or negative contribution to the output. Both CAT and AttCAT are developed based on GradCAM. It also studied a lot of previous approaches and mainly compared weight-based methods, gradient-based methods, and methods based on layer-wise relevance propagation. Results on diverse common benchmarks show the strength of the proposed method, including SST2, QQP, MNLI, Amazon, Yelp, and IMDB. Some qualitative results are included to help understand.", "questions": "The reviewer is mainly concerned with the missing of potential ablation studies and will adjust scores if it gets addressed properly.", "limitations": "The reviewer currently did not see sufficient ablation studies to support their claims. Also, the method currently did not provide evidence to support understanding of how transformers work in vision tasks.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The reviewer believes the studied direction is pretty much important for related communities where transformers gradually become the most common techniques we used in various tasks. The reviewer also agrees with the authors that we should also consider the magnitude of the features and the skip connection which play big roles in transformers. \n\nThey carefully studied a lot of literature and mainly compared with three categories including: (1) attention-weights-related works, like RawAtt and Rollout; (2) gradient-related works, like Grads and AttGrads; (3) layer-wise relevance propagation methods, like PartialLRP and TransAtt. Figure2 clearly tells the difference between the proposed method and all the previous approaches. \n\nExtensive quantitative experiments are conducted over various datasets including SST2, QQP, MNLI, Amazon, Yelp, IMDB, and SQuAD v1 and v2. Results show the proposed method generally achieves better performance compared with the previous approaches. The reviewer especially likes the red notation in the visualization of the proposed method which indicates the negative effect of the input texts.\n\n\nThe paper currently fused the introduction and related works into one section, which makes the introduction section pretty lengthy and a bit hard to read. Although the authors described the differences between the proposed method and previous methods in many places (intro, method, exps), the reviewer did not clearly understand the advantages of the proposed method and the key point leading to the good performance.\n\nAlso, the reviewer did not see related ablation studies related to two major claims that we should exploit (1) the magnitudes of the features and (2) the skip connections. There are huge missings, where the authors can not consider the skip connections when aggregating the information to see if considering skip connections really help the understanding.\n\nBesides, the abstraction shows the reviewer the proposed method address various different tasks while it mainly focused on NLP tasks and currently have no discussion to tasks in other domains. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DOTQceLuB2", "reviewer": "Reviewer_1Khh", "summary": "The paper proposes \"Attentive Class Activation Tokens\" (AttCAT), a post-hoc method for the explanation of Transformer models addressing NLP tasks.\n\nDifferent from existing related methods which only focus specific components of the models being explained (leading to reduced faithfulness) and/or heuristics, the proposed method stresses the use of a) the features encoded by the model, b) their gradients, and c) their associated attention weights as a complete package to address the weaknesses in existing explanation methods. \nThis enables not only the explanation of the parts of the input (tokens) that have a high impact on the prediction made by the model, but also whether this impact positively or negatively contributed to the prediction (directionality).", "questions": "N.A.", "limitations": "N.A.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "= Strengths\n\n+ The manuscript had a very good structure and organization. This led to clear content with a good flow. Overall I enjoyed reading this paper. I applaud the authors for the effort put on the presentation of this manuscript.\n\n+ The proposed method is relatively simple. The inclusion of each of the components (the features encoded by the model, their gradients, and their associated attention weights) is well motivated.\n\n+ On the qualitative side, to the best of my knowledge, the proposed method is novel and complementary to what is out there. On the quantitative side, it obtained state-of-the-art results w.r.t. existing methods.\n\n+ Empirical validation of the method was conducted considering a rich set of well known components including several transformer architectures, several NLP-related datasets, and several model explanation baselines from the literature.\n\n+ Observations made via the proposed method align with observation from previous efforts, e.g. [11] in Sec. 6.2. This already hints at the added value of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9Y6ZagbQTq", "reviewer": "Reviewer_ra8V", "summary": "The paper proposes a new method, termed AttCAT, for evaluating the importance of each input token in Transformers. The proposed method considers two perspectives for token importance, including the attention perspective and the gradient perspective. The authors introduce Class Activation Tokens, which is inspired by GradCAM [26]. And the Class Activation Tokens is combined with the Transformer attention to output AttCAT. The experimental results show that the resulting metric for token importance is superior to some other methods.", "questions": "- Instead of simply averaging the AttCAT over multiple heads and multiple layers, are there better methods for doing the aggregation of AttCAT here?\n- Could the author provide the statistics of the datasets, e.g., number of classes and dataset size?", "limitations": "The authors mention that they would extend the AttCAT method to explain generative and vision Transformer architectures as future works. But more discussions on limitations of this work would be useful.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The method is well motivated and clearly stated. It makes sense to combine the attention scores and the gradient weights to obtain a new metric for token importance in Transformers.\n- The experimental results clearly demonstrate the effectiveness of the proposed metric.\n\n## Weaknesses\n- The experiments could be extended. It would be helpful to see how the evaluation metrics, namely AOPC and LOdds, change against the corruption rate $k$ (removing the $k$% top-scored words). Also, it is interesting to see how the two metrics change when we remove the  $k$% lowest scored words. This is different from removing the $k$% top-scored words because the inputs to the model are different in the two cases. When removing the $k$% lowest scored words, we are caring the order of the less informative tokens, while in removing the $k$% lowest scored words, we are caring the order of the most informative tokens.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1k1wgU9-3bT", "reviewer": "Reviewer_pvDc", "summary": "The paper proposes a Transformer explanation technique via attentive class activation tokens, aka, AttCAT, leveraging encoded features, their gradients, and their attention weights to generate a faithful and confident explanation for Transformer’s output.\n", "questions": "In Table 1, when evaluating on QQP, TransAtt seems to have a better evaluation performance than the proposed method, AttCAT. Is there a reason why this happens?\n\nSimilar pattern can be observed in Table 2, AttGrads also has a better performance than the proposed method, AttCAT. Does this mean the evaluation performance heavily depends on the dataset and may not be consistent? Therefore, is the improvement of AttCAT also conditional?\n\nIn the qualitative comparison, the author does not compare AttCAT against CAT. I think this is the most important comparison, especially when the difference between AttCAT and CAT is very limited.\n", "limitations": "The work does not talk about its limitations.\n\nEspecially refer to above, it would be important to know in which dataset, AttCAT is better than former ones and in which dataset similar to QQP AttCAT may not perform better.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work clearly defines the problem and presents a detailed work-through with formulas of the proposed method.\n\nThe work also present experiments with obvious gains to demonstrate the effectiveness of the proposed method\n\nThe proposed method seems to have limited novel differences against the former method, CAT.\nThe work needs more qualitative and quantitative evaluation methods to prove why AttCAT is better than former methods in helping people understand Transformers instead of just showing performance numbers.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept