Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a Transformer explanation technique via attentive class activation tokens, aka, AttCAT, leveraging encoded features, their gradients, and their attention weights to generate a faithful and confident explanation for Transformer’s output.

strengths_and_weaknesses: The work clearly defines the problem and presents a detailed work-through with formulas of the proposed method.

The work also present experiments with obvious gains to demonstrate the effectiveness of the proposed method

The proposed method seems to have limited novel differences against the former method, CAT.
The work needs more qualitative and quantitative evaluation methods to prove why AttCAT is better than former methods in helping people understand Transformers instead of just showing performance numbers.


questions: In Table 1, when evaluating on QQP, TransAtt seems to have a better evaluation performance than the proposed method, AttCAT. Is there a reason why this happens?

Similar pattern can be observed in Table 2, AttGrads also has a better performance than the proposed method, AttCAT. Does this mean the evaluation performance heavily depends on the dataset and may not be consistent? Therefore, is the improvement of AttCAT also conditional?

In the qualitative comparison, the author does not compare AttCAT against CAT. I think this is the most important comparison, especially when the difference between AttCAT and CAT is very limited.

limitations: The work does not talk about its limitations.

Especially refer to above, it would be important to know in which dataset, AttCAT is better than former ones and in which dataset similar to QQP AttCAT may not perform better.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new method, termed AttCAT, for evaluating the importance of each input token in Transformers. The proposed method considers two perspectives for token importance, including the attention perspective and the gradient perspective. The authors introduce Class Activation Tokens, which is inspired by GradCAM [26]. And the Class Activation Tokens is combined with the Transformer attention to output AttCAT. The experimental results show that the resulting metric for token importance is superior to some other methods.
strengths_and_weaknesses: ## Strengths
- The method is well motivated and clearly stated. It makes sense to combine the attention scores and the gradient weights to obtain a new metric for token importance in Transformers.
- The experimental results clearly demonstrate the effectiveness of the proposed metric.

## Weaknesses
- The experiments could be extended. It would be helpful to see how the evaluation metrics, namely AOPC and LOdds, change against the corruption rate $k$ (removing the $k$% top-scored words). Also, it is interesting to see how the two metrics change when we remove the  $k$% lowest scored words. This is different from removing the $k$% top-scored words because the inputs to the model are different in the two cases. When removing the $k$% lowest scored words, we are caring the order of the less informative tokens, while in removing the $k$% lowest scored words, we are caring the order of the most informative tokens.

questions: - Instead of simply averaging the AttCAT over multiple heads and multiple layers, are there better methods for doing the aggregation of AttCAT here?
- Could the author provide the statistics of the datasets, e.g., number of classes and dataset size?
limitations: The authors mention that they would extend the AttCAT method to explain generative and vision Transformer architectures as future works. But more discussions on limitations of this work would be useful.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes "Attentive Class Activation Tokens" (AttCAT), a post-hoc method for the explanation of Transformer models addressing NLP tasks.

Different from existing related methods which only focus specific components of the models being explained (leading to reduced faithfulness) and/or heuristics, the proposed method stresses the use of a) the features encoded by the model, b) their gradients, and c) their associated attention weights as a complete package to address the weaknesses in existing explanation methods. 
This enables not only the explanation of the parts of the input (tokens) that have a high impact on the prediction made by the model, but also whether this impact positively or negatively contributed to the prediction (directionality).
strengths_and_weaknesses: = Strengths

+ The manuscript had a very good structure and organization. This led to clear content with a good flow. Overall I enjoyed reading this paper. I applaud the authors for the effort put on the presentation of this manuscript.

+ The proposed method is relatively simple. The inclusion of each of the components (the features encoded by the model, their gradients, and their associated attention weights) is well motivated.

+ On the qualitative side, to the best of my knowledge, the proposed method is novel and complementary to what is out there. On the quantitative side, it obtained state-of-the-art results w.r.t. existing methods.

+ Empirical validation of the method was conducted considering a rich set of well known components including several transformer architectures, several NLP-related datasets, and several model explanation baselines from the literature.

+ Observations made via the proposed method align with observation from previous efforts, e.g. [11] in Sec. 6.2. This already hints at the added value of the proposed method.
questions: N.A.
limitations: N.A.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed two methods to understand how transformers work: CAT and AttCAT. They are motivated to take care of the magnitudes of the features, the gradients, and the skip connection to examine (1) which tokens mostly influence the model's output and (2) whether a token pose a positive or negative contribution to the output. Both CAT and AttCAT are developed based on GradCAM. It also studied a lot of previous approaches and mainly compared weight-based methods, gradient-based methods, and methods based on layer-wise relevance propagation. Results on diverse common benchmarks show the strength of the proposed method, including SST2, QQP, MNLI, Amazon, Yelp, and IMDB. Some qualitative results are included to help understand.
strengths_and_weaknesses: The reviewer believes the studied direction is pretty much important for related communities where transformers gradually become the most common techniques we used in various tasks. The reviewer also agrees with the authors that we should also consider the magnitude of the features and the skip connection which play big roles in transformers. 

They carefully studied a lot of literature and mainly compared with three categories including: (1) attention-weights-related works, like RawAtt and Rollout; (2) gradient-related works, like Grads and AttGrads; (3) layer-wise relevance propagation methods, like PartialLRP and TransAtt. Figure2 clearly tells the difference between the proposed method and all the previous approaches. 

Extensive quantitative experiments are conducted over various datasets including SST2, QQP, MNLI, Amazon, Yelp, IMDB, and SQuAD v1 and v2. Results show the proposed method generally achieves better performance compared with the previous approaches. The reviewer especially likes the red notation in the visualization of the proposed method which indicates the negative effect of the input texts.


The paper currently fused the introduction and related works into one section, which makes the introduction section pretty lengthy and a bit hard to read. Although the authors described the differences between the proposed method and previous methods in many places (intro, method, exps), the reviewer did not clearly understand the advantages of the proposed method and the key point leading to the good performance.

Also, the reviewer did not see related ablation studies related to two major claims that we should exploit (1) the magnitudes of the features and (2) the skip connections. There are huge missings, where the authors can not consider the skip connections when aggregating the information to see if considering skip connections really help the understanding.

Besides, the abstraction shows the reviewer the proposed method address various different tasks while it mainly focused on NLP tasks and currently have no discussion to tasks in other domains. 
questions: The reviewer is mainly concerned with the missing of potential ablation studies and will adjust scores if it gets addressed properly.
limitations: The reviewer currently did not see sufficient ablation studies to support their claims. Also, the method currently did not provide evidence to support understanding of how transformers work in vision tasks.
review_title: null

Meta-review: metareview: This is an interesting paper with good contribution to the field. Most reviews are positive. 
confidence: Certain
Decision: Accept