Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces the concept of influence patterns which provides a novel way to quantify and analyse the flow of information through a sequence of model nodes in a neural network. Using influence patterns, the paper shows the end to end flow of information in a BERT architecture and experiments also show that the 
main_review: The paper is well written and easy to follow. The proposed technique of quantifying the flow of information in a bert model using influence patterns seems to be novel enough and can be very useful in better understanding the bert models and potentially identifying any redundancies/shortcomings. Moreover, as the experiments show is significantly better than the baselines. However, I am a little puzzled by the results in last three columns of Table-1, correct me if I am wrong but how is the original model performance worse than the method where you just retain the nodes from \pi_+ and make everything else zero?
Did the authors also try using these influence patterns to prune the network by only retaining the important nodes and make the model inference and possibly even training faster?
 
limitations_and_societal_impact: It would have been useful if authors included a discussion on the computational cost of finding the influence patterns.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This submission proposes a novel graph-based method to visualize the information model in transformer-based models such as BERT. The proposed method utilizes graph search algorithm to produce patterns which are generalized paths in graphs. The advantage of these patterns is that they are sparse and have fewer nodes than a conventionally defined path in the computation graph and are therefore potentially more interpretable. The authors demonstrate that such visualization can lead to interesting findings on several natural language processing tasks. Through quantitative evaluations, the authors also show that the proposed method is better than previous methods in terms of ablated accuracy and sparsity. 

That being said, my main concern is that I doubt such visualization can bring much *practical values* for model development and model understanding. I detail my concerns below.
main_review: 

Pros:

- The proposed method is novel
- The submission has done a good job comparing the proposed method to previous visualization methods through both qualitative and quantitative studies.
- The paper is well-written and clear

Cons:

I find it difficult to translate the kind of insights revealed by such visualization to any knowledge that is beneficial for model development or our understanding of BERT. I'll illustrate by the examples discussed in section 4.2

1. Finding I: Skip connection matters. While I find it interesting to see this behavior shown up in the visualization, it is hardly surprising. I believe we can safely say that without skip connections, it would be very difficult train these transformers if not impossible. Such fact already tells us the importance of skip connection. 
2. Finding II: First, I am not sure how much this finding is tied to these two specific instances. If the conclusion we want to have is that BERT may rely on ungrammatical correlations for prediciton, I think we need to have more observations than these three examples. Second, even if we can establish this conclusion, what does this tell us? Are there meaningful takeaways we can have to change our model.

Ultimately my position is that these visualizations should be means instead of ends in themselves. I understand that this comment may also be applied to many published previous work to this submission but I think it is important. I am happy to be convinced otherwise if the author can show other findings that can help us debug or improve our models.
limitations_and_societal_impact: I do not see any negative societal impact from this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In order to explain how information flows from input tokens to output predictions in transformer-style models such as BERT, this paper defined “influence patterns”, abstractions of sets of paths (of neural network nodes) through a transformer model. BERT_SMALL and BERT_BASE were used to perform the analysis under two groups of NLP tasks (1) subject-word agreement (SVA) and reflexive anaphora (RA); and (2) sentiment analysis (SA). 

The observations include (1) a significant portion of information flows in BERT go through the residual skip connections, instead of multi-head (self-)attention heads; (2) BERT could use grammatically incorrect (e.g., let “not” to be negative, without considering its context such as “not good” vs. “not bad”); (3) “influence patterns” account for information flows in BERT on average of 74% and is 25% more accurate than prior attention-based and layer-based explanation methods.

The definition of “influence patterns” using the idea of node paths and the major observations are the major contributions of this paper.

main_review: Understanding why “attention is all you need” in transformer-style models such as BERT is one of the important directions in NLP and deep learning, especially for large-scale pretrained models.

However, one important reference is missing in this paper, “Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth” (https://arxiv.org/abs/2103.03404 ) (opened Mar 5, 2021) where a quite similar finding of: skip connection also plays an important role compared with multi-head self-attentions. Also, in that paper, “paths” of nodes (skip connection and heads) were all included. That paper further gave a theoretical prove that why “skip connection” is essential and what will be the final consequence if transformer models without “skip connection”. Those results have a rather big overlap with the most important observation of this paper. Thus, would like to suggest the authors read that paper and make a clear “novelty/difference description”.


Another concern is that currently, BERT_SMALL and BERT_BASE are the two major models used for analyzing, taking three tasks and focused on MLM loss only. Since these two models only have 6 layers and 12 layers, it is also important to see the results in larger BERT models with L=24 and so on, and also with other GLUE/SuperGLUE tasks where sentence-order-prediction loss is required to capture paragraph-level linguistic information. “important information flow through skip connections 3 times more often than attentions” – is this conclusion/observation only on L=6 and L=12 BERT? Prefer deeper analysis of how the number of layers matter and if there is a trade-off or relation between L and importance of skip connections.

Detailed questions and comments:
1.	Line 10, as mentioned before, better mention existing path-based methods as well;
2.	Line 53, p_{-1}’s -1 is a bit ambiguous, possibly better mention that it’s the last element index of a list? (alike python?)
3.	Lines 61-63, we see “nodes”, “head nodes”, and “skip nodes”, possibly it is better to distinguish the nodes in one transformer encoder layer, such as multi-head self-attention, layer-norm, skip connection, and feed-forward layers’ nodes; also how they are selected for “path” construction. Some paths include only part of the nodes, if there are deeper reasons of this “skip”/”jump” paths, I would like to learn.
4.	Figure 1, “a, s, QoI” are used in the figure and the explaining of them are far away in the paper, possibly better give quick annotation of them in the caption;
5.	Figure 2, not quite understand why only one “node” is selected for analysis, since the connections are fully connected, I am wondering if ideas alike “beam” (top-N) best paths make sense to the final explanation;
6.	Lines 160-162, as mentioned before, possibly you can try L=24 or larger number of layers, or other types of pretraining tasks besides MLM (line 154);
7.	Line 219, I am wondering how many “not” included words/subwords were masked during MLM training? Does the masking techniques matter to this finding?
8.	Table 1, direct comparison of these quantitative results with baselines (attention/layer) is a bit difficult to follow. Are there any intuitive examples that show the merits of “influence patterns”? that is, I am wondering what will be the conclusions drawn by baselines (attention/layer) from your existing examples.
9.	Table 3 of appendix, is/are, himself/themselves are selected as examples here. Just wondering how often these words appeared in the original texts for training BERT? And any further comparison of their respective frequencies in the training/fine-tuning datasets?

Typos:
1.	Line 49, “are are” -> “are”;
2.	Line 150, “two group of” -> two groups of?
3.	Line 104, 105, since you used “1) firstly”, possibly you can only use 1) or only firstly;

limitations_and_societal_impact: No related negative societal impacts of this work. 
Also, if the work goes deeper, it can be a good example of explaining “how/why BERT and related large-scale pretraining language models work”.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: As BERT achieves promising results in many NLU tasks (e.g., GLUE benchmark and SquAD benchmark), providing explanation for BERT has also received increasing attention recently. Some literature examines pre-trained BERT encoders, through using probes to analyze their semantic and syntactic knowledge. Some other work focuses on fine-tuned BERT, and uses gradient-based explanation (or variants of gradient) to provide local explanation for each input sentence. This work novelly proposes influence patterns, to abstract sets of paths through the BERT model, as to to illustrate the information flowing process within BERT. Experimental results shed light on the decision making process of BERT models.

main_review: Advantages:

1.The proposed method is novel. The abstracted paths could illustrate the information flowing process within BERT in an friendly manner to users.

2. The paper is well written and I enjoy reading it.

3. Lots of interesting findings are found and the findings are convincing to me. For example, skip connections are important, and this could explain why lots of attention heads could be pruned without influencing the model performance. The decision making logic for phrases “not good” and “not bad” is novel.

I have the following minor concerns.

1. The proposed GPR model seems to work well with input texts with short length. Will the model work with longer input and with a larger model (e.g., BERT-large)?

2. Do the findings work for more complex NLU tasks? For example, the finding that skip connection is more important compared to attention heads, could be attributed to the simplicity of the tasks examined in this work, e.g, SVA and RA. For more complex NLU tasks, such as MNLI and SquAD, does the finding still hold?

3. Although lots of works use sparsity to evaluate the explanation quality, I do not think this is a good metric.  Suppose that the underlying model to be explained using dense patterns to make decisions, then the provided sparse explanations would be low fidelity to its decision making process. 

limitations_and_societal_impact: Yes, the limitations are adequately addressed by the authors.
review_title: null

Meta-review: All reviewers agree that this paper proposes a novel method to understand functioning of Transformer models through influence patterns. 3/4 reviewers find the contributions interesting and substantial and voted for acceptance. One reviewer raised concerns about the purpose of such techniques and how they can help in design better architectures. Authors responded by saying such understanding is important for building trust and debugging. I completely agree with Authors on this and think such analysis studies are very important and am glad to recommend acceptance. I encourage Authors to include suggested changes/new comparisons brought up by reviewers in the final version.
Decision: Accept (Poster)