Influence Patterns for Explaining Information Flow in BERT
Authors: Kaiji Lu
Zifan Wang
Piotr Mardziel
Anupam Datta
Venue: NeurIPS 2021 Poster (2021)
Keywords: BERT
interpretability
attention
information flow
Transformers

Abstract: While attention is all you need may be proving true, we do not know why: attention-based transformer models such as BERT are superior but how information flows from input tokens to output predictions are unclear.  We introduce influence patterns,  abstractions of sets of paths  through a transformer model. Patterns quantify and localize the flow of  information to paths passing through a sequence of model nodes. Experimentally, we find that significant portion of information flow in BERT goes through skip connections instead of attention heads. We further show that consistency of patterns across instances is an indicator of BERT’s performance. Finally, we demonstrate that patterns account for far more model performance than previous attention-based and layer-based methods.

Influence Patterns for Explaining Information Flow in BERT
Kaiji Lu; Zifan Wang, Piotr Mardziel, Anupam Datta Electrical and Computer Engineering Carnegie Mellon University
Mountain View, CA 94089
Abstract
While "attention is all you need" may be proving true, we do not know why: attention-based transformer models such as BERT are superior but how information flows from input tokens to output predictions are unclear. We introduce influence patterns, abstractions quantify and localize the flow of information to paths passing through sequence model nodes. Experimentally, we find that significant portion of information flow in BERT goes through skip connections instead of attention heads. We further show that consistency of patterns across instances is an indicator of BERT's performance. Finally, we demonstrate that patterns account for far more model performance than previous attention- -aatid and layer-based methods.
sets of paths through transformer model. Patterns
Introduction
Previous works show that transformer models such BERT 7 encode various linguistic concepts 24 [0,13), some of which can be associated with internal components of each layer, such as internal embeddings or attention weights (24] 40 15 33). However, exactly how information flows through transformer from input tokens to the output predictions remains an open question. Recent attempts methods, where attention weights are used as to answer this question include using attention-
indicators of flow of information [ 2 49), or layer-based approaches 15 14 33 which identify important network units in each layer.
In this paper, we examine the information flow question through an alternative lens of gradient-based attribution methods. We introduce influence patterns, abstractions of sets of gradient-based paths through a transformer's entire computational graph. We also introduce a greedy search procedure for efficiently and effectively finding patterss representative of concept-critical information flow. Figure [ provides an example of an influence pattern in BERT. We conduct an extensive empirical study of influence patterns for several NLP tasks: Subject-Vert Verb Agreement (SVA), Reflexive Anaphora (RA), and Sentiment Analysis (SA). Our findings are summarized below.
A significant portion of information flows in BERT go through skip connections and not attention heads, indicating that attention weights alone are not sufficient to characterize information flow. In our experiment, we show that average, important information flow through skip connections 3 times more often than attentions.
By visualizing the extracted patterns, we show how information flow of words interact inside the model and BERT may use grammatically incorrect cues to make predictions.
The consistency of influence patterns across instances of task reflects BERT's performance for that task.
"Correspondence to kai jaijjl 10ndreww cmu, edu
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

"the"
Agg
fg
A6
hg
Vo
11g
02
"uunyy"
A
A
Av
V1
"bay"
20g
Ag
N
h2
A1
h2
[MASK]"
A3
Ag Ag
A M0 Qof(va)
hg
BERT architecture (left) and details of a transformer layer (right) for an instance of the SVA task, which evaluates whether the model chooses the correct verb form is over are for [MASK] to agree with the subject. An example of a pattern is highlighted with red nodes. æ and h are input and internal embeddings, a attention heads, y output logits, Qol the function for computing quantity of interest.
Figure I:
Through ablation experiments, we find that influence patterns account for information flows in BERT on average 74% and 25% more accurately than prior attention-based and layer-based explanation methods 23 91, respectively.
2
Background
We begin this section by introducing notations and architecture of BERT in Sec. 2.1 We then introduce distributional infiuence as an axiomatic method to explain the output behavior of any deep model Sec. 2.2 which serves as building block for our method follow in the next section.
2.1 BERT
Throughout the paper we to denote vector and |S| to denote the cardinality of set S or number of nodes in a graph S. We begin with the basics the BERT architecture [4] (presented in Fig. T Let L be the number of layers in BERT, H the hidden dimension of embeddings at each layer, and A the number of attention heads. The list of input word embeddings is x [x1, X2, xN]. x; E IR. We denote the output of the I-th layer h First layer inputs are h def X1:N We use af denote the j-th attention head from the i-th embedding 1-th layer and s/ to denote the skip connection that is "copied" from the input embedding from the previous layer then combined with the attention output. The output logits are denoted by y.
Computation Graph of BERT deep network be viewed computational graph g def set of nodes, activation functions, and edges, respectively. In this paper, we assume the graph is directed, acyclic, and does not contain more than one edge per adjacent pair of nodes. A path in G is a sequence of graph-adjacent nodes [PI, P2. .P]; ]r]: is the output node. Thus, the as per chain rule. We further denote the Jacobian of the output of node n; w.r.t the output of connected (not necessarily directly) predecessor node ni evaluated
Jacobian passing through a path evaluated at input x is I[
(x)//n,(3)))
For computational and interpretability reasons, an ideal graph would contain as few nodes and edges possible while exposing the structures of interest. For BERT we propose two graphs: embedding-level graph Ge corresponding to the nodes and edges shown in Fig. (left) explain how the influence of input embeddings flow from one layer of internal representations to another and to the eventual prediction; and atlention-level gruph G. 3 G. that additionally includes attention head nodes and skip connection nodes in Fig. [ (right), finer decomposition to demonstrate how influence from the input embedding flows through the attention block (or skip connections) within each layer.

---

2.2 Explaining Deep Neural Networks
are well-studied in explaining the behavior of a deep model by attributing the model's output to each input feature as its feature importance score. While existing approaches often explain with the most important features for single class distributional Influence (23) generalizes gradient-based approaches to answer broader set of questions, e.g. why [MASK] should be IS instead of ARE (Figure by introducing quantity of interest (Qol). Suppose general network f R - IR... a Qol is a differentiable function (f)())) that outputs a scalar to incorporate the subject of an explanation. For example, to answer the aforementioned question, we can define q(f(x)) = f(x) - {(T)ARE where fz)), is the logit output of class *, Formally, we introduce Distributional Influence.
mundient-based explanations 39
Definition 1 (Distributional Influence) Given model f R R", input x, user-defined distribution D(x), and a user-defined Qol 9, Distributional Influence g(x: q, D) is defined as:
0g(f(z))
g q, D) def Ex)D(X)
Oz
Remark 1 Distributional Influence also leverages a user- distribution D(x) capture the network's s behavior in a neighborhood of the input of interest x. By introducing D(x), we can prove that several popular attribution methods are specific cases of the distributional influence; e.g- when D(x) is a Gaussian Distribution, 9(x: q, 'D) reduces to Smooth Gradiemt when D(x) is uniform distributions over a path {x + a(xx xx), a E [0, 1]} from user-defined baseline input X5, to x, D(x) reduces to Integrated Gradient (37..
We use D(x) as the uniform distribution over a linear path described in Remark in the rest of the paper because it provides several provable properties [39] to ensure the faithfulness of our explanations. We approximate the expectation in Def. by sampling discrete points in the uniform distribution.
3 Tracing Influence Flow with Patterns
To explain how different concepts in the input flow to final predictions in BERT, it is important to show how the information from each input word flows through each intermediate layer and finally reaches the output embedding of interest, e.g. [MASK] for pretraining or [CLS] for fine-tuned models. Prior approaches have use the attention weights to build directed graphs from one embedding to another [2, 49]. These approaches use heuristics to treat high attention weights as indicators of important information flow between layers. However, as more work starts to highlight the axiomatic justifications of gradient-based methods over attentions weights as an explanation approach [2), we therefore explore an orthogonal direction in applying distributional influence in BERT to trace the information flow. Since distributional influence only attributes the output behavior over the input features, in this section, we generalize it to find important internal components that faithfully account for the output behavior.
Tracing Influence by Patterns By viewing BERT as a computation graph (G. or G. in 2.1 we restate the problem: given source node and target node I, we seek significant pattern of nodes from s to that shows how the influence from s traverses from node to node and finally reaches t. An exhaust way to rank all paths by the amount of influence flowing from to is possible in smaller networks, as is done by Lu et al. 277. However, the similar approach lacks scalability to large models like BERT. Therefore, we propose way to greedily narrow down the searching space from all possible paths to specific patterns. That is, our approach is two-fold: 1) we employ abstractions of sets of paths as the localization and influence quantification instrument; 2) discover influential patterns with a greedy search procedure that refines abstract patterns into more concrete ones, keeping the influence high. We begin with the formal definition of pallern.
Definition 2 (Pattern) A pattern π is a sequence of nodes [TT, T2.. 1- such that for any pair of nodes n. T+41 adjacent in the sequence (not necessarily adjacent in the graph), there exists a path from T; and T,41-
pattern π abstracts a set of paths, written (Tr) that follow the given sequence of nodes but are free to traverse the graph between those nodes in any way. Interpreting paths and patterns sets, we define
3

---

x [6,4] 5
= [5, 8 = (2)
(5,ag [y.4]
Figure 2: A visual illustration of Guided Pattern Refinement (GPR) in toy example. start with pattern [s containing only the source and the target node. At each step define guided set E and E respectively and find the node the guided set that maximize the pattern influence (Def. GPR finally returns pattern π [s, az b1. that abstracts single path.
(()) C SonSC p} where P is the set of all paths from --1- If every sequence-adjacent pair nodes is directly connected then the pattern abstracts a single path. To quantify the amount of information that flows from the input node to the target node over particular pattern, we propose Pattern Inffuence, motivated by distributional influence.
Definition (Pattern influence) Given a computation graph and a user-defined distribution D, the influence an influence pattern written [xx, π) is the total inftuence of all the paths abstracted by the pattern: I(x, π def
Eper(H) Ezz)D(x) I (x)
(p))
Proposition 1 (Chain Rule) I(x; R) ((x) I[
(TL-) for any distribution D(xx).
(w,(x)
Prop. (proof in Appendix A simplifies the evaluation of the pattern influence by specifying the exact set internal nodes through which influence flows computational graph. We hereby introduce a greedy way of finding the most influential pattern in both G. and G...
Guided Pattern Refinement Starting with source and target nodes and along with initialized pattern x² {s, t} representing all paths between s and t, we construct n' by adding see@sseeene-aajaeee node e" (there direct path between and e") from a guiding set E that maximizes the influence of the resulting pattern such that:
0 arg max [I(x, [s, t)), = (s c)
(1)
This procedure is iterated until we exhaust the last guiding set. We show an example of GPR in graph Ge. each guiding set E' includes all embeddings
toy graph in Fig. 2 For an embedding hf.x at layer l. For the attention-level graph Ga. we refine on the embedding-level pattern x by only expanding л from addition nodes in G,, compared with G.: we perform GPR iterations between Ti, "441 E x² with the guiding set Ey which includes attention heads a' and the skip node s' ('Fig. I Right), until we reach the same last node in π. The returned attention-level pattern n thus abstracts single path from the source the target G.. As the attention-level analysis refines the embedding-led
analysis, the produced aitention-leve pattern T abstracts a strict subset of the paths of the attention- graph that the embedding-leve level pattern n" abstracts. That
while (77") C (xx"). The detailed algorithm of GPR and analysis of its optimality can be found in
Appendix B. B.2
Experiment
Experiments in this section demonstrate our method tool for interpreting end-to-end information flow in BERT. Specific visualizations exemplify these interpretations including the importance of skip connections and BERT's encoding of grammatical concepts included Sec. 4.2 Sec.
'The algorithm can be easily adapted to include more nodes per layer. However, we found one node per layer retain reasonable proportion of influence for the tasks evaluated in this paper(See S.44.4)

---

explores the consistency of patterns across instances and template positions and how they relate to task performances and influence magnitudes in. Sec demonstrates two advantages of patterns in explaining the information flow of BERT over baselines: 1) abstracted patterns, with much fewer nodes compared to the whole model, carry sufficient information for the prediction. That is, without the information outside the refined pattern, the model shows no significant performance drop: 2) At the same time, patterns are sparse but concentrated in BERT's components.
4.1 Setup
Tasks. We consider two groups of NLP tasks: (1) subject-word agreement (SVA) and reflexive anaphora (RA). We explore different forms of sentence stimuli(subaask) within each task: object relative clause (Obj.), subject relative clause (Subj.). within sentence complement (WSC), and across prepositional phrase (APP) in SVA [28;; number agreement (NA) and gender agreement (GA) in RA (24]. Both datasets are evaluated using masked language model (MLM) as is used in Goldberg [I3.. We sample 1000 sentences from each subtask evenly distributed across different sentence types (e.g. singular/plural subject & singular/plural intervening noun) with fixed sentence structure; (2) sentiment analysis(SA) (SA): we use 220 short examples (sentence length 17) from the evaluation set of the 2-class GLUE SST-2 sentiment analysis dataset [4]. More details and examples of each task can be found in Appendix[D.]
Models. For linguistic tasks, We evaluate our methods with pretrained BERT(L. = 6, 4 8) referred hereby as SMALL For SST-2 we fine-tuned on the pretrained BERTBASE with
L = 12, A = 12. The models are similar in sizes compared to the transformer models used in 2 All computations are done with a Titan on a machine with 64 GB of RAM. See Appendix D.2 for more details.
Implementation of GPR. Let the target node for SVA and RA tasks be the output of the Qol score ( Vcorrect Murong. For instance, Vıs - VARE for the sentence she [MASK] happy. Similarly, we use Spositive Inegative for sentiment analysis. We choose an uniform distribution over linear path from X, to x as the distribution D in Def. where the x, is chosen as the the input embedding of [MASK] because it can viewed a word with no information. For given input token x,, we apply GPR differently depending on the sign of distributional influence g(x; q. D): if g(x; q, DD 2 0, we maximize the pattern influence towards ((y) at each iteration of the GPR otherwise we maximize pattern influence towards -q(y). We use as the extracted patterns for individual input word i. When explaining the whole input sentence, we collect all refined patterns for each word and use Il U, mi Further, we denote II, as the set of patterns for all positively influential words. Both terms may be further decorated by a or e to denote attention- or embedding- levell results.
4.2 Visualizing Influence Patterns
This section does not serve as an evaluation but an exploration of insightful results and succinet conclusions an human user can learn from our proposed technique. We visualize the information flow identified by patterns found by GPR, and compare with those generated by attention weights as explored the literature 2 49
We therefore use Imn to denote pattern of nodes by maximizing the product of the corresponding attention weights between each pairs of nodes from adjacent layers. Implementation details are included in Appendix C.I]
We first focus on instances of the subtask SVA across object relative clauses (SVA-obj), which are generated from the template: the SUBJECT that the ATTRACTOR VERB [MASK] (is/ aree) ADJ.. We observe in Figure that the subject words exert positive input influences on the correct choice of the verb, and the intervening noun (attractor) exerts negative influence, which is true for both I(x;, TT) and Z(x., TT (blue and purple bars in Figure aaland 3b). While I1 (Fig, 3c) does not distinguish between positively influential words and negative ones, nor do they show an interpretable pattern. Our other main findings are discussed as follows.
Finding I: Skip Connection Matters. Horizontal dashed lines in Figure 3 iindicate that influence can flow through layers at the same word position via skip connections, which is not isolated as separate nodes Im (shown in Fig. 3cp. Fig. 3a and 3b also show that the influence from subject
'The two works compute attribution from input to internal embeddings by aggregating attention weights(averaged across heads) across layers while attribution to individual paths is implicit.
5

---

[CLS
[CLS|]
[CLS]]]
All bosein I(% Tlswsie x)
fammera
rag dki
farmer
neg xcp
thel
rag aln
neg an
farmet
s(%)
the
21%. x
a(%c
thei
assidan
2
the
loves
love
assistants
(MASK]
IMASK
love
old
[MASI]
old o
0
[SEP]]
[SEP]
[sen]]
0.02
(a) attractor(PS)
plural
suhject+singular (b)
singular
suhject+plural
(c) attention- based baseline 11am.
Sttracton(SP)
Figure 3: (a)(b) Patterns for two instances of (easelh pattern aain- For each plot: Left: bar plots of the distributional influence ((yeloo)l) I(x;, π{) (purple) and [(x,, x&) (blue) (or I(x,, Thaseline.))) for each word position i. Right: Extracted patterns II from selective words. Square nodes and circle nodes denote input and internal embeddings, respectively. In (a) and (b), influence flowing through skip connections represented by dashed lines and attention heads in solid lines; the edges are are marked with the corresponding attention head number (ranging from 1 to A) in I1
Line colors represent the sign of influence (red as negative and green as positive).
love
PANSI|
[CLS]]
not
good
hote MANSQO
[CLS]
He
not
MASS
(a) Interactions between clausal [MASK] for SVA-Obj
verbs &
(b) Interactions between not adjectives
Figure 4: (a) Patterns on three clausal verbs from SP, SVA-obj.(b) Patterns for two instances in the SA
task. Legend follows Figure 3
words farmer [s] travels through skip connections across layers before transfers into the attention head 5 in the last layer, indicating the "number information" from the subject embedding flows directly to the output. Interestingly, this also explains why attentions can be pruned effectively without compromising the performance [3] as important information may not flow through attentions at all. Namely, they are simply "copied"" to the next layer through the skip connection. In fact, attention heads are traversed far less often than skip connections, which account for 75.4% of nodes in I across all tasks evaluated.
Finding II: that as an Attractor. Fig. Ba and 3b shows that the singular subject is less influential than the plural subject, especially when compared to the large negative influence from the attractor. Besides, the word that behaves like a singular pronoun in the singular subject case (Figure (3b), flowing through the same pattern the subject (skip connections attention head 5), whereas that more like a grammatical marker (relativizer) in the plural subject case (Figure the pattern from that converge the subject in the second last layer via different attention head 1, An explanation to the observed difference is that that can either be used as a singular pronoun in English, marker that encodes the syntactic boundary of the clause to help identify the subject and ignore the attractor. This finding reveals that although the latter one is always the correct usage for that across all instances, BERT may resort to the "easier" (while ungrammatical) encoding of that as pronoun when the number happens to be singular.
Finding III: love, hate and like Figure 4a shows the pattern across three instances containing different clausal verbs by replacing love in Figure 33b with like or hate. We observe that hate and love are more influential, with a distinct and more concentrated pattern compared to that of

---

In INtter -0
Fattem Entrayy Oet. Fatterm
0.64
0.56
Saad ea Natter 10.5 pndd
If Fatlars m m2.2 p-
Patk mm)
0.52
NA
0.50
2.15
phj
2.10
0.58
3.00
0.56
AP2
0.55
Emrr
Pattern
3.0-
0.54
0.52
Sub
JSCC
0.38
WSC
355
4.0
Normalized Influence Magnitude
p(correct) p(incoreect)
(a) Pattern Entropy vs. Average Influence Magnitude for template positions in SVA-Obj
(b) Pattern Entropy Average Qol for linguistic tasks (SVA & RA)
Figure 5: Relationship between Task Performance, Influence Magnitude and Pattern Entropy
the word like, while one comparable to that of the subject or noun attractor. Therefore, hate and love are treated more like singular nouns than like, contributing positively to the correct prediction of the verb's number (but for the wrong grammatical reason). This discrepancy also corroborates different accuracy for instances in the three subsets: hate: .99; love: 1.0; like: .82. fact, the number of misclassifications containing like accounts for more than 33% of all misclassifications in SVA-Obj(SP).
Finding IV: Interactions with not. Two instances of sentiment analysis which BERT classifies correctly (negative for not good. and positive for not bad.) in Fig. 4b shows that: in not good., not and good does not interact internally and not carries much more positive influence than good towards the correct class (negative sentiment); 2) in not bad., on the contrary, not contributes negatively to the correctly class (positive sentiment) and bad interacts with the not in an internal layer and exhibit large positive influence. This disparity shows the model may treat not as a word with negative sentiment by default, and only when the subsequent adjective is negative, can it be used as a negation marker to encode the correct sentiment.
The asymmetries and oddities of patterns of Finding II to IV show how BERT may use ungrammatical correlations for its predictions. We include more examples of patterns in Appendix B
4.3 Consistency of Patterns Across Instances
The prior section offered examples building connections between information flow and patterns. Since patterns are computed for each word of each instance individually, a global analysis of consistency/variation of patterns across instances can help us make deeper insights and make more general conjectures. We quantify this variation with pattern entropy on the linguistic tasks (SVA & RA) with fixed templates, where for each instance, each template position are chosen from group of words. Interpreting the collection of instances, or template positions, in a task as a distribution, pattern entropy is the entropy of the binary probability that node is part of pattern, averaged over all nodes (minimally this is O if all patterns incorporate the exact same set of nodes). With a node in graph G, a of K instances, T&K the pattern of instance by abuse of notation and H as the entropy function: pattern entropy of a pattern π is defined as:
(Tr Eneg H (pn (P)( |G|
(P((TT) Ee 1( K
Figure 5a shows an inverse linear relation between influence magnitude and the pattern entropy of each template position(T.) in SVA-Obj. For example, the subject position has on average high influence magnitude and low pattern entropy, while grammatically unimportant template positions such as [CLS] and the period token has relatively low influence and high entropy. Note that this fit is not perfect: attractors, for example, have high influence magnitude but also high entropy's due to their disparate behaviors among instances shown in Sec
Figure 5b demonstrates more global inverse relation between the pattern entropy of II, and task- specific performance(average Qol) for 6 linguistic subtasks studied. This indicates that the more consistent pattern is for concept, i.c the more consistent how model locates the positively
7

---

Patterns
Obj.
Subj.
SVA
WSC
APP
NA
RA
GA
SA
Metrics
Obj. Subj.
SVA
WSC
APP
NA
RA
GA
SA
I (ours)
.96
.99
1.0
96
.98
.99
.97
conc.(1l;
.33
.30
.27
.33
.31
.22
.07
11 and
.55
.60
.55
.55
.55
.56
.52
conc.
.29 .30
.38
.31
.31
.23
.05
.49
.63
.56
.65
.47
share([1S]
.06
.09
.03
.06
.04
.02
.01
IIf
cond
.93
.91
.88
.90
.71
.95
.94
conc.
.21
.18
.16
.18
.18
.14
.01
in
.66
.71
.50
.56
.68
.50
.49
conc.
.17
.16
.25
.15
.17
.15
.01
I1 (ours)
.56
.65
.78
.89
share([11]
1.8
1.7
1.6
1.5
1.3
1.3
3e-2
I
.50
.50
.50
.50
.50
.50
.49
(-10-6)
repl_skip skip
.50
.58
.54
.52
.5
.50
.48
Alignment
.58
.57
.57
-59
.46
.53
.34
original
.96
1.0
1.0
1.0
83
.73
.92
Rate
Table I: Summary of quantitative results, with {e, denoting embedding & attention-level
metrics. Left: Ablated accuracies of II, and baseline patterns. Right: Sparsity metries; conc.(II..) positive/negative concentration; share([T")) path shure of pattern II".
influential signals, the better the model is at capturing that concept. In both figures, the baseline attention-basse patterns (Iamm does not indicate this relation. Together with Finding II to IV in Sec. 4.2 it demonstrates that patterns may offer alternative insights in formalizing and answering high- level questions such to what extent does BERT generalize to correct and consistent grammatical
rules or use spurious correlations which varies across instances 30 29
4.4 Evaluating Influence Patterns
This section serves as a formal evaluation of our proposed methods against existing baselines in tracing information flows. Except the attention baseline Imm defined in Sec. 4.2, we also include the following baselines (implementation details to follow in Appendix C.I
I rand: patterns with randomly sampled nodes.
That: patterns consisting of nodes that maximize the internal influence
23 for each guiding set.
Ieond: patterns consisting of nodes that maximize the conductance O for each guiding set.
compare various aspects extracted patterns against the following baselines, including: (1) ablation experiments showing how influence patterns account for model performance; (2) the sparsity of the patterns in the computational graph. Finally, we also discuss the relation between patterns and ceeuttionweeight.
Ablation. We extend the commonly-use ablation study for evaluating explanation methods 6
23 27
from input features to internal patterns as a sanity check of our method. That is, we ablate BERT down to simpler model: we only retain the nodes from II. (or II while replacing other nodes by zeros. The retained and replaced nodes together are forward passed to the next layer with the original model parameters until a new set of nodes are retained or replaced. Ablated Accuracy denotes the accuracy of the ablated model. We show the results for our methods II; and other baseline patterns in Table I Each baseline retains the same number of nodes as the corresponding IT, Although this process seems to be highly invasive, the model ablated with II; achieved the highest ablated accuracy uniformly over all tasks. All baseline patterns except IT cond gives random guesses. For the attention-level graph I.. we also add a counterfactual pattern lT repl_kip skip where replace each skip node in 11Ş with its all A corresponding attention heads. However, with much fewer nodes ablated, Ih _kpp still produces random ablated accuracies. This corroborates Finding I in Sec. 4.. that the skip connections relaying important information directly which attention block cannot replace. In summary, patterns refined by GPR account for the model's 's information flow more sufficiently compared to all baselines.
Sparsity of Patterns. A good explanation should not only account for the model performance, but also be sparse relative the entire model semantics so be interpretable to humans. In this section, we quantify the sparsity of extracted patterns using two metrics: path share and concentration. With
'we run Handd times per task average the ablated accuracy with no significant variation, the detailed statistics can be found in Appendix
We include discussions why H. has higher ablated accuracy than other baselines in Appendix C.2

---

on average more than 10 words per sentence, the embedding and attention-level graphs contain at least 10 and 10" individual paths, respectively. The totality of these paths represent the entire semanties of the BERT model. Path share(i.e. share(IT?)
7 (TT* ///(")), is defined as the number of paths pattern over the total number of paths in the entire computational graph. concentration, on the other hand, is defined as the proportion of negative/positiy pattern influence over the total positive/negative influence. It represents the average ratio between the blue/purple bars and the
orange bars in Figure 3
Table I Right) shows that the abstracted patterns contain only small share of paths while accounting for a large portion of both positive and negative influence. In linguistic tasks, the embedding level abstracted pattern has a concentration around 0.3 (conc. (II), )), indicating that the imput concept flows through single internal embeddings in each layer, instead of distributed to many words. Zooming graph, a relative high concentration cone.(TT+.) suggests that between the
in on the attention- internal embeddings of adjacent layers. influence is also more concentrated to either one attention head or the skip connection. We speculate the much lower conc.(IT++) ) in SA is due to (1) the much larger model for the SA task with more layers and attention heads (2) sentiment information may be more diverse and complex than the information needed to encode syntax agreements, therefore input information may flow in a more distributed way. However, despite low concentration of pattern influence, the extracted patterns for SA are still effective in capturing the model performance as shown in Table Left).
Attention heads in II vs. Attention weight value. Another observation is that the attention head nodes in IT found by GPR aligns, to greater extent than random (1/|A)). with the head of the largest attention weight along the corresponding embedding-level edge, as shown by alignment rate in Table Lefft. This partial alignment is expected since the Jacobian between two nodes correlates with the coefficients (weights) in the linear attention mechanism( meehanism(produet rule), while the opposite is also expected since (1) attention weights themselves
not fixed model parameters thus part of the gradient flow, (2) model components other than attention blocks come between two adjacent layers (e.g. dense layer & skip connections). In other words, attention weights are correlated, but not equivalent to gradient- braseent-based methods as explanations.
5 Related Work
Previous work has shown the encoding of syntactic dependencies such as SVA in RNN Language
models (25
16
22
19]. More extensive work has since been done on transformer-
architectures
such BERT. Probing classifiers has shown BERT encodes many types of linguistic knowledge [I]]
14 40
17
20
26
24
33 15). 133 discovers that SVA and RA in complex clausal structures is
better represented in BERT compared to an RNN model.
line of related works analyze self-attention weights of BERT 45 24 12], where attention heads
found to have direct correspondences with specific dependency relations. 2 and 49 with which we compare in this paper, propose using attentions to quantifyy information flows in BERT. Attention weights interpretation devices, however, have been controversial [36] and empirical analysis has shown that attention can be perturbed or pruned while retaining performance [2 46). Our work demonstrates that attention mechanisms only part of the computation graph, with each attention block complemented by other model components such as dense layer and skip connections; axiomatically justified influence patterns, however, can attribute to the whole computation graph. The strong influence passing through skip connections also corroborates the findings of [4] which finds input tokens mostly retain their identity. Besides pruning attentions, other works 32 35 [8]
also show that BERT is overparametrized and can be greatly compressed. Our work to some extent corroborates that point by pointing the sparse gradient flow, while employing ablation studies only to verify the sufficiency of the extracted patterns.
A closely related and concurrent work[T0] also shows the importance of skip connections and dense(MLP) layers by decomposing and analyzing aeward-pass computations in self-attention modules. Our work, in comparison, introduces a gradient-basee based method that can be generalized to any model as long as they are differentiable. We also focus on how influential patterns can help us understand information flow of specific NLP tasks.
Recent work introducing influence paths 277 offers another form of explanation. Lu al. [2 decomposed the attribution path-specific quantities localizing the implementation of the given

---

concept to paths through model. The authors demonstrated that for LSTM models, a single path is responsible for most of the input-output effect defining SVA. Directly applying individual paths to transformer-based models like BERT, however, results in intractable number of paths to enumerate due to the huge number of computation edges in BERT.
Limitations and Future Work
We believe there are three limitations to address in future work. (I) GPR algorithm does not guarantee absolute optimality, however, we provide empirical evidence in support of the searching algorithm in Appendix B.2 (2) For interpretability reasons, compute GPR by picking one node per guiding set, while more complicated information may distribute to multiple internal embeddings or attention heads. (3) The findings in Sec. 4.2 would benefit from more quantitative analysis to support more general claims of speculations) on BERT's handling of various finguistic/smmanic concepts. However, we will explore these limitations in future work and release code and hope the proposed methods will serve as an insightful tool in future exploration.
Conclusion
We demonstrated influence patterns for explaining information flow in BERT. We highlighted the importance of skip connections and BERT's potentially mishandling of various concepts through visualized patterns. We inspected the relation between consistency of patterns across instances with model performance and quantitatively validated pattern's 's sufficiency in capturing model performance.
Acknowledgement This work was developed with the support of NSF grant CNS-1704845. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation thereon. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the National Science Foundation or the U.S. Government. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this work.
10

---

References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large- scale machine learning heterogeneous systems, 2015. Software available from tensorflow.org.
[2] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, 2020.
[3] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018
[4] Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identifibility in transformers. In International Conference on Learning Representations, 2020.
[5] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACI. Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP,, 2019.
[6] Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. NIPS, 2017.
[7] Jacob Devlin, Ming-Woo Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
[8] Jay Younu Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. Eraser: A benchmark to evaluate rationalized nip models. Transactions of the Association for Computational Linguistics, 2020.
[9] Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. How important is neuron. In International Conference on Learning Representations, 2019.
[10] Yihe Dong. Jean-Band Bann-Bttistt Cordonnier, and Andreas Loukas Attention is not all you need: pure attention loses rank doubly exponentially with depth. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2793-2803. PMLR, 18-24 Jul 2021.
[11] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. When bert forgets how pos: Amnesic probing of linguistic properties and mlm predictions. arXiv preprint arXiv:2006.00995, 2020.
[12] Kawin Ethayarajh and Dan Jurafsky. Attention flows are shapley value explanations. arXiv preprint arXiv:2105.14653 2021.
[13] Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint arXiv:1901.05285: 2019.
[14] John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMMNP)-IJCNIP), 2019.
[15] John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
the 2019 Conference of the North American Chapter of the
11

---

[16] Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926, 2018.
[17] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[18] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 10351, 2019.
[19] Jaap Jumelet, Willem Zuidema, and Dieuwke Hupkes. Analysing neural language models: Contextual decomposition reveals default reasoning in number and gender assignment. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL.),
pages 111,,0199
[20] Josef Klafka and Allyson Ettinger. Spying on your neighbors: Fine- Fine- grained probing of contextual embeddings for information about surrounding words. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
[21] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 4365-4374, 2019.
[22] Yair Lakretz, Germán Kruszewski, Théo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and syntax units in Istm language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers),
2019.
[23] Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. felluence-directer explanations for deep convolutional networks. In 2018 IEEE International Test Conference (ITC). IEEE, 2018.
[24] Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside bert's linguistic knowledge. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019.
[25] Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 2016.
[26] Nelson F Fiu,, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. Linguistic knowledge and transferability of contextual representations. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
[27] Kaiji Lu, Piotr Mardziel, Klas Leino, Matt Fredrikson, and Anupam Datta. Influence paths for characterizing subject-verb number agreement in LSTM language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
[28] Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
[29] Thomas McCoy, Junghyun Min, and Tal Linzen. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. In Proceedings of the Third BlackboxNI.P Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 217-227, 2020.
12

---

[30] Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristies in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428 3448,,444422 2019.
[31] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, 2019.
[32] Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are winning, arXiv preprint arXiv:2005.000 00561, 2020.
[33] Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems 32. NeurIPS, 2019.
[34] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh, Beyond accuracy: Behavioral testing of nip models with checklist. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Lingusssccs pages 4902-4912, 2020.
[35] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: /910.0/ 108, 2019.
[36] Sofia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[37] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 1706.00700638255 2017.
[38] Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation, 2020.
[39] Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Saarning-Volumne 70, 2017.
[40] Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[41] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[42] Marcos Treviso and André FT Martins. The explanation game: Towards prediction explainability through sparse communication. In Proceedings of the Third BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, pages 107-118, 2020.
[43] Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.0888 0896222., 2019.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the st International Conference on Neural Information Processing Systems, 2017.
[45] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in transformer language model. In Proceedings of the 2019 ACI. Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019.
[46] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association Computational Linguistics, 2019.
[47] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pages 353-355, 2018.
13

---

[48] Zifan Wang, Haofan Wang, Shakul Ramkumar, Matt Fredrikson, Piotr Mardziel, and Anupam Datta, Smoothed geometry for robust attribution. In Advances in Neural Information Processing Systems. NeurIPS, 2020.
[49] Zhengxuan Wu, Thanh-Son Nguyen, and Desmond Ong. Structured self-attention weights encodes semantics in sentiment analysis. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 255-264, 2020.
14


Reviews:
[{"review_id": "yRYXEW8KwMJ", "reviewer": "Reviewer_NGL2", "summary": "This submission proposes a novel graph-based method to visualize the information model in transformer-based models such as BERT. The proposed method utilizes graph search algorithm to produce patterns which are generalized paths in graphs. The advantage of these patterns is that they are sparse and have fewer nodes than a conventionally defined path in the computation graph and are therefore potentially more interpretable. The authors demonstrate that such visualization can lead to interesting findings on several natural language processing tasks. Through quantitative evaluations, the authors also show that the proposed method is better than previous methods in terms of ablated accuracy and sparsity. \n\nThat being said, my main concern is that I doubt such visualization can bring much *practical values* for model development and model understanding. I detail my concerns below.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nPros:\n\n- The proposed method is novel\n- The submission has done a good job comparing the proposed method to previous visualization methods through both qualitative and quantitative studies.\n- The paper is well-written and clear\n\nCons:\n\nI find it difficult to translate the kind of insights revealed by such visualization to any knowledge that is beneficial for model development or our understanding of BERT. I'll illustrate by the examples discussed in section 4.2\n\n1. Finding I: Skip connection matters. While I find it interesting to see this behavior shown up in the visualization, it is hardly surprising. I believe we can safely say that without skip connections, it would be very difficult train these transformers if not impossible. Such fact already tells us the importance of skip connection. \n2. Finding II: First, I am not sure how much this finding is tied to these two specific instances. If the conclusion we want to have is that BERT may rely on ungrammatical correlations for prediciton, I think we need to have more observations than these three examples. Second, even if we can establish this conclusion, what does this tell us? Are there meaningful takeaways we can have to change our model.\n\nUltimately my position is that these visualizations should be means instead of ends in themselves. I understand that this comment may also be applied to many published previous work to this submission but I think it is important. I am happy to be convinced otherwise if the author can show other findings that can help us debug or improve our models.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see any negative societal impact from this work.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not have any ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "xRdi4UOMCY7", "reviewer": "Reviewer_pApo", "summary": "In order to explain how information flows from input tokens to output predictions in transformer-style models such as BERT, this paper defined “influence patterns”, abstractions of sets of paths (of neural network nodes) through a transformer model. BERT_SMALL and BERT_BASE were used to perform the analysis under two groups of NLP tasks (1) subject-word agreement (SVA) and reflexive anaphora (RA); and (2) sentiment analysis (SA). \n\nThe observations include (1) a significant portion of information flows in BERT go through the residual skip connections, instead of multi-head (self-)attention heads; (2) BERT could use grammatically incorrect (e.g., let “not” to be negative, without considering its context such as “not good” vs. “not bad”); (3) “influence patterns” account for information flows in BERT on average of 74% and is 25% more accurate than prior attention-based and layer-based explanation methods.\n\nThe definition of “influence patterns” using the idea of node paths and the major observations are the major contributions of this paper.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Understanding why “attention is all you need” in transformer-style models such as BERT is one of the important directions in NLP and deep learning, especially for large-scale pretrained models.\n\nHowever, one important reference is missing in this paper, “Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth” (https://arxiv.org/abs/2103.03404 ) (opened Mar 5, 2021) where a quite similar finding of: skip connection also plays an important role compared with multi-head self-attentions. Also, in that paper, “paths” of nodes (skip connection and heads) were all included. That paper further gave a theoretical prove that why “skip connection” is essential and what will be the final consequence if transformer models without “skip connection”. Those results have a rather big overlap with the most important observation of this paper. Thus, would like to suggest the authors read that paper and make a clear “novelty/difference description”.\n\n\nAnother concern is that currently, BERT_SMALL and BERT_BASE are the two major models used for analyzing, taking three tasks and focused on MLM loss only. Since these two models only have 6 layers and 12 layers, it is also important to see the results in larger BERT models with L=24 and so on, and also with other GLUE/SuperGLUE tasks where sentence-order-prediction loss is required to capture paragraph-level linguistic information. “important information flow through skip connections 3 times more often than attentions” – is this conclusion/observation only on L=6 and L=12 BERT? Prefer deeper analysis of how the number of layers matter and if there is a trade-off or relation between L and importance of skip connections.\n\nDetailed questions and comments:\n1.\tLine 10, as mentioned before, better mention existing path-based methods as well;\n2.\tLine 53, p_{-1}’s -1 is a bit ambiguous, possibly better mention that it’s the last element index of a list? (alike python?)\n3.\tLines 61-63, we see “nodes”, “head nodes”, and “skip nodes”, possibly it is better to distinguish the nodes in one transformer encoder layer, such as multi-head self-attention, layer-norm, skip connection, and feed-forward layers’ nodes; also how they are selected for “path” construction. Some paths include only part of the nodes, if there are deeper reasons of this “skip”/”jump” paths, I would like to learn.\n4.\tFigure 1, “a, s, QoI” are used in the figure and the explaining of them are far away in the paper, possibly better give quick annotation of them in the caption;\n5.\tFigure 2, not quite understand why only one “node” is selected for analysis, since the connections are fully connected, I am wondering if ideas alike “beam” (top-N) best paths make sense to the final explanation;\n6.\tLines 160-162, as mentioned before, possibly you can try L=24 or larger number of layers, or other types of pretraining tasks besides MLM (line 154);\n7.\tLine 219, I am wondering how many “not” included words/subwords were masked during MLM training? Does the masking techniques matter to this finding?\n8.\tTable 1, direct comparison of these quantitative results with baselines (attention/layer) is a bit difficult to follow. Are there any intuitive examples that show the merits of “influence patterns”? that is, I am wondering what will be the conclusions drawn by baselines (attention/layer) from your existing examples.\n9.\tTable 3 of appendix, is/are, himself/themselves are selected as examples here. Just wondering how often these words appeared in the original texts for training BERT? And any further comparison of their respective frequencies in the training/fine-tuning datasets?\n\nTypos:\n1.\tLine 49, “are are” -> “are”;\n2.\tLine 150, “two group of” -> two groups of?\n3.\tLine 104, 105, since you used “1) firstly”, possibly you can only use 1) or only firstly;\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No related negative societal impacts of this work. \nAlso, if the work goes deeper, it can be a good example of explaining “how/why BERT and related large-scale pretraining language models work”.\n", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "VHNNWYyweZ1", "reviewer": "Reviewer_MnyX", "summary": "As BERT achieves promising results in many NLU tasks (e.g., GLUE benchmark and SquAD benchmark), providing explanation for BERT has also received increasing attention recently. Some literature examines pre-trained BERT encoders, through using probes to analyze their semantic and syntactic knowledge. Some other work focuses on fine-tuned BERT, and uses gradient-based explanation (or variants of gradient) to provide local explanation for each input sentence. This work novelly proposes influence patterns, to abstract sets of paths through the BERT model, as to to illustrate the information flowing process within BERT. Experimental results shed light on the decision making process of BERT models.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n\n1.The proposed method is novel. The abstracted paths could illustrate the information flowing process within BERT in an friendly manner to users.\n\n2. The paper is well written and I enjoy reading it.\n\n3. Lots of interesting findings are found and the findings are convincing to me. For example, skip connections are important, and this could explain why lots of attention heads could be pruned without influencing the model performance. The decision making logic for phrases “not good” and “not bad” is novel.\n\nI have the following minor concerns.\n\n1. The proposed GPR model seems to work well with input texts with short length. Will the model work with longer input and with a larger model (e.g., BERT-large)?\n\n2. Do the findings work for more complex NLU tasks? For example, the finding that skip connection is more important compared to attention heads, could be attributed to the simplicity of the tasks examined in this work, e.g, SVA and RA. For more complex NLU tasks, such as MNLI and SquAD, does the finding still hold?\n\n3. Although lots of works use sparsity to evaluate the explanation quality, I do not think this is a good metric.  Suppose that the underlying model to be explained using dense patterns to make decisions, then the provided sparse explanations would be low fidelity to its decision making process. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the limitations are adequately addressed by the authors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "RVaWU0NS3T", "reviewer": "Reviewer_LK45", "summary": "This paper introduces the concept of influence patterns which provides a novel way to quantify and analyse the flow of information through a sequence of model nodes in a neural network. Using influence patterns, the paper shows the end to end flow of information in a BERT architecture and experiments also show that the ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and easy to follow. The proposed technique of quantifying the flow of information in a bert model using influence patterns seems to be novel enough and can be very useful in better understanding the bert models and potentially identifying any redundancies/shortcomings. Moreover, as the experiments show is significantly better than the baselines. However, I am a little puzzled by the results in last three columns of Table-1, correct me if I am wrong but how is the original model performance worse than the method where you just retain the nodes from \\pi_+ and make everything else zero?\nDid the authors also try using these influence patterns to prune the network by only retaining the important nodes and make the model inference and possibly even training faster?\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would have been useful if authors included a discussion on the computational cost of finding the influence patterns.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this paper proposes a novel method to understand functioning of Transformer models through influence patterns. 3/4 reviewers find the contributions interesting and substantial and voted for acceptance. One reviewer raised concerns about the purpose of such techniques and how they can help in design better architectures. Authors responded by saying such understanding is important for building trust and debugging. I completely agree with Authors on this and think such analysis studies are very important and am glad to recommend acceptance. I encourage Authors to include suggested changes/new comparisons brought up by reviewers in the final version.
Decision: Accept (Poster)