Influence Patterns for Explaining Information Flow in BERT
Year: 2022

Abstract:
While “attention is all you need” may be proving true, we do not know why: attention-based transformer models such as BERT are superior but how information flows from input tokens to output predictions are unclear. We introduce influence patterns, abstractions of sets of paths through a transformer model. Patterns quantify and localize the flow of information to paths passing through a sequence of model nodes. Experimentally, we find that significant portion of information flow in BERT goes through skip connections instead of attention heads. We further show that consistency of patterns across instances is an indicator of BERT’s performance. Finally, we demonstrate that patterns account for far more model performance than previous attention-based and layer-based methods.

heading: 1 Introduction
text: Previous works show that transformer models such as BERT [7] encode various linguistic concepts [24, 40, 13], some of which can be associated with internal components of each layer, such as internal embeddings or attention weights [24, 40, 15, 33]. However, exactly how information flows through a transformer from input tokens to the output predictions remains an open question. Recent attempts to answer this question include using attention-based methods, where attention weights are used as indicators of flow of information [5, 2, 49], or layer-based approaches[15, 14, 33], which identify important network units in each layer.
In this paper, we examine the information flow question through an alternative lens of gradient-based attribution methods. We introduce influence patterns, abstractions of sets of gradient-based paths through a transformer’s entire computational graph. We also introduce a greedy search procedure for efficiently and effectively finding patterns representative of concept-critical information flow. Figure 1 provides an example of an influence pattern in BERT. We conduct an extensive empirical study of influence patterns for several NLP tasks: Subject-Verb Agreement (SVA), Reflexive Anaphora (RA), and Sentiment Analysis (SA). Our findings are summarized below.
• A significant portion of information flows in BERT go through skip connections and not attention heads, indicating that attention weights[2] alone are not sufficient to characterize information flow. In our experiment, we show that on average, important information flow through skip connections 3 times more often than attentions.
• By visualizing the extracted patterns, we show how information flow of words interact inside the model and BERT may use grammatically incorrect cues to make predictions.
• The consistency of influence patterns across instances of a task reflects BERT’s performance for that task.
∗Correspondence to kaijil@andrew.cmu.edu
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
• Through ablation experiments, we find that influence patterns account for information flows in BERT on average 74% and 25% more accurately than prior attention-based and layer-based explanation methods[2, 23, 9], respectively.

heading: 2 Background
text: We begin this section by introducing notations and architecture of BERT in Sec. 2.1. We then introduce distributional influence as an axiomatic method to explain the output behavior of any deep model in Sec. 2.2, which serves as a building block for our method to follow in the next section.

heading: 2.1 BERT
text: Throughout the paper we use x to denote a vector and |S| to denote the cardinality of a set S or number of nodes in a graph S. We begin with the basics of the BERT architecture [7, 44] (presented in Fig. 1). Let L be the number of layers in BERT, H the hidden dimension of embeddings at each layer, and A the number of attention heads. The list of input word embeddings is x def= [x1,x2, ...,xN ],xi ∈ Rd. We denote the output of the l-th layer as hl1:N . First layer inputs are h 0 1:N def = x1:N .We use a l,i j to denote the j-th attention head from the i-th embedding at l-th layer and slj to denote the skip connection that is “copied” from the input embedding from the previous layer then combined with the attention output. The output logits are denoted by y.
Computation Graph of BERT A deep network can be viewed as a computational graph G def= (V,F , E), a set of nodes, activation functions, and edges, respectively. In this paper, we assume the graph is directed, acyclic, and does not contain more than one edge per adjacent pair of nodes. A path p in G is a sequence of graph-adjacent nodes [p1, p2, · · · , pt]; pt is the output node. Thus, the Jacobian passing through a path p evaluated at input x is ∏−1 i=1 ∂pi(x)/∂pi−1(x) as per chain rule. We further denote the Jacobian of the output of node ni w.r.t the output of connected (not necessarily directly) predecessor node nj evaluated at x as ∂nj(x)/∂ni(x).
For computational and interpretability reasons, an ideal graph would contain as few nodes and edges as possible while exposing the structures of interest. For BERT we propose two graphs: embedding-level graph Ge corresponding to the nodes and edges shown in Fig. 1 (left) to explain how the influence of input embeddings flow from one layer of internal representations to another and to the eventual prediction; and attention-level graph Ga ⊃ Ge that additionally includes attention head nodes and skip connection nodes as in Fig. 1 (right), a finer decomposition to demonstrate how influence from the input embedding flows through the attention block (or skip connections) within each layer.

heading: 2.2 Explaining Deep Neural Networks
text: Gradient-based explanations [39, 9, 37] are well-studied in explaining the behavior of a deep model by attributing the model’s output to each input feature as its feature importance score. While existing approaches often explain with the most important features for a single class i, distributional Influence [23] generalizes gradient-based approaches to answer a broader set of questions, e.g. why [MASK] should be IS instead of ARE (Figure 1), by introducing quantity of interest (QoI). Suppose a general network f : Rd → Rn, a QoI is a differentiable function q(f(x)) that outputs a scalar to incorporate the subject of an explanation. For example, to answer the aforementioned question, we can define q(f(x)) = f(x)IS − f(x)ARE where f(x)∗ is the logit output of class ∗. Formally, we introduce Distributional Influence.
Definition 1 (Distributional Influence) Given a model f : Rd → Rn, an input x, a user-defined distribution D(x), and a user-defined QoI q, Distributional Influence g(x; q,D) is defined as:
g(x; q,D) def= Ez∼D(x) ∂q(f(z))
∂z
Remark 1 Distributional Influence also leverages a user-defined distribution D(x) to capture the network’s behavior in a neighborhood of the input of interest x. By introducing D(x), we can prove that several popular attribution methods are specific cases of the distributional influence; e.g. when D(x) is a Gaussian Distribution, g(x; q,D) reduces to Smooth Gradient[37]; when D(x) is a uniform distributions over a path c = {x + α(x− xb), α ∈ [0, 1]} from a user-defined baseline input xb to x, D(x) reduces to Integrated Gradient [37].
We use D(x) as the uniform distribution over a linear path described in Remark 1 in the rest of the paper because it provides several provable properties[39] to ensure the faithfulness of our explanations. We approximate the expectation in Def. 1 by sampling discrete points in the uniform distribution.

heading: 3 Tracing Influence Flow with Patterns
text: To explain how different concepts in the input flow to final predictions in BERT, it is important to show how the information from each input word flows through each intermediate layer and finally reaches the output embedding of interest, e.g. [MASK] for pretraining or [CLS] for fine-tuned models. Prior approaches have use the attention weights to build directed graphs from one embedding to another [2, 49]. These approaches use heuristics to treat high attention weights as indicators of important information flow between layers. However, as more work starts to highlight the axiomatic justifications of gradient-based methods over attentions weights as an explanation approach [42], we therefore explore an orthogonal direction in applying distributional influence in BERT to trace the information flow. Since distributional influence only attributes the output behavior over the input features, in this section, we generalize it to find important internal components that faithfully account for the output behavior.
Tracing Influence by Patterns By viewing BERT as a computation graph (Ge or Ga in 2.1), we restate the problem: given a source node s and a target node t, we seek a significant pattern of nodes from s to t that shows how the influence from s traverses from node to node and finally reaches t. An exhaust way to rank all paths by the amount of influence flowing from s to t is possible in smaller networks, as is done by Lu et al. [27]. However, the similar approach lacks scalability to large models like BERT. Therefore, we propose a way to greedily narrow down the searching space from all possible paths to specific patterns. That is, our approach is two-fold: 1) we employ abstractions of sets of paths as the localization and influence quantification instrument; 2) we discover influential patterns with a greedy search procedure that refines abstract patterns into more concrete ones, keeping the influence high. We begin with the formal definition of a pattern.
Definition 2 (Pattern) A pattern π is a sequence of nodes [π1, π2, · · · , π−1] such that for any pair of nodes πi, πi+1 adjacent in the sequence (not necessarily adjacent in the graph), there exists a path from πi and πi+1.
A pattern π abstracts a set of paths, written γ(π) that follow the given sequence of nodes but are free to traverse the graph between those nodes in any way. Interpreting paths and patterns as sets, we define
γ(π) def = {p ⊆ P : π ⊆ p} where P is the set of all paths from π1 to π−1. If every sequence-adjacent pair of nodes is directly connected then the pattern abstracts a single path. To quantify the amount of information that flows from the input node to the target node over a particular pattern, we propose Pattern Influence, motivated by distributional influence.
Definition 3 (Pattern influence) Given a computation graph and a user-defined distribution D, the influence of an influence pattern π, written I(x, π) is the total influence of all the paths abstracted by the pattern: I(x, π) def= ∑ p∈γ(π) Ez∼D(x) ∏−1 i=1 ∂pi(z) ∂pi−1(z) . Proposition 1 (Chain Rule) I(x, π) = Ez∼D(x) ∏−1 i=1 ∂πi(z) ∂πi−1(z) for any distribution D(x).
Prop. 1 (proof in Appendix A) simplifies the evaluation of the pattern influence by specifying the exact set of internal nodes through which influence flows in a computational graph. We hereby introduce a greedy way of finding the most influential pattern in both Ge and Ga. Guided Pattern Refinement(GPR) Starting with source and target nodes s and t along with a initialized pattern π0 = {s, t} representing all paths between s and t, we construct π1 by adding one2 sequence-adjacent node e0 (there is a direct path between s and e0) from a guiding set E0 that maximizes the influence of the resulting pattern such that:
e0 = arg max e∈E0 {I(x, [s, e, t]), π1 =
[ s, e0, t ] , (1)
This procedure is iterated until we exhaust the last guiding set. We show an example of GPR in a toy graph in Fig. 2. For an embedding-level graph Ge, each guiding set El includes all embeddings hl1:N at layer l. For the attention-level graph Ga, we refine on the embedding-level pattern πe by only expanding πe from addition nodes in Ga compared with Ge: we perform GPR iterations between πel , π e l+1 ∈ πe with the guiding set Ela which includes A attention heads al and the skip node sl(Fig. 1 Right), until we reach the same last node in πe. The returned attention-level pattern πa thus abstracts a single path from the source to the target in Ga. As the attention-level analysis refines the embedding-level analysis, the produced attention-level pattern πa abstracts a strict subset of the paths of the attention-level graph that the embedding-level pattern πe abstracts. That is, πe ⊂ πa while γ(πa) ⊂ γ(πe). The detailed algorithm of GPR and analysis of its optimality can be found in Appendix B.1 and B.2.

heading: 4 Experiment
text: Experiments in this section demonstrate our method as a tool for interpreting end-to-end information flow in BERT. Specific visualizations exemplify these interpretations including the importance of skip connections and BERT’s encoding of grammatical concepts are included in Sec. 4.2. Sec. 4.3
2The algorithm can be easily adapted to include more nodes per layer. However, we found one node per layer retain a reasonable proportion of influence for the tasks evaluated in this paper(See Sec.4.4).
explores the consistency of patterns across instances and template positions and how they relate to task performances and influence magnitudes in. Sec 4.4 demonstrates two advantages of patterns in explaining the information flow of BERT over baselines: 1) abstracted patterns, with much fewer nodes compared to the whole model, carry sufficient information for the prediction. That is, without the information outside the refined pattern, the model shows no significant performance drop; 2) At the same time, patterns are sparse but concentrated in BERT’s components.

heading: 4.1 Setup
text: Tasks. We consider two groups of NLP tasks: (1) subject-word agreement (SVA) and reflexive anaphora (RA). We explore different forms of sentence stimuli(subtask) within each task: object relative clause (Obj.), subject relative clause (Subj.), within sentence complement (WSC), and across prepositional phrase (APP) in SVA [28]; number agreement (NA) and gender agreement (GA) in RA [24]. Both datasets are evaluated using masked language model (MLM) as is used in Goldberg [13]. We sample 1000 sentences from each subtask evenly distributed across different sentence types (e.g. singular/plural subject & singular/plural intervening noun) with a fixed sentence structure; (2) sentiment analysis(SA): we use 220 short examples (sentence length≤ 17) from the evaluation set of the 2-class GLUE SST-2 sentiment analysis dataset [47]. More details and examples of each task can be found in Appendix D.1.
Models. For linguistic tasks, We evaluate our methods with a pretrained BERT(L = 6, A = 8) [43], referred hereby as BERTSMALL. For SST-2 we fine-tuned on the pretrained BERTBASE[7] with L = 12, A = 12. The models are similar in sizes compared to the transformer models used in [2]. All computations are done with a Titan V on a machine with 64 GB of RAM. See Appendix D.2 for more details.
Implementation of GPR. Let the target node for SVA and RA tasks be the output of the QoI score q(y) def = ycorrect − ywrong . For instance, yIS − yARE for the sentence she [MASK] happy. Similarly, we use ypositive − ynegative for sentiment analysis. We choose an uniform distribution over a linear path from xb to x as the distribution D in Def. 3 where the xb is chosen as the the input embedding of [MASK] because it can viewed a word with no information. For a given input token xi, we apply GPR differently depending on the sign of distributional influence g(x; q,D): if g(x; q,D) ≥ 0, we maximize the pattern influence towards q(y) at each iteration of the GPR otherwise we maximize pattern influence towards −q(y). We use πi as the extracted patterns for individual input word i. When explaining the whole input sentence, we collect all refined patterns for each word and use Π = ⋃ i πi. Further, we denote Π+ as the set of patterns for all positively influential words. Both terms may be further decorated by a or e to denote attention-level or embedding-level results.

heading: 4.2 Visualizing Influence Patterns
text: This section does not serve as an evaluation but an exploration of insightful results and succinct conclusions an human user can learn from our proposed technique. We visualize the information flow identified by patterns found by GPR, and compare with those generated by attention weights as explored in the literature [2, 49]3. We therefore use Πattn to denote a pattern of nodes by maximizing the product of the corresponding attention weights between each pairs of nodes from adjacent layers. Implementation details are included in Appendix C.1.
We first focus on instances of the subtask SVA across object relative clauses (SVA-obj), which are generated from the template: the SUBJECT that the ATTRACTOR VERB [MASK](is/are) ADJ.. We observe in Figure 3 that the subject words exert positive input influences on the correct choice of the verb, and the intervening noun (attractor) exerts negative influence, which is true for both I(xi, πei ) and I(xi, πai ) (blue and purple bars in Figure 3a and 3b). While Πattn(Fig. 3c) does not distinguish between positively influential words and negative ones, nor do they show an interpretable pattern. Our other main findings are discussed as follows.
Finding I: Skip Connection Matters. Horizontal dashed lines in Figure 3 indicate that influence can flow through layers at the same word position via skip connections, which is not isolated as separate nodes Πattn (shown in Fig. 3c). Fig. 3a and 3b also show that the influence from subject
3The two works compute attribution from input to internal embeddings by aggregating attention weights(averaged across heads) across layers while attribution to individual paths is implicit.
words farmer[s] travels through skip connections across layers before it transfers into the attention head 5 in the last layer, indicating the “number information” from the subject embedding flows directly to the output. Interestingly, this also explains why attentions can be pruned effectively without compromising the performance [21, 31, 46] as important information may not flow through attentions at all. Namely, they are simply “copied” to the next layer through the skip connection. In fact, attention heads are traversed far less often than skip connections, which account for 75.4% of nodes in Πa across all tasks evaluated.
Finding II: that as an Attractor. Fig. 3a and 3b shows that the singular subject is less influential than the plural subject, especially when compared to the large negative influence from the attractor. Besides, the word that behaves like a singular pronoun in the singular subject case (Figure 3b), flowing through the same pattern as the subject (skip connections + attention head 5), whereas that is more like a grammatical marker (relativizer) in the plural subject case (Figure 3a): the pattern from that converge to the subject in the second to last layer via a different attention head 1. An explanation to the observed difference is that that can either be used as a singular pronoun in English, or a marker that encodes the syntactic boundary of the clause to help identify the subject and ignore the attractor. This finding reveals that although the latter one is always the correct usage for that across all instances, BERT may resort to the “easier”(while ungrammatical) encoding of that as a pronoun when the number happens to be singular.
Finding III: love, hate and like Figure 4a shows the pattern across three instances containing different clausal verbs by replacing love in Figure 3b with like or hate. We observe that hate and love are more influential, with a distinct and more concentrated pattern compared to that of
the word like, while one comparable to that of the subject or noun attractor. Therefore, hate and love are treated more like singular nouns than like, contributing positively to the correct prediction of the verb’s number (but for the wrong grammatical reason). This discrepancy also corroborates different accuracy for instances in the three subsets: hate: .99; love: 1.0; like: .82. In fact, the number of misclassifications containing like accounts for more than 33% of all misclassifications in SVA-Obj(SP).
Finding IV: Interactions with not. Two instances of sentiment analysis which BERT classifies correctly (negative for not good. and positive for not bad.) in Fig. 4b shows that: 1) in not good., not and good does not interact internally and not carries much more positive influence than good towards the correct class (negative sentiment); 2) in not bad., on the contrary, not contributes negatively to the correctly class (positive sentiment) and bad interacts with the not in an internal layer and exhibit a large positive influence. This disparity shows the model may treat not as a word with negative sentiment by default, and only when the subsequent adjective is negative, can it be used as a negation marker to encode the correct sentiment.
The asymmetries and oddities of patterns of Finding II to IV show how BERT may use ungrammatical correlations for its predictions. We include more examples of patterns in Appendix E.

heading: 4.3 Consistency of Patterns Across Instances
text: The prior section offered examples building connections between information flow and patterns. Since patterns are computed for each word of each instance individually, a global analysis of consistency/variation of patterns across instances can help us make deeper insights and make more general conjectures. We quantify this variation with pattern entropy on the linguistic tasks (SVA & RA) with fixed templates, where for each instance, each template position are chosen from a group of words. Interpreting the collection of instances, or template positions, in a task as a distribution, pattern entropy is the entropy of the binary probability that a node is part of a pattern, averaged over all nodes (minimally this is 0 if all patterns incorporate the exact same set of nodes). With a node n in graph G, a set of K instances, πk as the pattern of instance k by abuse of notation and H as the entropy function: pattern entropy of a pattern π is defined as:
Ĥ(π,G) =
∑ n∈GH(pn(π))
|G| ; pn(π) =
∑ k 1(n ∈ πk)
K
Figure 5a shows an inverse linear relation between influence magnitude and the pattern entropy of each template position(πi) in SVA-Obj. For example, the subject position has on average high influence magnitude and low pattern entropy, while grammatically unimportant template positions such as [CLS] and the period token has relatively low influence and high entropy. Note that this fit is not perfect: attractors, for example, have high influence magnitude but also high entropy’s due to their disparate behaviors among instances shown in Sec 4.2.
Figure 5b demonstrates a more global inverse relation between the pattern entropy of Π+ and taskspecific performance(average QoI) for 6 linguistic subtasks studied. This indicates that the more consistent a pattern is for a concept, i.e, the more consistent how a model locates the positively
influential signals, the better the model is at capturing that concept. In both figures, the baseline attention-based patterns (Πattn) does not indicate this relation. Together with Finding II to IV in Sec. 4.2, it demonstrates that patterns may offer alternative insights in formalizing and answering highlevel questions such as to what extent does BERT generalize to correct and consistent grammatical rules or use spurious correlations which varies across instances[30, 29, 34].

heading: 4.4 Evaluating Influence Patterns
text: This section serves as a formal evaluation of our proposed methods against existing baselines in tracing information flows. Except the attention baseline Πattn defined in Sec. 4.2, we also include the following baselines (implementation details to follow in Appendix C.1):
Πrand: patterns with randomly sampled nodes.
Πinf: patterns consisting of nodes that maximize the internal influence [23] for each guiding set.
Πcond: patterns consisting of nodes that maximize the conductance [9] for each guiding set.
We compare various aspects of extracted patterns against the following baselines, including: (1) ablation experiments showing how influence patterns account for model performance; (2) the sparsity of the patterns in the computational graph. Finally, we also discuss the relation between patterns and attention-weights.
Ablation. We extend the commonly-used ablation study for evaluating explanation methods [8, 6, 3, 23, 27] from input features to internal patterns as a sanity check of our method. That is, we ablate BERT down to a simpler model: we only retain the nodes from Πe+ (or Π a +) while replacing other nodes by zeros. The retained and replaced nodes together are forward passed to the next layer with the original model parameters until a new set of nodes are retained or replaced. Ablated Accuracy denotes the accuracy of the ablated model. We show the results for our methods Π∗+ and other baseline patterns4 in Table 1. Each baseline retains the same number of nodes as the corresponding Π+. Although this process seems to be highly invasive, the model ablated with Π∗+ achieved the highest ablated accuracy uniformly over all tasks. All baseline patterns except Πecond
5 gives random guesses. For the attention-level graph Πa, we also add a counterfactual pattern Πarepl_skip where we replace each skip node in Πa+ with its all A corresponding attention heads. However, with much fewer nodes ablated, Πarepl_skip still produces random ablated accuracies. This corroborates Finding I in Sec. 4.2 that the skip connections relaying important information directly which attention block cannot replace. In summary, patterns refined by GPR account for the model’s information flow more sufficiently compared to all baselines.
Sparsity of Patterns. A good explanation should not only account for the model performance, but also be sparse relative to the entire model semantics so as to be interpretable to humans. In this section, we quantify the sparsity of extracted patterns using two metrics: path share and concentration. With
4we run Πrand 50 times per task and average the ablated accuracy with no significant variation, the detailed statistics can be found in Appendix C.3
5We include discussions why Πecond has higher ablated accuracy than other baselines in Appendix C.2.
on average more than 10 words per sentence, the embedding and attention-level graphs contain at least 106 and 1011 individual paths, respectively. The totality of these paths represent the entire semantics of the BERT model. Path share(i.e. share(Π∗+) def = |γ∗(Π∗+)|/|P∗|), is defined as the number of paths in a pattern over the total number of paths in the entire computational graph. concentration, on the other hand, is defined as the proportion of negative/positive pattern influence over the total positive/negative influence. It represents the average ratio between the blue/purple bars and the orange bars in Figure 3.
Table 1(Right) shows that the abstracted patterns contain only a small share of paths while accounting for a large portion of both positive and negative influence. In linguistic tasks, the embedding level abstracted pattern has a concentration around 0.3 (conc.(Πe±)), indicating that the input concept flows through single internal embeddings in each layer, instead of distributed to many words. Zooming in on the attention-level graph, a relative high concentration conc.(Πa+), suggests that between the internal embeddings of adjacent layers, influence is also more concentrated to either one attention head or the skip connection. We speculate the much lower conc.(Π+) in SA is due to (1) the much larger model for the SA task with more layers and attention heads (2) sentiment information may be more diverse and complex than the information needed to encode syntax agreements, therefore input information may flow in a more distributed way. However, despite low concentration of pattern influence, the extracted patterns for SA are still effective in capturing the model performance as shown in Table 1(Left).
Attention heads in Πa vs. Attention weight value. Another observation is that the attention head nodes in Πa found by GPR aligns, to greater extent than random (1/|A|), with the head of the largest attention weight along the corresponding embedding-level edge, as shown by alignment rate in Table 1(Left). This partial alignment is expected since the Jacobian between two nodes correlates with the coefficients (weights) in the linear attention mechanism(product rule), while the opposite is also expected since (1) attention weights themselves are not fixed model parameters thus part of the gradient flow, (2) model components other than attention blocks come between two adjacent layers (e,g. dense layer & skip connections). In other words, attention weights are correlated, but not equivalent to gradient-based methods as explanations.

heading: 5 Related Work
text: Previous work has shown the encoding of syntactic dependencies such as SVA in RNN Language models [25, 16, 22, 19]. More extensive work has since been done on transformer-based architectures such as BERT. Probing classifiers has shown BERT encodes many types of linguistic knowledge[11, 14, 40, 41, 17, 20, 26, 24, 33, 15]. [13] discovers that SVA and RA in complex clausal structures is better represented in BERT compared to an RNN model.
A line of related works analyze self-attention weights of BERT[5, 45, 24, 12], where attention heads are found to have direct correspondences with specific dependency relations. [2] and [49], with which we compare in this paper, propose using attentions to quantify information flows in BERT. Attention weights as interpretation devices, however, have been controversial [36] and empirical analysis has shown that attention can be perturbed or pruned while retaining performance [21, 31, 46]. Our work demonstrates that attention mechanisms are only part of the computation graph, with each attention block complemented by other model components such as dense layer and skip connections; axiomatically justified influence patterns, however, can attribute to the whole computation graph. The strong influence passing through skip connections also corroborates the findings of [4] which finds input tokens mostly retain their identity. Besides pruning attentions, other works[32, 35, 18] also show that BERT is overparametrized and can be greatly compressed. Our work to some extent corroborates that point by pointing to the sparse gradient flow, while employing ablation studies only to verify the sufficiency of the extracted patterns.
A closely related and concurrent work[10] also shows the importance of skip connections and dense(MLP) layers by decomposing and analyzing forward-pass computations in self-attention modules. Our work, in comparison, introduces a gradient-based method that can be generalized to any model as long as they are differentiable. We also focus on how influential patterns can help us understand information flow of specific NLP tasks.
Recent work introducing influence paths [27] offers another form of explanation. Lu et al. [27] decomposed the attribution to path-specific quantities localizing the implementation of the given
concept to paths through a model. The authors demonstrated that for LSTM models, a single path is responsible for most of the input-output effect defining SVA. Directly applying individual paths to transformer-based models like BERT, however, results in an intractable number of paths to enumerate due to the huge number of computation edges in BERT.

heading: 6 Limitations and Future Work
text: We believe there are three limitations to address in future work. (1) GPR algorithm does not guarantee absolute optimality, however, we provide empirical evidence in support of the searching algorithm in Appendix B.2. (2) For interpretability reasons, we compute GPR by picking one node per guiding set, while more complicated information may distribute to multiple internal embeddings or attention heads. (3) The findings in Sec. 4.2 would benefit from more quantitative analysis to support more general claims(instead of speculations) on BERT’s handling of various linguistic/semantic concepts. However, we will explore these limitations in future work and release our code and hope the proposed methods will serve as an insightful tool in future exploration.

heading: 7 Conclusion
text: We demonstrated influence patterns for explaining information flow in BERT. We highlighted the importance of skip connections and BERT’s potentially mishandling of various concepts through visualized patterns. We inspected the relation between consistency of patterns across instances with model performance and quantitatively validated pattern’s sufficiency in capturing model performance.
Acknowledgement This work was developed with the support of NSF grant CNS-1704845. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation thereon. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the National Science Foundation or the U.S. Government. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this work.

Review summaries:
This submission proposes a novel graph-based method to visualize the information model in transformer-based models such as BERT. The proposed method utilizes graph search algorithm to produce patterns which are generalized paths in graphs. The advantage of these patterns is that they are sparse and have fewer nodes than a conventionally defined path in the computation graph and are therefore potentially more interpretable. The authors demonstrate that such visualization can lead to interesting findings on several natural language processing tasks. Through quantitative evaluations, the authors also show that the proposed method is better than previous methods in terms of ablated accuracy and sparsity.   That being said, my main concern is that I doubt such visualization can bring much *practical values* for model development and model understanding. I detail my concerns below.

In order to explain how information flows from input tokens to output predictions in transformer-style models such as BERT, this paper defined “influence patterns”, abstractions of sets of paths (of neural network nodes) through a transformer model. BERT_SMALL and BERT_BASE were used to perform the analysis under two groups of NLP tasks (1) subject-word agreement (SVA) and reflexive anaphora (RA); and (2) sentiment analysis (SA).   The observations include (1) a significant portion of information flows in BERT go through the residual skip connections, instead of multi-head (self-)attention heads; (2) BERT could use grammatically incorrect (e.g., let “not” to be negative, without considering its context such as “not good” vs. “not bad”); (3) “influence patterns” account for information flows in BERT on average of 74% and is 25% more accurate than prior attention-based and layer-based explanation methods.  The definition of “influence patterns” using the idea of node paths and the major observations are the major contributions of this paper. 

As BERT achieves promising results in many NLU tasks (e.g., GLUE benchmark and SquAD benchmark), providing explanation for BERT has also received increasing attention recently. Some literature examines pre-trained BERT encoders, through using probes to analyze their semantic and syntactic knowledge. Some other work focuses on fine-tuned BERT, and uses gradient-based explanation (or variants of gradient) to provide local explanation for each input sentence. This work novelly proposes influence patterns, to abstract sets of paths through the BERT model, as to to illustrate the information flowing process within BERT. Experimental results shed light on the decision making process of BERT models.