ABSTRACTING INFLUENCE PATHS  FOR EXPLAINING (CONTEXTUALIZATION OF) BERT MODELS
Authors: Kaiji Lu
Zifan Wang
Piotr Mardziel
Anupam Datta
Venue:  (2021)
Keywords: interpretability
natural language processing
transformer
BERT

Abstract: While “attention is all you need” may be proving true, we do not yet know why: attention-based transformer models such as BERT are superior but how they contextualize information even for simple grammatical rules such as subject-verb number agreement(SVA) is uncertain. We introduce multi-partite patterns, abstractions of sets of paths through a neural network model. Patterns quantify and localize the effect of an input concept (e.g., a subject’s number) on an output concept (e.g. corresponding verb’s number) to paths passing through a sequence of model components, thus surfacing how BERT contextualizes information. We describe guided pattern refinement, an efficient search procedure for finding sufficient and sparse patterns representative of concept-critical paths. We discover that patterns generate succinct and meaningful explanations for BERT, highlighted by “copy” and “transfer” operations implemented by skip connections and attention heads, respectively. We also show how pattern visualizations help us understand how BERT contextualizes various grammatical concepts, such as SVA across clauses, and why it makes errors in some cases while succeeding in others.

Under review as a conference paper at ICLR 2021
ABSTRACTING INFLUENCE PATHS FOR EXPLAINING (CONTEXTUALIZATION OF) BERT MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
While "attention is all you need" may be proving true, we do not yet know why: attention-based transformer models such as BERT are superior but how they contextualize information even for simple grammatical rules such subject- verb number agreement (SVA) is uncertain. We introduce multi-partite pattern,, abstractions of sets of paths through neural network model. Patterns quantify and localize the effect of an input concept (e.g., a subject's number) on an output concept (e.g. corresponding verb's number) paths passing through sequence of model components, thus surfacing how BERT contextualizes information. We describe guided pattern refinement, an efficient search procedure for finding sufficient and sparse patterns representative of concept-critical paths. We discover that patterns generate succinct and meaningful explanations for BERT, highlighted by "copy" and "transfer" operations implemented by skip connections and attention heads, respectively. We also show how pattern visualizations help us understand how BERT contextualizes various grammatical concepts. such SVA across clauses, and why it makes errors in some cases while succeeding in others.
INTRODUCTION
Recent advancements in NLP have been spurred by contextualized representations created in deep neural models such as BERT (Devlin et al. 2019). These contextualized representations, which are designed to be sensitive the context which they appear (Ethayarajh (2019). are also shown to capture many grammatical concepts (Lin et al. 1019:Tenne et al. 2019a), including subject-verb agreements(SVA) and reflexive anaphora(RA) (Goldberg 2019). However, the exact mechanism of contextualization in BERT, i.e., the process of developing contextualized representations from representations of individual input words in the sentence context, remains unclear. For example, in the sentence the pilots that the architect likes is/are short, choosing the correct the verb is over are to agree with the subject requires contextualizing the verb with plurality information of the subject. In this paper, answer the central question: How contextualization realized in BERT for grammatical concepts such as SVA and RA? Specifically,
identify sub-components of BERT that are a) sufficient for representing those concepts but also b) sparse enough to legibly show how BERT contextualizes the concepts across layers and whether the contextualization follows correct grammatical rules?
Prior works on explaining contextualization in BERT rely on the analysis of layer representations and attention components. Representation analyses, either by training a probing classifier (Lin et al. 2019 Tenney et al., 2019a), or finding parse trees embedded in the representations (Hewitt & Manning 2019 Reif et al.) 2019), demonstrate that relevant linguistic concepts are associated with the activations of BERT components (i.c. subject's 's number associated with the activations of a certain head a certain layer, or subject's representation closer to that of the verb's 's under certain transformations), but do not tell us how representations come about inside the model. Meanwhile, inspection of attention weights as indicators of the flow information between BERT layers (Clark) et al. 2019), requires subjective inference of relevant function (i.e. inference that certain head may be involved because high attention weights between cells at the subject and cells at the verb), which are found to be problematic in other contexts (Brunner et al. 2020: Jain & Wallace 2019). Analysis of attention further disregards the role of skip connections that do not involve attention at all. Neither approach allows track concept causal chain from input output distinguish

---

Under review as a conference paper at ICLR 2021
helpful from hindering representations or flows (hindering information such contextualization of confounding inputs like unrelated nouns in a sentence lead to errors on SVA).
To answer the central question while overcoming these limitations, introduce multi-partite patterns, abstractions of sets of paths through a neural model (a graph). Patterns quantify and localize the effect of an input concept (e.g. subject's number) corresponding verb's number) to a collection of paths passing through a sequence of model nodes and/or edges. We describe guided pattern refinement, search procedure for finding patterns representative of concept-critical paths that let us selectively explore the importance of chosen aspects of model (e.g. in BERT, we can refine patterns showing criticality of certain heads to paths also showing whether this is due to skip connections or due to attention). To demonstrate the contextualization process, we further extend the experimental framework to integrate impacts of multiple words towards a given concept (as opposed to impact of single word, e.g. subject on SVA).
output concept (e.g.
Contributions: 1) We describe multi-partite patterns for explaining the model-wide contextualization in neural models like BERT and guided pattern refinement (GPR) to discover influential patterns focusing on model elements of interest. 2) We visualize BERT's contextualization grammatical concepts including subject-verb agreements(SVA)) and reflexive anaphora(RA) (RA), and qualitatively show how BERT encode these concepts using grammatically correct or incorrect cues. 3) We validate the sufficiency and sparsity of derived patterns with model compression and concentration metrics, respectively.
We begin with a summary of requisite techniques in Sec. We describe the core elements of our methodology in Sec. 3 and exemplify them for understanding BERT in Sec. We elaborate on related works in Sec. 5 conclude in Sec. 6
BACKGROUND
We introduce the basics of the BERT architecture (Fig. D and the learning task subject to our work. We then discuss existing explanation devices and how they motivate our methods that follow.
BERT In BERT, let L be the number of Transformer encoder layers, H be the hidden dimension of embeddings at each layer, and A be the number of attention heads. The list of input word embeddings x def XI.,x,....N.. X; Rdd. We denote the output of the (-th layer as b First !. layer inputs are hy.v XI:N. We use a; to denote the j-th attention head from the i-th embedding 1-th layer and s'y to denote the skip connection that "copied" from the input embedding from the previous layer then combined with the attention output. Probability scores for candidates of [MASK] are denoted by y, def softmax(Wh(", W R where is the vocabulary size. We denote the index of [MASK] as m. The layered architecture is presented in Fig. Ileft) and a detailed view of the transformer layer in Fig. rright). For further details, refer to Vaswani et al. (211)) and
Devlin et al. (2019).
We focus on Masked Language Modeling (MLM) used in BERT pretraining: predict a masked word represented by [MASK] in context sentence. The MLM task has been used to evaluate whether BERT learns linguistic concepts such as SVA by measuring if it assigns a higher probability for the correct verb (e.g. are in Fig. ( than the incorrect verb (e.g. iss at the [MASK] position (Goldberg) 2019).
Distributional Influence. To explain DNN's behavior, distributional Influence attributes each input measure of impact on model output. Saliency (Baehrens et al.) 2010), as an example, defines influence as the gradient of output w.r.t. the input. In a generalized framework Leino et al.) (2018), influence quantifies the impact of each input feature towards a concept (e.g. SVA) by instrumenting model's 's inputs with distribution of interest (Dol) and the output with a quantity of interest (Qol).
Definition (Distributional Influence) Given a model f -
- R", an input x, a Dol D(x), and Qol q: R + R, Distributional Influence (x(x) quantifies the impact of an input concept defined
parttite because patterns abstract sets of paths in neural models viewed multi-partite graphs.
2

---

Under review as a conference paper at ICLR 2021
2 "skaters"
Fi
1+1
A,
8
'Ilke
9m
A.
2
mm [MASK]
Nm
1+1
h
B
JFmm 'young"
E
mm
1
a
Figure I: BERT Transformer architecture (left) and details of a transformer layer (right).
by the Dol on the output concept defined by the Qol:
0g (z))
9g (x) def E Exx
On
Instantiations defining SVA and RA concepts in BERT models are found in Sec. Examples of Dol include Gaussian distributions with mean x (Smilkov et al. (017), or uniform distributions over path x - axx X6) , E [0, }]} from user-defined baseline input to the target input (Sundararajan al.] 2017). We use the latter the rest of paper; we approximate the expectation in Def. by sampling discrete points in the uniform distribution (Sundararajan et al. 2017) (see Appendix B. for an analysis of the accuracy of approximation).
Explaining Contextualization with Influence Paths. While it can highlight relevant inputs, Distributional Influence cannot show if how they are contextualized internally to form higher- level concepts. Influence Paths (Lu et al., 2020) localizes input influence measurement to paths in neural model and thus can be used to show how the influence of the input representations flows internally through one internal representation to another. A computation graph g def (V, is a set of nodes, activation functions, and edges, respectively. In this paper, we assume the graph is directed, acyclic, and does not contain more than one edge per adjacent pair of nodes") A path p in G is sequence of graph-adjacent nodes [PI, P2,,,,.... P_-]. We denote the Jacobian of the output of node n, w.r.t the output of connected (not necessarily directly) predecessor node n, evaluated as Ôn, (x)) We write VxP as the component of the Jacobian passing through path evaluated input as per chain rule: Xxp def
i== (opp(xx8pp.1155
Definition 2 (Individual Path Influence) Given path p of a computation graph G, the individual path influence for an input x, or x(xx,) p) is:
x(x.p) def ExxD(x) [Vp)]
Lu et al. (2020) uses individual path influence decompose distributional influence paths and explain internal LSTM behaviour under SVA via the most influential path arg maxpep x(x,p) where P are all paths from input a particular output (normally Qol). The influence path approach relies on enumerating P.
PATH ABSTRACTION
Directly applying individual influence paths of [Lu et al. (2020) to transformerrbased models like BERT has computational and conceptual problems. BERT denser in terms of model connections: each node at each layer integrates information from all nodes of the prior layer (as opposed to the pair of short-term and long-term connections in LSTM). This results in intractable number of influence paths to enumerate, even for processing the simplest BERT variants.
"The single edge restriction is for notational conveniences to follow: if given neural model does have more than one edge hetween adjacent nodes, replace duplicate edges with 2-length paths through dummy identity nodes satisfy this requirement without affecting its semantics.
3

---

Under review as a conference paper at ICLR 2021
Our approach is three-fold: first employ abstractions of sets of paths as the localization and influence quantification instrument; second, we discover influential patterns with a greedy search procedure that refines abstract patterns into more concrete ones, keeping the influence high; and third, we consider the collection of influence patterns from every word a sentence to the quantity of interest. We begin with the pattern abstraction:
Definition 3 (Multi-partiee pattern) A multi-partite pattern π is sequence of nodes [m,, ,2,-.. --1 such that for any pair of nodes Ts, R+41 adjacent in the sequence (not necessarity adjacent in the graph), there exists a path from ni and 1141-
A pattern иT abstracts set of paths, written (T)) that follow the given sequence of nodes but are free to traverse the graph between those nodes in any way. Interpreting paths and patterns as sets, we define 7(x) der {p S P S p} where P is the set of all paths from to T-1- If every sequence- adjacent pair of nodes is directly connected then the pattern abstracts a single path.
Definition 4 (Pattern influence) Given computation graph and a Dol D, the influence of a multi- partite influence pattern π, written I(x, т) is the total influence of all the paths abstracted by the patter::
(x)
I(x, R.) def Σ
x(x,p) = E(x II 1
DEY(M)
Also note that influence of individual paths may be positive negative cancellation in the influence of a pattern which aggregates paths is possible.
Computation Graphs for BERT A given DNN can be expressed by many computational graphs. For computational and interpretability reasons, an ideal graph would contain as few nodes and edges possible while exposing structures of interest. For BERT in particular, level graph G. corresponding to the nodes and edges shown in Fig. (left) to explain how the influence of input embeddings flow from one Transformer layer to another and to the eventual prediction of [MASK]; ; and altention-level graph Go C G. that additionally includes head nodes as Fig. (right), a finer decomposition to demonstrate how influence from the input embedding flows through the attention block within each layer. BERT's s semantics are modeled using computational graph's 's activation functions which we omit here.
propose embedding-
As the attention level graph contains superset of the nodes of the embedding level graph, we can interpret embedding level patterns as abstracting paths in both the embedding level graph and the attention level graph. Furthermore, concrete path in Ge is a pattern in Ga as it contains Go-non- adjacent nodes and thus abstracting multiple paths in Ga. For given pattern क of Ge we can thus write 7a (T) as the set of paths it abstracts in Ga with:
Va
del
(a(P)
(ETa(*)
Guided Pattern Refinement(GPRR Instead of enumerating the path space of P for discovering influential paths, we approximate a search by greedily refining patterns while maximizing their influence. Starting with sources and target nodes s and t along with a pattern TO = {s representing all paths between and that maximizes the influence of the resulting pattern. At the first iteration and subsequently, the guiding set defines cut of the (multi-partite) graph between two sequence adjacent nodes (initially just 8 and f). The procedure is repeated with additional refinement. At iteration i + 1, a guiding set ES defines a cut between nodes s¹ and t while the cut node that refines the pattern to maximal influence is selected:
construct x¹ by adding node from a guiding set E
del m' der
e', arg max I(x, N² [s t \ss t eEE
Above, π [a, c \ a, b, denotes the pattern π in which sequence adjacent nodes a, c are replaced with a, b, e, in their position in the sequence.
4

---

Under review as a conference paper at ICLR 2021
Repeating the procedure for some number of steps until some stopping criterion is reached produces a sequence of patterns with decreasing abstraction: ((**+1) S )*()) Once a pattern is produced that abstracts a single path, no more refinement be done though it might not be desirable to continue refinement until that point for interpretability reasons. Also, the choice of guiding sets F at each iteration their influence significance and computational requirements of iteration. Smaller sets require fewer options to enumerate but are likely to lead to less influential patterns.
have an impact the resulting patterns both in terms of
In our experiments we employ a layer-ordered strategy for the embedding-level pattern refinement and then refine the resulting pattern in the attention-leve graph. In the embedding analysis, at iteration i, we focus on layer i. The guiding set E' is the cut:
(embedding-lewe guiding set) E def {4)),,
The refinement thus proceeds for iterations (the input layer can be skipped). If the input node denoted as v and the quantity of interest is denoted as q, the refinement process results in a pattern Te { x. f1 j.. ... .hf, 9} where j, are indices designating which embeddings at each level i the abstracted paths traverse.
The attention-level refinement starts with the embedding-level pattern x and exposes the attention heads to cut the flow of influence in that starting pattern, also in order of the layers. At iteration i, the cut E is:
(attention-level guiding set) E
C.k J.
U (
That is, the cut separates embedding nodes hi, and h hit1 with the attention heads ( i.k and skip analysis, the
ji+n
edge modeled as a node s'.- As the attention-level analysis refines the embedding produced attention-level pattern лO abstracts strict subset of the paths of the attention-level graph that the embedding-level pattern n" abstracts. That is, xe C TT and therefore (a C (TT").
In our experiments, we perform GPR independently for each input word, and refine with most positively influential cut nodes for positively influential words (g, (x,) 2 0)) but refine with the most negatively influential nodes for negative (x.) < 0) words. In the following section, use πy as the extracted patterns for individual input word i, IT as the set of patterns for all words, and I1 as the set of patterns for all positively influential words. Both terms may be further decorated by or e to denote attention-level or embedding-level results.
EVALUATION
We apply GPR to discover BERT patterns the level of embedding and attention. We begin with a summary of the linguistic tasks, datasets, models, and for their sparsity and sufficiency in Sec. 4.1 as measured by the metrics of concentration and compression accuracy (Lu et al.) 2020), respectively. Finally, in Sec. 4.2 we visualize example patterns and discuss how they help explain contextualization of SVA in BERT.
We evaluate patterns
Tasks, We consider two linguistic tasks: subject-word agreement (SVA) and reflexive anaphora (RA). We explore different forms of sentence stimuli each task: object relative clause (Obj.), subject relative clause (Subj.), within sentence complement (WSC), and across prepositional phrase (APP) in SVA; number agreement (NA) and gender agreement (GA) RA. SVA and RA datasets (Marvin & Linzen, 2018 Lin et al., 2019) are evaluated with MLM in same way as prior work (Goldberg 2019). We sample 200 sentences evenly distributed across different sentence types (e.g. singular/plural subject & singular/plural intervening noun) with fixed sentence structure from each task; sentence length and the word types in each position are consistent across samples. Examples of each task are found in Appendix A
Qol and Distributional Influence. We use the same Qol from Lu et al. (020) where q(ym) def Umrcervect Dmn, wrongs e.g. Vm,1s Ama for she [MASK] happy. We select D as uniform distribution over a linear path from X2 to x in the input space for each word with the baseline x; defined as the the input embedding of [MASK]; we view it neutral word with no information.
Model. We evaluate our methods with a BERT model with L = 6, = 8, referred hereby as BERT SMALL, of Ture et al. (2019) instead larger models such as BERT used in the original BERT
5

---

Under review as a conference paper at ICLR 2021
Pg
C
C:II:
acc.( f
acc. *)
Task
[P*]
(cc.()ri.)
a
a
e
a
e
e
a
SVA
Obj.
14.4
27.6
0.34
0.22
0.29
0.16
0.06
2.0e-7
0.99
0.74
0.50
0.51
0.96
Subj.
14.4
27.6
0.30
0.18
0.31
0.17
0.09
2.0e-7
0.74
0.56
0.50
0.49
1.00
WSC
13.8
27.0
0.27
0.16
0.38
0.24
0.03
1.8c-7
1.00
0.69
0.52
0.49
1.00
APP
27.6 14.4
0.33
0.18
0.31
0.15
0.06
1.6e-7
0,92
0.62
0.55
0.60
1.00
RA
NA
14.4
27.6
0.31
0.18
0.32
0.18
0.04
1.4e-7
0.68
0.49
0.54
0.52
0.83
GA
15.4
28.6
0.22
0.14
0.23
0.15
0.02
1.4e-7
0.76
0.72
0.65
0.55
0.73
Table I: Pattern sufficiency, sparsity, and related metrics on various linguistic tasks. Metrics are shown in the 1st row while the 2nd row indicate graph levels: * denotes e corresponding to abstracted embedding-level patterns xe or attention-level patterns n", respectively. [g]P=*
natural log of the number of possible paths; C4 c positive/negative concentrations; (x-
percentage of paths in the abstracted patterns over the total number of paths; acc.( ): the compressed accuracy of abstracted patterns; (rand.. *): the compressed accuracy of randomly
compressed models; (or.() the accuracy of the original model BERTSMALLE
paper (Devlin et al. 2019) because 1) we find BASE is not significantly better than BERT SMALL in the tasks of interests as shown in Appendix A 2) when approximating the expectation in Def. 2 with finite points, we find more than 2000 samples are required in BASE for an acceptable margin, while 50 samples suffices for BERT SMALL (see Appendix. [B.],, and 3) visualizations of abstracted patterns from BERTsMALL are easier on human interpreters.
4.1 QUANTITATIVE ANALYSIS
Recall that and T denote the abstracted pattern returned by GPR with an embedding- graph and an attention-leve graph, respectively. The quantitative evaluation in this section aims to verify that influential patterns in BERT are 1) sparse: inputs influence the Qol largely through nf or n and 2) sufficient: BERT retains high task accuracy if only nt Tº are evaluated at inference time. Firstly, we introduce concentration to evaluate sparsity:
Definition (Concentration) Given an input with N words/embeddings, their distributional influences 9g (x) and pattern influences I(x, Ti) for a set of patterns II { the concentration of the positive (and negative) pattern influence C (and CTI-) are the patterns' share of positive (or negative) influences as compared to the total positive (or negative) distributional influence:
OlI+ def E. (I(x, mi); [x m.); 2 0}} C def ΣN (I(x, mi); I[((x, < 0}}
Σ; (x) (x), 0]}
(x): (x), < 0]}
To evaluate sufficiency, we employ a compression study previously used to verify other explanation devices (Dabkowski Gal 2017 Ancona et al., 20188 Leino et al. 2018 Lu al. 2020 For each example, we compress BERT down to a specific pattern: we only retain the nodes from 11 (or 114) while replacing all other nodes, layer by layer. Starting from the first layer, the embedding nodes not in II; (or the attention/skip connection nodes not in II;)) are replaced by the embedding of [MASK] (or zero vectors for attention/skip nodes), while the nodes IIŞ (or I1)) remain untouched. The retained and replaced node together are forward passed to the next layer using the original model parameters until new set nodes needs to be retained or replaced. We then compare the accuracies of predicting labels of [MASK] between the original model and the compressed model, the former of which we refer to as compressed accuracy.
Explanations of Results. Concentration and compressed accuracy of л" and n" are shown in Table I [usiing the setup in Sec. Replacing * in the second row with e or a corresponds to the results for the abstracted embedding- patterns n" and the attention-level patterns л², respectively.
Sparsity with concentration. Per the gradient chain rule, the total influence all individual paths from the input to Qol equals to the distributional influence, therefore, < CII+,CIL. < 1. As

---

Under review as a conference paper at ICLR 2021
[CLS]]
[CLS]]
the
ngg
the
neg am
neg dkp
plots
s(%)
plol
4(%) Ils, r2)
Ils.
1hat
thet
Il..
architecl
ministens
-
lkk
[MASK]]
[MASK]]
[SPP]
[SEP]
-1.0
-2 0
(a) plural subject + singular intervening noun(PS)
(b) singular subject + plural intervening noun(SP)
Figure 2: Significant patterns x" extracted by GPR from the attention- graph for task SVA across Object Relative Clauses (Goldberg, 2019; Marvin & Linzen 2018), in two exmaples with attractors.
Left: bar plots of the distributional influence g( (xx)(yelloow I [(x;, m (purple) and I(x;, TT) (blue) for each word at position i. Right: significant patterns x from each input word at position i to quantity of interest (verb number correctness). The square nodes denote the input embeddings and circles denote internal contexualized embeddings. Dashed lines correspond skip connection in the attention block while solid lines correspond to connection through (any) attention heads. Attention connections with high influence flow are marked with the corresponding attention head number (ranging from 1 to 8). Line colors represent the sign of influence (red as negative and green as positive).
shown in the first two columns of Table [ there are about exp 14 and exp 27 individual patterns in the embedding and attention-level graphs, respectively. However, the abstracted patterns (shown by the 3rd to 6th columns) account for large portion of both positive and negative influence across all tasks. The embedding level (* e) abstracted pattern contributes around 30% of the total influence indicating that the concept is concentrated to individual contextualized embeddings in each layer, instead of dispersed to many words. Zooming on the attention-level (* = a), concentration suggests that between the contextualized embeddings of adjacent layers, influence is also more concentrated to either one attention head or the skip connection.
Sufficiency with model compressed accuracy. We show the original accuracies of the model on different tasks in the last column. The compressed model retains a high accuracy (as shown in 9th and thth columns) with tiny portion of the models (shown in 7t and 8th columns) retained. As comparison, we denote the compressed accuracy with random patterns in the th and 12 columns, which compresses the model by retaining the same number of nodes πς but has performance close to 50%, effectively random guess; randomly chosen patterns of the same size do not abstract the concept at all.
4.2 EXPLAINING CONTEXTUALIZATION SVA ACROSS OBJECT RELATIVE CLAUSE
In this section, we explain contextualization between internal representations of BERT by visualizing the significant patterns n. and n found by GPR for two examples of SVA across object relative clauses as seen in Figure Results on other tasks are included in the Appendix, B.2
First we observe that in both sentences of Figure 2 both words the subject phrase ("the"" and the nouns) exert positive input influence on the correct prediction of the verb, and the intervening noun(attractor) exerts negative influence, which also true for both Z(x, Tς ); and I(x, mf)
"Copy and Transfer" We observe many horizontal dashed lines in Figure 2 indicating significant influence flows through layers at the same word position using skip connections. Zooming in on कद and Tf. we observe that the subject phrase travels through skip connections across the lower layers, and only through attention head in the last layer. This "copy and transfer" procedure indicates that BERT mostly picks up the signal from the subject input embedding without much contextualization,

---

Under review as a conference paper at ICLR 2021
however, exactly how it overcomes(or not) the comparable signals from the attractors is explained the next section. In addition, we speculate that the reason that attentions can be effectively pruned without compromising the performance in prior works (Michel et al. 2019), is that some concepts does not travel through attention block at all: they are simply "copied" to the next layer through the skip connections. In Appendix [B.2 we observe that all the above conclusions are also prevalent in other tasks (such as the contextualization of propositions in WPP task), though different heads might be used for the "transfer" operations in different tasks.
The Role of that Comparing two sentences of Figure 2. we do observe that the influence from the singular subject is weaker than that of the plural subject, especially compared the negative influence from attractors. The key difference is that that behaves differently for singular and plural subject. that in Figure [4a behaves as a singular noun (since that t. also means a singular pronoun in English), flowing through the same straightforward pattern as the subject (skip connections + attention head 5); that in Figure 2a] however, behaves more like a grammatical marker(relattvvizer). the pattern from that travel from itself to the subject in the second to last layer through a different attention head. We speculate that that in plural subject sentences encodes the syntactic boundary of the clauses and help identify the main subject and ignore the intervening noun. As result, attractors in PS have smaller negative influence, compared to the high negative influence from attractors in SP (a similar comparison is also observed in PP and SS cases). This discrepancy in the behavior of that also corroborates lower SVA accuracy in SP case than in PS case (See Appendix [A]. We observe this difference consistently across all instances as shown in an aggregated visualization in Appendix B.2 in other tasks as well.
RELATED WORK
Previous work has shown the encoding of syntactic dependencies such as subject verb agreements(SVA) in RNN Language models (Linzen et al. 2016), as well as the explanations for such encoding Hupkes et al. 2018 Lakretz et al.) 2019, Jumelet et al. 2019). More extensive work has since been done transtormer-based architectures such BERT(Devlin et al.. 2019).
Diagnostic classifiers trained on output and internal representations discover that BERT encodes many types of linguistic knowledge(Elazze
al.] 2020; Hewitt & Liang 2019: Tenney et al.)
2019a|b; Jawahar et al.] 2019; Klafka & Ettinger] [2020] Liu et | Lin et al. 2019). ranging from syntactic concepts to more complicated semantic ones. Goldberg (2019) discovers that
SVA and RA in complex clausal structures is better represented in BERT compared to an RNN model. This partially explained by (Reif et al. 2019 Hewitt & Manning 2019) which show that contextual embeddings in BERT can encode syntactic structures hierarchically comparable to those represented in a dependency tree. However all these analyses are done on frozen contextual embedding layers; the exact causal mechanism concept is encoded from input to output is not explored.
Another line of work in interpreting BERT concerns analyzing the self-attention weights of Clark et al., 2019 Vig & Belinkov] 2019 Lin et al.] 2019), where attention heads are found to have direct correspondences with specific dependency relations. However, attention weights as interpretation devices has been controversial(Serrno & Smith 2019). and empirical analysis has shown that attention can be perturbed or pruned while having the same or even better Caarmance(Kovalery et al., 2019; Michel et al. 2019 Voita et al. 2019). More importantly, our work demonstrate that attention mechanisms are only part of BERT computation graph, with each attention block complemented by additional architecture such as dense layer and skip connections. The strong influence passing through skip connections also corroborates the findings of Brunner et al.] (2020) which find input tokens mostly retain their identity. Besides pruning attentions, other works((rasanna et al., 2020; (ann et al., 2019; Jiao et al.] (209)) also show that BERT overparametrized and can be greatly compressed. Our work to some extent corroborates that point by pointing to the sparse gradient flow, while employing model compression only to verify the sufficiency of the extracted patterns.
Recent work introducing influence paths (Lu et al. 2020) offers another form of explanation. et al. (2020) decomposed the attribution to path-specific quantities localizing the implementation of the given concept to paths through a model. The authors demonstrated that for LSTM models, a single path is responsible for most of the input-output effect defining SVA, and explored the effects
Lu
8

---

Under review as a conference paper at ICLR 2021
of unhelpful nouns which showed negative influence SVA. We describe the limitations of this methodology when applied to BERT in Sec.
6 CONCLUSION
We have demonstrated how multi-partite influence patterns to localize DNN model's handling of a concept of interest and along with a pattern refinement method we how BERT handles subject-verb number agreement and reflexive anaphora. We quantitatively validated the sufficieny and sparsity of influence patterns in BERT by way of compression experiments and the influence concentration of discovered patterns. We qualitatively and visually demonstrated BERT's contextualization in the two tasks using our methodology. Our formalism and methods are general enough to apply to the analysis of other aspects of BERT and other models.
REFERENCES
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference Learning Representations, 2018. URL https:/ ftttp:://opereeveee net www.:/rrtttrrrm
Sy21R9JAM
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831, 2010.
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identifiability in transformers. In International Conference on Learning Representations, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, 2019.
Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In NIPS, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. When bert forgets how to pos: Amnesic probing of linguistic properties and mlm predictions. arXiv preprint arXiv:2006.00 009995 2020.
Kawin Ethayarajh. How contextual contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 55-65, 2019.
Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint N00iv:191.052222 2019.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2733- 2743, 2019.
John Hewitt and Christopher Manning. structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4129-4133 2019.

---

Under review as a conference paper at ICLR 2021
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926, 2018.
Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3543-3556, 2019.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3651-3657, 2019.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 1909. 10351, 2019.
Jaap Jumelet, Willem Zuidema, and Dieuwke Hupkes. Analysing neural language models: Contextual decomposition reveals default reasoning in number and gender assignment. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL),
pp. 1-11, 2019.
Josef Klafka and Allyson Ettinger. Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words. arXiv preprint arXiv:2005.01810, 2020.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 4365-44 4374,, 2019.
Yair Lakretz, Germán Kruszewski, Théo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and syntax units in lstm language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pp. 11-20, 2019.
Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. influence-directed explanations for deep convolutional networks. In 2018 IEEE International Test Conference (ITC), pp. 1-8. IEEE, 2018.
Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside bert's linguistic knowledge, In Proceedings of the 2019 ACL Workshop BlackboxNLP: Anadlyin and Interpreting Neural Networks for NLP, pp. 241-253, 2019.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 2016.
Nelson Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. Linguistic knowledge and transferability of contextual representations.
In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume / (Long and Short Papers), pp. 1073-1094, 2019.
Kaiji Lu, Piotr Mardziel, Klas Leino, Matt Fredrikson, and Anupam Datta. Influence paths for characterizing subject-verb number agreement in LSTM language models. Association for Computational Linguistics, 2020.
Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1202-1202, 2018.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, pp. 14014-14024, 2019.
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When bert plays the lottery, all tickets are winning. arXiv preprint arXiv:2005.00661, 2020.
10

---

Under review as a conference paper at ICLR 2021
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bei Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8594-8603. 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Sofia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2931-2951 2019.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenherg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: (70600/70003825, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Vons Volume 70, pp. 3319- 3328. JMLR. org, 2017.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nip pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp- 4593 4601, 2019a.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, 2019b.
Iulia Turc, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better:
On the importance of pre-training compact models. arXiv preprint 1908 0896622, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 63-76, 2019.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, 2019.

---

Under review as a conference paper at ICLR 2021
A
APPENDIX: INTRODUCTION TO LINGUISTIC TASKS
BERT Small
BERT Base
Task
Type
Example
SVA
SS
Object
SP
the author that the guard likes [MASK(is/are) young
0.92 0.9 1
0.96
Relative
PS
0.98
Clause
PP
Subject
SP
the author that likes the guard [MASK(is/are)] young
1
0.96
Relative
1
0.98
Clause
PP
SS
Within
the mechanic said the author [MASK(is/are)) young
Sentence
PS
Complement
SS
0.99
Across
SP
the author next to the guard [MASK(is/are)] young
1
0.98
Prepositional
PS
0.98
0.98
Phrase
PP
1
Reflexive
Anaphora
SS
0.66
0.6
Number
SP
the author that the guard likes hurt (MASK(hinnll//heensllee))
0.66
0.74
Agreement
PS
0.83
0.83
PP
1
0.96
MM
0.78
Gender
MF
some wizard who can dress our man can
0.32
0.96
Agreement
FF
clean (MASK(himsel(heersttt]]
1
0.9
FM
0.8
0.66
Table Example of each agreement task and their performance on two BERT models, first tasks are sampled from Marvin & Linzen (2018), the last task is sampled from dataset in Lin et al. (019). all datasets are constructed as an MLM task according to Goldberg (2019).
12

---

Under review as a conference paper at ICLR 2021
B APPENDIX: EXPERIMENT DETAILS
B.1 CONVERGENCE CHECK
When the Dol of distributional inftuence (x(x) is uniform distribution on linear path from baseline input X6 to the target input x, the completeness (Sundararajan et al. 2017) axiom shows {(f(x)) - (x(())))
Σj (x),( where q is the selected Quantity of Interest. However, when summation is used to approximate the expectation in practice, the RHS of the completeness axiom does not always converges to LHS easily.
Fig. 3 we plot the percentage of difference (xxb)) against the resolution, the number
E, (x) of samples drawn from the distribution in the summation. Due to the limit of our GPU memory ( (22B)) and the computational cost, find the maximum number of batched samples to be 50. Therefore, BERTsMALL has lower approximation error compared to BERT BASE- The much harder approximation of the larger BERT model is likely due the complicated decision boundaries of larger BERT, masking the output sensitive to small perturbations.
[g(f(x)) - (xx))
BERTISAAL))
Percentage of Differene: Infuence to Qol
Ma
300
100
100
2000
Resalition
Figure 3: Convergence Analysis for Calculating the Distributional influence(IG) for SVA Across Object Relative Clauses for BERT (used in this paper) BERT from Devlin al. (2019) perturbations. X-axis is the number of samples used to approximate the influence, Y-axis is the percentage of deviation from approximation true influence value.
B.2 AGGREGATED INFLUENCE GRAPHI FOR ALL TASKS
In this section, show the aggregated visualization across all examples by superimposing the visualization of individual instances as the ones in Figure 2] while adjusting the line width to be proportional the frequency of flow across all examples. The words within parenthesis represent one instance of the word in that position.
13

---

Under review as a conference paper at ICLR 2021
[CLS]
po6,
(CLS]
pos, on
nag sep
(author
(author
that
that
the
the
(guards)
(guard)
(lke
(ikes]
[MASK]]
[MASK]
(young)
(young)
[SEP
(a) singular subject plural intervening noun(SP)
(b) singular subject singular intervening noun(SS)
[CLS]]
[CLS]
(authors)
(authors)
that
that
the
the
-
(uard)
[guards]
(ikes]
[MASK]]
[MASK]]
(youg))
[young)
[SEP
[SEP]
(c) plural subject + singular intervening noun(PS)
(d) plural subject + plural intervening noun(PP)
Figure 4: SVA Across Object Relative Clause.
14

---

Under review as a conference paper at ICLR 2021
[CLS]
poc,
(CLS]
pos, om
nng amn
(author
(author
that
that
(Ikas)
(Ikes)
100
[quards]
(guard)
[MASK]]
[MASK]
(young)
(young)
-0 -00
0
-
[SEP
[SEP]
(a) singular subject plural intervening noun(SP)
(b) singular subject singular intervening noun(SS)
[CLS]]
[CLS]]
(authorss
(authors)
that
that
[1ke
((ke)
(ward)
(quards)
[MASK]]
[MAS]]
(yong
(young)
[SEP
[SEP]
(c) plural subject + singular intervening noun(PS)
(d) plural subject + plural intervening noun(PP)
Figure SVA Across Subject Relative Clause.
15

---

Under review as a conference paper at ICLR 2021
[CLS]
p06,
(CLS]
pos, on
the
ngg dep
the
(author)
(author)
(sak)
(sakd)
the
[quards]
(guard)
[MASK]]
[MASK]
(young)
(young)
[SEP
(a) singular subject plural intervening noun(SP)
(b) singular subject singular intervening noun(SS)
[CLS]]
[CLS]
the
the
(authors)
(authors)
(said)
(said)
the
(quadd
(puards)
[MASK]]
[MASK]
(young)
(young)
[SEP
[SEP]
(c) plural subject + singular intervening noun(PS)
(d) plural subject + plural intervening noun(PP)
Figure 6: SVA Within Sentence Complements.
16

---

Under review as a conference paper at ICLR 2021
[CLS]
p06,
[CLS]
pos, an
nng dep
(author
(nex)
(nXx))
(00
(10)
(quards]
(guard)
[MASK]]
[MASK]]
(young)
-
(young)
-
[SEP]
(a) singular subject plural intervening noun(SP)
(b) singular subject singular intervening noun(SS)
[CLS]]
[CLS]
(authorss
(authors)
(nex))
(nexo]
000
[00
(gard)
(quards)
[MASK]]
[MASK]]
(young)
[young]
[SEP
[SEP]
(c) plural subject + singular intervening noun(PS)
(d) plural subject + plural intervening noun(PP)
Figure SVA Across Prepositional Phrase.
17

---

Under review as a conference paper at ICLR 2021
[CLS]
p00,
[CLS]
po6, om
nng sep
(authon
[author
that
that
the
the
2
(guards)
(guard)
(lke
(ikes]
(hur)
(hur)
[MASK]]
[MASK]
[SEP]
[SEP]
(a) singular subject plural intervening noun(SP)
(b) singular subject singular intervening noun(SS)
[CLS]]
[CLS]
(authors)
(authors)
that
1hat
the
the 0
(uard
(guards)
(ikes]
(hur 0
(hur)
[MASK]]
[ASK]
[SEP|]
=0
[SEP]
(c) plural subject + singular intervening noun(PS)
(d) plural subject + plural intervening noun(PP)
Figure 8: RA: Number Agreement
18

---

Under review as a conference paper at ICLR 2021
[CLS]]
500,
[CLS]
po6, am
(some)
nng dep
(some)
(kng)
(wzard)
(who)
(who)
[can]
(can)
(dress
(dress)
[our)
(our)
=a
[men]
(man)
[can)
[can]
20
(clean)
(cleen)
10
[MASK]
[MASK]
[SEP
[SEP]
(a) male subject male intervening noun(MM)
(b) female subject male intervening noun(FM)
[CLS]]
[CLS]]
(some)
(some)
(22rd)
(King)
-
(Who
50
r0
(who)
-o
(can)
(dress)
(dress)
[our]
(our]
(woman)
1
(man)
(can)
(can)
+0
(clan)
20 20
(clean)
20
00
90
[MASK]
0
[MAS]]]
[SEP
[SEP]
(c) female subject female intervening noun(FF)
(d) male subject + female intervening noun(MF)
Figure 9: RA: Gender Agreement
19


Reviews:
[{"review_id": "SbOZnrFMN5X", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\n\nThis paper investigates how BERT uses attention to contextualise information. To find an answer to this question, the authors use abstractions of sets of paths through a neural network model, to quantify and localise the effect of specific inputs to the output. They describe these paths -- multi-partite patterns, as well as a guided pattern refinement that can be used to discover influential paths. To showcase their proposed techniques, the authors consider subject-verb agreement and reflexive anaphora in BERT.\n\n*Review*\n\nThe proposed method is exciting: investigating the flow of information information in neural models is a promising research direction that is receiving more and more attention. The authors spend sufficient time explaining several aspects of their method, although I find it at times a bit confusing to understand why some information goes in the background section and some instead in the section \"path abstraction\". \n\nIn terms of the result section, I think the paper could be improved. It is not trivial to parse the numbers in Table 1 and 2, and not much explanation is given. The paper furthermore only reports results on one of the tasks given in the abstract, all the rest of the results are in the appendix. I am missing also some explicit comparisons with previous studies with similar aims (see appendix). In general, I find that a bit too much of information that is relevant to understand the story is put in the appendix.\n\n*Questions for the authors:*\n- Much work has been done on investigating how LSTM language models model subject verb agreement and anaphora resolution, and in the literature also  several comparisons between BERT and LSTM models can be found. To validate your technique, perhaps it could be interesting to run it also for LSTM models, for which it is easier to establish if the results make sense? _**Update after author response:** I understand that space might not permit running this also for an LSTM model, and that there are substantial differences between LSTM models and BERT, in terms of depth. Still, I would appreciate to have some baseline results, which would help to better assess the contribution. In my opion, LSTM models, for which much more is already known about the paths through the network, would be the easiest way to go._\n- I think your work is very interesting, but I am really struggling to understand the results. Could you elaborate a bit more, and provide a bit more structured explanation and interpretation? _**Update after author response**: I appreciate that you took into consideration my request to add a bit more explanation. I still find the structure a bit difficult to understand, but it has definitely improved since the last version_\n\n*A few notes and suggestions*\n_**Update after author response:** Thanks for incorporating most of my suggestions_\n- Figure 1 comes a bit early for me, at the point where it is referenced I cannot really parse it yet, because much of the info to understand follows only later \n- The font size of the legend in Figure 1 is very small, could you increase it?\n- You generate your own dataset with NA-tasks with fixed patterns, this was previously also done by Lakretz et al 2018 -- The emergence of number and syntax units in LSTM language models -- a dataset which was used by several others afterwards. For comparability with others that have investigated this phenomenon in BERT and other models, it might be nice to run some experiments using that dataset.\n- A very relevant reference is Jumelet et al (2020) -- Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment -- who use contextual decomposition to find how information flows in a recurrent LM, looking also at SVA and RA. In a more recent paper Jumelet (2020) repeated this analysis also for BERT.  It would be nice to see some comparisons, and also some contextualisation on how the method here relates to their work, and to contextual decomposition (Murdoch et al 2018 -- Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs) in general.\n- In related work, I am missing Hupkes et al 2018 (Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure) as a reference for diagnostic classification\n- bibliography should be checked, there are quite a few arxiv references that are actually published papers\n\n_**Update after author response**: I think the paper has improved substantially. The direction is exciting, but even with the updates I believe this paper needs quite some work to improve the presentation and clarity, which is why I have not updated my score._", "extra_scores": {}, "extra_text": {}}, {"review_id": "RKei0M_bnpk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes functioning of BERT by identifying gradient-based influence paths to track flow of influence between model inputs and outputs. Such analysis using influence paths has been established in prior work, but the present paper expands on this work with definition of \"multi-partite\" patterns, and by introducing a method for identifying strongly influential paths in the model. The authors evaluate on existing datasets for studying subject-verb agreement and reflexive anaphora, and they report measures of concentration of the influence flows, and performance of the model after compression based on the identified influence paths.\n\nOverall, I think there are certainly interesting analyses that could come out of this work, but the current paper does not provide a clear enough contribution to be ready for publication. \n\nThe reported results don't give us much to go on in terms of interpreting what we have learned from these analyses. The authors choose two measures: \"concentration\" of the influence paths, and drop in accuracy when compressing the model based on those paths. I'm not sure what we should be taking away from the concentration numbers, and the authors don't provide any substantive discussion of why this matters. There is also no meaningful baseline against which to compare these numbers. The second measure, accuracy of the compressed model, is a bit easier to interpret, in that it allows us to verify the extent to which critical information flow is indeed occurring within the identified paths, but while this verification serves to increase confidence in the method, it's not clear what it tells us about the functioning of the BERT model. (The authors also do not give a clear statement of what task these accuracies are in fact for, though I assume by default that they are referring to accuracy in assigning higher probability to the grammatical option over the ungrammatical option in the selected Marvin & Linzen datasets.) There is also no discussion of how we should interpret results from the attention-based versus the embedding-based influence paths.\n\nZooming out further, there isn't a clearly identifiable question being asked in this paper, and in particular there is no clear connection between the analysis method being used and the particular linguistic phenomena embodied by the chosen evaluation data. What question is being asked about the model's handling of these linguistic phenomena, and why is this method appropriate for asking it? What output of the analysis will be used as an answer to the key questions? These connections should be defined clearly and from the start. As it is, I'm unclear on what these results tell us about the chosen linguistic phenomena (or, conversely, why this set of phenomena was chosen to showcase utility of this method). The measures reported have some variation between sentence types -- but what does it mean that there is higher positive embedding influence concentration for sentences with subject relative clauses, or higher negative embedding influence concentration for within sentence complement sentences? What is the meaning of the accuracies dropping more for subject relative clauses and number agreement among the embedding influence paths, and all accuracies seemingly dropping for the attention influence paths?  \n\nIn Section 4.2 there are a couple of observations about the influence paths that make some concrete connections to potential questions about syntactic dependencies in the chosen datasets. However, because no formal connection between these things has been established prior to this point (no clear question, no clear linking hypothesis between the analysis and the phenomena) this set of observations comes a bit out of the blue, and leaves us still without clear takeaways. In sum, I think that there is potential for interesting insights about these phenomena to come out of this analysis method, but the paper would benefit from much more clearly defined questions and linking assumptions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AQ571UKrj_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\n\nThis paper proposes a new explanation method of DNNs by applying Distributional Influence analysis on the proposed Multi-partite patterns. A Multi-partite pattern $\\pi$ is basically a partial path; it \"abstracts\" a set of paths $\\gamma(\\pi)$ which consists of the paths that include all edges in the pattern. The influence of $\\pi$ is thus defined as the sum of path influences among $\\gamma(\\pi)$. The authors also propose Guided Pattern Refinement (GPR) algorithm to find the most influential patterns for a certain input word. It is a greedy algorithm that adds an edge per step to maximize the current pattern influence.\n\nAlthough this method is in principle applicable to any DNNs, this paper is mostly focused on BERT. Experiments are done to evaluate and showcase the usefulness of the proposed method. The experiment results on Concentration (of influence) and Model Compression illustrated that the extracted patterns from BERT indeed capture our quantity of interest (QoI). The authors also utilize the extracted patterns to shed more light on BERT behaviors.\n\n\n#### Strength\n\n- This paper is targeted at BERT explanation, which is very important and valuable. Model explanation is gaining more and more attention, especially for BERT, given its dominant popularity and yet hardness in interpretations.\n- The proposed explanation method is reasonable and it proved useful in the experiments. The Concentration and Compression results are promising (except that comparisons to other methods are not provided, which is a weakness). Also, the behaviors discovered and discussed are also interesting.\n\n\n#### Weakness\n\n- It looks like the proposed method is largely based on (Lu et al, 2020); the major difference is the introduction of Multi-partite patterns, which basically expands the objects of influence analysis from paths to patterns (partial paths). This doesn't look like a significant novelty.\n- Several parts of the paper are very confusing to me, especially the background section (maybe because my primary research area is not model interpretation). I tried to read several references but still have many questions unsolved. I will put specific questions in the next section.\n\n\n#### Questions & Suggestions\n- Confusions\n\t- Section 2, Distribution Influence (paragraph): \"Influence can quantify a concept ...\" might be misleading. Influence is not quantifying a concept but quantifying the impact of an input concept (subject S/P) on an output concept (verb S/P), as stated in Definition 1.\n\t- Section 2, Definition 1: \"... a DoI D(x) parameterized by an input, ...\" Here \"parameterized by an input\" confused me a lot when I first read it. After checking references and the experiments, I guess you are referring to a distribution like the uniform distribution from original embedding to [MASK] embedding, where the original embedding is the \"parameter\". However, this sense is somewhat unusual... I always thought about \"parameterize\" like $\\mu,\\theta$ in Gaussian distributions and thus cannot understand. Also, I think the word \"input\" here is ambiguous: you have an \"analysis input\", which is $x$; you also have \"model input\", which are samples from $D(x)$. It might be better to explicitly distinguish them.\n\t- Section 2, Line 22-23: \"... in our use cases DoI are distributed along a one dimensional path $c:[0,1]\\to R^d$.\" This does not look valid mathematically; how can a distribution be over a function? I guess what you mean is that DoI are distributed over $c=\\\\{x_m+\\alpha(x-x_m),\\alpha\\in[0,1]\\\\}$.\n\t- Section 3, Guided Pattern Refinement(GPR): \"One could repeat the procedure until an individual path is produced though for reasons of interpretability, it might be preferable to reason about patterns instead of single paths.\" How are you actually extracting patterns? As I see in Figure 1 and 3, you are illustrating paths. Do you mean \"repeat the procedure until an individual path, but (mentally) reason about patterns in the path\"? It looks like so according to your experiments discussed in Table 1.\n\n- Others\n\t- Section 4: (a minor one) It is kind of inconvenient for readers that you frequently refer to Figure 1 in your discussion, which is many pages away. Would it be better if you move or copy Figure 1 and put it together with Figure 3?\n\n\n#### Typos\n- Figure 1, Line 1: \"GRP\" -> \"GPR\"\n- Page 3, Line 2: extra space\n- Page 4, Def 4: in the middle term, $x$ -> $z$\n- Page 4, Def 5, Line 4: $n_w,n_w$ -> $n_w,n_q$\n- Page 7, Line 2: \"retain a even\", \"a\" -> \"an\"\n- Page 7, Line -9: \"In ,\" -> (In all?)\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "91JPqzxxIc1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\n\nThis paper addresses the problem of how BERT contextualizes information. Specifically, this paper tackles a computational limitation of influence paths used to analyze LSTM models by introducing multi-partite patterns and guided pattern refinement (GPR). A multi-partite pattern, $\\pi$, defines a set of paths that follow a sequence of edges. The more edges it has in the sequence, the fewer paths it defines. Then, GPR is a greedy algorithm that iteratively adds an edge to $\\pi$ such that the influence of the pattern is maximized. For each iteration, the search is constrained to a set of edges within a single layer of BERT. As a result, an influence path is constructed (or rather approximated). The author validates the proposed method using concentration and model compression experiments for multiple agreement tasks. The concentration results show that influence paths discover by GPR have a very high influence. This is true for both embedding-level paths and attention-level paths. For model compression, the results show that most of the accuracy is maintained when retaining the identified influence paths and \"churning\" the others. Finally, the author explains how BERT contextualizes input words by visualizing the influence paths. The author discusses the four examples of the SVA across Obj task. The influence of the subject and the intervening noun is similar to that of LSTM (Lu et al., 2020). Insights from the visualization of the influence paths are discussed such as \"copy and transfer\" and a conflicting role of \"that\" (which might explain the common mistake of BERT).\n\n\n### Recommendation:\nOverall, I incline toward rejecting. This paper provides an instrument to explain BERT, but I have a hard time understanding the result itself (influence paths or patterns). I also have a major concern with the final analysis and its findings. \n\n#### Pros:\n- Novel method to compute influence paths that can help us interpret how BERT utilizes input words.\n- The quantitative experiments conducted on many agreement tasks validate the proposed methods.\n- Interesting findings that BERT likes to \"copy\".\n\n#### Cons:\n- Interpreting the influence paths to understand how BERT works is quite difficult and arguably subjective. It is also confusing why Figure 1 is so different from Figure 3. \n- Although the result influence paths include the skip connections, it is not quite clear how it is compared to the attention approach in the other aspects. For example, the author could use the model compression experiment on the high attention weight embedding (or hidden representation).\n- It is unclear how the main finding of section 4.2 came to be. For example, how many sentences that the author observed before concluding that \"copy and transfer\" are common. The speculation of \"that\" influence paths can somewhat explain the result discrepancies of SS and PS. Are there other words that behave similarly (such as \"who\")? I think the author should provide a more rigorous analysis on the interpretability of the proposed method, as well as an explanation of BERT contextualization.\n- It is quite common to study how the model works through this kind of task. But, most of the time, we BERT of encoding sentences without [MASK] word. I am not sure how this finding will generalize to other tasks.\n- This paper is very hard to read. The math expressions are clear, but the explanations and intuitions are often confusing.\n\n\n### Questions:\n\n1. I am not quite sure that \"churning\" means. Could you please provide more detail?\n1. What are the numbers on the \"Significant patterns\" figures?\n1. I am a bit confused with Figures 1 and 3. In Figure 1, there is a path from 1 word to [MASK], but in Figure 3 (and others), there are multiple paths. Could you explain why is there a difference?\n1. Would you have the same result using Lu et al., average embedding instead of [MASK] embedding as a base embedding?\n\n### Minor comments:\n- I think Section 4.2 onward is a bit rushed. There are many typos, such as \"... connections.  . We speculate ...\" or \"we how BERT\".\n- Definition 5, typo on \"$n_w,n_w$\"?\n- So much useful figures and tables are in Appendices.\n- I am not sure the point of the convergence analysis of the BERT base vs small. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper propose a method to explain the contextualization of BERT by identifying a set of influence paths from the input to the output.  Although all reviewers give overall score 6, their comments are pointing to the negative direction.  The following excerpts summarize the general sentiment of the reviews:
R2: Overall, I incline toward rejecting. This paper provides an instrument to explain BERT, but I have a hard time understanding the result itself (influence paths or patterns). I also have a major concern with the final analysis and its findings.  
R4: I think the paper has improved substantially. The direction is exciting, but even with the updates I believe this paper needs quite some work to improve the presentation and clarity, which is why I have not updated my score.
R1: Overall, I think there are certainly interesting analyses that could come out of this work, but the current paper does not provide a clear enough contribution to be ready for publication.
R3: It looks like the proposed method is largely based on (Lu et al, 2020); the major difference is the introduction of Multi-partite patterns, which basically expands the objects of influence analysis from paths to patterns (partial paths). This doesn't look like a significant novelty.

Decision: Reject