SlowGuess/ACL_2020_OCR huggingface dataset:
Kaiji Lu
Carnegie Mellon University kaijil@andrew.cmu.edu
Piotr Mardziel
Carnegie Mellon University  
piotrm@gmail.com
Carnegie Mellon University  
kleino@cs.cmu.edu
Matt Fedrikson
mfredrik@cmu.edu
LSTM-based recurrent neural networks are the state-of-the-art for many natural language processing (NLP) tasks. Despite their performance, it is unclear whether, or how, LSTMs learn structural features of natural languages such as subject-verb number agreement in English. Lacking this understanding, the generality of LSTMs on this task and their suitability for related tasks remains uncertain. Further, errors cannot be properly attributed to a lack of structural capability, training data omissions, or other exceptional faults. We introduce influence paths, a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network. The approach refines the notion of influence (the subject's grammatical number has influence on the grammatical number of the subsequent verb) into a set of gate-level or neuron-level paths. The set localizes and segments the concept (e.g., subject-verb agreement), its constituent elements (e.g., the subject), and related or interfering elements (e.g., attractors). We exemplify the methodology on a widely-studied multi-layer LSTM language model, demonstrating its accounting for subject-verb number agreement. The results offer both a finer and a more complete view of an LSTM's handling of this structural aspect of the English language than prior results based on diagnostic classifiers and ablation.
Traditional rule-based NLP techniques can capture syntactic structures, while statistical NLP techniques, such as n-gram models, can heuristically integrate semantics of a natural language. Modern RNN-based models such as Long Short-Term Memory (LSTM) models are tasked with incorporating both semantic features from the statistical associations in their training corpus, and structural features generalized from the same.
Anupam Datta
danupam@cmu.edu
Figure 1: Subject-verb agreement task for a 2-layer LSTM language model, and primary paths across various LSTM gates implementing subject-verb number agreement. A language model assigns score  $s$  to each word. Agreement is the score of the correctly numbered verb minus that of the incorrectly numbered verb.
Despite evidence that LSTMs can capture syntactic rules in artificial languages (Gers and Schmidhuber, 2001), it is unclear whether they are as capable in natural languages (Linzen et al., 2016; Lakretz et al., 2019) in the context of rules such as subject-verb number agreement, especially when not supervised for the particular feature. The incongruence derives from this central question: does an LSTM language model's apparent performance in subject-verb number agreement derive from statistical heuristics (like n-gram models) or from generalized knowledge (like rule-based models)?
Recent work has begun addressing this question (Linzen et al., 2016) in the context of language models: models tasked with modeling the likelihood of the next word following a sequence of words as expected in a natural language (see Figure 1, bottom). Subject-verb number agreement dictates that the verb associated with a given subject
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4748-4757 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
should match its number (e.g., in Figure 1, the verb "run" should match with the subject "boys"). Giulianelli et al. (2018) showed that the subject grammatical number is associated with various gates in an LSTM, and Lakretz et al. (2019) showed that ablation (disabling activation) of an LSTM model at certain locations can reduce its accuracy at scoring verbs of the correct grammatical number.
Influence offers an alternate means of exploring properties like number agreement. We say an input is influential on an outcome when changing just the input and nothing else induces a change on the outcome. In English grammar, the number of a subject is influential on the number of its verb, in that changing the number of that subject while keeping all other elements of a sentence fixed would necessitate a change in the number of the verb. Algorithmic transparency literature offers formal definitions for empirically quantifying notions of influence for systems in general (Datta et al., 2016) and for deep neural networks specifically (Leino et al., 2018; Sundararajan et al., 2017).
The mere fact that subject number is influential on verb number as output by an LSTM model is sufficient to conclude that it incorporates the agreement concept in some way but does not indicate whether it operates as a statistical heuristic or as a generalized rule. We address this question with influence paths, which decompose influence into a set of paths across the gates and neurons of an LSTM model. The approach has several elements:
1. Define an input parameter to vary the concept-specific quantity under study (e.g., the grammatical number of a particular noun, bottom-left node in Figure 1) and a concept-specific output feature to measure the parameter's effect on (e.g, number agreement with the parameterized noun, bottom-right node in Figure 1).
2. Apply a gradient-based influence method to quantify the influence of the concept parameter on the concept output feature; as per the chain rule, decompose the influence into model-path-specific quantities.
3. Inspect and characterize the distribution of influence across the model paths.
The paths demonstrate where relevant state information necessitated by the concept is kept, how it gets there, how it ends up being used to affect
the model's output, and how and where related concepts interfere.
Our approach is state-agnostic in that it does not require a priori an assumption about how or if the concept will be implemented by the LSTM. This differs from works on diagnostic classifiers where a representation of the concept is assumed to exist in the network's latent space. The approach is also time-aware in that paths travel through cells/gates/neurons at different stages of an RNN evaluation. This differs from previous ablation-based techniques, which localize the number by clearing neurons at some position in an RNN for all time steps.
- We introduce influence paths, a causal account of the use of concepts of interest as carried by paths across gates and neurons of an RNN.
- We demonstrate, using influence paths, that in a multi-layer LSTM language model, the concept of subject-verb number agreement is concentrated primarily on a single path (the red path in Figure 1), despite a variety of surrounding and intervening contexts.
- We show that attractors (intervening nouns of opposite number to the subject) do not diminish the contribution of the primary subject-verb path, but rather contribute their own influence of the opposite direction along the equivalent primary attractor-verb path (the blue path in the figure). This can lead to incorrect number prediction if an attractor's contribution overcomes the subject's.
- We corroborate and elaborate on existing results localizing subject number to the same two neurons which, in our results, lie on the primary path. We further extend and generalize prior compression/ablation results with a new path-focused compression test which verifies our localization conclusions.
Our results point to generalized knowledge as the answer to the central question. The number agreement concept is heavily centralized to the primary path despite the varieties of contexts. Further, the primary path's contribution is undiminished even amongst interfering contexts; number errors are not attributable to lack of the general number concept but rather to sufficiently influential contexts pushing the result in the opposite direction.
LSTMs Long short-term memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997) have proven to be effective for modeling sequences, such as language models, and empirically, this architecture has been found to be optimal compared to other second-order RNNs (Greff et al., 2017). LSTMs utilize several types of gates and internal states including forget gates  $(f)$ , input gates  $(i)$ , output gates  $(o)$ , cell states  $(c)$ , candidate cell state  $(\tilde{c})$ , and hidden states  $(h)$ . Each gate is designed to carry out a certain function, or to fix a certain drawback of the vanilla RNN architecture. E.g., the forget gate is supposed to determine how much information from the previous cell state to retain or "forget", helping to fix the vanishing gradient problem (Hochreiter, 1998).
Number Agreement in Language Models The number agreement (NA) task, as described by Linzen et al. (2016), is an evaluation of a language model's ability to properly match the verb's grammatical number with its subject. This evaluation is performed on sentences specifically designed for the exercise, with zero or more words between the subject and the main verb, termed the context. The task for sentences with non-empty contexts will be referred to as long-term number agreement.
"Human-level" performance for this task can be achieved with a 2-layer LSTM language model (Gulordava et al.), indicating that the language model incorporates grammatical number despite being trained only for the more general word prediction task. Attempts to explain or localize the number concept within the model include (Lakretz et al., 2019), where ablation of neurons is applied to locate specific neurons where such information is stored; and (Giulianelli et al., 2018; Hupkes et al., 2018), where diagnostic classifiers are trained on gate activations to predict the number of the subject to see which gates or timesteps the number concept exhibits itself. These works also look at the special cases involving attractors—intervening nouns with grammatical number opposite to that of the subject (deemed instead helpful nouns if their number agrees with the subject)—such as the word "tree" in Figure 1. Both frameworks provide explanations as to why attractors lower the performance of NA tasks. However, they tend to focus on the activation patterns of gates or neurons without justifying their casual relationships with the concept of gram-
matical number, and do not explicitly identify the exact temporal trajectory of how the number of the subject influences the number of the verb.
Other relevant studies that look inside RNN models to locate specific linguistic concepts include visualization techniques such as (Karpathy et al., 2015), and explanations for supervised tasks involving LSTMs such as sentiment analysis (Murdoch et al., 2018).
Attribution Methods Attribution methods quantitatively measure the contribution of each of a function's individual inputs to its output. Gradient-based attribution methods compute the gradient of a model with respect to its inputs to describe how important each input is towards the output predictions. These methods have been applied to assist in explaining deep neural networks, predominantly in the image domain (Leino et al., 2018; Sundararajan et al., 2017; Bach et al., 2015; Simonyan et al., 2013). Some such methods are also axiomatically justified to provide a causal link between inputs (or intermediate neurons) and the output.
As a starting point in this work, we consider Integrated Gradients (IG) (Sundararajan et al., 2017). Given a baseline,  $x_0$ , the attribution for each input at point,  $x$ , is the path integral taken from the baseline to  $x$  of the gradients of the model's output with respect to its inputs. The baseline establishes a neutral point from which to make a counterfactual comparison; the attribution of a feature can be interpreted as the share of the model's output that is due to that feature deviating from its baseline value. By integrating the gradients along the linear interpolation from the baseline to  $x$ , IG ensures that the attribution given to each feature is sensitive to effects exhibited by the gradient at any point between the baseline and instance  $x$ .
Leino et al. (2018) generalize IG to better focus attribution on concepts other than just model outputs, by use of a quantity of interest (QoI) and a distribution of interest (DoI). Their measure, Distributional Influence, is given by Definition 1. The QoI is a function of the model's output expressing a particular output behavior of the model to calculate influence for; in IG, this is fixed as the model's output. The DoI specifies a distribution over which the influence should faithfully summarize the model's behavior; the influences are found by taking an expected value over DoI.
Definition 1 (Distributional Influence). With quantity of interest,  $q$ , and distribution of interest,  $D$
the influence,  $\chi$ , of the inputs on the quantity of interest is:
$$
\chi (q, D) = \underset {\vec {x} \sim D} {\mathbb {E}} \left[ \frac {\partial q}{\partial x} (\vec {x}) \right]
$$
The directed path integral used by IG can be implemented by setting the DoI to a uniform distribution over the line from the baseline to  $\vec{x}$ :  $D = \text{Uniform}(\vec{x}_0\vec{x})$ , for baseline,  $\vec{x}_0$ , and then multiplying  $\chi$  by  $\vec{x} - \vec{x}_0$ . Conceptually, by multiplying by  $\vec{x} - \vec{x}_0$ , we are measuring the attribution, i.e., the contribution to the QoI, of  $\vec{x} - \vec{x}_0$  by weighting its features by their influence. We use the framework of Leino et al. in this way to define our measure of attribution for NA tasks in Section 3.
Distributional Influence can be approximated by sampling according to the DoI. In particular, when using  $D = \mathrm{Uniform}\left(\overrightarrow{x_0\vec{x}}\right)$  as noted above, Definition 1 can be computationally approximated with a sum of  $n$  intervals as in IG:
$$
\chi \approx \sum_ {i = 1} ^ {n} \frac {\partial q}{\partial x} \left(\frac {i}{n} \vec {x} + \left(1 - \frac {i}{n}\right) \vec {x} _ {0}\right)
$$
Other related works include Fiacco et al. (2019), which employs the concept of neuron paths based on cofiring of neurons instead of influence, also on different NLP tasks from ours.
Our method for computing influence paths begins with modeling a relevant concept, such as grammatical number, in the influence framework of Leino et al. (Definition 1) by defining a quantity of interest that corresponds to the grammatical number of the verb, and defining a component of the input embedding that isolates the subject's grammatical number (Section 3.1). We then decompose the influence measure along the relevant structures of LSTM (gates or neurons) as per standard calculus identities to obtain a definition for influence paths (Section 3.2).
3.1 Measuring Number Agreement
For the NA task, we view the initial fragment containing the subject as the input, and the word distribution at the position of its corresponding verb as the output.
Formally, each instance in this task is a sequence of  $d$ -dimensional word embedding vectors,  $\mathbf{w} \stackrel{\mathrm{def}}{=} \langle \vec{w}_i \rangle_i$ , containing the subject and the corresponding verb, potentially with intervening words
in between. We assume the subject is at position  $t$  and the verb at position  $t + n$ . The output score of a word,  $w$ , at position  $i$  will be written  $s_i(w)$ . If  $w$  has a grammatical number, we write  $w^+$  and  $w^-$  to designate  $w$  with its original number and the equivalent word with the opposite number, respectively.
Quantity of Interest We instrument the output score with a QoI measuring the agreement of the output's grammatical number to that of the subject:
Definition 2 (Number Agreement Measure). Given a sentence,  $\mathbf{w}$ , with verb,  $w$ , whose correct form (w.r.t. grammatical number) is  $w^{+}$ , the quantity of interest,  $q$ , measures the correctness of the grammatical number of the verb:
$$
q \left(\mathbf {w}\right) \stackrel {\mathrm {d e f}} {=} s _ {t + n} \left(w ^ {+}\right) - s _ {t + n} \left(w ^ {-}\right)
$$
In plain English,  $q$  captures the weight that the model assigns to the correct form of  $w$  as opposed to the weight it places on the incorrect form. Note that the number agreement concept could have reasonably been measured using a different quantity of interest. E.g., considering the scores of all vocabulary words of the correct number and incorrect number in the positive and negative terms, respectively, is an another alternative. However, based on our preliminary experiments, we found this alternative does not result in meaningful changes to the reported results in the further sections.
Distribution of Interest We also define a component of the embedding of the subject that captures its grammatical number, and a distribution over the inputs that allows us to sensitively measure the influence of this concept on our chosen quantity of interest. Let  $\vec{w}^0$  be the word embedding midway between its numbered variants, i.e.,  $\frac{\vec{w}^{+} + \vec{w}^{-}}{2}$ . Though this vector will typically not correspond to any English word, we interpret it as a number-neutral version of  $\vec{w}$ . Various works show that linear arithmetic on word embeddings of this sort preserves meaningful word semantics as demonstrated in analogy parallelograms (Mikolov et al., 2013). Finally, given a sentence,  $\mathbf{w}$ , let  $\mathbf{w}_t^0$  be the sentence  $\mathbf{w}$ , except with the word embedding  $\vec{w}_t$  replaced with its neutral form  $\vec{w}_t^0$ . We see that  $\mathbf{w} - \mathbf{w}_t^0$  captures the part of the input corresponding to the grammatical number of the subject,  $\vec{w}_t$ .
Definition 3 (Grammatical Number Distribution). Given a singular (or plural) noun,  $w_t$ , in a sentence,  $\mathbf{w}$ , the distribution density of sentences,  $D_{\mathbf{w}}$ , exercising the noun's singularity (or plurality) linearly
interpolates between the neutral sentence,  $\mathbf{w}_t^0$ , and the given sentence,  $\mathbf{w}$ :
$$
D _ {\mathbf {w}} \stackrel {{\mathrm {d e f}}} {{=}} U n i f o r m \left(\overline {{\mathbf {w} _ {t} ^ {0} \mathbf {w}}}\right)
$$
If  $\vec{w}_t$  is singular, our counterfactual sentences span  $\mathbf{w}$  with number-neutral  $\vec{w}_t^0$  all the way to its singular form  $\vec{w}_t = \vec{w}_t^+$ . We thus call this distribution a singularity distribution. Were  $w_t$  plural instead, we would refer to the distribution as a plurality distribution. Using this distribution of sentences as our DoI thus allows us to measure the influence of  $\mathbf{w} - \mathbf{w}_t^0$  (the grammatical number of a noun at position  $t$ ) on our quantity of interest sensitively (in the sense that Sundararajan et al. define their axiom of sensitivity for IG (Sundararajan et al., 2017)).
Subject-Verb Number Agreement Putting things together, we define our attribution measure.
Definition 4 (Subject-Verb Number Agreement Attribution). The measure of attribution,  $\alpha$ , of a noun's grammatical number on the subject-verb number agreement is defined in terms of the DoI,  $D_{\mathbf{w}}$ , and  $QoI$ ,  $q$ , as in Definitions 3 and 2, respectively.
$$
\alpha \left(\mathbf {w}\right) = \left(\mathbf {w} - \mathbf {w} _ {t} ^ {0}\right) \chi (q, D _ {\mathbf {w}})
$$
Essentially, the attribution measure weights the features of the subject's grammatical number by their Distributional Influence,  $\chi$ . Because  $D_{\mathbf{w}}$  is a uniform distribution over the line segment between  $\mathbf{w}$  and  $\mathbf{w}_t^0$ , as with IG, the attribution can be interpreted as each feature's net contribution to the change in the QoI,  $q(\mathbf{w}) - q(\mathbf{w}_t^0)$ , as  $\sum_{i} \chi(\mathbf{w})_i = q(\mathbf{w}) - q(\mathbf{w}_t^0)$  (i.e., Definition 4 satisfies the axiom Sundararajan et al. term completeness (Sundararajan et al., 2017)).
In Figure 1, for instance, this definition measures the attribution from the plurality of the subject ("boys"), towards the model's prediction of the correctly numbered verb ("run") versus the incorrectly numbered verb ("runs"). Later in this paper we will also investigate the attribution of intervening nouns on this same quantity. We expect the input attribution to be positive for all subjects and helpful nouns, and negative for attractors, which can be verified by the  $P^{+}$ columns of Table 1 (the details of this experiment are introduced in Section 4).
3.2 Influence Paths
Input attribution as defined by IG (Sundararajan et al., 2017) provides a way of explaining a model by highlighting the input dimensions with large attribution towards the output. Distributional Influence (Leino et al., 2018) with a carefully chosen QoI and DoI (Definition 4) further focuses the influence on a concept at hand, grammatical number agreement. Neither, however, demonstrate how these measures are conveyed by the inner workings of a model. In this section we define a decomposition of the influence into paths of a model, thereby assigning attribution not just to inputs, but also to the internal structures of a given model.
We first define arbitrary deep learning models as computational graphs, as in Definition 5. We then use this graph abstraction to define a notion of influence for a path through the graph. We posit that any natural path decomposition should satisfy the following conservation property: the sum of the influence of each path from the input to the output should equal the influence of the input on the QoI. We then observe that the chain rule from calculus offers one such natural decomposition, yielding Definition 6.
Definition 5 (Model). A model is an acyclic graph with a set of nodes, edges, and activation functions associated with each node. The output of a node,  $n$ , on input  $x$  is  $n(x) \stackrel{\mathrm{def}}{=} f_n(n_1(x), \dots, n_m(x))$  where  $n_1, \dots, n_m$  are  $n$ 's predecessors and  $f_n$  is its activation function. If  $n$  does not have predecessors (it is an input), its activation is  $f_n(x)$ . We assume that the domains and ranges of all activation functions are real vectors of arbitrary dimension.
We will write  $n_1 \to n_2$  to denote an edge (i.e.,  $n_1$  is a direct predecessor of  $n_2$ ), and  $n_1 \to {}^* n_2$  to denote the set of all paths from  $n_1$  to  $n_2$ . The partial derivative of the activation of  $n_2$  with respect to the activation of  $n_1$  will be written  $\frac{\partial n_2}{\partial n_1}$ .
This view of a computation model is an extension of network decompositions from attribution methods using the natural concept of "layers" or "slices" (Dhamdhere et al., 2018; Leino et al., 2018; Bach et al., 2015). This decomposition can be tailored to the level of granularity we wish to expose. Moreover, in RNN models where no single and consistent "natural layer" can be found due to the variable-length inputs, a more general graph view provides the necessary versatility.
Definition 6 (Path Influence). Expanding Definition 4 using the chain rule, the influence of input
node,  $s$ , on target node,  $t$ , in a model,  $G$ , is:
$$
\begin{array}{l} \chi_ {s} = \underset {x \sim D (x)} {\mathbb {E}} \left[ \frac {\partial t}{\partial s} (x) \right] \\ = \mathbb {E} _ {x \sim D (x)} \left[ \sum_ {p \in (s \rightarrow^ {*} t)} \prod_ {(n _ {1} \rightarrow n _ {2}) \in p} \frac {\partial n _ {2}}{\partial n _ {1}} (x) \right] \\ = \sum_{p\in (s\to^{*}t)}\underbrace{\mathbb{E}_{x\sim D(x)}\left[\prod_{(n_{1}\to n_{2})\in p}\frac{\partial n_{2}}{\partial n_{1}}(x)\right]}_{\chi_{s}^{p}} \\ \end{array}
$$
Note that the same LSTM can be modeled with different graphs to achieve a desired level of abstraction. We will use two particular levels of granularity: a coarse gate-level abstraction where nodes are LSTM gates, and a fine neuron-level abstraction where nodes are the vector elements of those gates. Though the choice of abstraction granularity has no effect on the represented model semantics, it has implications on graph paths and the scale of their individual contributions in a model.