SlowGuess/ACL_2020_OCR huggingface dataset:
Gate-level and Neuron-level Paths We define the set of gate-level nodes to include:  $\{f_t^l, i_t^l, o_t^l, c_t^l, \tilde{c}_t^l, h_t^l : t < T, l < L\}$ , where  $T$  is the number of time steps (words) and  $L$  is number of LSTM layers. The node set also includes an attribution-specific input node  $(\mathbf{w} - \mathbf{w}_t^0)$  and an output node (the QoI). An example of this is illustrated in Figure 2. We exclude intermediate calculations (the solid nodes of Figure 2, such as  $f_t \odot c_{t-1}$ ) as their inclusion does not change the set of paths in a graph. We can also break down each vector node into scalar components and further decompose the gate-level model into a neuron-level one:  $\{f_{ti}^l, i_{ti}^l, o_{ti}^l, c_{ti}^l, \tilde{c}_{ti}^l, h_{ti}^l : t < T, i < H, l < L\}$ , where  $H$  is the size of each gate vector. This decomposition results in an exponentially large number of paths. However, since many functions between gates in an LSTM are element-wise operations, neuron-level connections between many neighboring gates are sparse.
Path Refinement While the neuron-level path decomposition can theoretically be performed on the whole network, in practice we choose to specify a gate-level path first, then further decompose that path into neuron-level paths. We also collapse selected vector nodes, allowing us to further localize a concept on a neuron level while avoiding an explosion in the number of paths. The effect of this pipeline will be empirically justified in Section 4.
Figure 2: Influence path diagram in a NA task for the 2-layer LSTM model. The red path shows the path with the greatest attribution (the primary path) from the subject; The blue path shows the primary path from the intervening noun.
In this section we apply influence path decomposition to the NA task. We investigate major gate-level paths and their influence concentrations in Section 4.2. We further show the relations between these paths and the paths carrying grammatical number from intervening nouns (i.e. attractors & helpful nouns) in Section 4.3. In both we also investigate high-attribute neuron along primary paths allowing us to compare our results to prior work.
4.1 Dataset and Model
We study the exact combination of language model and NA datasets used in the closely related prior work of Lakretz et al. (2019). The pre-trained language model of Gulordava et al. and Lakretz et al. is a 2-layer LSTM trained from Wikipedia articles. The number agreement datasets of Lakretz et al. are several synthetically generated datasets varying in syntactic structures and in the number of nouns between the subject and verb.
For example, nounPP refers to sentences containing a noun subject followed by a prepositional phrase such as in Figure 1. Each NA task has subject number (and intervening noun number if present) realizations along singular (S) and plural (P) forms. In listings we denote subject number (S or P) first and additional noun (if any) number second. Details including the accuracy of the model on the NA tasks are summarized by Lakretz et al. (2019). Our evaluation replicates part of Table 2 in
said work.
4.2 Decomposing Number Agreement
We begin with the attribution of subject number on its corresponding verb, as decomposed per Definition 6. Among all NA tasks, the gate-level path carrying the most attribution is one following the same pattern with differences only in the size of contexts. With indices  $t$  and  $t + n$  referring to the subject and verb respectively, this path, which we term the primary path of subject-verb number agreement, is as follows:
$$
x _ {t} (D o I) \cdot \tilde {c} ^ {0} \cdot c ^ {0} \cdot h ^ {0} \cdot \tilde {c} ^ {1} \cdot \left(c ^ {1}\right) ^ {*} \cdot h ^ {1} \cdot Q o I
$$
The primary path is represented by the red path in Figure 2. The influence first passes through the temporary cell state  $\tilde{c}^0$ , the only non-sigmoid cell states capable of storing more information than sigmoid gates, since  $i, f, o \in (0,1)$  while the tanh gate  $\tilde{c} \in (-1,1)$ . Then the path passes through  $c^0$ ,  $h^0$ , and similarly to  $c^1$  through  $\tilde{c}^1$ , jumping from the first to the second layer. The path then stays at  $c^1$ , through the direct connections between cell states of neighbouring time steps, as though it is "stored" there without any interference from subsequent words. As a result, this path is intuitively the most efficient and simplistic way for the model to encode and store a "number bit."
The extent to which this path can be viewed as primary is measured by two metrics. The results across a subset of syntactic structures and number conditions mirroring those in Lakretz et al. (2019) are shown in Table 1. We include 3 representative variations of the task. The metrics are:
1.  $t$ -value: probability that a given path has greater attribution than a uniformly sampled path on a uniformly sampled sentence.
2. Positive/Negative Share  $(\pm \mathrm{Share})$  : expected (over sentences) fraction of total positive (or negative) attribution assigned to the given positive (or negative) path.
Per Table 1 (From Subject, Primary Path), we make our first main observation:
Observation 1. The same one primary path consistently carries the largest amount positive attribution across all contexts as compared to all other paths.
Even in the case of its smallest share (nounPPAdv), the  $3\%$  share is large when taking into account
more than 40,000 paths in total. Sentences with singular subjects (top part of Table 1) have a slightly stronger concentration of attribution in the primary path than plural subjects (bottom part of Table 1), possibly due to English plural (infinitive) verb forms occurring more frequently than singular forms, thus less concentration of attribution is needed due to the "default signal" in place.
Primary Neurons We further decompose the primary path into influence passing through each neuron. Since only connections between second layer cell states are sparse, we only decompose the segment of the primary path from  $c_{t}^{1}$  to  $c_{t + n}^{1}$ , resulting in a total of 650 (the number of hidden units) neuron-level paths. (We leave the non-sparse decompositions for future work). The path for neuron  $i$ , for example, is represented as:
$$
x _ {t} (D o I) \cdot \tilde {c} ^ {0} \cdot c ^ {0} \cdot h ^ {0} \cdot \tilde {c} ^ {1} \cdot \left(c _ {i} ^ {1}\right) ^ {*} \cdot h ^ {1} \cdot Q o I
$$
To compare the attribution of an individual neuron with all other neurons, we employ a similar aforementioned  $t$ -value, where each neuron-level path is compared against other neuron-level paths.
The results of the neuron-level analysis are shown in Table 1 (From Subject, Primary Neuron). Out of the 650 neuron-level paths in the gate-level primary path, we discover two neurons with consistently the most attribution (neurons 125 and 337 of the second layer). This indicates the number concept is concentrated in only two neurons.
Comparison with Lakretz et al. (2019) Uncoincidentally, both neurons match the units found through ablation by Lakretz et al., who use the same model and dataset (neurons 988 and 776 are neurons 125 and 337 of the second layer). This accordance to some extent verifies that the neurons found through influence paths are functionally important. However, the  $t$ -values shown in Table 1 show that both neuron 125 and 337 are influential regardless of the subject number, whereas Lakretz et al. assign a subject number for each of these two neurons due to their disparate effect in lowering accuracy in ablation experiments. One possible reason is that the ablation mechanism used in (Lakretz et al., 2019) assumes that a "neutral number state" can be represented by zero-activations for all gates, while in reality the network may encode the neutral state differently for different gates.
Another major distinction of our analysis from Lakretz et al. (2019) regards simple cases with no
Task	C	From Subject	From Intervening Noun
P+	|P|	Primary Path	Primary Neuron	P+	|P|	Primary Path	Primary Neuron
+Share	t	t125	t337	± Share	t	t125	t337
Simple	S	1.0	16	0.47	1.0	0.99	1.0	-	-	-	-	-	-
nounPP	SS	1.0	6946	0.1	1.0	1.0	1.0	0.82	16	0.31(+)	0.9	0.78	0.98
nounPP	SP	1.0	6946	0.1	1.0	1.0	1.0	0.23	16	0.24(-)	0.23	0.06	0.15
nounPPAdv	SS	1.0	41561	0.07	1.0	1.0	1.0	0.92	152	0.09(+)	0.96	0.85	1.0
nounPPAdv	SP	1.0	41561	0.07	1.0	1.0	1.0	0.32	152	0.09(-)	0.14	0.13	0.01
Simple	P	1.0	16	0.33	0.93	0.97	0.99	-	-	-	-	-	-
nounPP	PS	1.0	6946	0.05	0.91	0.99	1.0	0.06	16	0.28(-)	0.21	0.22	0.12
nounPP	PP	1.0	6946	0.05	0.92	0.99	1.0	0.95	16	0.31(+)	0.9	0.97	0.79
nounPPAdv	PS	1.0	41561	0.03	0.93	0.99	1.0	0.32	152	0.04(-)	0.28	0.41	0.16
nounPPAdv	PP	1.0	41561	0.03	0.92	0.99	1.0	0.83	152	0.07(+)	0.92	0.99	0.84
word between subjects and verbs. Unlike Lakretz et al., who claim that the two identified neurons are "long-term neurons", we discover that these two neurons are also the only neurons important for short-term number agreement. This localization cannot be achieved by diagnostic classifiers used by Lakretz et al., indicating that the signal can be better uncovered using influence-based paths rather than association-based methods such as ablation.
4.3 Decomposing from Intervening Nouns
Next we focus on NA tasks with intervening nouns and make the following observation:
Observation 2. The primary subject-verb path still accounts for the largest positive attribution in contexts with either attractors or helpful nouns.
A slightly worse NA task performance (Lakretz et al., 2019) in cases of attractors (SP, PS) indicates that they interfere with prediction of the correct verb. In contrast, we also observe that helpful nouns (SS, PP) contribute positively to the correct verb number (although they should not from a grammar perspective).
Primary Path from the Intervening Noun We adapt our number agreement concept (Definition 2) by focusing the DoI on the intervening noun, thereby allowing us to decompose its influence on the verb number not grammatically associated with it. In Table 1 (From Intervening Noun) we discover a similar primary path from the intervening noun:
Observation 3. Attribution towards verb number from intervening nouns follows the same primary path as the subject but is of lower magnitude and
Table 1: Statistics for attribution of primary paths and neurons from the subject/intervening noun:  $P_{+}$  is the percentage of sentences with positive input attribution. Task and  $C$  columns refer to sentence structures in Lakretz et al. (2019).  $|P|$  is the total number of paths;  $t$  and  $\pm$  Share are t-values and positive/negative share, respectively. For calculating  $t_{125}$  and  $t_{337}$  of primary neurons (125 and 337), we exclude these two neurons to avoid comparing them with each other.
Task	C	Compression Scheme
\( \overline{C_{si}} \)	\( \overline{C_s} \)	\( \overline{C_i} \)	\( C_{si} \)	\( C_s \)	\( C_i \)	\( C \)
nounPP	SS	.66	.77	.95	.93	.71	.77	.95
nounPP	SP	.64	.36	.94	.64	.75	.40	.74
nounPP	PS	.34	.24	.92	.40	.69	.18	.80
nounPP	PP	.39	.66	.91	.76	.68	.58	.97
nounPP	mean	.51	.51	.93	.68	.70	.48	.87
nounPPAdv	SS	.70	.86	.98	.73	.56	.43	1.0
nounPPAdv	SP	.70	.43	.99	.50	.60	.27	.88
nounPPAdv	PS	.38	.22	.98	.76	.79	.56	.96
nounPPAdv	PP	.39	.67	.98	.84	.83	.76	1.0
nounPPAdv	mean	.54	.55	.99	.71	.69	.50	.96
Table 2: Model compression accuracy under various compression schemes.  $C$  is the uncompressed model.
reflects either positive or negative attribution in cases of helpful nouns or attractors, respectively.
This disparity in magnitude is expected since the language model possibly identifies the subject as the head noun through the prepositions such as "behind" in Figure 1, while still needing to track the number of the intervening noun in possible clausal structures. Such need is comparably weaker compared to tracking numbers of subjects, possibly because in English, intervening clauses are rarer than intervening non-clauses. Similar arguments can be made for neuron-level paths.
4.4 Model Compression
Though the primary paths are the highest contributors to NA tasks, it is possible that collections of associated non-primary paths account for more of the verb number concept. We gauge the extent to which the primary paths alone are responsible for the concept with compression/ablation experi
periments. We show that the computations relevant to a specific path alone are sufficient in maintaining performance for the NA task. We compress the model by specifying node sets to preserve, and intervene on the activations of all other nodes by setting their activations to constant expected values (average over all samples). We choose the expected values instead of full ablation (setting them to zero), as ablation would nullify the function of Sigmoid gates. For example, to compress the model down to the red path in Figure 2, we only calculate the activation for gates  $\tilde{c}_t^0$  and  $\tilde{c}_t^1$  for each sample, while setting the activation of all other  $\tilde{c}, f, o, i$  to their average values over all samples. In Table 2, we list variations of the compression schemes based on the following preserved node sets:
$$
\begin{array}{l} C \stackrel {\text {d e f}} {=} \left\{f _ {t} ^ {l}, i _ {t} ^ {l}, o _ {t} ^ {l}, \tilde {c} _ {t} ^ {l}: t _ {\text {s u b}} <   t <   t _ {\text {v e r b}}, l \in \{0, 1 \} \right\} \\ C _ {s} \stackrel {\mathrm {d e f}} {=} \left\{\tilde {c} _ {t _ {\mathrm {s u b}}} ^ {0}, \tilde {c} _ {t _ {\mathrm {s u b}}} ^ {1} \right\} C _ {i} \stackrel {\mathrm {d e f}} {=} \left\{\tilde {c} _ {t _ {\mathrm {i n t}}} ^ {0}, \tilde {c} _ {t _ {\mathrm {i n t}}} ^ {1} \right\} \\ C _ {s i} \stackrel {\text {d e f}} {=} C _ {s} \cup C _ {i} \\ \end{array}
$$
For example, column  $C_{si}$  in Table 2 shows the accuracy when the compressed model only retains the primary path from both the subject and the intervening noun while the computations of all other paths are set to their expected values; while in  $\overline{C_{si}}$ , all paths but the paths in  $C_{si}$  are kept.
We observe that the best compressed model is  $\overline{C_i}$ , where the primary path from the intervening noun is left out; it performs even better than the original model; the increase comes from the cases with attractors (PS, SP). This indicates that eliminating the primary path from the attractor improves the model. The next best models apart from  $C$  are  $C_s$  and  $C_{si}$ , where primary paths are kept. Compressed models without the primary subject-verb path  $(\overline{C_{si}},\overline{C_s},C_i)$  have performances close to random guessing.
Observation 4. Accuracy under path-based model compression tests corroborate that primary paths account for most of the subject number agreement concept of the LSTM.
By comparing the SP and PS rows of  $\overline{C_{si}}$ ,  $\overline{C_s}$ ,  $C_s$ , and  $C_i$ , we observe the effect of attractors in misguiding the model into giving wrong predictions. Similarly, we see that helpful nouns (SS, PP) help guide the models to make more accurate predictions, though this is not grammatically justified.
The combination of finely-tuned attribution and gradient decomposition lets us investigate the handling of the grammatical number agreement concept attributed to paths across LSTM components. The concentration of attribution to a primary path and two primary cell state neurons and its persistence in a variety of short-term and long-term contexts, even with confounding attractors, demonstrates that the concept's handling is, to a large degree, general and localized. Though the heuristic decisioning aspect of an LSTM is present in the large quantities of paths with non-zero influence, their overall contribution to the concept is insignificant as compared to the primary path. Node-based compression results further corroborate these conclusions.
We note, however, that our results are based on datasets exercising the agreement concept in contexts of a limited size. We speculate that the primary path's attribution diminishes with the length of the context, which would suggest that at some context size, the handling of number will devolve to be mostly heuristic-like with no significant primary paths. Though our present datasets do not pose computational problems, the number of paths, at both the neuron and the gate level, is exponential with respect to context size. Investigating longer contexts, the diminishing dominance of the primary path, and the requisite algorithmic scalability requirements are elements of our ongoing work.
We also note that our method can be expanded to explore number agreement in more complicated sentences with clausal structures, or other syntactic/semantic signals such as coreference or gender agreement.
Acknowledgement This work was developed with the support of NSF grant CNS-1704845 as well as by DARPA and the Air Force Research Laboratory under agreement number FA8750-15-2-0277. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of DARPA, the Air Force Research Laboratory, the National Science Foundation, or the U.S. Government. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this work.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. *PloS one*, 10(7):e0130140.
Anupam Datta, Shayak Sen, and Yair Zick. 2016. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In 2016 IEEE symposium on security and privacy (SP), pages 598-617. IEEE.
Kedar Dhamdhere, Mukund Sundararajan, and Qiqi Yan. 2018. How important is a neuron? arXiv preprint arXiv:1805.12233.
James Fiasco, Samridhi Choudhary, and Carolyn Rose. 2019. Deep neural model inspection and comparison via functional neuron pathways. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5754-5764.
F. A. Gers and E. Schmidhuber. 2001. LSTM recurrent networks learn simple context-free and context-sensitive languages. Trans. Neur. Netw., 12(6):1333-1340.
Mario Giulianielli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information.
Klaus Greff, Rupesh K Srivastava, Jan Koutnik, Bas R Steunebrink, and Jürgen Schmidhuber. 2017. Lstm: A search space odyssey. IEEE transactions on neural networks and learning systems, 28(10):2222-2232.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Colorless green recurrent networks dream hierarchically.
Sepp Hochreiter. 1998. The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 6(02):107-116.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. 2018. Visualisation and diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078.
Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in lstm language models. arXiv preprint arXiv:1903.07435.
Klas Leino, Shayak Sen, Anupam Datta, Matt Fredrikson, and Linyi Li. 2018. Influence-directed explanations for deep convolutional networks. In 2018 IEEE International Test Conference (ITC), pages 1-8. IEEE.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
W James Murdoch, Peter J Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from lstms. arXiv preprint arXiv:1801.05453.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3319-3328. JMLR.org.