Mapping the Timescale Organization of Neural Language Models
Authors: Hsiang-Yun Sherry Chien
Jinhan Zhang
Christopher Honey
Venue: ICLR 2021 Poster (2021)
Keywords: natural language processing
LSTM
timescale
hierarchy
temporal context

Abstract: In the human brain, sequences of language input are processed within a distributed and hierarchical architecture, in which higher stages of processing encode contextual information over longer timescales. In contrast, in recurrent neural networks which perform natural language processing, we know little about how the multiple timescales of contextual information are functionally organized. Therefore, we applied tools developed in neuroscience to map the “processing timescales” of individual units within a word-level LSTM language model. This timescale-mapping method assigned long timescales to units previously found to track long-range syntactic dependencies. Additionally, the mapping revealed a small subset of the network (less than 15% of units) with long timescales and whose function had not previously been explored. We next probed the functional organization of the network by examining the relationship between the processing timescale of units and their network connectivity. We identified two classes of long-timescale units: “controller” units composed a densely interconnected subnetwork and strongly projected to the rest of the network, while “integrator” units showed the longest timescales in the network, and expressed projection profiles closer to the mean projection profile. Ablating integrator and controller units affected model performance at different positions within a sentence, suggesting distinctive functions of these two sets of units. Finally, we tested the generalization of these results to a character-level LSTM model and models with different architectures. In summary, we demonstrated a model-free technique for mapping the timescale organization in recurrent neural networks, and we applied this method to reveal the timescale and functional organization of neural language models

Published as a conference paper at ICLR 2021
MAPPING THE TIMESCALE ORGANIZATION OF NEURAL LANGUAGE MODELS
Hsiang- Yun Sherry Chien, Jinhan Zhang & Christopher. J. Honey Department of Psychological and Brain Sciences Johns Hopkins University Baltimore, MD 21218, USA aeberry.chiem jzhan205, chris honey) jhu. edu
ABSTRACT
In the human brain, sequences of language input are processed within dis- tributed and hierarchical architecture, in which higher stages of processing en- code contextual information over longer timescales. In contrast, in recurrent neu- ral networks which perform natural language processing, we know little about how the multiple timescales of contextual information are functionally organized. Therefore, we applied tools developed in neuroscience to map the "processing timescales" of individual units within word-level LSTM language model. This timescale-mapping method assigned long timescales to units previously found to track long-range syntactic dependencies. Additionally, the mapping revealed a small subset of the network (less than 15% of units) with long timescales and whose function had not previously been explored. We next probed the functional organization of the network by examining the relationship between the processing timescale of units and their network connectivity. We identified two classes long-timan tigeealeee units: "controller" units composed a densely interconnected sub- network and strongly projected to the rest of the network, while "integrator" units showed the longest timescales in the network, and expressed projection profiles closer to the mean projection profile. Ablating integrator and controller units af- fected model performance at different positions within sentence, suggesting dis- tinctive functions of these two sets of units. Finally, we tested the generalization of these results character-level LSTM model and models with different archi- tectures. In summary, we demonstrated a model-free technique for mapping the timescale organization in recurrent neural networks, and we applied this method to reveal the timescale and functional organization of neural language models
INTRODUCTION
Language processing requires tracking information over multiple timescales. To be able to predict the final word "timescales" in the previous sentence, must consider both the short-range context (e.g. the adjective "multiple") and the long-range context (e.g. the subject "language processing") How do humans and neural language models encode such multi-scale context information? Neuro- scientists have developed methods to study how the human brain encodes information over multiple timescales during sequence processing. By parametrically varying the timescale of intact context, and measuring the resultant changes in the neural response, series of studies (Lerner et al.) [011]] Xu al.] 2005; Honey et al., 2012) showed that higher-order regions are more sensitive range context change than lower-order sensory regions. These studies indicate the existence of "hierarchy of processing timescales" in the human brain. More recently, [Cien & Honey (2020) used a time-resolved method to investigate how the brain builds shared representation. when two groups of people processed the same narrative segment preceded by different contexts. By directly mapping the time required for individual brain regions to converge response to shared input, we confirmed that higher-order regions take longer to build a shared repre- sentation. Altogether, these and other lines of investigation suggest that sequence processing in the
long-
shared representation in
'The code and dataset to reproduce the experiment can be found httpp::/giitee tttp:///gghhubcoooooo timecattmaaaaaammmmm

---

Published as a conference paper at ICLR 2021
brain is supported by a distributed and hierarchical structure: sensory regions have short process- ing timescales and are primarily influenced by the current imput and its short-range context, while higher-order cortical regions have longer timescales and track longer-range dependencies (Hasson
et al. 2015 Honey et al. 2012, Chien & Honey 2020 Lerner et al. 2011 Baldassano et al. 2017
Runyan
2017 Fuster 1997).
How are processing timescales organized within recurrent neural networks (RNNs) trained to per- form natural language processing? Long short-term memory networks (LSTMs) (Hochreiter & Schmidhuber 1997) have been widely investigated in terms of their ability to successfully solve se- quential prediction tasks. However. long-range dependencies have usually been studied with respect to a particular linguistic function (e.g. subject-verb number agreement, Linzen et al. [016 Gulor- dava et al. 2018: Lakretz et al. 2019), and there has been less attention on the broader question of how sensitivity to prior context- - broadly construed - is functionally organized within these RNNs. Therefore, drawing on prior work in the neuroscience literature, here we demonstrate a model-free approach to mapping processing timescale in RNNs. We focused existing language models that were trained to predict upcoming tokens at the word level (Gulordava et al. 2018) and at the char- acter level (Hahn & Baroni 2019). The timescale organization of these two models both revealed that the higher layers of LSTM language models contained a small subset of units which exhibit long-range sequence dependencies; this subset includes previously reported units (e.g. "syntax" unit, Lakretz et al. 2019) as well as previously unreported units.
After mapping the timescales of individual units, asked: does the processing timescales of each unit in the network relate to its functional role, as measured by its connectivity? The question is mo- tivated by neuroscience studies which have shown that in the human brain, higher-degree nodes tend to exhibit slower dynamics and longer context dependence than lower-degree nodes (Baria el al. 2013). More generally, the primate brain exhibits core periphery structure in which a relatively small number of "higher order" and high-degree regions (in the prefrontal cortex, in default- mode regions and in so-called "limbic" zones) maintain a large number of connections with one another, and exert a powerful influence over large-scale cortical dynamics (Hagmann et al. 2008: Mesulam 1998 Gu et al. 2015p. Inspired by the relationships between timescales and network structure in the brain, we set out to test corresponding hypotheses in RNNs: (1) Do units with longer- timesclle tend to have higher degree in neural language models? and (2) Do neural language models also ex- hibit "core network" composed of functionally influential high-degree units? Using an exploratory approach, we found that units with longer timescales tend to have more projec- tions to other units. Furthermore, we identified a set of medium-to-long timescale "controller" units which exhibit distinct and strong projections to control the state of other units, and a set of long- timescale "integrator units" which showed influence on predicting words where the long context is relevant. In summary, these findings advance our understanding of the timescale distribution and functional organization of LSTM language models, and provide method for identifying important units representing longerange contextual information in RNNs.
2 RELATED WORK
Linguistic Context in LSTMs. How do LSTMs encode linguistic context at multiple timescales? Prior work suggested that the units sensitive information that requires long-range dependencies are sparse. By ablating one unit at a time, Lakretz et al. (2019) found two units that encode infor- mation required for processing long-range subject-verb number agreement (one for singular and one for plural information encoding). They further identified several long- range "syntax units" whose activation was associated with syntactic tree-depth. Overall, [Lkretz et al. (2019) suggests that a sparse subset of units tracks long-range dependencies related to subject-verb agreement and syntax. If this pattern is general - i.e. if there are very few nodes tracking long-range dependencies in gen- eral - this may limit the capacity of the models process long sentences with high complexity, for reasons similar to those that may limit human sentence processing (Lakretz et al., 2020). To test whether long-range nodes are sparse in general, we require model-free approach for mapping the context dependencies of every unit in the language network.
Whole-network context dependence. Previous work by Khandelwal et al. (2018) investigated the duration of prior context that LSTM language models use to support word prediction. Context- dependence was measured by permuting the order of words preceding the preserved context, and
2

---

Published as a conference paper at ICLR 2021
observing the increase in model perplexity when the preserved context gets shorter. Khandelwal jet al. (2018) found that up to 200 word-tokens of prior context were relevant to the model m perplexity,
but that the precise ordering of words only mattered within the most recent 50 tokens. The token- based context-permutation method employed in this study was analogous to the approach used to measure context-dependence in human brain responses movies (Hasson et al. 2008) and to audi- tory narratives (Lerner et al. 2011).
Inspired by the findings of Khandelwal et al. (2018) and Lakretz al. (2019). in the present study we set out to map the context-dependence across all of the individual units in the LSTM model. This en- abled us to relate the timescales to the effects of code-specific ablation and the network architecture itself. In addition, our context manipulations included both context-swapping tive meaningful contexts) and context-shuffing (permuting the words in the prior context to disrupt inter-word structure), which allowed us to better understand how individual words and syntactically structured word- sequenesss contribute to the the context representation of individual hidden units.
(substituting alterna-
3 METHODS
3.1 LANGUAGE MODELS AND CORPUS
We evaluated the internal representations generated by a pre-trained word-level LSTM language model (WLSTM, Gulordava al. 2018) as well as a pre-trained character-level LSTM model (CLSTM, Hahn & Baroni] 2019) as they processed sentences sampled from the 427804-word (1965719-charact)) novel corpus: Anna Karenina by Leo Tolstoy (Tolstoy] 2016), translated from Russian to English by Constance Garnett.
For the WLSTM, we used the model made available by Gulordava et al. (2018). The WLSTM has a 650-dimensional embedding layer, two 650-dimensional hidden layers and an output layer with vocabulary size 50,000. The model was trained and tested on Wikipedia sentences and was not fine-tuned to the novel corpus. Therefore, we only used sentences with low perplexity from the novel in our main timescale analysis. We performed the same analysis using the Wikipedia test set from Gulordava et al. (2018) and obtained similar results (See Section 5.3, Figure A.4A, Appendix A.2.1. For the CLSTM, we used the model made available by by|Hahn& & Baroni (211). The CLSTM has 200-dimensional embedding layer, three 1024-dimensional hidden layers and an output layer with vocabulary size 63, The model was trained on Wikipedia data with all characters lower-cased and whitespace removed. We tested the model with sentences sampled from Anna Karenina as the WLSTM model, and we obtained bits-per- per-character (BPC) similar to what Hahn & Baroni (2019) reported in their original work.
3.2 TEMPORAL CONTEXT CONSTRUCTION PARADIGM
In order to determine the processing timescales of cell state vectors and individual units, we mod- ified the "temporal context construction" method developed by Chien & Honey (2020). Thus, the internal representations of the model were compared across two conditions: (1) the Intact Context condition and (2) the Random Context condition. In both conditions, the model was processing the same shared sequence of words (for example, segment B), but the preceding sentence differed across the two conditions. In the Intact Context condition, the model processed segment B (the shared seg- ment) preceded by segment A, which was the actual preceding context from the original text. In the current study, for example, segment A and B are connected by ", and" within long sentences from the novel corpus (Figure to ensure the temporal dependencies between and B. In the Random Context condition, however, the model processed the same shared input (segment B), but the context was replaced by segment X, which was a randomly sampled context segment from the rest of the corpus. Segment X was therefore not usually coherently related to segment B. For the WLSTM timescale analysis, we chose long sentences in the Intact Context condition that satisfied the follow- ing constraints: (1) mean perplexity across all words in the sentence < 200, (2) the shared segment was longer than 25 words, and (3) the context segment was longer than 10 words. 77 sentences are included as trials in our analyses. In the Random Context condition, segments and randomly sampled 30 context segments (each longer than 10 words) from other parts of the novel. For the CLSTM timescale analysis, we used the same 77 long sentences in the Intact
preserved the same shared
3

---

Published as a conference paper at ICLR 2021
Context condition, and randomly sampled 25 context segments (with length > 33 characters) for the Random Context condition.
A
Different context
Shared input
Intact
He would into the nursery several times day
and sit there for long while, that the nurses, who were at first afraid of him, got quite used to his presence.
condition
Random
She positively shrank together and flushed the point of tears
condition
B
Intact context
He would 80
into
nursery
Several times
day
and and sit
sit there
for
long while
50
the
nurses
who were
first afraid
of
LSTM cell state activation
Activation difference |Intact- - Random)
She positively
shrank together and flusedd
Random context
the of tears
for long
while
8o that
the
nurses
who
were
at
first
afraid
of
Long-timescale unit Short-limescale unit
activation difference
Different context
Shared input
Figure 1: Method for mapping processing timescales of individual units. Example sentences for the model to process in the Intact Context and Random Context condition. In the Intact Context condition, the shared segment preceded by an intact context from the corpus; while in the Ran- dom Context condition, this preceding context segment is replaced by randomly sampled context segments. Schematic hidden state activation process the shared segment preceded by different context between the two context conditions, the hidden unit activation difference (i.e. the mean absolute difference of unit activation between the two conditions) decreases time with different rates. The expected decreasing pattern of acti- vation difference of a long-timescad unit and a chort-timescale unit are shown schematically in the green and red curves, respectively.
the neural network. When the model starts to
In brief, the model processing the same input (the shared segment) with different preceding context (the intact vs. random context). We can now measure the context dependence of individual units by examining how the cell state activations differ between the two conditions, while the network is processing the shared segments with identical input. Any difference in internal representations must arise from the context manipulation, since the current input the same. A decrease activation difference over time implies that the units exposed in the Intact context and Random context start to build a similar representation as they process the shared input. For a long-timescall unit, whose current state is dependent information in the far-preceding context, difference is preserved across contexts (Figure DB green curve), even while the unit processing the shared input. On the other hand, for short-timescalee unit whose activation is driven largely by the current imput, we will see that the activation difference drops quickly (Figure IB, red curve) as the unit processes the shared input.
will see that the activation

---

Published as a conference paper at ICLR 2021
4 HIERARCHICAL ORGANIZATION OF TIMESCALES ACROSS LAYERS
Do higher levels of the LSTM model exhibit greater context-dependencee Lakretz al. 2019 ob- served that long-range functional units were more common in higher layers, and in general, higher- levels of hierarchical language model exhibit longer range context-dependence (Jain al. 2019 Jain & Huth 2018). Therefore, to validate our stimuli and the sensitivity of our methods, we first compared the processing timescales of different hidden layers in both of the LSTMs, by correlating the cell state vectors, column by column, between the Intact condition and Random condition.
We found that both layers showed near-zero correlation when processing the different context, and the correlation increased they began to process the shared input. In the WLSTM, the correlation increased more slowly for second-level cell state vectors than for first-level cell state vectors. Thus, the representation of second-level cell state is more sensitive to the different context than the first level. Similarly, for the CLSTM model, the third-level cell state exhibited longer-lasting context sensitivity than lower levels (Figure 2). This observation of longer context-dependence in higher stages processing is consistent with prior machine learning analyses (Lakretz et al., 2019: Jain & Huth (2118) and is also analogous what is seen in the human brain (Hasson et al. 2015 Chien
& Honey 20200] Lerner et al., 2011 Jain et al. 2019). Based on the finding of longer context dependence in higher layers, examined single units in the highest level hidden units, i.e. the second level of WLSTM (n=650) and the third level of CLSTM (n=1024).
A
1.0
B
1.0
Intact-Random Correlation
Intact-Random Correlation
0.8
0.6
0.6
0.4
0.2
0.2
Cell state level
Cell state level
cell state level level
0.0
0.0
Cell level
-10
Different context
Shared input
Different context
Shared input
Figure 2: Context effect measured by cell-state vector correlation at different layers in word-level LSTM (WLSTM) and character-level LSTM (CLSTM). A. Correlation curves the WLSTM cell- state vectors across the Intact Context condition and Random Context condition as a function of input token. In both models, the correlation increased as the models began to process the shared segment. Higher-level cell states exhibited slower increase in correlation, compared to lower-level cell states, indicating that the higher-levels retain more of the prior context information for longer. B. As for A. but applied to the three levels of CLSTM. Similar the WLSTM, higher-level cell state the CLSTM showed more context sensitivity than the lower-level cell state.
5 PROCESSING TIMESCALES OF INDIVIDUAL UNITS WITHIN LSTM LAYERS
5.1 QUANTIFYING SINGLE UNIT TIMESCALES
We examined the absolute single unit activation difference when processing the shared segments preceded by different context. As expected, most of the hidden units showed different activation when the imput tokens were different (i.e. while processing the non-shared context in the Intact Context and Random Context conditions). However, once the shared input tokens begin (at 0) the intact-Randie activation differences drop (Figure A.TA
We used the rate at which the curves drop quantify the processing timescale, as this is a measure of how quickly the responses align across different context conditions. To quantify the timescale of individual units, we fit the activation difference curves with logistic function:
L
Y())
1+ k(z-2o) +
5

---

Published as a conference paper at ICLR 2021
As shown in Figure A.IA and Figure A.IB, the logistic function fit the raw activation difference curves. We then computed the "timescale" each unit as the time-tohall--me malf-ma of the logistic decay. In particular, for the WLSTM shared segment, and at the end of the shared segment ((44) (797 for the CSLTM) to calculate the taae-ttohhff--aimmmeee of unit as:
used the activation difference Y (0) at the beginning of the
timescale, = --- ((()) (244
2
where the inverse function (v) identifies the largest integer 1, for which (()) < y. We included 635 units in WL.TTM and 1012 units in CLSTM for further analysis after excluding the units which could not be accurately fit by logistic function (See Appendix A.I).
5.2 DISTRIBUTION OF UNIT TIMESCALES IN LSTM LANGUAGE MODELS
The results showed that of the 635 WLSTM units whose processing timescale we mapped, approx- imately 70% of the units were insensitive long-range context (processing timescale < 3 words): their activation difference dropped immediately at onset of the shared segment. In contrast, only approximately 13% of the units had timescales > 7 words (Figure A.2 Figure 3/ shows the absolute activation difference of all units in WLSTM sorted by timescale (long to short). Some of the longer-timescale units continued to exhibit a large activation difference even when processing the shared segments for more than 20 tokens.
As we were testing the same word-level LSTM previously studied by Lakretz et al. (2019), we began by examining the timescales of hidden-state units that were already known to be involved in processing context-dependence language information: a "singular number unit" 988, a "plural number unit" 776, and "syntax unit" 1150. We found that, compared to other units, both "number" units had medium timescales (~ words, ranked 129 of 635 units), while the "syntax" unit had a long timescale (~ 7 words, ranked 64 of 635 units) [iguure/A.I.
We repeated the timescale mapping in the CLSTM model, and again identified a small subset of long-timescale units (Figure 3. Figure A.2 Although there were overall more units in CLSTM, over 63% of the units were insensitive to the context (timescale < characters). Fewer than 15% of the units exhibited timescale 10 characters, and the unit with the longest timescale only dropped to its malf-maximum activation-difffrence after 50 characters of shared input.
5.3 TIMESCALE VARIATION ACROSS DATASETS AND CONTEXT CONDITIONS
To ensure that the timescales we measured were robust across datasets, we conducted the same analysis WLSTM using the Wikipedia testing dataset used in Gulordava al. (201)). The mapped timescales were highly correlated (r=0.82, p<0..00) across the Anna Karenina dataset and the Wikipedia dataset (Appendix A.2.I Figure A.4A).
Similarly, to confirm that the timescales measured were not specific to our testing using the ", and" conjunction point, also measured timescales an alternative segmentation point, and found that the timescales were largely preserved (r=0.83, p00000)), notwithstanding there were a small set of A.222 Figure FigureA.44B)
notable exceptions (Appendix A.2.2
Although we measured the timescales of context dependence using "token distance", these measures not invariant to changes in the the "syntactic distance".. For example, if one were replace comma with a "full stop". then the token distance would be unaltered but the syntactic distance could be greatly altered. Indeed, we found that most units showed little context dependence when the preceding context segment ended with a "full stop", which served as a clear signal for the end of a sentence (Appendix A.2.3] Figure A.4C).
Finally, we examined whether the contextual information retained by the language models (and the associated timescales measurement) was sensitive to linguistic structure in the context, or whether was primarily driven simply by the presence or absence of individual words. To this end, gener- ated text for the Random Context condition by shuffling the order of words from the Intact segment. We found that while the presence of individual words did play important role in determining the context representations (and thus the timescales), several units showed a longer timescale when the prior context was composed of coherently structured language (Appendix A.2.4 Figure(..4D).

---

Published as a conference paper at ICLR 2021
WLSTM
CLSTM
Long-range
Long-rango
Otymtex unto
Calume 776 (singuaar uni9958
Inaen Dondoom ald bustion Binunoo
Frauc- Ruinda volton oifirnnno
Lingt scrted by Fimaee Perooae
Uit by tiescollss
Shortirange
Short-range
Shared
(word)
Diferrnce contect (Intact Random)
Shared input
(characters)
Figure 3: Timescale organization in word-level LSTM (WLSTM) and character-level LSTM (CLSTM) language model. A. Absolute activation difference for each WLSTM hidden unit over time, with units (rows) sorted by timescales. A small set of long-timescale units (top) sustain an activation difference during shared segment processing, but most (bottom) are context-insensitive short- timessliee units. B. Absolute activation difference for each CLSTM unit over time, with units sorted by timescales. Similar to the WLSTM, small set of long-timescale CLSTM hidden units maintain long-range contextual information.
CONNECTIVITY MEDIUM- TO LONG-TIMESCALES UNITS IN LSTMs
Having mapped the timescales of each processing unit, we next asked: how does the processing timescale of unit relate to its functional role within the network? More specifically, are units with longer timescales also units with high degree in the connectivity network? To answer these questions, we analyzed (1) the projection strength of each unit and (2) the similarity the overall projection pattern (hidden-to- to-gates) across different units. The projection patterns were defined using the direct weight projections from one hidden unit at time f to the input and forget gate of other hidden units time i
In LSTMs, the amount of contextual (Q-1) and input (o)) information stored in the cell state (c:) determined by the forget gate (fr) and input gate (i.) activation (Eq. 3); and the activation of the gates i, and ft are determined by the current input at time and the hidden units at time (- 1 through weight matrices U and W (Eq. 4, 5).
G: f a-1 +
(3)
i, = -U,,, W,h, bj) f: o(U,2, + wyht by)
(4)
(5)
Here, we were interested understanding how the contextual information over different timescales is projected from the hidden units to the input and forget gates of other units, and further influence the update of cell states. Thus, we analyzed the network connectivity focusing on the weight matrices W. and Wy within the highest layer of the WLSTM or CLSTM.
6.1 STRONG PROJECTIONS FROM LONG-TIMESCALL HIDDEN UNITS TO GATE UNITS
Units with longer processing timescales made larger number strong projections ([zsccore) > 5, Appendix A.3) to the input and forget gates of other units in both WLSTM (r=0.31, p<00.01, Figure 4 and CLSTM models (r=0.24, p<0.001, Figure A.5 Furthermore, we found that the "syntax" unit (Unit 1150) reported by Lakretz et al. (2019) in the WLSTM model possessed the largest number of strong projections to the input and torget gates of all other units, and the major recipients from Unit 1150 were units with medium- to long-timescale units (Fi

---

Published as a conference paper at ICLR 2021
6.2 IDENTIFY CONTROLLER UNITS IN LSTM LANGUAGE MODELS
The presence of strong projections from the "syntax" unit to other long-timescale units motivated us to further explore whether high-degree, long-timescale units in the LSTM also densely interconnect to form a "core network". perhaps analogous to what is seen in the brain (Hagmann et al.) 2008 Mesulam 1998, Baria et al. 2013). If so. this set of units may have an especially important role in controlling how prior context is updated and how it is used to gate current processing, analogous to the controller system in the brain (Gu et al. 2015). To identify these putative "controller units", we binarized the network by identifying the top 258 projection weights from the weight matrices (see Appendix (A.3), which provided the edges for a network analysis. We then used k-core analysis (Batagelj & Zaversnik 2003) to identify the "main network core" (the core with the largest degree) of the network (Figure A.3). At the maximal & = 5, the k-core analysis yielded a set of densely interconnected nodes, composed of many tingtiiesccal and timessleee units (Figure AA.)) also labeled in red in Figure A)). We (tentatively) refer to this set as the "controller" set of the net- work. Performing the same k-core analyses the CLSTM model, we observed that the main core network was again composed of disproportionately many medium and long-timescale "controller" units (Figure (15gureA.55A) A.5
Iniegnater une
Contro unia
B
Weigett on Parger gate
Cantroler
Hit irpul pata polection
a3et OST1
200
400
600
Targe unite scrtect Emercil
Dilurencc of cookgedd probedilic largd words stuadd mode- iradd mame)
All tolonne
Pardon straed
Final sentanco
5 2
3
36
Nurberr strong projectiona hicden to inpud gato forget gate
Figure 4: Timescale and connectivity organization in a word-level LSTM. A. exhibited stronger projections from the hidden state at time the forget gate and input gate at time + 1. B. Strength of hidden-forget gate and hidden-inpee indenn-npputeee gate projections from high-degree "syntax" unit to all other units. The units receiving strong projections ([z-score}> > 5) are labeled. C. Ablating the two sets of long-timescale units results in different impact to the LSTM performance. Specifically, ablating "controller" units impaired overall word prediction (upper panel), while ablat- ing "integrator" units impaired prediction of words in the later part of the sentences (bottom panel). D. Multi-dimensional scaling representation of network connectivity. The distance between two nodes indicates the similarity of their hidden-to-gan connection patterns. The size of each node indicates its degree (the number of strong projections from that node to the gate units). An edge between nodes indicates significant hidden-to-gate projection between them.
units
6.3 DISTINCTIVE ROLES OF LONG-TIMESCAA CONTROLLER AND INTEGRATOR UNITS
We used multi-dimeenional scaling (MDS) to visualize the similarity of projection patterns across L.STM units. We recovered a 2-dimensional MDS embedding, in which the inter-unit distances was defined based on the similarity of their hidden-to-gate projection patterns (i.c., similarity of values in the unthresholded LSTM weight matrices W; and Wy). We visualized the MDS solution as a graph structure, in which each node is unit, and the edges reflect connection properties of that unit. Figure shows the resulting 2-D space, with units color-coded by their timescale.
"Controller units" (labeled Figure 4D) were positioned around the periphery of the MDS space, suggesting that these units expressed projection patterns that were distinct from other "controller" units and also from the rest of the network. In contrast, we observed several long-timescale units
8

---

Published as a conference paper at ICLR 2021
positioned in the center of the MDS space, suggesting that the projection patterns of these units were similar to the mean projection pattern. We refer to this more MDS- set as the "integrator units" (labeled in green in Figure 4A). Similar to the WLSTM, the projection patterns of the "controller units" in the CLSTM were distinct from other units in the network, according to the MDS results (Figure A.5()) However, did not observe "integrator units" positioned in the center of the MDS space of the CLSTM.
Are the "controller" and "integrator" units particularly important for the model's ability to predict the next token? To test the functional importance of these subsets of units, we conducted group ablation analyses (See Appendix A..4.. Ablating controller units reduced the accuracy of token prediction overall, while ablating integrator units only reduced prediction accuracy for the last words of the sentences (Figure 4C). The results confirm that the putative controller and integrator nodes are functionally significant, with distinctive roles in the WLSTM language model.
Finally, to test the generalization of the timescale and connectivity analyses to a different model architecture, we conducted preliminary analyses on Gated Recurrent Unit (GRU) language model (Cho et al. 2014) and another word-level LSTM model with a smaller hidden size (100 units) per layer. The models were trained using similar parameter settings as in Gulordava et al. (2018) un- til they converged without any model-specific optimization. We found similar sparsity of long- timescale units in both models, but did not observe the same relationship between timescales and connectivity (Appendix A.5 A.6 Figure A.7
A.8 A.9 A.10
7 DISCUSSION
We demonstrated a new method for mapping the timescale organization in recurrent neural language models. Using this method, we mapped the timescale distributions of units within word-level and character-level LSTM language models, and identified a small set of units with long timescales. We then used network analyses to understand the relationship between the timescale of unit and its connectivity profile, and we distinguished two subsets of long-timescale units with seemingly dis- tinctive functions. Altogether, we proposed methods combining timescale and connectivity analyses for discovering timescale and functional organization in language models.
The units with longer processing timescales included some units whose role in long-range lan- guage dependencies had already been established (Lakretz et al. 2019), but almost all of the long timescale units are of unknown function. The timescale mapping procedure described here provides model-free method for identifying nodes necessary for long-range linguistic and discursive pro- cesses (e.g. tracking whether a series of words constitutes an assertion or a question). Future studies of these neural language models could focus the specific linguistic information tracked by the timesiile units, especially the "controller" units which control the information flow of other units in the network.
The current study measured unit timescales using a simple token distance, and so the method may be applied understanding recurrent neural nets beyond language models. It will be insightful for future studies to investigate whether the processing timescales characterized via token distance comparable to those measured using functional measures, such as syntactic distance. Relat- edly, while we explored the timescale variance under several context conditions, more thorough investigation will be needed examine how the timescales of individual units may vary different positions within sentence, both in terms of token location and syntactic location.
Processing timescales may exhibit an analogous hierarchical organization in LSTMs and in the human cerebral cortex: in both cases, subset of nodes with high degree and high inter-connectivity express unusually long timescales. More detailed testing of this apparent correspondence is required, however, because units within an LSTM layer are not spatially embedded and constrained as in biological brains, and thus the LSTM units do not express spatially graded timescale topography.
ACKNOWLEDGMENT
C.J.H and H-Y.S.C gratefully acknowledge the support of the National Institutes of Mental Health (grant RMIMHI 19099)

---

Published as a conference paper at ICLR 2021
REFERENCES
Christopher Baldassano, Janice Chen, Asieh Zadbood, Jonathan W Pillow, Uri Hasson, and Ken- neth Norman. Discovering event structure in continuous narrative perception and memory. Neuron, 95(3):709-721, 2017.
2017.
Alex Baria, A Mansour, Lejian Huang, Marwan N Baliki, Guillermo A Cecchi, M-Marsel Mesu- lam, and A Vania Apkarian. Linking human brain local activity fluctuations to structural and functional network architectures. Neuroimage, 73:144-155, 2013.
Vladimir Batagelj and Matjaz Zaversnik. An (m) algorithm for cores decomposition of networks.
arXiv preprint cs/031004, 2003.
Hsiang- Yun Sherry Chien and Christopher Honey. Constructing and forgetting temporal context in the human cerebral cortex. Neuron, 2020.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mn pecoder-decodee for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, 2014.
Joaquín Fuster. Network memory. Trends in neurosciences, 20( 20100)451-453) 1997.
Shi Gu, Fabio Pasqualetti, Matthew Cieslak, Qawi K Telesford, B Yu Alfred, Ari E Kahn, John D Medaglia, Jean M Vettel, Michael B Miller, Scott Grafton, et al. Controllability of structural brain networks. Nature communications, 6(1):1-10, 2015.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1195-1205, 2018.
Patric Hagmann, Leila Cammoun, Xavier Gigandet, Reto Meuli, Christopher J Honey, Van J Wedeen, and Olaf Sporns. Mapping the structural core of human cerebral cortex. PLoS Biol, 6(7):e159, 2008
Michael Hahn and Marco Baroni. Tabula nearly rasa: Probing the linguistic knowledge of character- level neural language models trained unsegmented text. Transactions of the Association for Computational Linguistics, 7:467-484, 2019.
Uri Hasson, Eunice Yang. Ignacio Vallines, David J Heeger, and Nava Rubin. A hierarchy of tem- poral receptive windows in human cortex. Journal of Neuroscience, 28(10):2539-22 2550, 2008.
Uri Hasson, Janice Chen, and Christopher J Honey. Hierarchical process memory: memory integral component of information processing. Trends in cognitive sciences, 196))300-3133, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Christopher Honey, Thomas Thesen, Tobias H Donner, Lauren Silbert, Chad Carlson, Orrin Devinsky, Werner K Doyle, Nava Rubin, David J Heeger, and Uri Hasson. Slow cortical dynamics and the accumulation of information over long timescales. Neuron, 76(2):423 434, 2012.
Shailee Jain and Alexander Huth. Incorporating Context into Language Encoding Models for fMRI. Advances in Neural Information Processing Systems, pp. 6628-6637, 2018.
Shailee Jain, Amanda LeBel, and Alexander Huth. Improving language encoding of fMRI responses with transformers. Annual Meeting of the Society for Neuroscience., 2019.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 284-294. 2018.
10

---

Published as a conference paper at ICLR 2021
Yair Lakretz, Germán Kruszewski, Théo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence number and syntax units in Istm language models. Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1-20,, 2019.
Yair Lakretz, Stanislas Dehaene, and Jean-Rémi King. What limits our capacity to process nested long-range dependencies in sentence comprehension? Entropy, 22(4):446. 2020.
Yulia Lerner, Christopher Honey, Lauren Silbert, and Uri Hasson. Topographic mapping of hierarchy of temporal receptive windows using a narrated story. Journal of Neuroscience, 31(8); 2906-2915, 2011.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability Istms to learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521- 535, 2016.
M-Marsel Mesulam. From sensation to cognition. Brain: a journal of neurology, 121(6):1013-1052, 1998.
Caroline Runyan, Fugenio Piasini, Stefano Panzeri, and Christopher Harvey. Distinct timescales of population coding across cortex. Nature, 348076655)92-96622010 2017.
Leo Tolstoy. Anna karenina. Lulu. com, 2016.
Jiang Xu, Stefan Kemeny, Grace Park, Carol Frattali, and Allen Braun. Language in context: emergent features of word, sentence, and narrative comprehension. Neurolmage, 2503):0002-
1015, April 2005. ISSN 10538119. doi: J0.10016...uurrmmge2220001111110 URL http:
7/linkinghe iighb. eeeeelseeeee com

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 UNITS EXCLUDED FROM TIMESCALE ANALYSIS
We excluded I unit in the WLSTM model and 5 units in CLSTM model which were not properly fit using the logistic function; we further excluded 14 units in the WLSTM model and units in the CLSTM model which either did not show a non-zero activation difference before the shared segment started, or whose activation differences increased when started to process the shared segment. After these exclusions, 635 units remained in the WLSTM and 1012 units remained in the CLSTM for further analysis.
A.2 TIMESCALE ANALYSES ACROSS DIFFERENT DATASETS AND CONTEXT CONDITIONS
A.2.1 WIKIPEDIA TEST DATASET
The Anna Karenina corpus used in the current study has a different linguistic structure from the Wikipedia corpus which the WLSTM and CLSTM models trained. Although we analyzed only the Anna Karenina sentences with low perplexity, it was important to test the robustness of our results across datasets. Thus, we mapped the timescale of each unit using the Wikipedia test set, as used by Gulordava et al. (018). Specifically, we sampled 500 long sentences containing ". and" for the Intact Context condition. As before, we generated sentences by preceding the "shared input" segment (after the conjunction) with either the original prior context segment, or a randomly chosen prior context segment. Same as the original analysis, we then replaced the context segment with 30 context segments randomly sampled from other parts of the test set for generating the Random Context condition. The mapped timescales using the Wikipedia test set were highly correlated with the novel corpus, suggesting the robustness of unit timescales (Figure A.4A).
A.2.2 TIMESCALES MEASURED IN THE MIDDLE OF A SENTENCE
To examine how the timescales of individual units may vary across different positions in a sentence, we varied the location of the segmentation point. Instead of using the conjunction (", and"") as the segmentation point, we chose an arbitrary segmentation point: the 15th token of a long sentence, to separate context segment and shared segment. In the Random Context condition, we replaced the context segment with the first 15 tokens from other sentences of the corpus. We found that the unit timescales were highly correlated with the condition where used the conjunction the segmentation point with several units shift their timescales to either directions (Figure A.43). This analysis was conducted using Wikipedia test set.
A.2.3 TIMESCALE RESET AT THE BEGINNING OF SENTENCE
To examine if the timescales of individual units can flexibly reset at the beginning of sentence, we conducted the same timescale analysis but using a "full stop" as the segmentation point instead of the conjunction and". Thus, if the original test string was "The girl kicked the call, and the boy caught it", then the full-stop version of the test string would be "The girl kicked the ball. The boy caught it." In this setting, the context segment and shared input segment in the Intact Context condition are two consecutive sentences. To ensure the temporal dependence between the context segment and shared input segment, we sampled 100 consecutive sentence pairs from the Anna Karenina corpus. Note that this is not possible using the Wikipedia test set from Gulordava al. (2018), because that set is composed of unrelated sentences. The Random Context condition was generated by replacing the first sentence with randomly sampled sentences from other parts of the novel. We found that when using "full stop" to segment context and shared input, most units in the network showed timescale near O, indicating near-zero dependence on the linguistic context from the text preceding the full stop (Figure (iguree..4)) A.4C). This suggests that the units in LSTM tend to "reset" their context representation the beginning of sentence.
A.2.4 CONTEXT REPRESENTATION SHAPED BY INDIVIDUAL WORDS
Inspired by the siken-shuffing procedure of Khandelwal et al. (2018), we explored whether the context representations of individual units in the L.TTM were shaped by individual words, rather
12

---

Published as a conference paper at ICLR 2021
than coherent sequences of words. For this analysis, instead of replacing the context with syntac- tically structured segments from other part of the corpus, we generated the "random context" by shuffling the order of words within the context segment. We then mapped the unit timescales as before, by examining the unit activation difference as a function of the distance from the onset of shared input. Intriguingly,
found that most of the units showed similar timescales across the context-rellacement and context-shuflling procedures (Figure (A.4p). This suggests that the context representations in LSTMs largely depend on the presence of individual words in the context, rather than their appearance within coherent linguistic sequences. However, we did observe a subset of units (labeled in the Figure, and almost all tong-iime units) whose timescales were longer when context was replaced rather than shuffled. For this subset of units, the ability to maintain a represen- tation of prior context over many tokens depends on that prior context being coherent linguistic sequence. This subset of units are a promising target for future studies of syntactic representations in LSTMs.
A.3 IDENTIFYING STRONG HIDDEN-TO-GATE PROJECTIONS
First, for each hidden unit, we concatenated the corresponding rows in the Wh: and Way matrices, to generate single "hidden-to-ge projection vector for that hidden unit. Next we we z-scored the vector to get standardized projection values from that unit to all other units in the network. Using |z-score|> 5 as criterion, we identified a total of 258 "strong projections" from all hidden units to the input gate and forget gate in the WLSTM. The projection strength of each unit was then calculated based on its number of "strong projections" (Figure 4 Although the criterion Iz- score/> was selected to better visualize the results in Figure 4 different criteria did not change the results that units with longer timescales have more strong projections. For example, using [z-score|> 3 as threshold we obtained (imeesiee, projections) = 0.30, p<0.001; |z-score| |> 4 we obtained corsttimescale, projections) 0.35, p=00.001.
Next, we identified the edges corresponding to the top 258 magnitude weight-values within the combined Whi and Whs matrices. Together, these edges formed a "strong-projectio network". Finally, we used k-core analysis to identify the main core of the strong-projectoo network. This main core composed our "controller units" (Figure A.3).
Using the same criteria and method, we identified a total of 390 "strong projections" from all hidden units to the input gate and forget gate in the CLSTM. We then extracted the top 390 weight values from the weight matrices to construct a "strong- projection network" and again identified the main core network, composed the "controller units" for the CLSTM model (Figure A. A.5
A.4 ABLATION ANALYSES ON PUTATIVE CONTROLLER AND INTEGRATOR UNITS
To examine the non-trivial roles of the controller and integrator units identified in the word-level LSTM model, we performed a preliminary group ablation analysis to look at how ablating the con- troller units influences model performance random set of units. Specifically, since Long-timescale integrator units should have most effect pre- dicting tokens the later part of the sentences (i.e., when more context is integrated), we examined the model performance on predicting tokens at two different positions: (1) all the tokens regard- less of their positions in the sentences ("All tokens" condition), and (2) the last tokens of sentences ("Final tokens" condition).
predicting the token, relative the ablation of
We evaluated the effects of ablation on model performance by measuring the differences of proba- bilities (AP) assigned to the target words (AP probability of target word in ablated model minus probability of target word in original model). Ablation effects for controller units (N=9) and inte- grator units (N=10) were compared against bascline of ablating the same number of randomly- selected units from layer of the LSTM (Figure 4C). We used the test corpus used by /ulordava et al. (2018) and measured the average performance of each model across 100 text- text-batches, ran- domly sampled from the Wikipedia test dataset. Each text-batch was composed of 1000 tokens that start at the beginning of sentence.
In the "All tokens" condition, we calculated the AP for every token in the tested text, while in the "Final tokens" condition, we caleulated AP only at the last token of every sentence (i.e. the token
13

---

Published as a conference paper at ICLR 2021
right before the full stop"." of each sentence). We then average the AP in both conditions across text-batches to get a mean performance difference between the ablated model and the intact model.
Ablating controller units reduced the probabilities assigned to the target words, more so than ablating random units (Figure |c, controller vs. random across 100 text batches: Cohen's d = -4.85, t 34.28, p=00.00).. In contrast, ablating integrator units reduced the probabilities less than ablating random units (integrator vs. random: Cohen's d = 2.50, t t= 17.67, p=00.001). We hypothesized that that the integrator units mostly influence the model's prediction performance for tokens where long-range information is especially relevant, such as in the later portions of clauses and sentences. Consistent with this, we found that, when we examined the ablation effects only for tokens in the final position of a sentence, ablating integrator units reduced the probabilities more than ablating random units (Cohen's s d -0.34, = -2.41, p = 0.017). Interestingly, ablating controller units reduced the probability of sentence- final targets less than random units (Cohen's 's d 0.67, t 4.74, p<00.011.
In summary, these ablation results indicate a non-trivial functional role for the controller and in- tegrator units, despite the fact that each subset of units is composed of only 10 amongst 650 total hidden units. Also, the putative controller and integrator sets appear to have distinctive roles within the WLSTM, with the controllers supporting accurate predictions overall, while the integrator units appear to boost accurate predictions at the end of sentences.
A.5 MAPPING THE TIMESCALE ORGANIZATION IN A GRU LANGUAGE MODEL
A.5.1 TRAINING
To explore whether the timescale mapping methods, and our findings, may generalize other model architectures, we trained and studied word-level GRU language model (Cho et al. 2014). As far as possible, we applied similar parameters in the GRU as were used for the LSTM by Gulordava et al. (2018:: the same Wikipedia training corpus, the same loss function (i.e. cross- cross-entropy loss), and the same hyperparameters except for learning rate initialized 0. which we found more optimal to train the GRU. The GRU model also had two layers, with 650 hidden units in each layer.
We trained the GRU model for 30 epochs, at which point the GRU converged to a validation per- plexity of 1 118.36. Note that since we adapted similar training settings were used for training the LSTM model by Gulordava al. without model-specific optimization, the perplexity is higher than that of the LSTM model reported in Gulordava et al. (2018) (perplexity = 52.1 in the English corpora, after training for 40 epochs and selecting the model with the lowest perplexity out of 68 combinations of different hyperparamters). We then analyzed the timescale of its hidden units using the same method as was used for analyzing the LSTMs, and using the test data derived from the training Wikipedia corpus.
A.5.2 TIMESCALE ORGANIZATION OF A GRU MODEL
Similar to the LSTM model of Gulordova et al, the majority of the units in the GRU also showed shorter timescales. More specifically, we found: (1) the second layer of the GRU model was more sensitive to prior context than the first layer, as in the I.STM (A. A): (2) the distribution of timescales across units was similar in the GRU and LSTM, although the GRU showed a more right-skewed distribution with larger proportion of short-timescale units A. B, C).
A.5.3 TIMESCALE VERSUS NETWORK CONNECTIVITY IN A GRU MODEL
We also performed the timescale vs. network connectivity analyses on the GRU model. Because the update of hidden states in GRU are controlled by the reset and update gate, we measured the projec- tion patterns of hidden units by analyzing the matrix of combined mndden-to-update- -ate and hidden- to-reset-gate weights. contrast to the LSTM models, hidden units in the GRU that we trained did not show a relationship between longer timescales and stronger hidden-to-gate projections (A.8A). Moreover, when using k-core analysis to identify subumits of interconnected high-degree units, the core network in the GRU contained many units with long to short timescales. Interestingly, when we visualized the position of the k-core units in the MDS space, they tended to locate at the edge of the space, similar what found in LST. This indicates that, in the LSTM, the core units
14

---

Published as a conference paper at ICLR 2021
in the GRU have distinctive profiles, distant from one another and from other units in the network A. However, we did not observe the pattern of "integrator units" in the GRU as in the LSTM.
These apparent similarities and differences between LSTM and GRU are intriguing, but we empha- size that (1) the perplexity of this GRU model is much higher than the LSTM, due to the sub-optimal parameter settings, and that (2) comparing the LSTM and GRU connection patterns is not straight- forward, as the overall distribution of weights is different. Further work will be required to determine comparable thresholds for "strong" projections and "high-degree units" in each case. As noted in the manuscript and above, the connectivity results are exploratory; however, we believe that the GRU analysis demonstrates how these methods can be extended to map and compare the functional organization of language models of different architecture.
Finally, we note that when conducting the timescale analysis on an incompletely trained GRU model (trained ~ 10 epochs. validation perplexity ~ 350), the timescale distribution was more right-skewed (Figure A.6B) than the better-trained GRU (Figure (A.7 Altogether, these results suggest that the long-timescale units in GRU were gradually formed during the training process.
A.6 MAPPING THE TIMESCALE ORGANIZATION IN A WORD-LEVEL LSTM WITH DIFFERENT IIDDEN SIZE
To examine whether the number of hidden units in the model would affect the timescale organization in an LSTM, we trained another 2-layer word-level LSTM model with the same Wikipedia corpus and similar parameter settings as in Gulordava al. 2018), , but with only 100 hidden units in each layer. We called this model LSTM-100, We trained the model for 56 epochs until the model converged a validation perplexity 98. 75, and conducted the same analysis as described in the main text to map the timescales of LSTM-100. Because 1SST--000 have overall less weight connections, we use |z-score>> 3 as criteria to determine the "strong" hidden-to-ger -o-gaee projections for connectivity analyses.
Regarding the timescale distribution in LSTM-100, we found that the results were similar to the 650-unit word-level LSTM model, in that: (1) the second layer of LSTM-100 showed more context sensitivity than the first layer, and (2) although it was difficult to quantitatively compare the unit- level timescale distribution between the LSTM-100 model and the LSTM with 650 units, they both contain a similarly small subset of long-tinn timescale units. (A.).
We did not observe significant correlation between the unit timescale and number of strong projec- tions generated by each unit in the LSTM-100 model: the long-timescall units the LSTM-100 did not have connections than short-timescale units. When visualizing the MDS space of connec- tivity similarity of LS0M- the "controller units" identified using the k-core analysis were located in the edge of the space, similar to the 650-unit LSTTM model. Interestingly, we observed subset of -imeescle units in the center of the MDS space, analogous to the "integrator units" found in the 650-unit LSTM model. Altogether, the pattern of "integrator units" might be a commonly evolved feature that is shared between LSTM model architectures, but not with GRU architectures.
15

---

Published as a conference paper at ICLR 2021
WLSTM
CLSTM
1.4
1.4
Unit 1150 ("syntax' unit)
Unit 283
1.2
Unit 776 ("plural' unit)
Intact-Randomt activation difference
Unit 823
Intact-Random
1.0
Unit 988 ("singular" unit)
1.0
Unit 256
0.8
0.8
activation
0.6
0.6
0.4
0.4
0.2
0.2
0.0
-10
0
24 (word)
Different context
Shared segment
Different context
Shared segment (character)
|Intact vs. Randomi
(Intact vs. Random)
Figure A.1: Example single units activation differences across tokens in word-level LSTM (WL- STM) and character-level LSTM (CLSTM) language models. A. Single units activation differences across tokens in the WLSTM, with logistic curve fits overlaid. The error bars indicate 95% con- fidence interval across trials. The example units include the functional units reported by (Lakretz) et al., 2019). B. Example units activation differences in CLSTM model. The three units were ran- domly selected from long-, medium- and thors-timescales units.
A
B
short
long
1.75
Timescale
1.4
activation difference
1.50
activation difference
Intact- Intact-Randon Random
1.25
Intact Intact-Randon Random
1.0
1.00
0.8
0.6
0.50
0.4
0.25
0.00
-10
0
-33
Difference context (Intact vs. Random)
Shared input
(word)
Difference context (Intact Random)
Shared input (character)
Figure A.2: Logistic fitted curves of activation difference in individual units in word-level and character-level LSTM language model, colored by timescale. A. Logistic fitted curves of activation difference over time of all units in word-level LSTM. The color indicates the timescale measured by full-width half-maximum (FWHM) of the curve. B. Logistic fitted curves of activation difference time of all units in character-level LSTM.
16

---

Published as a conference paper at ICLR 2021
941
1.0
771
Normalized timescale
0..8
1125
1190
-0.6
1287
1150
0..
0.2
760
1102
0.0
1183
Figure A.3: The core network consists of "controller units" identified in word-level LSTT model.
17

---

Published as a conference paper at ICLR 2021
A
B
LINE teranca (eeaeto the 195 150 teteen sentenres)
de HAlkppdio TEsT
1comontt
17 16
0.1 6
17
timescale (Anna Karenine
timescale
(etrtestisshhee wagetme
liom
D
14
114
UntEmocaa Smscoaa (WCC0S))
190
Sconteen saging
OWEE
2
10 12 16 18 UER Bieescale (wands)
(conboxt.oom Shorce Seger enteddlddll www.erteddedoooooo
"comma")
Figure A.4: Mapped timescales across different datasets and context conditions. A. Timescales mea- sured using Wikipedia test set Gulordava al. (2018) are highly correlated with the timescales measured using Anna Karenina (r=0.82, p=00.00). B. Timescales measured in the middle of sen- tence (based on the token number) are highly correlated with timescales measured at the conjunction of sentence(", and") (r=0.83, 0<0.001). C. Timescales measured between sentences (i.e., using "full stop" to segment context and shared segments), vs. within sentence (i.e., using "comma" to segment context and shared segments). Most of the units showed little context dependence when the segmentation performed at the beginning of a sentence, suggesting a "reset" context representa- tion in these units. D. Timescales measured by replacing the context with syntactically structured segments vs. with the shuffled context tokens. Most of the units showed similar timescales under the conditions, and several units showed longer timescales (i.e. preserved more context) when the context segments were syntactically structured.
18

---

Published as a conference paper at ICLR 2021
331
10
40
216
02
125
530
Figure A.5: Timescale and connectivity organization in character-level LSTM (CLSTM). A. Longer-timen timescale units have overall stronger projections from the hidden units at time to the forget gates and input gates at time /++1 1. B. The main network (k: 4) formed by the controller units in the CLSTM. C. The multidimensional scaling space of the hidden-to-gate connection pattern of all units. The distance between nodes indicates their hidden-to-gate connection similarity; the size of the node indicates the number of strong projections from the node; and the line between two nodes indicates significant projections between them.
Timescale organization of an incompletely trained GRU network
A
over
Long
1.0
0.8
Intec--lundimm
0.4
0.2
Unit sorted by timescale
stase levl
Merrealtoo difference
Offferent context
Shared
250
GRU timescale distribution
200
150
Number
100
Short timescale
8
12
15
18
-10
0
24
Unit timescale
Diffeent contest
Shered input
Figure A.6: Mapped timescale organization in an incompletely trained GRU language model. A. Correlation curves of GRU hidden-state vectors across the Intact Context condition and Random Context condition as function of input token. Similar to the word level LSTM, the second layer of the GRU model was more sensitive to prior context than the first layer. Both GRU and LSTM models showed sparse long-timen aiggtimesccaee units compared to rich short-timeseas unit timescale distribution in the incompletely trained GRU is more right-skeeed, indicating that the long-time timescaee units have not emerged due to insufficient training. C. Absolute activation difference for each second-layer (incompletely trained) GRU unit over time.
units. However, the
19

---

Published as a conference paper at ICLR 2021
A
C
1.0
Unit activation difference over time
Comelation
Long timescale
0.6
Intact-famdem
0.4
0.2
sorted tmesscaee
hidden state level state level
0.0
actvvetton diffreen Inferencc
-10
24
Diffeent context
input
100
LSTM timescale distriibution
timescale distribution
60
60
40
20
Short timescale
0
6
10
12
14
16
18
20
100
24
Unit timescale
Different context
Figure A.7: Mapped timescale organization a GRU language model. A. Correlation curves of the GRU hidden-state vectors across the Intact Context condition and Random Context condition function of input token. The second layer of the GRU model was more sensitive to prior context than the first layer. B. Distributions of unit timescales are similar GRU in LSTM. Both models showed sparse long-timg long-timescale units compared to rich short-timescale units. C. Absolute activation difference for each second-laye layer GRU unit over time.
20

---

Published as a conference paper at ICLR 2021
A
B
hidden update projection
resat gate projection
1004
3007
Connoee and sosh Prte by 100000
Normalzzed timescale
0.8
08
1071
1
2
3
784
Number strong projections from hiddon to gate units
Figure A.8: Timescale-connectiivee analyses in GRU language model. A. Different from LSTM, the GRU we analyzed did not show the pattern that units with longer timescales exhibited more strong projections (r=-0.10, p=0.01). B. Multi-dimensional scaling (MDS) of connectivity similarity in GRU. The size of each node indicates its degree. An edge between nodes indicates significant hidden-to to-gee projection between them. The "controller units" identified in GRU using k-core analysis (labeled on the graph) tend to locate at the edge of the MDS space, similar to the LSTM. However, we did not observe the "integrator units" in the center of the MDS space. The long- timescale units in the GRU are more distributed in the MDS space compared to those in the LSTM.
21

---

Published as a conference paper at ICLR 2021
A
C
1.0
Unit activation difference over time
mmscctaaadoo Correction
Long timescale
0.6
0.7
state
Unit sorted by limescale
sicivcttion difter difference ennc
Different
Shared Input
B
LSTM-650 distribution
100
LSTM 100 timescale distribution
60
Number
20
Short
0
2
10
12
14
16
18
20
-10
24
Unit timescale
Different context
Shared Input
Figure A.9: Mapped timescale organization in an LSTM language model with 100 hidden units per layer (LSTM-100), A. Correlation curves of the LSTM cell-state vectors across the Intact Context condition and Random Context condition as function of input token. Similar to the word-level LSTM reported in the main text, the second layer of the LSTM-100 model was more sensitive to prior context than the first layer. B. Distributions of unit timescales in LSTM with 100 units and in LSTM with 650 units. Both models showed sparse long-timescale units compared to rich short- timescale units. C. Absolute activation difference over time of LSTM-100 100 second-layer units sorted by timescale.
22

---

Published as a conference paper at ICLR 2021
A
B
Hidden to inpul gate projection
200
forgei projection
Controliar units
2344
134
Bauc unte cont D9 Aimneeaa
279
Noralized timescale
135
129
strong projections from hidden to gate units
Figure A.10: Timescale-comeectvvin results in an LSTM language model with 100 hidden units per layer (LSTM-100). Different from the L.STM with 650 units, LSTM-100 did not show the pattern that units with longer timescales exhibited more strong projections (r=00.05, p=0.64). B. Multi- dimensional scaling (MDS) results LSTM-100, Similar to LSTM with 650 units, the "controller units" identified in LSTM-100 using k-core analysis (labeled red on the graph) tend to locate the edge of the MDS space, similar to the LSTM. Furthermore, there are several song-timescale units located in the center of the MDS, analogous to the "integrator units" in the LSTM with 650 units.
23


Reviews:
[{"review_id": "g6fkLwE-cm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper looks at LSTMs with the intention of understanding their functional connectivity. I am not sure exactly what the relationship between the brain and LSTMs is being assumed or proposed herein — however I understand the need to understand complex neural networks regardless of their relationship to biological systems. \n\nI would have liked to have a discussion with respect to what the hierarchical organisation is due to. Is this merely a repercussion of the connectivity, for example? What do the authors think?\n \nIn terms of work that looks at ablation (i.e., damage), it might be useful to bear in mind limitations of such work if various (seemingly, perhaps) extraneous factors are not taken into account, see: https://doi.org/10.1007/s42113-020-00081-z\n\nI think this paper can be polished to the level of a solidly good paper if the authors can sketch out a bit more their rationale and syllogisms with respect to my above questions. \n\nMinor:\n* Figures are very hard to read, is it possible to redesign them slightly to make the text bigger? \n* In LaTeX to open double quotes you need to use two backticks. Also the \\cite and \\citep commands should be used appropriately in terms of places where \\citep is needed as well as use of optional arguments to avoid double parentheses.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YsssyVYMlpr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper applies tools from neuroscience to understand how language models integrate across time. The basic approach is to present a phrase, preceded by two different context phrases: one that is natural (i.e. the phrase that actually preceded it in the corpus) and one that is randomly selected. The authors then measure how long it takes for the unit activations to become similar for the two different contexts, which provides a measure for how long the context impacts the representation. They find that (1) timescales increase at later layers of the language model (2) that only a small fraction of units exhibit long timescales (3) that long/medium-timescale units appear to come in two forms which they try and characterize using graph-style analyses. \n\n--\n\nPros:\n\nHow language models integrate across time is clearly important, and this paper describes interesting first steps in characterizing the analysis of time using relevant tools from the neuroscience literature. \n\nThe method presented is simple and broadly applicable. \n\nThe graph-style results seem intriguing if a little hard to make sense of.  I also think that the sparsity of the long-timescale units is cool and interesting.\n\n--\n\nLimitations and questions:\n\n1.\tIt’s not clear to me if the notion of time is a meaningful one in a language model. For example, the duration of contextual effects on a unit that codes syntactic number will presumably be highly variable and depend upon the details of the particular sentence being encoded. Thus a natural question is how variable are these timescales from moment-to-moment? What’s being plotted is the average across a bunch of sentences, segmented at a particular moment (a conjunction). How robust are these results if one examines a different point in a sentence? Are the timescales of some units more variable than others? -- Update: the authors have repeated their analysis for a different sentence point (after the 10th word) and report similar results. This analysis is helpful, though of course the 10th word is not a very principled break point, and there presumably is a lot of variation in timescales that are being averaged across. I continue to wonder how meaningful the notion of an absolute timescale is. -- \n\n2.\tNone of the steps in the graph analyses seemed particularly natural or well-motivated to me. Why were the graph edges thresholded at z>5 and why was k-core analysis performed? I find it hard to make sense of what this analysis tells us about how language information is processed. Is there some reason why medium timescale “controller” units and long-timescale “integrator” units should help with language processing? If these results are purely exploratory and lack a clear interpretation, then perhaps the authors could help the reader by explaining the thought process behind the exploration. Perhaps starting with the MDS plot would be useful rather than the k-core analysis, because the MDS plot clearly shows some interesting structure. -- The authors have motivated some of their analyses by discussing brain research reporting that longer-timescale regions are more densely connected. Of course, the relationship between connectivity between large-scale brain regions and the units in a LSTM remains highly speculative. But having some motivation is helpful. --\n\n3.\tIt would be interesting to know how dependent these findings are on the model’s architecture. Would similar results be found for a Transformer or a simpler GRU-style RNN? -- The authors have attempted to address this point, but with limited time were not able to train a network to a high level of performance. --\n\n--\n\nMinor points:\n\nIn Figure 4, it would be helpful if the absolute timescale was labeled in all plots rather than the rank of the unit or the “normalized timescale”. The absolute timescale seems much more meaningful to me (and the units can of course still be ranked, just the axis labels changed or augmented). \n\nThe legend for Figure 4c is incorrect. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "QxnC9gJQbz2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "_**Update after author response**_: I think this is a very promising paper, and I am really excited about seeing techniques from neuroscience employed to answer questions about neural network models. The authors have further conducted several additional experiments after reviewer comments, which I appreciate. However, my most fundamental concern -- the mismatch between the method and the way that it is validated -- unfortunately still stands, which is why I would encourage the authors to further pursue this line of work, but recommend to reject it for ICLR.\n\n**Summary**\n\nThis paper proposes to apply time-scale methods from neuroscience to investigate the timescale organisation in neural language models. More specifically, the authors test the timescale of individual units in a word- and character-level LSTM by comparing the units' activations values on the same sentence, but with different contexts. Using this method, the authors first  show that the higher layers on average have longer timescales. They then, for all units, they fit a logistic function to the \"recovery\" curves and use the half-times of this curves as an indication of the time scale of these units. They test the syntax unit and two long-distance units found by Lakretz et al and show that the number units have similar time-scales, while the syntax unit have a longer time scale. Lastly, the authors analyse the connectivity between the longer time scale units and find that the units with longer processing timescales make a larger number of strong projections. Within these units, the authors identify two sets of units in the word-level LSTM: \"controller units\", that play a role in how the connectivity of the network is updated, and \"integrator units\", that instead integrate information.\n\n**Strong points**\n- Neuroscience has long been asking questions about the brain that are very similar to the questions we now ask about neural networks, cross-pollination between these fields is extremely important, and this paper contributes to this\n- Aside from the main technique, the paper introduces some interesting and useful methods, such as projectivity analysis and k-core analysis. I think these methods can be useful for other researchers as well\n- Time scale analysis of LSTMs is a very relevant and interesting topic, that deserves more attention than it is currently getting\n\n*Concerns*\n- My main concern is that there seems to be a mismatch between the \"language time scales\" on which the authors operate: their experiment is designed to investigate the impact of extra-sentential context, but the Lakretz et al results they keep coming back to concern syntactic phenomena that are only relevant *within* a sentence, which is a different scale. In other words, the units found by the authors of this paper are long-distance when it comes to integrating context, but the syntax and number units found by Lakretz et al are not really related to that: they model relationships *within* sentences. Theoretically speaking, they should be reset at the beginning of every new sentence and they should thus be completely independent from the content. That the authors find this to be untrue is interesting, but inconsistent with what Lakretz et al describe these unit do. Since this is not addressed at all in the paper, it makes the results in general a bit difficult to interpret. _**Update after author response**: In their response the authors clarified that the they have only analysed single sentences, where two distinct subsentences are combined with a conjunction. This, unfortunately, does not make a difference for the argument: whether two sentences are split by a full stop or instead concatenated with \"and\" does not make any difference for the argument above, since the subject-verb agreement relationships that the units the authors look at model do not cross these boundaries either. Furthermore, in their response the authors state that the find that the context representations of units was 'reset' at sentence boundaries, as I asked before. I appreciate that the authors did these additional experiments, but I find the result somewhat worrisome: since the units they are looking at are syntactic units that encode number across long distance subject verb relationships, they should be reset both when a new sentence starts, as well as when a new conjunct with a new relationship starts. In terms of SV relationships, there should be no difference between \"The boy kicked the ball and the girl caught it\" and \"The boy kicked the ball. The girl caught it.\" That the authors do find a difference points to a potential flaw in methodology._\n\n- Relatedly, the authors say that  their result that the syntax unit is a long distance unit, while the number units are not. This is not consistent with what they say in the related work of the section, but also not with the results reported by Lakretz et al, who hypothesise that the syntax units represent the depth of the syntactic dependency. This is something that changes with every new incoming word, whereas the number units are the ones that have to keep their activation constant across time. \n\n- While, as I said before, I think it is great that the authors try to use methods from neuroscience into the field, I do think that in this case  the main method they propose is only very marginally different from earlier work (in particular Khandelwal et al. Perhaps it would make more sense to put a bit more stress on the rest of the methods as well (btw, also Lakretz et al do connectivity analysis).\n- The results are a bit underexplained, and understanding them requires many back and forths to the appendix. I would have appreciated a bit more motivated interpretation of several aspects. For instance: why is there such a large difference in activation differences in different units in the \"pre-shared segment\" part, and is this related to the half-time (it seems so from the plots)? What is the difference between character and word-level models in terms of expectations (we'd expect there to be an additional level of time-hierarchy, perhaps?) How do assessing activation differences and correlations differ in terms of conclusions? These things should, in my opinion, all be worked out a bit better.\n- Lastly, there are a few unsupported claims, the most important of which that their method recovers the previously discovered units of Lakretz et al, while (as far as I understand), they actually only *use* their method to analyse those neurons, but did not find them independently. (for other suggestions and comments, see below).\n\nTo summarise, while I think the idea is very nice and definitely worth working out further, I do think that some work is needed to make this a publishable paper.\n\n\n*Suggestions/comments for authors*\n_Typographic_:\n- If you use quotes in latex, you should use different ones for left (`) and right ('), for them to appear correctly (check for instance line three in the introduction)\n- To prevent additional spaces after abbreviations like e.g. and i.e., put a backslash: \"e.g.\\ \"\n- Lerner et al --> put all references within parenthesis\n- Introduction switches from present tense to paste tense in the last paragraph\n- \"we measure the time-taken for the effect of this prior context to ”decay” (see Methods)\" --> I don't really understand what this means, you measure how long it takes for these changes to not be measurable anymore?\n- Try to avoid double parethesis with abbreviations, e.g.: (WLSTM Gulordava et al. (2018)) should be: (WLSTM, Gulordava et al; 2018). You can do this with \\citep[text before][text after]{citation}.\n- \"has an 650-dimensional\" --> \"has a 650-dimensional\"\n- \"without fine-tuning to the novel\" --> I first thought this sentence was unfinished until I read back and realised that \"the novel\" is your corpus. This is a bit confusing perhaps you could rephrase.\n- \"how the cell state activation differ\" --> \"how the cell state activations differ\"\n- \"we will see that the activation difference drop quickly' --> drops quickly / see the activation difference drop quickly\n- There are several references that were published at ACL* conferences that are listed as arxiv papers in the reference list (Lakretz et al, Gulordava et al, Khandelwal et al)\n\n_Content_\n- I would say that the conclusion that \"Overall, prior works suggests that a small subset of units track long-range dependencies\" is rather overstated: Lakretz et al found that the units representing long distance number information were sparse, but this does not imply that long range information in general is represented sparsely. Their method also focusses quite exclusively on finding sparsely distributed properties, as more distributed properties cannot be found with ablation. Furthermore, this is just one study, focusing on one syntactic aspect. I would suggest to rephrase this a bit.\n- Lakretz at all actually identified several syntax units, but only one of them was interpretable.\n- I find it a bit confusing that in 3.2, second paragraph, you first talk about comparing cell state activation, then say that you compare hidden state activations and then talk again about the cell state activation\n\n- Figure 1 C & D: I don't think these figures add much to the paper, for the following reasons i) They show only individual units and no average, making it difficult to interpret the values ii) while, as pointed out in 5.1, the *rate* of decay is the most important, the cut-off point is not indicated in the figure, which puts a stress on irrelevant aspects: the actual difference between the two lines.\n- I would appreciate to have Figure A.1 in the main text, it is important for the story.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PQyeL7IFvVL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the application of innovative methods to track the flow of linguistic information in LSTM language models. In particular, the overarching question is how contextual information might be encoded in the network at the level of single units, and how context disruption might alter the LSTM dynamics and thus impact its predictive ability.\nThe paper is clear and it tackles an interesting question. The approach is well motivated, and the authors give a brief survey of the most recent applications of this kind of methodology in linguistics and cognitive neuroscience studies.\nThe methodology is generally appropriate, though some details and parameters (e.g., numerical thresholds) seem to be chosen arbitrarily. Also, the analysis could be improved by applying statistical testing in order to better quantify the strength of the observed effects.\nOverall, I think this is a nice paper, though it might be especially relevant to the linguistics community rather than to the ICLR community. Moreover, I think that further analyses are required in order to better clarify some important aspects. In particular, I think that ablation studies should be performed in order to better identify the functional role of the “controller” and “integrator” units, whose actual functional role remains a bit speculative (and mostly based on structural / connectivity information). It would also strengthen the paper to have some more controlled simulations, where the contextual information is defined according to specific linguistic constraints, in order to better characterize what the target units are actually encoding. Indeed, as also noted by the authors almost “all the long timescale units are of unknown function”. Finally, I think that it would be important to establish whether these findings are generally applicable to LSTM models, regardless of the specific architecture under investigation (e.g., What happens if we force the LSTM to rely on fewer units? Does the hierarchical organization of the context improve by adding more layers?).\n\nOther comments:\n- Why did the author choose to test the model on a different corpus (Anna Karenina novel) rather than considering a test set from the same corpus from which the training set was derived? The Tolstoy book might have a quite different linguistic structure from that of the corpora used to train the LSTMs.\n- It might be informative to also include a third condition in-between “Intact” and “Random” context, where the same context words are maintained with scrambled order. This would allow to better understand the role of individual words in shaping context representation and activating the LSTM units.\n- In Fig. 1D, it is interesting to note that the Unit 823 (green line) actually exhibits a sharp increase in difference after the shared segment starts. Do the authors have a possible explanation for this kind of phenomena? Was it observed systematically in other units?\n- In relation to the results shown in Fig. 3A, I did not understand how the thresholds and parameters for the k-core analysis were chosen.\n- Pg. 3: there is a typo regarding the size of the output layer (5,0000)\n- In Fig. A1, error bars would help in better understanding the actual difference between the curves.\n- In order to improve reproducibility, it would be very helpful to share the source code used for these analyses.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies methods inspired by neuroscience to analyze the inner workings of LSTM language models. In particular, a simple and clever approach is proposed, in which a sentence is presented in its observed context vs. a random one. The time for a unit activation to become similar in the two contexts is used as a probe of the timescale of contextual effects. The main results are that timescales increase with layer and that there are two classes of long-timescale units with different graph-theoretical properties. The functionality of syntax-sensitive units previously identified in the literature is confirmed. Finally, the analysis is replicated for a character-level model.

The paper received detailed and insightful reviews, and there was a lively (but always respectful) discussion between authors and reviewers.

Overall, the reviewers liked the topic of the paper and the overall methodology, however they had several issues with it. One of the issue pertained to the "holistic" approach to time in the paper, which is measured in number of tokens, rather than in terms of syntactic distance. More in general, there was a feeling that the paper was somewhat short on actual insights on the exact functional role of units in a linguistic context. The reviewer who assigned the most severe score was mostly concerned about one specific instance of this, namely the fact that the authors focus on syntax-tracking and number agreement units whose scope should not really extend across sentences. Moreover, the reviewer was surprised that the syntax-tracking units maintain information across longer distances than the number-agreement units, that should, by definition, keep track of long-distance relations.

I am divided. I welcome work that focuses on novel qualitative and quantitative analyses of an existing model. I wished there were clearer take-home messages on how LSTMs process language, but I recognize that our knowledge of deep-learning models is very preliminary, and I am thus not surprised that the conclusions are not entirely clear.  The reviewers raised important concerns, but I would not confidently claim that we know enough about the relevant units to be genuinely surprised by some of the results. For example, can we really say that number-agreement units are only limited to clause-internal agreement tracking? Couldn't it be, say, that we will discover in the future they also play a role in tracking discourse-determined pronominal number (going out on a random limb, here, of course)?

Overall, I would like to see this at least as a poster at the conference, but I am assigning low confidence to my recommendation as I respect the reviewers' point of view.

Decision: Accept (Poster)