Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper looks at LSTMs with the intention of understanding their functional connectivity. I am not sure exactly what the relationship between the brain and LSTMs is being assumed or proposed herein — however I understand the need to understand complex neural networks regardless of their relationship to biological systems. 

I would have liked to have a discussion with respect to what the hierarchical organisation is due to. Is this merely a repercussion of the connectivity, for example? What do the authors think?
 
In terms of work that looks at ablation (i.e., damage), it might be useful to bear in mind limitations of such work if various (seemingly, perhaps) extraneous factors are not taken into account, see: https://doi.org/10.1007/s42113-020-00081-z

I think this paper can be polished to the level of a solidly good paper if the authors can sketch out a bit more their rationale and syllogisms with respect to my above questions. 

Minor:
* Figures are very hard to read, is it possible to redesign them slightly to make the text bigger? 
* In LaTeX to open double quotes you need to use two backticks. Also the \cite and \citep commands should be used appropriately in terms of places where \citep is needed as well as use of optional arguments to avoid double parentheses.  

review_title: Towards understanding the internal organisation of LSTMs

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper applies tools from neuroscience to understand how language models integrate across time. The basic approach is to present a phrase, preceded by two different context phrases: one that is natural (i.e. the phrase that actually preceded it in the corpus) and one that is randomly selected. The authors then measure how long it takes for the unit activations to become similar for the two different contexts, which provides a measure for how long the context impacts the representation. They find that (1) timescales increase at later layers of the language model (2) that only a small fraction of units exhibit long timescales (3) that long/medium-timescale units appear to come in two forms which they try and characterize using graph-style analyses. 

--

Pros:

How language models integrate across time is clearly important, and this paper describes interesting first steps in characterizing the analysis of time using relevant tools from the neuroscience literature. 

The method presented is simple and broadly applicable. 

The graph-style results seem intriguing if a little hard to make sense of.  I also think that the sparsity of the long-timescale units is cool and interesting.

--

Limitations and questions:

1.	It’s not clear to me if the notion of time is a meaningful one in a language model. For example, the duration of contextual effects on a unit that codes syntactic number will presumably be highly variable and depend upon the details of the particular sentence being encoded. Thus a natural question is how variable are these timescales from moment-to-moment? What’s being plotted is the average across a bunch of sentences, segmented at a particular moment (a conjunction). How robust are these results if one examines a different point in a sentence? Are the timescales of some units more variable than others? -- Update: the authors have repeated their analysis for a different sentence point (after the 10th word) and report similar results. This analysis is helpful, though of course the 10th word is not a very principled break point, and there presumably is a lot of variation in timescales that are being averaged across. I continue to wonder how meaningful the notion of an absolute timescale is. -- 

2.	None of the steps in the graph analyses seemed particularly natural or well-motivated to me. Why were the graph edges thresholded at z>5 and why was k-core analysis performed? I find it hard to make sense of what this analysis tells us about how language information is processed. Is there some reason why medium timescale “controller” units and long-timescale “integrator” units should help with language processing? If these results are purely exploratory and lack a clear interpretation, then perhaps the authors could help the reader by explaining the thought process behind the exploration. Perhaps starting with the MDS plot would be useful rather than the k-core analysis, because the MDS plot clearly shows some interesting structure. -- The authors have motivated some of their analyses by discussing brain research reporting that longer-timescale regions are more densely connected. Of course, the relationship between connectivity between large-scale brain regions and the units in a LSTM remains highly speculative. But having some motivation is helpful. --

3.	It would be interesting to know how dependent these findings are on the model’s architecture. Would similar results be found for a Transformer or a simpler GRU-style RNN? -- The authors have attempted to address this point, but with limited time were not able to train a network to a high level of performance. --

--

Minor points:

In Figure 4, it would be helpful if the absolute timescale was labeled in all plots rather than the rank of the unit or the “normalized timescale”. The absolute timescale seems much more meaningful to me (and the units can of course still be ranked, just the axis labels changed or augmented). 

The legend for Figure 4c is incorrect. 
review_title: Exploratory analyses of timescales in neural language models

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper explores the application of innovative methods to track the flow of linguistic information in LSTM language models. In particular, the overarching question is how contextual information might be encoded in the network at the level of single units, and how context disruption might alter the LSTM dynamics and thus impact its predictive ability.
The paper is clear and it tackles an interesting question. The approach is well motivated, and the authors give a brief survey of the most recent applications of this kind of methodology in linguistics and cognitive neuroscience studies.
The methodology is generally appropriate, though some details and parameters (e.g., numerical thresholds) seem to be chosen arbitrarily. Also, the analysis could be improved by applying statistical testing in order to better quantify the strength of the observed effects.
Overall, I think this is a nice paper, though it might be especially relevant to the linguistics community rather than to the ICLR community. Moreover, I think that further analyses are required in order to better clarify some important aspects. In particular, I think that ablation studies should be performed in order to better identify the functional role of the “controller” and “integrator” units, whose actual functional role remains a bit speculative (and mostly based on structural / connectivity information). It would also strengthen the paper to have some more controlled simulations, where the contextual information is defined according to specific linguistic constraints, in order to better characterize what the target units are actually encoding. Indeed, as also noted by the authors almost “all the long timescale units are of unknown function”. Finally, I think that it would be important to establish whether these findings are generally applicable to LSTM models, regardless of the specific architecture under investigation (e.g., What happens if we force the LSTM to rely on fewer units? Does the hierarchical organization of the context improve by adding more layers?).

Other comments:
- Why did the author choose to test the model on a different corpus (Anna Karenina novel) rather than considering a test set from the same corpus from which the training set was derived? The Tolstoy book might have a quite different linguistic structure from that of the corpora used to train the LSTMs.
- It might be informative to also include a third condition in-between “Intact” and “Random” context, where the same context words are maintained with scrambled order. This would allow to better understand the role of individual words in shaping context representation and activating the LSTM units.
- In Fig. 1D, it is interesting to note that the Unit 823 (green line) actually exhibits a sharp increase in difference after the shared segment starts. Do the authors have a possible explanation for this kind of phenomena? Was it observed systematically in other units?
- In relation to the results shown in Fig. 3A, I did not understand how the thresholds and parameters for the k-core analysis were chosen.
- Pg. 3: there is a typo regarding the size of the output layer (5,0000)
- In Fig. A1, error bars would help in better understanding the actual difference between the curves.
- In order to improve reproducibility, it would be very helpful to share the source code used for these analyses.

review_title: A promising approach to explore the emergent structure of LSTM models, which needs further development to shed light on emergent function

final_score: rating: 3: Clear rejection
review_content: _**Update after author response**_: I think this is a very promising paper, and I am really excited about seeing techniques from neuroscience employed to answer questions about neural network models. The authors have further conducted several additional experiments after reviewer comments, which I appreciate. However, my most fundamental concern -- the mismatch between the method and the way that it is validated -- unfortunately still stands, which is why I would encourage the authors to further pursue this line of work, but recommend to reject it for ICLR.

**Summary**

This paper proposes to apply time-scale methods from neuroscience to investigate the timescale organisation in neural language models. More specifically, the authors test the timescale of individual units in a word- and character-level LSTM by comparing the units' activations values on the same sentence, but with different contexts. Using this method, the authors first  show that the higher layers on average have longer timescales. They then, for all units, they fit a logistic function to the "recovery" curves and use the half-times of this curves as an indication of the time scale of these units. They test the syntax unit and two long-distance units found by Lakretz et al and show that the number units have similar time-scales, while the syntax unit have a longer time scale. Lastly, the authors analyse the connectivity between the longer time scale units and find that the units with longer processing timescales make a larger number of strong projections. Within these units, the authors identify two sets of units in the word-level LSTM: "controller units", that play a role in how the connectivity of the network is updated, and "integrator units", that instead integrate information.

**Strong points**
- Neuroscience has long been asking questions about the brain that are very similar to the questions we now ask about neural networks, cross-pollination between these fields is extremely important, and this paper contributes to this
- Aside from the main technique, the paper introduces some interesting and useful methods, such as projectivity analysis and k-core analysis. I think these methods can be useful for other researchers as well
- Time scale analysis of LSTMs is a very relevant and interesting topic, that deserves more attention than it is currently getting

*Concerns*
- My main concern is that there seems to be a mismatch between the "language time scales" on which the authors operate: their experiment is designed to investigate the impact of extra-sentential context, but the Lakretz et al results they keep coming back to concern syntactic phenomena that are only relevant *within* a sentence, which is a different scale. In other words, the units found by the authors of this paper are long-distance when it comes to integrating context, but the syntax and number units found by Lakretz et al are not really related to that: they model relationships *within* sentences. Theoretically speaking, they should be reset at the beginning of every new sentence and they should thus be completely independent from the content. That the authors find this to be untrue is interesting, but inconsistent with what Lakretz et al describe these unit do. Since this is not addressed at all in the paper, it makes the results in general a bit difficult to interpret. _**Update after author response**: In their response the authors clarified that the they have only analysed single sentences, where two distinct subsentences are combined with a conjunction. This, unfortunately, does not make a difference for the argument: whether two sentences are split by a full stop or instead concatenated with "and" does not make any difference for the argument above, since the subject-verb agreement relationships that the units the authors look at model do not cross these boundaries either. Furthermore, in their response the authors state that the find that the context representations of units was 'reset' at sentence boundaries, as I asked before. I appreciate that the authors did these additional experiments, but I find the result somewhat worrisome: since the units they are looking at are syntactic units that encode number across long distance subject verb relationships, they should be reset both when a new sentence starts, as well as when a new conjunct with a new relationship starts. In terms of SV relationships, there should be no difference between "The boy kicked the ball and the girl caught it" and "The boy kicked the ball. The girl caught it." That the authors do find a difference points to a potential flaw in methodology._

- Relatedly, the authors say that  their result that the syntax unit is a long distance unit, while the number units are not. This is not consistent with what they say in the related work of the section, but also not with the results reported by Lakretz et al, who hypothesise that the syntax units represent the depth of the syntactic dependency. This is something that changes with every new incoming word, whereas the number units are the ones that have to keep their activation constant across time. 

- While, as I said before, I think it is great that the authors try to use methods from neuroscience into the field, I do think that in this case  the main method they propose is only very marginally different from earlier work (in particular Khandelwal et al. Perhaps it would make more sense to put a bit more stress on the rest of the methods as well (btw, also Lakretz et al do connectivity analysis).
- The results are a bit underexplained, and understanding them requires many back and forths to the appendix. I would have appreciated a bit more motivated interpretation of several aspects. For instance: why is there such a large difference in activation differences in different units in the "pre-shared segment" part, and is this related to the half-time (it seems so from the plots)? What is the difference between character and word-level models in terms of expectations (we'd expect there to be an additional level of time-hierarchy, perhaps?) How do assessing activation differences and correlations differ in terms of conclusions? These things should, in my opinion, all be worked out a bit better.
- Lastly, there are a few unsupported claims, the most important of which that their method recovers the previously discovered units of Lakretz et al, while (as far as I understand), they actually only *use* their method to analyse those neurons, but did not find them independently. (for other suggestions and comments, see below).

To summarise, while I think the idea is very nice and definitely worth working out further, I do think that some work is needed to make this a publishable paper.


*Suggestions/comments for authors*
_Typographic_:
- If you use quotes in latex, you should use different ones for left (`) and right ('), for them to appear correctly (check for instance line three in the introduction)
- To prevent additional spaces after abbreviations like e.g. and i.e., put a backslash: "e.g.\ "
- Lerner et al --> put all references within parenthesis
- Introduction switches from present tense to paste tense in the last paragraph
- "we measure the time-taken for the effect of this prior context to ”decay” (see Methods)" --> I don't really understand what this means, you measure how long it takes for these changes to not be measurable anymore?
- Try to avoid double parethesis with abbreviations, e.g.: (WLSTM Gulordava et al. (2018)) should be: (WLSTM, Gulordava et al; 2018). You can do this with \citep[text before][text after]{citation}.
- "has an 650-dimensional" --> "has a 650-dimensional"
- "without fine-tuning to the novel" --> I first thought this sentence was unfinished until I read back and realised that "the novel" is your corpus. This is a bit confusing perhaps you could rephrase.
- "how the cell state activation differ" --> "how the cell state activations differ"
- "we will see that the activation difference drop quickly' --> drops quickly / see the activation difference drop quickly
- There are several references that were published at ACL* conferences that are listed as arxiv papers in the reference list (Lakretz et al, Gulordava et al, Khandelwal et al)

_Content_
- I would say that the conclusion that "Overall, prior works suggests that a small subset of units track long-range dependencies" is rather overstated: Lakretz et al found that the units representing long distance number information were sparse, but this does not imply that long range information in general is represented sparsely. Their method also focusses quite exclusively on finding sparsely distributed properties, as more distributed properties cannot be found with ablation. Furthermore, this is just one study, focusing on one syntactic aspect. I would suggest to rephrase this a bit.
- Lakretz at all actually identified several syntax units, but only one of them was interpretable.
- I find it a bit confusing that in 3.2, second paragraph, you first talk about comparing cell state activation, then say that you compare hidden state activations and then talk again about the cell state activation

- Figure 1 C & D: I don't think these figures add much to the paper, for the following reasons i) They show only individual units and no average, making it difficult to interpret the values ii) while, as pointed out in 5.1, the *rate* of decay is the most important, the cut-off point is not indicated in the figure, which puts a stress on irrelevant aspects: the actual difference between the two lines.
- I would appreciate to have Figure A.1 in the main text, it is important for the story.
review_title: Great to see some methods from neuroscience applied to interpretability research for a relevant question, results and setup could be improved

Meta-review: This paper applies methods inspired by neuroscience to analyze the inner workings of LSTM language models. In particular, a simple and clever approach is proposed, in which a sentence is presented in its observed context vs. a random one. The time for a unit activation to become similar in the two contexts is used as a probe of the timescale of contextual effects. The main results are that timescales increase with layer and that there are two classes of long-timescale units with different graph-theoretical properties. The functionality of syntax-sensitive units previously identified in the literature is confirmed. Finally, the analysis is replicated for a character-level model.

The paper received detailed and insightful reviews, and there was a lively (but always respectful) discussion between authors and reviewers.

Overall, the reviewers liked the topic of the paper and the overall methodology, however they had several issues with it. One of the issue pertained to the "holistic" approach to time in the paper, which is measured in number of tokens, rather than in terms of syntactic distance. More in general, there was a feeling that the paper was somewhat short on actual insights on the exact functional role of units in a linguistic context. The reviewer who assigned the most severe score was mostly concerned about one specific instance of this, namely the fact that the authors focus on syntax-tracking and number agreement units whose scope should not really extend across sentences. Moreover, the reviewer was surprised that the syntax-tracking units maintain information across longer distances than the number-agreement units, that should, by definition, keep track of long-distance relations.

I am divided. I welcome work that focuses on novel qualitative and quantitative analyses of an existing model. I wished there were clearer take-home messages on how LSTMs process language, but I recognize that our knowledge of deep-learning models is very preliminary, and I am thus not surprised that the conclusions are not entirely clear.  The reviewers raised important concerns, but I would not confidently claim that we know enough about the relevant units to be genuinely surprised by some of the results. For example, can we really say that number-agreement units are only limited to clause-internal agreement tracking? Couldn't it be, say, that we will discover in the future they also play a role in tracking discourse-determined pronominal number (going out on a random limb, here, of course)?

Overall, I would like to see this at least as a poster at the conference, but I am assigning low confidence to my recommendation as I respect the reviewers' point of view.

Decision: Accept (Poster)