Short-term memory in neural language models
Authors: Kristijan Armeni
Christopher Honey
Tal Linzen
Venue: ICLR 2022 Submitted (2022)
Keywords: short-term memory
language models
transformer
lstm
GPT-2

Abstract: When a language model is trained to predict natural language sequences, its prediction at each moment depends on a representation of prior context. Thus, language models require mechanisms to maintain and access memory. Although we design the architectural features of these models, we do not know how their memory systems are functionally organized via learning: what kind of information about the prior context can they retrieve? We reasoned that access to arbitrary individual tokens from the past could be computationally powerful, akin to the  working memory which is important for flexible cognition in humans, and we therefore tested whether language models could ``retrieve'' the exact words that occurred previously in a text. In particular, we tested how the ability to retrieve prior words depended on (i) the number of words being retrieved, (ii) their semantic coherence, and (iii) the length and quality of the intervening text. We evaluated two particular architectures of neural language models: the attention-based transformer and the long short-term memory network (LSTM). In our paradigm, language models processed English text in which a list of nouns occurred twice. We operationalized retrieval as the reduction in surprisal from the first presentation of the list to its second presentation. We found that the transformer models retrieved both the identity and ordering of nouns from the first list. The transformer was successful even when the noun lists were semantically incoherent, and this effect was largely robust to the type or length of the intervening text. Further, the transformer’s retrieval was markedly enhanced when it was trained on a larger corpus and with greater model depth. Lastly, its ability to index prior tokens was dependent on learned attention patterns. In contrast, the LSTM models exhibited less precise retrieval (smaller reductions in surprisal). The LSTM’s retrieval was limited to list-initial tokens, and occurred only across short intervening texts. Moreover, the LSTM's retrieval was not sensitive to the order of nouns and this non-specific retrieval improved when the list was semantically coherent. In sum, the transformer, when trained to predict linguistic tokens, implements something akin to a working memory system, as it could flexibly retrieve individual token representations across arbitrary delays. Conversely, the LSTM maintained a coarser and more rapidly-decaying semantic gist of prior tokens, weighted heavily toward the earliest items.  Thus, although the transformer and LSTM architectures were both trained to predict language sequences, only the transformer learned to flexibly index prior tokens.

Under review as a conference paper at ICLR 2022
SHORT-TERM MEMORY IN NEURAL LANGUAGE MOD- ELS
Anonymou authors Paper under double-bling review
ABSTRACT
When a language model is trained to predict natural language sequences. its prediction each moment depends on a representation of prior context. Thus, language models require mechanisms to maintain and access memory. Although we design the architectural features of these models, we do not know how their memory systems are functionally organized via learning: what kind of information about the prior context can they retrieve? We reasoned that access to arbitrary individual tokens from the past could be computationally powerful, akin to the working memory which is important for flexible cognition in humans, and we therefore tested whether language models could "retrieve" the exact words that occurred previously in a text. In particular, we tested how the ability to retrieve prior words depended on (i) the number of words being retrieved, (ii) their semantic coherence, and (iii) the length and quality of the intervening text. We evaluated two particular architectures of neural language models: the attention-based transformer and the long short-term memory network (LSTM). In our paradigm, language models processed English text which a list of nouns occurred twice. We operationalized retrieval as the reduction in surprisal from the first presentation of the list to its second presentation. We found that the transformer models retrieved both the successful identity and ordering of nouns from the first list. The transformer even when the noun lists were semantically incoherent, and this effect was largely robust the type length of the intervening text. Further, the transformer's retrieval was markedly enhanced when it was trained on a larger corpus and with greater model depth. Lastly, its ability index prior tokens dependent on learned attention patterns. In contrast, the LSTM models exhibited less precise retrieval (smaller reductions in surprisal). The LSTM's retrieval was limited list-initial tokens, and occurred only across short intervening texts. Moreover, the LSTM's retrieval was not sensitive the order of nouns and this non- specific retrieval improved when the list was semantically coherent. In sum, the transformer, when trained predict linguistic tokens, implements something akin to a working memory system, as it could flexibly retrieve individual token representations across arbitrary delays. Conversely, the LSTM maintained a coarser and more rapidly- decaying semantic gist of prior tokens, weighted heavily toward the earliest items. Thus, although the transformer and LSTM architectures were both trained to predict language sequences, only the transformer learned to flexibly index prior tokens.
INTRODUCTION
Language models (LMs) are computational systems trained predict upcoming tokens based on past context. To perform this task well, they must construct a coherent representation of the text, which requires establishing relationships between words that occur at non-adjacent time points. Despite their simple learning objective, LMs based on contemporary artificial neural network architectures perform well on language processing tasks that require maintenance and retrieval of context spanning multiple words in a sentence. For example, LMs learn to correctly match the grammatical number of the subject and a corresponding verb across multiple intervening words; for example, they prefer the correct The girls standing at the desk are tall, to the incorrect The girls standing the desk tall (Linzen et al.) 2016 Marvin & Linzen 2018: Gulordava et al., 2018 Futrell et al., 2018). Moreover, language representations learned by large-scale LMS can be fine-tuned directly used

---

Under review as a conference paper at ICLR 2022
Memory for language in neural LMs
How detailed is LM memory of past nouns (token identity and ordering)?
How resilient is LM memory to size and content of intervening text?
How content-invariat LM memary past nouns during processing?
context shart-term memory
token retrieval
Intact
repeated Bragon
semantics: (arbitrary, coherent)
pillow
..*
justice
danger
scrambled
permutted
cos
W%
dragon
pillow ...
justice Banget
X
...
justice
aspect
dragon
pillow
BOG
onion
attack
prison
N
(26, 99, 194, 4353
N (3,5, 7, 10)
preface text
first list
intervening text
second list
LM input sequence
Figure 1: Assessing memory retrieval in neural language models. In language, memory of past context is required for ongoing comprehension. How are memory systems of neural LMs organized, via learning, for processing? In our paradigm, language models processed English text in which list of nouns occurred twice. We operationalized retrieval as the reduction in surprisal from the first presentation of the list its second presentation. The two lists of nouns were embedded in a vignette which consisted of a context string, first list, an intervening text, and the second noun list. To probe the properties of neural LM memory. we measured LM surprisal on the second list while varying: a) set size, b) the structure of the second list, c) the length of the intervening text, and d) the content and structure of the intervening text.
for solving downstream natural language processing (NLP) tasks (Devlin et al. 2019 Brown et al.) 2020). Together, these results suggest that LMs extract linguistically meaningful signals and that, over the course of learning, they develop a short-term memory capacity: the ability to store and access recent past context for processing. But what is the nature of the memory processes that LMs learn? Are these memory processes able access individual tokens from the recent past verbatin, or is the memory system more implicit, that only aggregate gisl of the prior context is available to subsequent processing? The ability to access arbitrary individual tokens from the past could be computationally powerful, akin to the "working memory" systems thought to enable flexible human cognitive capacities (Baddeley 2003).
In this work, we introduce paradigm for characterizing the short-term memory abilities of language model. We apply it to two particular architectures of neural language models that possess the architectural ingredients to hold past items in memory: attention based transformers (Vaswani let al., (017) and long short-term memory networks (Hochreiter & Schmidhuber, 1997] LSTM). Whereas LSTM incorporates the past by reusing the results of processing from previous time steps and through dedicated memory cells, the transformer receives the internal representations of each of the previous tokens as input. These architectural ingredients alone, however, are not sufficient for a model to have memory. Whether or not the model puts this memory capacity to use depends on whether the training task (next word prediction) requires it - the parameters controlling the activation of context representations and subsequent retrieval computations are in both cases learned.
Our goal was determine whether and when the LMs we study maintain and retrieve verbatim representations of individual prior tokens. First, we measured the detail of the context representation: does the LM maintain a verbatim representation of all prior tokens and their order, or does instead combine multiple prior tokens into summary representation, like semantic gist? Second, considered the resilience of the memory to interference: after how many intervening tokens do the representation of prior context become inaccessible? Third, we considered the the context representations: does the resilience of prior context depend on semantic coherence of the prior information, can arbitrary and unrelated information sequences be retrieved?
of
Our paradigm, which is inspired by benchmark tasks for models of human short-term memory (Oberauer al. 2018), reveals whether not and under what conditions the LM can encode and
2

---

Under review as a conference paper at ICLR 2022
retrieve lists of nouns embedded in vignettes (see Fig. D By comparing LM surprisal on repeated lists of nouns, we found that:
The transformers we tested can perform verbatim recall of past nouns and are robust to interference. The transformers retrieved precise representation of nouns (word identity and word order) from the context and did so over long-range intervening text (> 400 tokens).
The LSTM we tested stores a semantic gist of past nouns over short distances. The LSTM's noun list retrieval capacity was limited to semantic summary of the earlier nouns in the list, did not preserve the order or identity of the original nouns, and only succeeded across small number of intervening tokens.
Verbatim recall in transformers improves with training corpus size. The transformer trained on larger dataset showed much larger effects compared transformer trained on smaller corpus.
Verbatim recall in transformers requires depth. Memory retrieval capacity emerged with 6- and 12- layer models, but smaller and 3-layer models.
Order-sensitivity of transformers relies on learned attention patterns. After randomly permuting attention weights, the transformer no longer showed ordering retrieval capacity.
2 RELATED WORK
Previous studies examined how properties linguistic context influenced average LM performance. Khandelwal et al. (2018) showed that LSTM LMs use window of approximately 200 tokens of past context and word order information of the past 50 words, in the service of predicting the next token in natural language sequences. Subramanian al. (2020) applied similar analysis to a transformer LM and showed that LM loss on test-set sequences was not sensitive to context perturbations beyond 50 tokens. These studies focused on how specific features past context affect LM performance on novel input in the test set. In contrast, our paradigm tests for the ability of LMs to exactly retrieve nouns that are repeated from prior context.
METHODS
3.1 PARADIGM: LISTS NOUNS IN CONTEXT
Noun lists were embedded in brief vignettes (Figure I A and B). Each vignette opened with preface string (e.g. "Before the meeting, Mary wrote down the following list of words:"). This string was followed by list of nouns first list), which were separated by commas; the list-final noun was followed by a full stop (e.g. "county, muscle, vapor.). The first list was followed by an intervening text, which continued the narrative established by the preface string ("After the meeting, she took break and had a cup of coffee."). The intervening text was followed by short prompt string (c.g. "After she got back, she read the list again:") after which another list of nouns, either identical the first list or different from it, was presented (we refer to this list as the second list).
3.2 SEMANTIC COHERENCE OF NOUN LISTS
We used two types of word lists: arbitrary and semantically coherent (all word lists are shown in Tables [ the Appendix). Arbitrary word lists (e.g. "device, singer, picture") were composed of randomly sampled nouns from the Toronto word pool Semantically coherent word lists were sampled from the categorized noun word pool which contains 32 lists, each of which contains 32 semantically related nouns (e.g. "robin, sparrow, heron,.." ...").
We ensured that the nouns in each list were part of the LSTM's vocabulary. After ensuring there were at least 10 valid, in-oocabulary nouns per semantic set (as this was the maximal list length we considered), we were able to construct 23 nouns lists. Finally, to reduce the variance attributable to tokens occurring in specific positions, we generated 10 "folds" of each list by circularly shifting the
3

---

Under review as a conference paper at ICLR 2022
tokens in the first list 10 times. In this way, each noun in each list was tested in all possible ordinal positions. This procedure resulted in a total of 23 x 10 = 230 noun lists.
3.3 LANGUAGE MODELS
LSTM We used an LSTM released by van Schijndel et al. (2119) which had hidden layers with 400-dimensional hidden states each and had vocabulary of 28,439 entries. The model contained
25.3 million trainable parameters. It was trained on a 40-million word subset of the Wikitext- 103 corpus (Merity al., (016) and had test-set perplexity of 79.6. The model's vocabulary contained 28,439 entries. We selected this checkpoint as our starting point because van Schijndel et al.] (2019) showed that increasing the number of model parameters and size of training set beyond this point did not significantly improve the model's performance on the subject-verb agreement task. To test for the potential role of model/training set size on our task, we also evaluated larger, 2-layer LSTM with 1600-dimensional hidden states (132 million trainable parameters), trained by van Schijndel et al. (2019) on 80 million tokens which had a test-set perplexity of 108.5.
Transformer We trained a transformer LM on 40 million subword tokens of the Wikitext-103 benchmark (i.e. comparable to the corpus size used for training the LSTM checkpoint). We ensured that the vocabulary size was matched the size of the LSTM vocabulary (28,439 entries) by retraining the tokenizer (see Appendix section se.4ioo[A.4.2).. We trained both the 12-layer GPT-2 architecture (known "GPTT2 small", 107.7 million trainable parameters) and, point of comparison, smaller, 1-, 3-, and 6-layer transformers (29.7,4 43.9, and 65.2 million trainable parameters, respectively). The context window was set to 1024 tokens and embedding dimension was kept 768 across the architectures (see Section A..4 for training details). The perplexities for the 12-, 6-, 3- and I-layer models on the Wikitext- 103 test set were 40.3, 46.7, 60.1, and 93.2, respectively. Unless indicated otherwise, the label "Wikitext- 103 transformer" refers to the 12 12-layer checkpoint in the figures.
We also evaluated the pretrained transformer LM released by Radford al. (2019), which we refer to simply as GPT-2. This model was trained on the WebText corpus, which consists of approximately 8 million online documents. We used the GPT-2-small checkpoint which has 12 attention layers and 768-dimensional embedding layer. The model contains 124 million parameters and has a vocabulary of 50,257 entries. We used the maximum context size of 1024 tokens.
3.4 SURPRISAL
For each token we in our sequence, we computed the negative log likelihood (surprisal): surprisall (w;) = log, P(wa/wi.....W In cases when the transformer byte-pair encoding tokenizer split a noun into multiple tokens-e.2 "sparrow" might be split into "sp" and "arrow"- summed the surprisals of the resulting tokens.
Quantifying retrieval: repeat surprisal To quantify how the memory trace of the first list affected surprisal on the second list, we measured the ratio between the surprisal on the second list and the surprisal on the first list: repeat surpris sal s(La) 100 where ((T,) refers to mean surprisal
s(L1)
across non-initial nouns in the first list and s ((2)) to median surprisal across all non-initial nouns in
the second list. We take a reduction surprisal on second lists to indicate the extent which an LM has retrieved tokens from the first list.
4 TRANSFORMER RESULTS
4.1 THE TRANSFORMERS RETRIEVED PRIOR NOUNS AND THEIR ORDER; THIS CAPACITY IMPROVED WHEN THE MODEL WAS TRAINED ON A LARGER CORPUS
We tested whether the transformers could retrieve the identity and order of 10-token noun lists (arbitrary or semantically coherent). To this end, we constructed vignettes in which the second list
Models are available at: https:://cot /ttp:::///oo..r///000555 / )
4

---

Under review as a conference paper at ICLR 2022
Verbatim retrieval of words and their ordering as function position list
A)
LSTM-400 (Wikilext-103)
Arbitrary list
Semantically coherent list
Second list
surprisal
Repeated
(bit)
Permuted
Novel
B)
Transformer (Wikitext-108
surprisal
(61
Transformer fRadford et al, 2019)
surprball
10
(660
-2
tolen position relative list onset
taken position relative to list onset
Figure 2: Median surprisal (over Nlist 230) broken down per token position in second lists of arbitrary nouns and semantically coherent nouns. Negative values on x-axis represent 4 tokens of prompt string that introduced the second list: "(she) read the list again". The O-index marks the first noun in the list. Line style and hue denote manipulation of the second list relative to the first list. Error bands denote 95% confidence interval around the median (bootstrap estimate).
was either a) identical to the first list, b) a permutation of the first list, or c) a list of novel nouns not present in the first list We then measured retrieval as reduction surprisal from first second list.
When the two transformers were presented with second lists that were repeated version of the first, token-by-token. (blue in Fig. 2 and C) surprisal decreased compared to novel tokens, suggesting that the transformer was able to access verbatim representation of past nouns from context. When the second list was a permutation of the first one, surprisal was higher compared to when it was repeated, indicating that the transformer expected the nouns to be ordered as in the first list. Importantly, surprisal differences were considerably smaller for the transformer trained on the 40 million Wikitext- 103 corpus (Fig. 2 B) compared GPT--2 (Fig. C). This suggests that the training set size plays an important role in supporting verbatim recall.
In order to contextualize the magnitude of these retrieval effects, computed the relative surprisal (Section 3.4 across all tokens in lists except the first one (Fig. 3 left). When the first and second 80% of the first list, lower
lists identical, the Wikitext- 103 transformer relative surprisal than the ~ 85% for the permuted lists, and ~ 100% for the novel list. In GPT-2, repeat surprisal was 2% of the first list, much lower than the 60% for the permuted lists, and 100% of the novel list.
Retrieval in GPT-2 was robust to the exact phrasing of the text that introduced the lists. Replacing the subject 'Mary' with 'John' in the vignette, replacing the colon with a comma or randomly permuting the preface or the prompt strings did not affect the results (see Fig. 9 in A.8). The same control analyses show that the retrieval effects for Wikitext- 103 transformer were reduced under these variations (see Fig. g in A confirming that larger corpus size contributes to robustness of the transformer retrieval.
4.2 TRANSFORMER RETRIEVAL WAS ROBUST TO THE NUMBER OF ITEMS BEING RETRIEVED
In studies of human short-term memory, performance degrades as the number of items that need to be retained increases ("set-size effects", Oberauer et al. 2018). Is our LMs' short-term memory
'Novel nouns in the string were introduced by randomly selecting list of nouns from one the 22, remaining lists in the noun pool. In semantically coherent lists, novel nouns were drawn from a different semantic category than the nouns in the first list.
5

---

Under review as a conference paper at ICLR 2022
Verbatim retrieval of words as a function of increasing set size and intervening text
set size
Second list
Ii MM **. DWV Blagen
Repeated
preface
first list
intervening text second list
Permuted
Novel
A)
LSTM (Wikitext-103)
Arbitrary list
Semantically coherent
Arbitrary list
Semantically coherent
list
repeat surprisal
110
90
80
Transformer (Wikitext-103)
repest surprisal
(%)
60
Transformer (Radford et al, 2019)
100
50
0-
99
size
intervening len. (n. tokens)
intervening len. (n. tokens)
(n. tokens)
(n. tokens)
Figure 3: Verbatim token retrieval for varying number of tokens being retrieved (left) and the length of the intervening text (right). Reported is proportion of list-median surprisal on second relative to first list nouns. Points show group median (over N = 230). Error bars denote 95% confidence interval around the median (bootstrap estimate). For set size manipulation, intervening text is fixed at 26 tokens. For intervening text manipulation, set size is fixed at 10 tokens.
similarly taxed by increasing the set size? We varied the number of tokens to be held in memory with Niokems E {3, 5, 7, 10). For this comparison, Results reported in Fig. show that for both the smaller Wikitext-103 transformer and the larger GPT-2, verbatim recall was, for the most part, consistent across the different set sizes. For GPT-2, repeat surprisal increased monotonically with set size only when the order of nouns in second list, either semantically coherent or arbitrary, was permuted. This increase surprisal with set size for permuted sequences occurred because, if the model has perfect memory of the list of tokens, but cannot predict the order in which they will reoccur, then its probability of guessing the next item in permuted list of possible tokens will be 1/n,, and the mean value of is larger for larger set sizes.
length intervening text was kept at 26 tokens.
4.3 TRANSFORMER WORD RETRIEVAL WAS ROBUST TO THE SIZE AND CONTENT OF INTERVENING TEXT, BUT SCRAMBLING THE INTERVENING TEXT REDUCED RETRIEVAL OF ORDER INFORMATION
For how long are individual items retained in the memory of the LM? We tested this by varying the length of the intervening text for Nioken E {26, 99, 194, 435} (see Fig. I panel B). To generate longer intervening text samples, we continued the narrative established by the initial preface string ("Before the meeting, Mary wrote down the following list of words:"). For complete texts,

---

Under review as a conference paper at ICLR 2022
LM verbatim word retrieval function of intervening text type
A)
LSTM (Wikitext-103)
B)
Transformer (Wikitext-103
Arbitrary
Semantically coherent
Arbitrary list
Sernantically coherent
list
list
repea surprissl (%5
60
intact
sctambled
incongruent
intact
scrambled
incongruent
intact
actambled
incongruent
intact
actambled
incongruent
C)
Transformer (Radford et al, 2019)
Arbitrary list
Semantically coherent
list
repeat repeat surprical
100
Second list
Repested
50-
Permuted
Novel
intact
scrambied
incongruent
intact
scrambled
incongruent
intervening text
intervening text
Figure 4: LM memory retrieval for different intervening texts. We plot relative list-median surprisal over all non-initial tokens in lists. Points show group median (over N 230). Error bars denote 95% confidence interval around the median (bootstrap estimate). Note that in the top-row plots y-axis starts at 60%.
Section A.2 in Appendix A All intervening text strings ended with the same prompt string ("When she got back, she read the list again:") which introduced the second list.
Memory retrieval in the transformer models, whether trained on Wikitext- -03 or a much larger corpus size, was largely invariant to the size of the intervening text between the first and second lists (Fig. 3 and C, respectively). The results suggest that the two transformers were retrieving prior nouns using form of direct indexing of the relevant words from the input buffer, rather than implementing a generic memory heuristic, such as predicting that the nouns that have occurred in the most recent 20 tokens will recur.
The results previous sections suggest that increasing the length of well-formed, semantically coherent intervening text does not interfere with memory retrieval in the transformer. In models of human memory, current context, such as immediately preceding text, can indeed be used as a cue for recalling the encoded items (Kahana 2020). Does the transformers' capacity retrieve copies of past nouns rely on the content and structure of the intervening text? We tested this by creating incongruent and scrambled versions of the original intervening text (435 tokens). An incongruent condition was created by using intervening text that was syntactically well-formed but semantically incongruent with respect to the preface (see Appendix A.2.2). The scrambled version was created by randomly permuting the tokens of the intervening text.
The transformers' retrieval of past tokens was largely unaffected by the specific content of the intervening text, long as the intervening text was coherent/well-foree (Fig. 4). However, in GPT-2, median surprisal across permuted arbitrary lists nouns increased by ~ 8% when the intervening text scrambled (Fig. 4 bottom) compared to well-formed text. This suggests that GPT-2 relied on narrative coherence of the intervening text, rather than its aggregate semantic content alone, cue for retrieving the ordering information of arbitrary word lists.
7

---

Under review as a conference paper at ICLR 2022
4.4 TRANSFORMER VERBATIM RECALL IS LEARNED, GUIDED BY ATTENTION, AND REQUIRES DEPTH
Having shown that the transformer LMS could flexibly and robustly retrieve words and their ordering verbatim from short-term memory (Figs. 3 and we next asked: is this ability learned, or does derive directly from the architecture? To address this question, we re-ran the experiment with varying number of tokens in lists with a randomly initialized transformer model (architecture as in Section
3.3..
This random-weiss weights model was unable to retrieve words or their order: surprisal remained at 100% relative to first lists regardless of whether or not the nouns in the second list have appeared before (Fig. left).
Next we tested whether the transformers' ability to recall past tokens depended on the attention mechanism (Bahdanau et al., [014] Vaswani al.) 2017) which allows it, in principle, to use all past words, weighted according to their relevance, for next word prediction. The attention function
atin so fofamax x OK") V is supported by the query and key matrices Q, K E 708x To test for the role of attention in verbatim retrieval, we randomly permuted the rows and columns in each of the 12 attention layers of GPT-2 and reran the experiment with varying number of tokens in lists. model retained some capacity to retrieve past nouns (Fig. [5] right), but the effect was greatly reduced (surprisal reduction of ~ 10% for shuffled-attention compared with 100% for the intact model). Intriguingly, this chuffled-attention model showed the same surprisal for repeated and permuted lists, indicating that was no longer accessing word order information from the original list. Thus, the attention mechanism is necessary for transformers to index past nouns and their order from memory.
vds."
The
Finally, deep layered architecture is a key characteristic of transformers and performance typically scales with model size (Radford et al. 2019); Kaplan et al. 2020). Does the capacity to perform verbatim recall depend on model depth? To address this question, we trained 1-, 3-, 6- and 12-layer transformers on our 40-million subset of Wikitext-103. Consistent with the hypothesis that depth is crucial, the smaller 1- and 3-layer models show modest verbatim recall capacity, but were not sensitive to order (~~ 90% repeat surprisal for repeated and permuted lists, Z Sensitivity to order progressively emerged in 6- and 12-layer models, where relative surprisal levels were ~ 5-10% lower for repeated relative to permuted lists (Fig. in A.7
5 LSTM RESULTS
5.1 THE LSST RETRIEVES GIST-LIKE MEMORIES OVER SHORT INTERVENING DISTANCES, FACILITATED BY SEMANTIC COHERENCE
The LSTM generated the expectation that the nouns the second list should belong to the same semantic category and this weighted towards carlier nouns in the first list. We established this using the same paradigms as described in preceding sections to test memory retrieval capacities of an LSTM LM. If the intervening text was longer than 26 tokens, LSTM repeat surprisal across non- initial token positions (Figure 3 A, right) revealed a modest decrease (~ 5%) relative to first list, but only when the nouns in the first and second lists were from the same semantic category. Examining surprisal values broken down by token position in the list (Fig. 2 shows that in semantically coherent lists of nouns, surprisal was higher for novel lists than for repeated permuted lists, but this memory effect was only present for tokens near the beginning of the list.
In light of this evidence for retrieval in the LSTM across 26 intervening tokens, we examined whether the LSTM retrieves more successfully over much shorter intervals. by reducing the intervening text to 4 tokens of coherent text (see Appendix A.2.4.. In this short-range retrieval setting, we now observed small reduction of relative repeat surprisal of ~ 5% for arbitrary lists of 3 or 5 nouns as well as stronger ~ 15% reduction for semantically coherent lists (Figure in A.6). Overall, the reduction in surprisal was comparable for repeated and permuted lists, indicating that the LSTM did not predict that words would occur in their original order. We did not observe a change in retrieval even with an LSTM that was trained on twice as many tokens and had an approximately 5-fold increase in parameter count (see Fig. A.6) Taken together, the experiments described in the section suggest that the LSTM retrieves a semantic gist of the prior list. Consistent with this notion of aggregate
8

---

Under review as a conference paper at ICLR 2022
semantic memory, we found that retrieval was stronger for semantically coherent lists, for which an aggregated semantic representation would be closer to each of the individual words in the list.
6 DISCUSSION
Short-term memory- the capacity temporarily store and access recent context for current processing is a crucial component of language prediction. In this paper, we introduced a paradigm for characterizing a language model's s short-term memory capabilities, based on retrieval of verba- tim content (sequences of nouns) from prior context, and used this paradigm to analyze LMs with transformer and LSTM architectures.
The transformers tested were able to access verbatim information individual tokens and their order from past context. Furthermore, this verbatim retrieval was learned and largely rexilient to interference from intervening context. This indicates that the models (especially those trained on the largest corpora) implemented, via learning, a nigh-resolution memory system. The ability to access individual tokens may in support functions that rely token indexing, akin the functionality of a
working memory ( (WM) buffer proposed in cognitive science (Baddeley) 2003). On the one hand, such flexible WM could subserve the reported ability of transformers to rapidly generalize to new tasks (Brown et al. 2020). Indeed, past work suggests that such meta-learning traditional RNNs requires dynamic short-term memory mechanism known as fast weights Schmidhuber 1992 Ba et al. 2016) which can be thought of as analogous to self-attention in transformers (Schlag et al. 2021). On the other hand, highly resilient verbatim memory system could also be disadvantageous if it causes the LM to place too much confidence on verbatim features of prior context for next-word prediction. Indeed, text generated from a transformer LM's predictions can be highly repetitive (Holtzman et al. 2020) it is possible that an over-reliance on accessing short-term memory may underlie this tendency.
In contrast to the transformers, the LSTM model (in spite of it's larger parameter count) only retrieved a coarse semantic category of previous lists, without fine- ane-grained information about word order, and was only able to do when the intervening text was short. The tendency of LSTMs to rely on the fuzzy representation of past context for next-word prediction has been reported previously (Khandelwal et al., 2018). Whereas in sequence-to-sequenee to-sequeeeee tasks requiring recall of short lists of pseudowords, recurrent neural networks are a good model of human short-term memory (Botvinick) Plaut (006),, the copying capacity of LSTMs did not generalize longer sequences of symbols (Grefenstette et al.] 2015). Is tracking a shallow representation of context always a limitation? Not necessarily. Humans frequently maintain a ggood-enough (i.e. gist-like) representation of context (Ferreira & Patson, (007). When the potential for memory capacity is limited (e.g. when context must be compressed single hidden state as in an RNN) maintaining a broad, gist-like as opposed to token-specific- memory of context may be more efficient overall.
Because the transformer models were better overall LMs than the LSTM models (i.e. they exhibited lower perplexity). future work must replicate these results on more powerful LSTM LMs. For example, future work should test LSTMs augmented with attention (Bahdanau et al. 2014) or the copy-mechanism (Gu et al. 2016) trained with more sophisticated regularization (Merity et al. 2017). That said, it remains challenging to discern which aspects of a model need to be matched across architectures in order to make the comparison fair: while matching corpora is fairly straightforward, is not clear whether the number of layers or parameters has a common meaning across such distinct architectures.
CONCLUSIONS
Pretrained language models, and self-supervised predictive learning broadly, have received increased attention terms of their (in)sufficiency as a framework for achieving feats of human-like language processing (Kaplan et al.) 2020 Linzen & Baroni, 2020). Here, we tested the ability of language models to perform an important aspect of human intelligence for natural language - flexibly accessing items from short-term memory and show that the transformer model, even though not trained with a short-term memory objective, retrieved remarkably detailed representations of past context for predicting new input.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
The research reported here addresses a specific, basic research question about the cognitive orga- nization of short-lerm memory in contemporary language processing algorithms. Although from a broader perspective, the nature of memory is key question in developing human-like artificial intelligence, it is, our opinion, unlikely that the results reported here could pose novel societal challenges as we are primarily trying to better the understanding of the already developed systems.
REPRODUCIBILITY STATEMENT
Anonymized repository hosting the code used in current experiments is avail- able the following link:
attcss:/anooymmous www.en.ccc...... accence/rrrec...coo / short-term-mentt memory-mooro----ee por-innneural-nne 31F9 The codehase in the repository itself is not self-contained, however, we used the popular scientific python and NLP libraries - any such external dependencies with cursory documentation (including installation) are clearly indicated and provided in the repository's s README.md.
All our scripts were used GPUs were used for transformer training. For single job, single GPU device was used and single job typically required ~6 hours of core-walltime (12-layer model). We tended request least 44GB of RAM per job (12-layer model). For a single transformer evaluation, we typically required 13:30 (hh:mm) of fore-walltime and 4GB RAM per GPU job. A single LSTM evaluation job typically required ~ 06:00 (hh:mm) or less of core-walltime and ~ 8GB RAM per GPU job and these parameter values were sufficient prevent overflows.
segh-performnce computing (HPC) cluster. Nvidia Quadro RTX 8000
REFERENCES
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. Using fast [cs, stat], December 2016. URL weights to attend to the recent past. arXiv: 1610.06258.
//arxiv..or wwtpiii.org//bs//11111 6258
Alan Baddeley. Working memory: looking back and looking forward. Nature Reviews Neuroscience,
4(10):829 839, October 2003. ISSN 1471-003X. 1471-0048. doi: 10.1038/nm1121. URL
http:/ /ttt::/www.. http::/wwwww.att....et.t natura. com. com
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473 2014. URR.itttp:/arxiw.orgg abs/
1409. 04
Matthew M. Botvinick and David C. Plaut. Short-term memory for serial order: A recurrent neural network model. Psychological Review, 113(2):01--33, 2006. ISSN 1939-1471, 0033-295X.
doi: 10.10300033-2955..2 113.2.001. Uttht::///do: http:/dii.ppp..ooo wpw.ppp.p..o......p///pp.p/po/.. wwt.::ii..cmcommmmmmmm
1037/003: 295X. 111.2.21.2000 2.20
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv:2005. [cs], July 2020. URL http//axivvvv.. org./ivoorrrreeooo as 2200 141 arXiv: 2005.14165.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep [cs], May 2019, URL
bidirectional transformers for language understanding. arXiv: 1810.04805.
http:://arxiii htttp://arxiiw.rr/abbbs.. wwtg/bs.////ooo..ooooooo..os.. 1810 04805
Fernanda Ferreira and Nikole D. Patson. The 'Good Enough' approach to language comprehension.
Language and Linguistics Compass, 1(1-2):71-83, March 2007. ISSN 1749818X. doi: 10. 111 074 2007 00007 /RRtatoo///oooooneieeeaaaaaaaa
wiley. com/ doi/10
1111/ 749 2000
10

---

Under review as a conference paper at ICLR 2022
Richard Futrell, Ethan Wilcox. Takashi Morita, and Roger Levy. RNNs psycholinguistic subjects: Syntactic state and grammatical dependency. arXiv: 0099009990119 [cs]. September 2018. URL
httt:://axiv.. org/ abs 1809 019
arXiv: 1809.01329.
Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to
transduce with unbounded memory. arXiv:1506.0257 [cs], November 2015. URL http:
02516
arXiv: 1506.02516.
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. Incorporating copying mechanism in sequeence-t--sequence learning. arXiv: (603/ 1603306333 [cs], June 2016. URL
orq/abs/1603 06393 arXiv: 1603.06393.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Colorless green recurrent networks dream bierarchically. arXiv: 11138 [cs], March 2018. URL
http://arxiv org/ hbtp://b/////// 111880331111111 11138
arXiv: 1803. 11138.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 98))
1780, November 1997. ISSN 0899-7667, 1530-888X doi: 10.1162/neco.. 1997. 1735. URL
http:// http://wwwww.o wtw.:/imtpresssssjooooo
org. wttp://ss....///////oo////////// 116.110...111111 newo. 1997.9997777 9.8...9... 1735
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv: 090499049990 0975/ [cs], February 2020. URL http:://axxiv.org/aatet
09751 arXiv: 1904.09751.
Michael J. Kahana. Computational models memory search. Annual Review of Psy-
chology, 71(1):107-138, January 2020. ISSN 0066-4308, 1545-2085. doi: 10.1146/
munurev-psych-0104113. 103358. URL wttp:s///ww..mmmneeeeeeeeee. wrtp:evviiiiisog/oo//////
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.
arXiv:2001.0836 [cs, star), January 2020. URL httt:::/aw....org////// arXiv: 2001.08361.
08361
Urvashi Khandelwal, He Hc, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. arXiv: 1805, 0462300433330 [cs], May 2018. URL /ttpt/pp///xxi abs / 1805.10000 643
arXiv: 1805.04623.
Tal Linzen and Marco Baroni. Syntactic structure from deep learning. arXiv:2004.10827 /cs), April 2020. doi: 10. hww.uuurre---innuiiiie--02222-20 2004 10827 arXiv: 2004.10827.
URL http::/aaxiv.org//
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs learn syntax-sensitive dependencies. arXiv: 16/ 1.01368 [cs/, November 2016. URL http:://tt:rttt org/abs/1611, 01368
arXiv: 1611 1.01368.
Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. arXiv:1808.09031 1808 09031 arXiv: 1808.09031.
[cs], August 2018. URL. UR..ittp:://axxi..orr////
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv: 07843 [cs], September 2016. URL
07843
arXiv: 1609.07847
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. arXiv:1708 02182 [cs]. August 2017. URL http::://aai..oo////aaaa/ 1708.02182. arXiv: 1708.02182.
Klaus Oberauer, Stephan Lewandowsky, Edward Awh, Gordon D. A. Brown, Andrew Conway,
Nelson Cowan, Christopher Donkin, Simon Farrell, Graham J. Hitch, Mark J. Hurlstone, Wei Ji
Ma, Candice C. Morey, Derek Evan Nee, Judith Schweppe, Evie Vergauwe, and Geoff Ward. Benchmarks for models of short-term and working memory. Psychological Bulletin, 144(9): 985-958, September 2018. ISSN 1939- 1455, 0033-2909. doi: 10. 10.10377bu10000153, 1037/bu 0000153
URL
http: //do: apa. wtt.ppp...p...o... www.o:...mmm..omm.mom 1...:3//11111111111

---

Under review as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019, URL http:::://pppprrsenuriissss....
ibrary.
pdf
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Lan- guage models are unsupervised multitask learners. 2019. URL http:tttss///mmcppppiii
nouufroot.nee/bbtteer
models/ /aaeelllllaaggeeeeeeaa models. pdf
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers secretly fast weight programmers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings Machine Learning Research, pp. 9355-9366. PMLR, July 2021. URL hnttps:/pproceecingg wtt.s//gss.......s/111.1.333
sch aanllag2aa.htr html
Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, January 1992. ISSN 0899-7667, 1580-888X. doi: n0.1162/neco.19//9 1992.4.1.131. URL /ttpthtt/////// rect witt wtw.it.ee/.eeeo//attilliooo htt.pp/eeeeaieeetct...eeeeeeeeee
7/131-139/ 39/5620
Sandeep Subramanian, Ronan Collobert, Mare' Aurelio Ranzato, and Y.-Lan Boureau. Multi-scale transformer language models. arXiv:2005.0058 [cs], May 2020. URL http: org/ abs/2005. 00581
arXiv: 2005.00581.
Marten van Schijndel, Aaron Mueller, and Tal Linzen. Quantity doesn't buy quality syntax with neural language models, November 2019. URL http:////eeooooooorrrrrroo////// type: dataset.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser. and Illia Polosukhin. Attention is all you need. arXiv: 03762 [cs], June 2017. URL arXiv: 1706.03762.
http:://axxiiii wwt.oorg/aaoo.............. abs / 03762
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-A language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. doi: 10. 1e2118663/1/2220.cco http:///00000mmmmmmmmmmo.... URL https:/ /www
natural
ac www.orrg/atthhtt.. http:r///aanthoooo//2200000 o2y2 wmnlp demos.
A APPENDIX
A.1
CODE AND DATA AVAILABILITY
The data will be shared on the project OSF repository: [TBA].
Code with documentation is available at the following anonymized Github repository:
A.2 VIGNETTES
A.2.1 INTACT
Before the meeting, Mary wrote down the following list of words:
12

---

Under review as a conference paper at ICLR 2022
W1. W2..., WN
intervening_eee _erriingggtrrtt After the meeting, she took a break and had a cup of coffee. When she got back, she read the list again: W1, W2.... WN
intervening_tett
After the meeting, Mary went for a walk. It was a busy day and she needed a break. Outside was really beautiful and warm and the flowers in the park were blooming. When she got back, she read the list again: W1, W,,.., WN
intervening terty: After the meeting, Mary went for walk. It was busy day and she needed break. Outside was really beautiful and warm and the flowers in the park were blooming. While she was walking, she listened the wonderful bird songs. During the walk, Mary could not stop thinking about the meeting. She was thinking about the discussions she had with her coworkers. Luckily, she met her neighbors Sarah and Ryan and they talked briefly. When she got back, she read the list again: W1, W....., WN
intervening_eende After the meeting, Mary went for a walk. It was a busy day and she needed break. Outside was really beautiful and warm and the flowers in the park were blooming, While she was walking, she listened the wonderful bird songs. During the walk, Mary could not stop thinking about the meeting. She was thinking about the discussions she had with her coworkers. Luckily, she met her neighbors Sarah and Ryan and they talked briefly. The couple has just moved to the area from a different city. Mary thought they were very a lovely couple and made good company. They were just getting to know the neighborhood and this was their first time in the park. Mary was curious what were their first impressions of the town. The neighborhood felt very safe to them and they absolutely loved the park. This was only their second time visiting the park. There was so much to discover, many winding paths and hidden gardens. When she got back, she read the list again: W1, W2,...Wx
intervening_ter _estt: After the meeting, Mary went for walk. It was busy day and she needed break. Outside was really beautiful and warm and the flowers in the park were blooming. While she was walking, she listened the wonderful bird songs. During the walk, Mary could not stop thinking about the meeting. She was thinking about the discussions she had with her coworkers. Luckily, she met her neighbors Sarah and Ryan and they talked briefly. The couple has just moved to the area from a different city. Mary thought they were very lovely couple and made good company. They were just getting to know the neighborhood and this was their first time in the park. Mary was curious what were their first impressions of the town. The neighborhood felt very safe to them and they absolutely loved the park. This was only their second time visiting the park. There was so much to discover, so many winding paths and hidden gardens. It was not big park by any means, but it offered quiet refuge where one can escape the worries of everyday life. It also offered opportunities to do sports of all kinds. Young people from around the area played basketball, football, volleyball. Others took part in outdoor workout sessions. Young families were going on a stroll with their children. Finally, there were so many people who brought their dogs for walk. It was incredibly satisfying see the joy our animal friends get when you throw them a ball. All this diversity of people and activities made a walk in this park truly rewarding and relaxing daily routine. In fact, Sarah and Ryan were thinking of getting a dog. They have not fully decided yet but they really wanted to spend more time outdoors. Mary liked dogs as well, but she was more of a cat person herself. She and her husband had two cats. One was two and the other four years old. They were very independent and spent most of their time outdoors. Mary thought having an animal was a great idea. They talked for little bit and then Sarah and Ryan invited her to come over for cup of coffee. Mary said she had time over the weekend. When she got back, she read the list again: WI, W2,... WN
A.2.2 INCONGRUENT
Before the meeting, Mary wrote down the following list of words: W1. Wz..., WN
intervening_tent There is voice in the waters of the great sea. It calls to man continually. When she got back, she read the list again: W1, W WNy
intervening terty: Sometimes it thunders in the tempest, when the waves leap high and strong and the wild winds shriek and roar. Sometimes it whispers in the calm, small voice, as if to solicit our regard. When she got back, she read the list again: Wi, W2 .... WN
13

---

Under review as a conference paper at ICLR 2022
intervening_tett terty: After the meeting, Mary went for walk. It was busy day and she needed break. Outside was really beautiful and warm and the flowers in the park were blooming. The sea has much say; far more than could possibly be comprehended in one volume, however large. It tells us of the doings of man on its broad bosom, from the day in which he first ventured to paddle along shore to the day when he launched his great iron ship, and rushed out sea. When she got back, she read the list again: W1, W WN
unterveninggteree After the meeting, Mary went for a walk. It was a busy day and she needed break. Outside was really beautiful and warm and the flowers in the park were blooming. The sea has much to say; far more than could possibly be comprehended in one volume, however large. tells us of the doings of man on its broad bosom, from the day in which he first ventured to paddle along shore to the day when he launched his great iron ship, and rushed out to sea. Before proceeding to the consideration of the wonders connected with and contained in the sea, we shall treat of the composition of the sea itself and of its extent, depth, and bottom. What is the sea made of? Salt water, is the ready reply that rises naturally to every lip. But to this we add the question, what is salt water? To these queries we give the following reply, which, we doubt not, will rather surprise some of our readers. The salt of the ocean varies considerably in different parts. When she got back, she read the list again: W1, W2..., WN
intervening texts: After the meeting, Mary went for walk. It was busy day and she needed break. Outside was really beautiful and warm and the flowers in the park were blooming. The sea has much to say; far more than could possibly be comprehended in one volume, however large. It tells us of the doings of man on its broad bosom, from the day in which he first ventured to paddle along shore to the day when he launched his great iron ship, and rushed out to sea. Before proceeding to the consideration of the wonders connected with and contained in the sea, we shall treat of the composition of the sea itself and of its extent, depth, and bottom. What the sea made of? Salt water, is the ready reply that rises naturally to every lip. But to this we add the question, what is salt water? To these queries we give the following reply, which, we doubt not, will rather surprise some of our readers. The salt of the ocean varies considerably in different parts. Near the equator, the great heat carries up a larger proportion of water by evaporation than in the more temperate regions. Thus, as salt is not removed by evaporation, the ocean in the torrid zone is salter than in the temperate or frigid zones. The salts of the sea, and other substances contained in it, are conveyed there by the fresh water streams that pour into it from all the continent of the world Here, these substances cannot be evaporated, they creatures. The operations of the ocean are manifold. But we cannot speak of these things without making passing reference to the operations of water, as that wonder-working agent of which the ocean constitutes but a part. Nothing in this world is ever lost or annihilated. As the ocean receives all the water that flows from the land, so it returns that water, fresh and pure, in the shape of vapour, to the skies. where, in the form of clouds, it conveyed to those parts of the earth where its presence is most needed. After having gladdened the heart of man by driving his mills and causing his food to grow, finds its way again into the sea: and thus the good work goes on with ceaseless regularity. When she got back, she read the list again: Wi, W2,.., WN
accumulate to such a degree as to render the ocean uninhabitable by living
The incoongruent int intervening text was sampled from: "The ocean and its wonders" by R. M. Ballantyne. http:://wwww.guutebberrrrr///ee/
A.2.3 SCRAMBLED
Before the meeting, Mary wrote down the following list of words:
W1, W2,.., WN
intervening fesptterttt After a break, a cup and coffee of had she the took meeting, When she got back, she read the list again: W1, W2.... WN
interveninggterte Outside the the beautiful and park flowers blooming were in and was warm really. After, walk for Mary the went meeting. It needed busy break she day was and a. When she got back, she read the list again: W1, W,,..,, WN
intervening httreennnnngtntt:ieeeeee Luckily and and met Sarah they Ryan briefly talked her, neighbors she. Thinking during, stop meeting the not about Mary the could walk. The while walking to songs bird listened wonderful, she she was. After, walk for Mary the went meeting. Had she about she coworkers her
14

---

Under review as a conference paper at ICLR 2022
with the was discussions thinking. Outside the the beautiful and park flowers blooming were in and was warm really. It needed busy break she day was a and a. When she got back, she read the list again: Wi, 2 .... WN
intervening_tette First they their was neighborhood getting and the in park the this to were time know just. There paths much, and many gardens hidden winding to was discover so. The while walking to songs bird listened wonderful, she she was. Had she about she coworkers her with the was discussions thinking. From the just area city different the moved couple to has. The absolutely and very them loved they park the safe neighborhood to felt. Outside the the beautiful and park flowers blooming were in and was warm really. And Mary were couple company good lovely made very thought a they. Luckily and and met Sarah they Ryan briefly talked her, neighbors she. Thinking during, stop meeting the not about Mary the could walk. After, walk for Mary the a went meeting. Their this park visiting second was the time only. Impressions Mary what first town the of were was their curious, needed busy break she day was and a. When she got back, she read the list again: W1, w,.,., Wn
intervening_tere It needed busy break she day was a and a. First they their was neighborhood getting and the in park the this to were time know just. Had she about she coworkers her with the was discussions thinking. Of they independent most outdoors time their and were spent very. Get it friends them our joy satisfying when the throw ball a animal to was you incredibly. The while walking to songs bird listened wonderful, she she was. Weekend had time Mary said the over she. An Mary idea a animal thought great was having. Mary she was but cat of herself person more well liked, dogs. It of opportunities kinds sports to also all do offered. Cats husband had she and two her. They spend they really fully but more outdoors time wanted decided have yet not. A a and of rewarding park all in made this activities relaxing routine daily truly walk people this and diversity. There paths so much, and many gardens hidden winding to was discover so. Finally dogs who were people for brought walk a their so, many there. Luckily and and met Sarah they Ryan briefly talked her, neighbors she. The absolutely and very them loved they park the safe neighborhood Outside the the beautiful and park flowers blooming were in and was warm really. Young football basketball around played., volleyball or the people area from. Their this park visiting second was the time only. To Sarah for her they Ryan then invited and cup coffee of over come and little talked bit for. From the just area city different the a moved couple to has. And Mary were couple company good lovely made very thought they. Young with going children stroll families their a were. Worries a means escape where a offered but one refuge can it by any it the quiet of life everyday. big was park not. Of in Sarah thinking dog were getting and fact Ryan, Thinking during, stop meeting the not about Mary the could walk. After, walk for Mary the a went meeting, And one old four the was years other two. Impressions Mary what first town the of were was their curious. Sessions in outdoor others took workout part. When she got back, she read the list again: W1, W2, WN N
felt.
A.2.4 SHORT
Before the meeting, Mary wrote down the following lists of words. One was:
W1, W2,..,, WN
intervening_terte And the other: W1, W WN
A.3 NOUN LISTS
Arbitrary and semantic word lists are shown in Tables 1 and respectively.
A.4
TRANSFORMER TRAINING
All models were trained using the PyTorch (Paszee et al. 2019 v1.6) and HuggingFace (Woll et al. 2020 14.6.1) 1) python libraries.
A.4.1
DATASETS
Models were trained on 40 million token subset of the Wikitext- 103 training set (Merity et al. 2016). To create the training set, we selected the first 40 million tokens in the original training set. We then retokenized the dataset with the trained BPE tokenizer (see A.4.2 below). The original
15

---

Under review as a conference paper at ICLR 2022
Table 1: Lists of arbitrary nouns used in present experiments.
list
1 patience, notion, movie, women, canoe, novel, folly, silver, eagle, center.
2 pleasure, pattern, leader, culture, worker, master, meadow, writer, apple, costume.
3 paper, belief, factor, total, comrade, angle, battle, pistol, nothing, riches.
4 cabin, doorway, candle, parent, monarch, kindness, lover, copy, soldier, kingdom.
5 future, legend, problem, flavor, prairie, forehead, illness, planet, canvas, chamber. oven, patient, daughter, bubble, colour, product, echo, pepper, fountain, music. village, shipping, beauty, football, merit, autumn, lumber, research, resort, rival.
6
7
8
county, muscle, vapor, shepherd, sickness, herald, value, mission, finger, building. iron, onion, opera, attack, prison, butter, interest, colonel, commerce, beggar. blanket, marriage, ticket, baby, treasure, event, weakness, cottage, cotton, judgment.
9
10
11
summer, bottom, meaning, campaign, voyage, cannon, helmet, thunder, hatred, stanza. effort, province, parcel, temple, river, major, meeting, career, bargain, chimney. acre, fortune, motive, question, service, minute, tiger, author, sorrow, parlor. motor, lawyer, powder, habit, mountain, district, learning, leather, hero, water.
12
13
14
15
orange, letter, acid, stocking, olive, garden, feeling. motion, compass, model. island, theory, person, season, supper, reason, patent, picture, custom, twilight.
16
17
dragon, pillow, aspect, chairman, marble, horror, justice, danger, bedroom, canal. writing, pocket, training, circuit, cousin, chapter, quarter, button, turkey, surface.
18
19
sailor, matter, darkness, scatter, captain, tunnel, method, wagon, effect, arrow.
20
image, butcher, anchor, scholar, compound, tribute, victim, lily, witness, widow. candy, window, detail, ocean, program, traffic, feather, array, pilot, silence.
21
22
vessel, robber, banner, kitten, lemon, failure, princess, painter, bullet, rifle.
23
engine, timber, harbour, party, level, money, single, system, unit, traitor.
Table 2: Lists of semantic nouns used in present experiments.
list
1
window, door, roof, wall, floor, ceiling, room, basement, hearth, hall.
2 leg, arms, head, eye, foot, nose, finger, ear, hand, toe.
3
sailboat, destroyer, battleship. cruiser, submarine, yacht, canoe, freighter, tugboat, steamship.
4 robin, sparrow, heron, eagle, crow, hawk, parrot, pigeon, woodpecker, vulture.
5 apple, pear, banana, peach, grape, cherry, plum, grapefruit, lemon, apricot.
6 hammer, saw, nails, level, plane, chisel, ruler, wrench, drill, screws.
7
hurricane, tornado, rain, snow, hail, storm, wind, cyclone, clouds, sunshine. oxygen, hydrogen, nitrogen, carbon, sodium, sulphur, helium, chlorine, calcium, potassium. chemistry, physics, psychology, biology, zoology, botany, astronomy, mathematics, geology, microbiology. piano, drum, trumpet, violin, clarinet, flute, guitar, saxophone, trombone, oboe.
8
9
10
11
knife, spoon, fork, pan, pot, stove, bowl, mixer, cup, dish.
12
trout, shark, herring, perch, salmon, tuna, goldfish, cod, carp, pike.
13
football, baseball, basketball, tennis, swimming, soccer, golf, hockey, lacrosse, badminton.
14
doctor, lawyer, teacher, dentist, engineer, professor, carpenter, salesman, nurse, psychologist.
15
oak, maple, pine, elm, birch, spruce, redwood, walnut, fir, hickory. shirt, socks, pants, shoes, blouse, skirt, coat, dress, hat, sweater.
16
17
cancer, measles, tuberculosis, polio, malaria, leukemia, pneumonia. smallpox, influenza, encephalitis. mountain, hill, valley, river, rock, lake, canyon, tundra, ocean, cave.
18
19
murder, rape, robbery, theft, assault, arson, kidnapping, larceny, adultery, battery.
20
log, cat, horse, cow, lion, tiger, elephant, pig, bear, mouse.
21
fly, ant, bee, mosquito, spider, beetle, wasp, moth, flea, butterfly.
22
blue, red, green, yellow, black, purple, white, pink, brown, blonde.
23
cotton, wool, silk, rayon, linen, satin, velvet, denim, canvas, felt.
16

---

Under review as a conference paper at ICLR 2022
Table 3: Architecture and model parameters.
layer
3 layer
6 layer
12 layer
activation function
gelu_nes
gelu_ new
gelu_ned
gelu_nes
n layer
6
12
head
3
6
12
n ctx
1024
1024
1024
1024
positions
1024
1024
1024
1024
vocab size
28439
28439
28439
28439
per device train batch size
12
12
12
12
per device eval batch size
12
12
12
12
learning rate
0.00007
0.00007
0.00007
0.00006
adam betal
0.6
0.6
0.6
0.6
adam beta2
0.05
0.05
0.05
0.1
params (M)
29.7
43.9
65.2
107.7
Verbatim retrieval of words in randomly initialized and shuffled attention models
Transformer (random init.) Semantically coherent
Transformer (shuffled attn. weights) Semantically
Arbitrary
Arbitrary
Hist
list
repeat surprisal
Second list
Repeated
100
Permuted
Novel
10
10
set size (n. tokens)
set sizee (n. tokens)
set size (n. tokens)
set size tokens)
Figure 5: Repeat surprisal for randomly initialized transformer LM and a transformer with permuted attention weights. Reported is relative list-median surprisal over all non non-initial tokens in lists only. Points show group median (over Nlist = 230). Error hars denote 95% confidence interval around the median (bootstrap estimate). Note that in these plots y-axis starts at 70%.
validation and test sets were used for model validation and testing, respectively. The retokenized datasets (i.e. indices marking vocabulary items) for the train, validation, and test set are available under the files tab of our OSF repository (see Section A.I)
A.4.2 TOKENIZER
To control the vocabulary size, retrained the BPE tokenizer the concatenated Wikitext-100 103 training, evaluation and test sets. Vocabulary size was set to 28,439 entries and a minimum token occurrence frequency of 2 was used inclusion criterion. The resulting merges.. txt and vocabu- lary (vocab.json) files are available under the files tab of the project OSF repository 5 To perform training, used the tokenizer. wtw.niieetttann....... module by HuggingFace (Wolf al. 2020) library (see dataset. py).
A.5 RANDOMLY INITIALIZED TRANSFORMER, TRANSFORMER WITH SHUFFLED ATTENTION
The results for randomly initialized transformer and the transformer with shuffled attention are shown in Fig.
A.6
LSTM VERBATIM RECALL OVER 3-TOKEN CONTEXT
The results for LSTM verbatim recall over short context (3 tokens) are shown in Fig.
link TBA
17

---

Under review as a conference paper at ICLR 2022
Verbatim retrieval words after intervening text of tokens
LSTM (400-dim, 40 million tokens)
LSTM (1600-dim, million tokens)
Arbitrary list
Semantically coherent
Arbitrary list
Semantically coherent
list
list
repeat surprisa
Second list
100
00
Repeated
Permuted
80
80
Novel
10
10
10
size (n.. tokens)
(n. tokens)
(n. tolens)
(n. tokens)
Figure 6: LSTM verbatim token retrieval for varying number of tokens being retrieved short (3-token) intervening text. Reported is proportion of list-median surprisal on second relative to first list of nouns (repeat surprisal). Points show group median (over Nrist 230). Error bars denote 95% confidence interval around the median (bootstrap estimate).
Verbatim retrieval of words with increasing transformer depth
Transformer (1 layer)
Transformer (3 layer)
Arbitrary list
Semantically coherent
Arbitrary list
Semantically coherent
11.
repeat surprisal
Second list
Repeated
90
Permuted
Novel
70
Transformer layer)
Transformer (12 layer)
Arbitrary
Semantically
Arbitrary list
Semantically coherent
list
list
repeat- surptical
100
(%)
80
70
10
10
10
set
size (nt. tokens)
set size (n. tokens)
set size
(n. tokens)
tokens)
Figure 7: LM memory retrieval for models of different depths. Reported is relative list-median surprisal over all non-initial tokens in lists. Points show group median (over A 230). Error bars denote 95% confidence interval around the median (bootstrap estimate). Note that in these plots y-axis starts at 70%.
A.7 VERBATIM RECALL REQUIRES DEPTH
The results for transformers at different number of layers (depth) are shown in Fig.
A.8 RESULTS ON CONTROL VIGNETTES
The results with additional control vignettes are shown in Figures g (Wikitext-10) 103 transformer) and (GPT-2).
18

---

Under review as a conference paper at ICLR 2022
Verbatim retrieval per token position in control vignettes (Wikitext-10 103 transformer)
A) A''Jhhn instead of 'Mary' (Wikitext-103 transformer)
B) Commaa as cue (Wikitext-103 transformer)
Arbitrary list Semantically coherent
Arbitrary
Semantically coherent
list
list
repeat surprisal
Second list
Repeated
80
Permuted
Novel
60
7
10
5
10
10
C) Shuffled preface string (Wikitext-103 transformer)
D) Shuffled prompt string (Wikitext-105 transformer)
Arbitrary Semantically cohenent
Arbitrary list
Semantically coherent
list
Tist
apppatt surpeied
100
80
60
60
7
10
5
30
(n.. tokens)
(n. tokens)
(n. tokens)
(n. tokens)
Figure 8: Wikitext-103 transformer memory retrieval results for control vignettes. Top left: this vignette the subject 'Mary" is replaced with 'John'. Top right: in this vignette the colon introducing the list is replaced with a comma. Bottom: the preface string is randomly permuted. Reported is relative list-median surprisal over all non-initial tokens in lists. Points show group median (over Aftist 230). Error bars denote 95% confidence interval around the median (bootstrap estimate). Note that these plots y-axis starts 70%.
19

---

Under review as a conference paper at ICLR 2022
Verbatim retrieval function of token position in control vignettes (GPT-2)
A)
'John' instead of 'Mary' (GPT-2)
B)
Comma as cue (GPT-2)
Arbitrary list
Semantically coherent
Arbitrary list
Semantically coherent
list
list
repeat surprisal
100
Second list
Repeated
Permuted
Novel
Shuffled preface string (GPT-2)
Shuffled prompt string (GPT-2)
Arbitrary list
Semantically coherent
Arbitrary list
Semantically coherent
list
list
100
repeat surprisal
90
0-
3
10
3
10
set sizee (n.. tokens)
set size
(n.. tokens)
(n. tokens)
(n. tokens)
Figure 9: Results for paradigms with additional control vignettes: A) the subject 'Mary' replaced with 'John', B) the colon token introducing the list is replaced with a comma, C) the tokens in the preface string are randomly permuted and, D) the tokens in the prompt string are randomly permuted. Reported is relative list-median surprisal over all non-initial tokens in lists. Points show group median (over lisist 230). Error bars denote 95% confidence interval around median (bootstrap estimate).
20

---

Under review as a conference paper at ICLR 2022
Analysis using the mean as the estimator of group statistic
LSTM (Wikitext-103) 103)
Transformer (Wikitext-103)
Arbitrary list Sementically coherent
Arbitrary list
Semantically coherent
list
list
repoat surptiaa
130
80
90
60
3
10
10
3
10
size
set sizee
set
size
(n. tokens)
(n. tolenss)
(n. tokens)
(n. tokens)
Transformer (Radford al, 2019)
Arbitrary list
Semantically coherent
list
repest surprisal
Second list
Repeated
50-
Permuted
Novel
0-
10
10
scc sizze
set sizee
(n. tokens)
(n. tokens)
Figure 10: Verbatim token retrieval for varying number of tokens being retrieved (left) and the length of the intervening text (right). Reported is proportion of list- mist-mediin surprisal on second relative to first list of nouns. Points show group mean (over Niist 230). Error bars denote 95% confidence interval around the mean (bootstrap estimate). For set size manipulation, intervening text is fixed at 26 tokens. For intervening text manipulation, size is fixed at 10 tokens.
Table 4: LSTM-400 (Wikitext-103) surprisal values for four initial token positions, list type and second list condition.
Token position
2
List
Second list
arbitrary
Novel
18.5 (18.0-18.8)
16.1 (15.5-16.5))
15.6 (15.2-116)))
15.5 (15.1-115.))
Permuted
18.1 (17.5-18.5)
15.3 (15.0-155)
15..
15.2 (14.9-15.))
Repeated
18,0 (17.7-1888)
(1..(152))6600
15.5 (15.1-11)))))
15.5 (15.1-15.8)
semantic
Novel
(19.1-19.9)
(12.
(11.4-1))))
(10.5-11.33
Permuted
17.6 (17.2-178)
12.7 (11.-1313))
9.9 (944-110.4))
9.7 (9.1-101)))
Repeated
17.5 (17.1-1778)
12.0 121.2-1222
10.5 (9.9-10.9)
10.0 (9.4--0))))
A.9
USING THE MEAN AS THE ESTIMATOR
Results of the analyses using the mean, rather than median, as the estimator are shown in Fig.
10
A.10
RESULTS TABLES
A.10.1
TABLES FOR SURPRISAL TIME-COURSES (FIGURE
Numerical values for data shown in Fig. 2 are shown in Tables (LSTM), 5 (Wikitext- 103 trans-
former), and 6 (Radford al. 2019).
A. 10.2 TABLES FOR SET-SIZE MANIPULATION (FIGURE
Numerical values for data shown in Fig.
(left) are shown Tables 7 (LSTM), 8 (Wikitext- 103
transformer), and (Radford al. 2019
21

---

Under review as a conference paper at ICLR 2022
Table 5: Transformer (Wikitext-103) surprisal values for four initial token positions, list type and second list condition.
Token position
2
List
Second list
arbitrary
Novel
12.3 (12.1-12.5)
12.3 (11.9-133)))
11.8 (11.4-12.3)
12.3 (11.8-133))
Permuted
12.0 (11.6-1222)
11..
(11.001))))))
12.1 (11.6-12)))
Repeated
(4.2-6.)))))
(10.9-1.))11
10.8 (10.3-11.3)
10.7 (10.1-11.3)
semantic
Novel
12.8 (12.6-13.6)
(32.(12))114.1)
13.2 (12.3-133))
12.6 (12.001)))33
Permuted
12.0 (11.611))))))
(1.6((12221222)
11.7 (11.2-1222)
11.8 (11.4-1222
Repeated
5.1 (4.4-6.4)
(1..8-11))))
11.1 (10.6-11.7)
11.0
Table Transformer (Radford et al, 2019) surprisal values for four initial token positions, list type and second list condition.
Token position
2
3
List
Second list
arbitrary
Novel
10.4 (10.2-10.6)
9.6 (9.3-98)))
9.4 (9.1-9.))
9.0 (8.8-92))
Permuted
7.9 (7.5-8.0)
(5.6-66))))
5.6 (5.2-558))
5.6 (..4-66))
Repeated
1,9 (1.--.0)
(0.5-0.)
0.3 (0.3-03))
0.2 (0.--.22)
semantic
Novel
11.5 (11.1-11.9)
(..-6..))
(4.7--5))))
4.7 (..(4444
Permuted
5.2 (4.9-555
3..5-.))))
(3.1-33))
3.3 (3.0-3.4)
Repeated
(1.8-.)))
0.3 (0.3-0)))
0.2 (0.2-0.3)
0.2 (0.2-0.2)
Table LSTM (Wikitext-103) word list surprisal as a function of set size. We report the percentage of list-median surprisal second relative to first lists. Ranges 95% confidence intervals around the observed median (bootstrap estimate, resample 1000). The length intervening text is fixed at 26 tokens.
Set-Size
3
5
7
10
List
Condition
Arbitrary
Novel
103.4% (100.1-105.9)
102.6% (100.7-104.7)
101.8% (100.4-103.0)
101.3% (99.9-102.1)
Permuted
101.2% (99.4-102.5)
100.7% (99.9-101.8)
99.2% (98.7-100.8)
100.1% (99.6-100.5)
Repeated
101.1% (100.6-102.))
100.8% (100.2-101.3)
100.6% (100.1-101.0)
100.4% (100.0-100.7)
Semantic
Novel
103.2% (98.4-108.4)
102.1 (97.5-108.7)
102.5% (96.5-109.5)
107.1% (95.0-110.7)
Permuted
98.4% (95.0-101.2)
96.1% (94.5-97.8)
95.1% (93.5-96.7)
95.1% (94.3-96.2)
Repeated
98.1% (96.6-99.2)
95.8% (95.4-97.0)
95.6% (95.1-96.3)
95.2% (94.6-96.1)
Table 8: Transformer (Wikitext-100 103) word list surprisal as function of set size. We report the percentage of list-median surprisal on second relative first lists. Ranges are 95% confidence intervals around the observed median (bootstrap estimate, Nreesmple 1000). The length of intervening text fixed at 26 tokens.
Set-Size
10
List
Condition
Arbitrary
Novel
96.3% (03.5-100.7)
100.4% (95.5-103.7)
100.7% (97.9-105.2)
101.4% (97.8-104.0)
Permuted
84.8% (82.7-88.7)
86.0% (83.7-88.0)
87.0% (85.3-89.5)
86.8% (84.7-89.2)
Repeated
75.0% (71.4-79.1)
78.1% (76.4-80.5)
79.7% (77.2-82.1)
81.3% (79.4-83.4)
Semantic
Novel
100.0% (94.0-107.3)
97.5% (94.3-106.1)
100.5% (97.1-104.2)
101.3% (97.3-103.4)
Permuted
79.9% (76.4-83.6)
78.4% (76.2-82.0)
80.6% (78.0-81.9)
81.2% (80.2-82.6)
Repeated
71.4% (69,8-75 (69.8-73.8)
70.4% (68.9-72.8)
72.2% (70.6-74.5)
74.7% (73.1-75.9)
22

---

Under review as a conference paper at ICLR 2022
Table 9: Transformer (Radford al, 2019) word list surprisal function of set size. We report the percentage of list-median surprisal on second relative to first lists. Ranges are 95% confidence intervals around the observed median (bootstrap estimate, Nresample 1000). The length of intervening text is fixed at 26 tokens.
Set-Size
7
List
Condition
Arbitrary
Novel
96.3% (93.9-97.9)
95.3% (93.4-96.9)
95.1% (94.4-96.1)
95.7% (94.7-96.6)
Permuted
27.2% (25,4-29.5)
39.0% (37.6-41.0)
48.3% (47.3-49.3)
58.2% (57.2-59.4)
Repeated
3.8% (3.4-4.2)
3.6% (3.3-3.9)
3.3% (2.9-3.4)
2.7% (2.5-2.9)
Semantic
Novel
102.0% (94.8-105,
97.9% (92.5-101.0)
99.1% (95.2-102.0)
99.6% (96.0-103.8)
Permuted
32.6% (30.1-36.2)
47.7% (45.1-49.5)
59.9% (57.7-61.7)
72.4% (70.7-74.2)
Repeated
4.5% (4.2-4.7)
4.6% (4.2-5.1)
4.1% (3.7-4.8)
3.3% (3.0-3.6)
Table 10: LSTM (Wikitext-103) word list surprisal as a function of intervening text size. We report the percentage of list-median surprisal on second relative to first lists. Ranges 95% confidence intervals around the observed median (bootstrap estimate, Aresample = 1000). The list length is fixed at 10 tokens.
Intervening text len.
26
99
194
435
Condition
Arbitrary
Novel
101.3% (99.9-102.1)
103.2% (102.5-104.1)
101.2% (99.7-102.)) 1)
103.4% (102,3-103
Permuted
(90.5-100.5)
(102.111003
(102.1-103.8
Repeated
100.4% (100.0 100.7)
103.2% (102.8. 103.6)
101.2% (100.8 101.5
103.2% (102.9 103
Semantic
107.1% (95.0-110.7)
107.7% (99.7-111.6)
101.0% (94.0-108.9)
106.9% (99.8-113.2)
Permuted
95.1% (94.3-96.2)
102.1% (100.7-102.6
100.3% (99.3-101.1)
104.9%
(103.6-106.)
Repeated
95.2% (94.6-96.1)
101.5% (100.8-1024)
99.4% (98.3-100.3)
104.4%
(103.6-105.5
101..3 TABLES FOR INTERVENING TEXT LENGTH MANIPULATION (FIGURE
Numerical values for data shown Fig. (right) are shown in Tables 10 (LSTM). (Wikitext-103
transformer), and 12 (Radford et al. 2019).
A.10.4 NUMERICAL DATA FOR TRANSFORMER DEPTH RESULTS (FIGURE
Numerical values for data 1-, 6-, and 12-layer transformer (Fig, 7 are shown in Tables 13
14 15
and 16 respectively.
A.11 COMPUTE RESOURCES
Nvidia K80 GPUs were used to run jobs involved in GPT-2 evaluation. For single job, single GPU device was used and a single job typically required ~12 hours of walltwalt and ~ 1.5 GB of RAM. We tended to request least 13:00 (hh:mm) of walltime and 5GB RAM per job.
Table 1I: Transformer (Wikitext-103) word list surprisal as a function of intervening text size. We report the percentage of list-median surprisal on second relative first lists. Ranges are 95% confidence intervals around the observed median (bootstrap estimate, Nresample = 1000). The list length is fixed 10 tokens.
Intervening text len.
26
99
194
435
List
Condition
Arbitrary
Novel
101.4% (97.8-104.0)
100.7% (98.0-103.9)
101.7% (98.8-105.1)
101.1% (98.6-103.7)
Permuted
86.8% (84.7-89.2)
87.6% 186.1-89.1)
90.4% (88.7-91.9) .9.9
91.0% (89.2-92.0)
Repeated
81.3% (79.4-83.4)
81.7% (79.9-83.7)
84.5% (82.8-86.)
85.7% (83.7-87.1)
Semantic
Novel
101.3% 197.3-103.4)
100.7% (96.6-103) 103.3)
101.5% (97,6-105)
101.1% (97.9- (97.9-104.3)
Permuted
81.2% (80.2-82.6)
81.0% (79.8-83.2)
83.2% (81.7-84.6)
83.7% (81.4-85.2)
Repeated
74.7% (73.1-75.9)
74.4% (72.4-75.9)
76.5% (74.7-78.4)
77.3% (75.9-78.8)
23

---

Under review as a conference paper at ICLR 2022
Table 12: Transformer (Radford al, 2019) word list surprisal as function of intervening text size. We report the percentage of list-median surprisal on second relative to first lists. Ranges are 95% confidence intervals around the observed median (bootstrap estimate, Nresample 1000). The list length is fixed at 10 tokens.
Intervening text len.
26
99
194
435
List
Condition
Arbitrary Novel
95.7% (94.7-96.6)
97.5% (96.7-98.4)
98.3% (97.3-98.9)
99.9% (99.2-100.9)
Permuted
58.2% (57.2-59.4)
59.7% (58.8-60.0
61.8% (60.6-63.1)
62.0% (61.0-62.8)
Repeated
2.7% (2.5-2.9)
1,7% (1.6-1.8)
1.7% (1.5-1.8)
1.8% (1.6-1.9)
Semantic
Novel
99.6% (96.0-103
101.7% (97.2-104.9)
103.5% (98. 1-105.7)
103.1% (99 (99.4-107.6)
Permuted
72.4% (70.7-74.2)
73.7% (72.1-75.1)
75.0% (73.3-77.2)
76.1% (73.8-78.9)
Repeated
3.3% (3.0-3.6)
2.8% (2.6-3.1)
2.9% (2.6-3.2)
4.1% (3.8-4.6)
Table 13: Transformer (1 layer) word list surprisal as function of set size. We report the percentage of list-median surprisal on second relative to first lists. Ranges are 95% confidence intervals around the observed median (bootstrap estimate (Nresample 1000).
Set-Size
7
10
List
Condition
Arbitrary
Novel
98.9% (96.0-102.1)
98.9% (96.--100.2)
97.7% (95.9-99.0)
97.9% (96.2-99.9)
Permuted
93.9% (92.5-94.8)
94.0% (93.1-94.7)
93.3% (92.6-93.7)
94.2% (93.8-94.7)
Repeated
94.1% (93.6-9..5)
93.9% (93.7-94.2)
93.3% (93.0-93.8)
94.2% (94.1-94.4)
Semantic
Novel
100.6% (96.2-106.3)
97.8% (94.2-102.2)
95.8% (92.8-98.7)
96.9% (95.0-100.1)
Permuted
93.2% (91.9-94.0)
92.7% (91.6-93.9)
92.1% (91.5-92.6)
93.2% (92.4-93.7)
Repeated
93.0% (92.6-93.6)
92.5% (92.1-93.1)
92.3% (92.0-92.7)
93.2% (92.9-93.6)
Table 14: Transformer (3 layer) word list surprisal function of set size. We report the percentage of list-median surprisal on second relative to first lists. Ranges are 95% confidence intervals around the observed median (bootstrap estimate (Nressample 1000).
Set-Size
5
10
List
Condition
Arbitrary
Novel
98.0% (96.4-100.8)
97.4% (95.3-99.6)
96.6% (94.8-98.7)
94.9% (93.6-96.9)
Permuted
92.8% (91.8-94.2)
91.8% (90.2-92.3)
90.0% (88.9-91.1)
89.8% (89.3-90.6)
Repeated
92.7% (92.1-93.2)
91.4% (90.9-91.9)
90.3% (89.7-91.1)
89.8% (89.1-90.4)
Semantic
Novel
98.8% (95.1-103.4)
97.2% (93.1-100.8)
97.3% (94.6-99.6)
96.2% (93.5-98.0)
Permuted
91.4% (89.0-92.9)
88.3% (87.0-89.9)
87.9% (87.0-89.0)
87.7% (86.7-88.6)
Repeated
91.4% (89.8-92.2)
89.2% (88.2-89.9)
88.0% (87.2-88.7)
87.6% (87.2-88.2)
Table 15: Transformer (6 layer) word list surprisal function of set size. We report the percentage of list-median surprisal on second relative to first lists. Ranges are 95% confidence intervals around the observed median (bootstrap estimate (Nresample 1000).
Set-Size
7
10
List
Condition
Arbitrary
Novel
100.9% (93.5-103.5)
98.4% (95.8-101.8)
100.2%
103.4)
100.1 (97.6-102.2)
Permuted
89.0% (86.4-92.1)
82.4% (80.0-85.5)
84.8% (83.6-85.9)
86.2% (85.0-87.3)
Repeated
83.5% (79.1-88,9)
78.0% (75.4-80.0)
79.4% (78.0-80.4)
80.5% (79.4-81.3)
Semantic
Novel
100.7% (95.0-105,
97.1% (93.6-104.1)
95.6% (93.5-100.9)
97.3% (92.5-101.3)
Permuted
85.0% (81 (81.3-88.6)
79.0% (76.5-82.2)
78.4% (77.1-81.6)
81.0% (79.6-82.1)
Repeated
79.2% (76.0-83.9)
74.3% (72.5-77.1)
75.0% (72.9-77.0)
76.8% (75.3-78.0)
24

---

Under review as a conference paper at ICLR 2022
Table 16: Transformer (12 layer) word list surprisal function of set size. We report the percentage of list-median surprisal on second relative to first lists. Ranges are 95% confidence intervals around the observed median (bootstrap estimate (Arresample 1000).
Set-Size
7
10
List
Condition
Arbitrary
Novel
98.3% (94.8-105.2)
100.9% (98.6-105.1)
101.1% (97.4-104.2)
100.4% (98.8-103.1)
Permuted
86.6% (84.1-90.7)
85.3% (82.8-88.0)
85.9% (83.5-89.5)
85.8% (83.1-87.7)
Repeated
79.9% (76.9-82.8)
78.2% (75.7-80.0)
79.7% (77.1-81.5)
80.7% (78.5-82.1)
Semantic
Novel
101.0% (94.2-110.7)
100.0% (93.3-106.1)
102.0% (97.3-104.9)
101.6% (97.1-104.8)
Permuted
81.2% (77.4-87.44
79.3% (76.0-82.1)
79.7% (78.2-80.8)
81.1% (79.7-82.8)
Repeated
72.5% (70.9-74.7)
71.7% (69.6-73.5)
72.5% (70.3-74.1)
74.2% (72.5-75.3)
CPUs were use to evaluate RNN models. A single job typically required ~00:30 (hh:mm) of core- walltime and 1.5GB RAM per job. To our experience requesting 06:00 (hh:mm) of walltime and 4GB is typically sufficient to avoid overflows.
25


Reviews:
[{"review_id": "nMA7xTTEEV8", "reviewer": "Reviewer_M2A9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The experimental paradigm is interesting and translates well to this setting\n- The experiments are well-executed, and many alternative hypotheses are considered. Their conclusions seem justified from the experiments.\n- The conclusions are interesting, particularly those contrasting LSTMs with Transformers.\n\nWeaknesses:\n- The conclusions apply within the (interesting) experimental paradigm, but it's not clear how they translate to natural language and other tasks. For example, the experiments convinced me that Transformers are better than LSTMs at recalling verbatim context (which makes sense, given the way self-attention works), while LSTMs only maintain a gist of the context that they've processed. However (thinking about practical applications, which admittedly may be beyond the scope of this work), it's not clear to me when \"verbatim recall\" is necessary vs. it's better to have a semantic gist of context. Perhaps some more discussion about this would be useful.", "summary_of_the_paper": "This paper studies how LSTMs and Transformers represent prior context. In particular, this work adapts benchmark tasks for human working memory to neural language models. In this paradigm, the model is presented with <preface text> <a list of words> <intervening text> <the same list of words>, and perplexity / surprisal is measured for each of the words when the list of words appears a second time at the end of the input. Their results lead to a variety of insights about how LSTMs and Transformers trained on varying amounts of data with varying amounts of depth handle context.", "summary_of_the_review": "This paper thorough studies LSTMs and Transformers on a recall-based probing task. The work raises several interesting conclusions about LSTMs and Transformers, and they are well-supported by the experiments. However, it is somewhat unclear to me how these experiments might translate to other settings, and I'd like to hear more from the authors about this. Overall, I thought the paper was interesting and well-executed, if a bit narrow, and would be an interesting paper to the ICLR audience.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OYLVCt4UbA1", "reviewer": "Reviewer_r2TC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n - The question explored by the paper is interesting and important given the ubiquity of these models. The extent to which models such as GPT-2 can retrieve prior information looks very interesting and the difference in performance between Transformers and LSTMs is also worth noting.  \n\n - The results also add to possible reasons why in the space of large-scale LMs, Transformers have been more successful than LSTMs. \n\n - The paper is very well written and the narrative is clearly presented.  \n\nWeaknesses and Questions:\n\n - All the experiments seem to be intended on LMs trained on large/medium size datasets particularly Wikitext-103. Since the overall goal is to understand the ability of these models to access prior context, it would be interesting to compare models trained on smaller datasets or maybe even synthetic datasets, to check whether LSTMs perform worse/comparable/better than Transformers in that setting. It seems a bit surprising that LSTMs perform so poorly and it could be useful to characterize, at what stage the difference in performance between Transformers and LSTMs occur. \n\n - It seems like the dataset on which the models are trained also plays a significant role, looking at the difference in results between GPT-2 and other transformer results. Do you think the results for LSTMs could differ if trained on some other larger dataset? I see that when both the models are trained on the same Wikitext dataset, LSTMs perform poorly compared to Transformers but I wanted to understand why GPT-2 performs much better, if it is because of the dataset, and whether that could also affect LSTMs. \n\n- The experimental design seems a bit weakly motivated. It could be useful if the authors could discuss why the setup is more suitable for evaluating the ability of LMs to access memory as compared to some other arbitrary choices. \n\n  - The approach to compare LMs based on reduction in surprisal has been used by several recent works (as cited in the paper) but it is not clear if it is the best way to test a model's ability to access memory. Did you consider any alternatives? Any reason why this seemed like the best option? \n\n  - Also, why did you choose to go with noun lists and also in particular those noun lists? Since some models like GPT-2 did quite well with noun lists of length 10, I was curious to what extent they could maintain such a list. Did you try longer lists? \n\n  - In section 3.4, why did you consider the median surprisal? \n\n \nThe first experiment in section 4.4 does not make sense to me. How or why would a randomly initialized Transformer (or any LM model) reproduce (or assign a high probability to) a list of nouns based on English-like prompts? \n\nIt is mentioned that the depth of a Transformer seems to be a key attribute. Does the number of layers also affect LSTMs or does their performance saturate after a few layers? \n ", "summary_of_the_paper": "The paper attempts to investigate the ability of Transformer and LSTM based LMs to retrieve information about the prior context. To do so, they provide the LMs some prompts, a list of nouns and check their ability to reproduce the list by comparing the probabilities assigned to the same list of nouns (checking the reduction in surprisal).  \n\nThe premise of the work is that making use of prior context is an important functionality of language models and while several prior works have demonstrated that LMs do so to capture certain types of dependencies, their work aims to understand the extent to which Transformers and LSTM based LMs can retrieve words from the prior context.  \n\nVia their experiments, they try to evaluate three things: (1) How well can such LMs reproduce the noun list from memory verbatim, (2) the effect of intervening tokens on an LM's ability to access, and (3) whether the ability to access memory depends on semantic coherence of the prior information.  \n\nThey considered two versions of LSTM models and a few versions of Transformer models including GPT-2. Based on their results, it seems like Transformer based LMs performed consistently better than LSTMs on the defined task. The GPT-2 model performs much better than others. They also did some experiments to evaluate the role of attention weights by comparing with a model with shuffled attention weights.  ", "summary_of_the_review": "Overall it is an interesting and well-written paper trying to explore an important question with some sound experiments. At the same time, it seems to have certain weaknesses in the experimental setup and design which diminishes the conclusiveness of some of their results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EcYzE7-CP-V", "reviewer": "Reviewer_WHFY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies a very important question: to what extent can NLP models retrieve past input? Memorization is an important skill for communicating in natural language, and I am not familiar with too many works that try to measure it explicitly (see [1] for one related example). The setup proposed by the authors generally makes sense, and the results, at least with respect to GPT-2, are non-trivial: GPT-2 memorizes list1 verbatim, even after as many as 435 words. Overall these results will be of interest to the ICLR community.\n\nMy main concern about this paper relates to the LSTM baseline and the interpretation of its results. The authors basically present one positive result in this paper (GPT-2, though arguably the small transformer is also somewhat positive), and one negative result (the LSTM). They then use the negative result to say (or at least hint) something general about LSTMs. I am not sure I would jump to conclusions so quickly here. Take fig2 for instance. LSTMs and Transformer trained on Wiki103 exhibit a rather similar trend: we see very small differences between novel, permuted and repeated. The main difference is the loss value, which is higher for LSTMs. But this does not indicate that transformers memorize better than LSTMs, just that the transformer LM is of higher quality (by the way, what was the perplexity of the LSTM model?). Further, the LSTM was trained with 2 layers. The depth analysis of transformers shows that shallow transformers (<= 3 layers) are unable to memorize, so this factor alone could explain why the LSTM model was unable to memorize. On a related note: how many layers did the wiki103 transformer presented in figures 2-4 have? They surely don't have the same number of layers as the LSTM (as the authors didn't train any transformer with 2 layers). There are other confounding factors that make it hard to compare against the two model, e.g., the tokenization (the LSTM uses words, while the transformer uses BPEs). This also relates to a somewhat surprising choice by the authors (as far as I understand): to match the vocabulary sizes, which is hard to interpret given the different tokenization. \n\n \nTo summarize, here are the concrete questions I think should be addressed in the next version:\n1. What is the perplexity of the LSTM model?\n2. How deep is the transform that was used in the main experiment?\n\nIn addition, I would consider reframing the story about LSTMs.\n\nReferences:\n[1] https://arxiv.org/abs/1805.11653\n", "summary_of_the_paper": "This paper studies the short-term memory of NLP models. The authors design a memorization task of the format: <prefix> <list1 of words> <infix> <list2 of words>, where list1 and list2 are either identical, a permutation of one another, or unrelated. They train several models (an LSTM and several transformer variants, including GPT-2) with the language modeling objective, and then measure the median LM loss of each model across all the words on list2. They find that their LSTM does not show any memorization skills, the medium size transformer is able to memorize to a certain extent, and that GPT-2 memorizes almost perfectly, at least when list1=list2. Further analysis studies the factors that influence this memorization, including the length of the list, the length of the infix, and the depth of the model.\n", "summary_of_the_review": "Pros: \nA very important research question, a nice experimental setup, interesting results regarding GPT-2.\n\nCons:\nPotentially inaccurate conclusions regarding the connection between LSTMs and transformers due to various confounding factors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8zzKIB8YKew", "reviewer": "Reviewer_ax86", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a task for pretrained language models: extract in verbatim a list of nouns occurred in previous context. The authors compare two families of models (i.e. transformers and LSTMs) using the constructed benchmark and show some fruitful results. The main conclusion seems to be that transformers can perform robust verbatim recall while LSTM tends to store a semantic gist. The other interesting results include that transformer performance on this task improves with corpus size and transformer depth, and the performance of transformers is linked to the attention patterns.\n\nHowever, the paper doesn’t provide a deep dive into the topics (e.g. why pretrained transformers excel at this exact extraction task), although the authors propose some research directions in general in the conclusion. In my humble opinion, there are at least two directions that might give insights to explain such superior performance for transformers (which I think also are related works to the paper that should be appropriately discussed, the third one might be remotely related):\n\n#Attention\nAttention can be integrated into recurrent neural networks (https://arxiv.org/abs/1409.0473). If attention is the key ingredient that makes transformers perform better at this task, it will be interesting to see if integrating attention in RNNs can help with its performance.\n\n#Copy mechanism\nThe copy mechanism is a standard architecture for neural autoregressive models (see pointer net https://arxiv.org/pdf/1506.03134.pdf and copy mechanism https://arxiv.org/abs/1603.06393 as examples). Different questions that can be asked such as how copy (in RNNs and transformers) can help this task; will transformers continue to benefit from having a copy mechanism? What is the difference between transformer only and transformer + copy mechanism etc.\n\n#Linguistic structures (optional)\nThe robustness of transformers to intervening text might be attributed to transformers' capability of getting the linguistic instruction to copy, maybe particular for large models like GPT-2 (https://nlp.stanford.edu/pubs/hewitt2019structural.pdf). The experiments don’t seem to support or argue against such possible explanations while this might be interesting and relevant.\n\nIn conclusion, I think the constructed benchmark sheds some light on the inner workings of transformers; but these results are not well discussed with existing literature, making the novelty hard to spot and the presented research look not deep enough for ICLR.\n\nMinor writing/clarification:\n- In section 4.3 final paragraph, I suggest adding \"absolute value\" for the part involving GPT-2 increased 8%.\n- In discussion, it is said that \"overly effective short-term memory may underlie the tendency for transformers to repeat\" and cite (Holtzman et al. 2020). I don't think such argument is put forward in (Holtzman et al. 2020) which mainly discusses sampling strategy to improve generation performance (which mitigates the repetition issue).\n- In discussion, \"Similar limitations of LSTMs in retrieving past context have been shown in past work\", I think readers expect some references at the end of this sentence (some of the references afterwards can fit here).\n\n", "summary_of_the_paper": "Leveraging the verbatim extraction task, the authors compare LSTM and transformer architectures showing transfomers perform robust verbatim recall while LSTM tends to store a semantic gist; the question of depth, attention and how long such memory are stored are also investigated.", "summary_of_the_review": "By using verbatim extraction as a task, the authors reveal the difference of two popular autoregressive neural architectures: LSTM and Transformers. The authors also show that the phenomenon is related to the attention pattern, and have answered different questions including model depth, training size, intervening text impacts on the results.\n\nHowever, many related works do not seem to be well compared or even discussed. These include at least attention in RNNs and copy mechanism, and maybe also related to linguistic structures that transformer might capture. These missings make some important research questions unanswered for this work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "510VET3me3h", "reviewer": "Reviewer_Tpb6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The paper investigates an interesting task.\n- It is understandable, well structured and well written.\n- Experiments conducted with varying list sizes, different list types (arbitrary vs. coherent words), and different sizes and content of intervening text to provide insights. \n- The work is reproducible as the code is provided.\n \n**Weaknesses**\n\n- The data is synthetic and developed based on predefined templates that represent an extremely narrow sequences of words in the following format:   *[she created the following list of words:] [\"1st list of nouns separated by commas.\"] [she took a break blah blah.] [when she got back, she read the list again:] [\"2nd list of nouns\"].*    In addition, this way of text generation can potentially lead to implausible texts. For example, in several samples provided in the appendix, it's not clear whether \"got back\" in the above template actually refers to the first list or to the things that happen in the intervening text. Given these reasons, I think the reported results are less likely to be generalizable to real world datasets. \n\n- Although language modeling has been used as a motivation for investigating memory in neural models, the task formulation is very different from language modeling, which is the task of predicting the probability of a sequence of words occurring in a language, not necessary predicting a sequence of nouns separated by commas. LMs should be able to learn syntactic structure such as subject verb agreement, long-range dependencies among words, the role of function and content words, etc.\n\n- It's not clear if the task is truly evaluating models' ability in retrieving/recalling words from context. The model may accurately predict some of the words in the 2nd list due to their correlation with the intervening words but not because of their occurrence in the first list. \n\n- A limited number of words (~ 230 unique words) and lists (~ 230 lists of length <=10 words) are used in experiments. These number of words and sequences are of small scale and may be simply memorized by the model. \n   \n- Surprisal of each list is estimated by the \"median\" instead of \"average\" of the surprisal scores obtained for non-initial nouns in the list. Although median might be a better metric than average in many settings, here, memory should be evaluated against the entire list but not one of the words in the lists. \n\n- The number of parameters significantly differs between transformers and LSTM networks used in experiments, which naturally affects their performance and makes the results not directly comparable.  \n\n- It's not clear why transformer models are compared against vanilla LSTMs but not bidirectional LSTMs with attention. \n\n\n\n\n  \n", "summary_of_the_paper": "This paper argues that language models (LMs) require mechanisms to retain information about context words for better next word prediction. Several free recall experiments have been conducted to determine if such models can exactly retrieve/recall words from context. Specifically, different LMs are given sequences that contain texts in the following order: a preface text, a list of $n$ target nouns ($n\\leq 10$), an intervening text, a short prompt string, and a second list of nouns, which are either identical to the first list or different from it (i.e. similar words but permuted or completely different words). Reduction in surprisal from the first lists to the second lists is used as a metric to quantify and compare memory of different LMs. Surprisal of each list is estimated by the \"median\" of the surprisal scores obtained for non-initial nouns in the list. Several transformer models and LSTM-based language models of different sizes are compared in terms of reduction in surprisal. Results show that transformer models are superior in retrieving word identity and order from small/long-range intervening text (from a few to more than 400 tokens). Also more training data and greater model depth lead to better performance in case of transformer models. ", "summary_of_the_review": "This submission investigates the interesting question of memory in language models, but the task formulation, data and experimental setup can be improved and therefore I don't recommend the paper for publication at this time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores the memorization of tokens in prior context in LSTM and Transformer based language models. While all reviewers agree this is an interesting and important direction worth studying, they raise several concerns about the validity of the experimental setup and the conclusions drawn about LSTMs. Primarily the shallow depth of the LSTM architecture seems to confound the main conclusion about their inferiority (Reviewer WHFY). Further exploration about what makes transformers better (e.g. attention) is also important to provide a more complete picture (Reviewer ax86). Other concerns include the use of synthetic data (Reviewer tpb6), a limited number of noun lists (Reviewer r2TC) and the lack of discussion about the practical significance of verbatim recall (Reviewer M2A9). Overall, while the paper takes a step towards an important insight about pretained LMs, it needs to be polished further and hopefully can be published at a future conference.
Decision: Reject