Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper argues that language models (LMs) require mechanisms to retain information about context words for better next word prediction. Several free recall experiments have been conducted to determine if such models can exactly retrieve/recall words from context. Specifically, different LMs are given sequences that contain texts in the following order: a preface text, a list of $n$ target nouns ($n\leq 10$), an intervening text, a short prompt string, and a second list of nouns, which are either identical to the first list or different from it (i.e. similar words but permuted or completely different words). Reduction in surprisal from the first lists to the second lists is used as a metric to quantify and compare memory of different LMs. Surprisal of each list is estimated by the "median" of the surprisal scores obtained for non-initial nouns in the list. Several transformer models and LSTM-based language models of different sizes are compared in terms of reduction in surprisal. Results show that transformer models are superior in retrieving word identity and order from small/long-range intervening text (from a few to more than 400 tokens). Also more training data and greater model depth lead to better performance in case of transformer models. 
main_review: **Strengths**

- The paper investigates an interesting task.
- It is understandable, well structured and well written.
- Experiments conducted with varying list sizes, different list types (arbitrary vs. coherent words), and different sizes and content of intervening text to provide insights. 
- The work is reproducible as the code is provided.
 
**Weaknesses**

- The data is synthetic and developed based on predefined templates that represent an extremely narrow sequences of words in the following format:   *[she created the following list of words:] ["1st list of nouns separated by commas."] [she took a break blah blah.] [when she got back, she read the list again:] ["2nd list of nouns"].*    In addition, this way of text generation can potentially lead to implausible texts. For example, in several samples provided in the appendix, it's not clear whether "got back" in the above template actually refers to the first list or to the things that happen in the intervening text. Given these reasons, I think the reported results are less likely to be generalizable to real world datasets. 

- Although language modeling has been used as a motivation for investigating memory in neural models, the task formulation is very different from language modeling, which is the task of predicting the probability of a sequence of words occurring in a language, not necessary predicting a sequence of nouns separated by commas. LMs should be able to learn syntactic structure such as subject verb agreement, long-range dependencies among words, the role of function and content words, etc.

- It's not clear if the task is truly evaluating models' ability in retrieving/recalling words from context. The model may accurately predict some of the words in the 2nd list due to their correlation with the intervening words but not because of their occurrence in the first list. 

- A limited number of words (~ 230 unique words) and lists (~ 230 lists of length <=10 words) are used in experiments. These number of words and sequences are of small scale and may be simply memorized by the model. 
   
- Surprisal of each list is estimated by the "median" instead of "average" of the surprisal scores obtained for non-initial nouns in the list. Although median might be a better metric than average in many settings, here, memory should be evaluated against the entire list but not one of the words in the lists. 

- The number of parameters significantly differs between transformers and LSTM networks used in experiments, which naturally affects their performance and makes the results not directly comparable.  

- It's not clear why transformer models are compared against vanilla LSTMs but not bidirectional LSTMs with attention. 




  

summary_of_the_review: This submission investigates the interesting question of memory in language models, but the task formulation, data and experimental setup can be improved and therefore I don't recommend the paper for publication at this time.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper attempts to investigate the ability of Transformer and LSTM based LMs to retrieve information about the prior context. To do so, they provide the LMs some prompts, a list of nouns and check their ability to reproduce the list by comparing the probabilities assigned to the same list of nouns (checking the reduction in surprisal).  

The premise of the work is that making use of prior context is an important functionality of language models and while several prior works have demonstrated that LMs do so to capture certain types of dependencies, their work aims to understand the extent to which Transformers and LSTM based LMs can retrieve words from the prior context.  

Via their experiments, they try to evaluate three things: (1) How well can such LMs reproduce the noun list from memory verbatim, (2) the effect of intervening tokens on an LM's ability to access, and (3) whether the ability to access memory depends on semantic coherence of the prior information.  

They considered two versions of LSTM models and a few versions of Transformer models including GPT-2. Based on their results, it seems like Transformer based LMs performed consistently better than LSTMs on the defined task. The GPT-2 model performs much better than others. They also did some experiments to evaluate the role of attention weights by comparing with a model with shuffled attention weights.  
main_review: Strengths: 

 - The question explored by the paper is interesting and important given the ubiquity of these models. The extent to which models such as GPT-2 can retrieve prior information looks very interesting and the difference in performance between Transformers and LSTMs is also worth noting.  

 - The results also add to possible reasons why in the space of large-scale LMs, Transformers have been more successful than LSTMs. 

 - The paper is very well written and the narrative is clearly presented.  

Weaknesses and Questions:

 - All the experiments seem to be intended on LMs trained on large/medium size datasets particularly Wikitext-103. Since the overall goal is to understand the ability of these models to access prior context, it would be interesting to compare models trained on smaller datasets or maybe even synthetic datasets, to check whether LSTMs perform worse/comparable/better than Transformers in that setting. It seems a bit surprising that LSTMs perform so poorly and it could be useful to characterize, at what stage the difference in performance between Transformers and LSTMs occur. 

 - It seems like the dataset on which the models are trained also plays a significant role, looking at the difference in results between GPT-2 and other transformer results. Do you think the results for LSTMs could differ if trained on some other larger dataset? I see that when both the models are trained on the same Wikitext dataset, LSTMs perform poorly compared to Transformers but I wanted to understand why GPT-2 performs much better, if it is because of the dataset, and whether that could also affect LSTMs. 

- The experimental design seems a bit weakly motivated. It could be useful if the authors could discuss why the setup is more suitable for evaluating the ability of LMs to access memory as compared to some other arbitrary choices. 

  - The approach to compare LMs based on reduction in surprisal has been used by several recent works (as cited in the paper) but it is not clear if it is the best way to test a model's ability to access memory. Did you consider any alternatives? Any reason why this seemed like the best option? 

  - Also, why did you choose to go with noun lists and also in particular those noun lists? Since some models like GPT-2 did quite well with noun lists of length 10, I was curious to what extent they could maintain such a list. Did you try longer lists? 

  - In section 3.4, why did you consider the median surprisal? 

 
The first experiment in section 4.4 does not make sense to me. How or why would a randomly initialized Transformer (or any LM model) reproduce (or assign a high probability to) a list of nouns based on English-like prompts? 

It is mentioned that the depth of a Transformer seems to be a key attribute. Does the number of layers also affect LSTMs or does their performance saturate after a few layers? 
 
summary_of_the_review: Overall it is an interesting and well-written paper trying to explore an important question with some sound experiments. At the same time, it seems to have certain weaknesses in the experimental setup and design which diminishes the conclusiveness of some of their results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how LSTMs and Transformers represent prior context. In particular, this work adapts benchmark tasks for human working memory to neural language models. In this paradigm, the model is presented with <preface text> <a list of words> <intervening text> <the same list of words>, and perplexity / surprisal is measured for each of the words when the list of words appears a second time at the end of the input. Their results lead to a variety of insights about how LSTMs and Transformers trained on varying amounts of data with varying amounts of depth handle context.
main_review: Strengths:
- The experimental paradigm is interesting and translates well to this setting
- The experiments are well-executed, and many alternative hypotheses are considered. Their conclusions seem justified from the experiments.
- The conclusions are interesting, particularly those contrasting LSTMs with Transformers.

Weaknesses:
- The conclusions apply within the (interesting) experimental paradigm, but it's not clear how they translate to natural language and other tasks. For example, the experiments convinced me that Transformers are better than LSTMs at recalling verbatim context (which makes sense, given the way self-attention works), while LSTMs only maintain a gist of the context that they've processed. However (thinking about practical applications, which admittedly may be beyond the scope of this work), it's not clear to me when "verbatim recall" is necessary vs. it's better to have a semantic gist of context. Perhaps some more discussion about this would be useful.
summary_of_the_review: This paper thorough studies LSTMs and Transformers on a recall-based probing task. The work raises several interesting conclusions about LSTMs and Transformers, and they are well-supported by the experiments. However, it is somewhat unclear to me how these experiments might translate to other settings, and I'd like to hear more from the authors about this. Overall, I thought the paper was interesting and well-executed, if a bit narrow, and would be an interesting paper to the ICLR audience.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the short-term memory of NLP models. The authors design a memorization task of the format: <prefix> <list1 of words> <infix> <list2 of words>, where list1 and list2 are either identical, a permutation of one another, or unrelated. They train several models (an LSTM and several transformer variants, including GPT-2) with the language modeling objective, and then measure the median LM loss of each model across all the words on list2. They find that their LSTM does not show any memorization skills, the medium size transformer is able to memorize to a certain extent, and that GPT-2 memorizes almost perfectly, at least when list1=list2. Further analysis studies the factors that influence this memorization, including the length of the list, the length of the infix, and the depth of the model.

main_review: This paper studies a very important question: to what extent can NLP models retrieve past input? Memorization is an important skill for communicating in natural language, and I am not familiar with too many works that try to measure it explicitly (see [1] for one related example). The setup proposed by the authors generally makes sense, and the results, at least with respect to GPT-2, are non-trivial: GPT-2 memorizes list1 verbatim, even after as many as 435 words. Overall these results will be of interest to the ICLR community.

My main concern about this paper relates to the LSTM baseline and the interpretation of its results. The authors basically present one positive result in this paper (GPT-2, though arguably the small transformer is also somewhat positive), and one negative result (the LSTM). They then use the negative result to say (or at least hint) something general about LSTMs. I am not sure I would jump to conclusions so quickly here. Take fig2 for instance. LSTMs and Transformer trained on Wiki103 exhibit a rather similar trend: we see very small differences between novel, permuted and repeated. The main difference is the loss value, which is higher for LSTMs. But this does not indicate that transformers memorize better than LSTMs, just that the transformer LM is of higher quality (by the way, what was the perplexity of the LSTM model?). Further, the LSTM was trained with 2 layers. The depth analysis of transformers shows that shallow transformers (<= 3 layers) are unable to memorize, so this factor alone could explain why the LSTM model was unable to memorize. On a related note: how many layers did the wiki103 transformer presented in figures 2-4 have? They surely don't have the same number of layers as the LSTM (as the authors didn't train any transformer with 2 layers). There are other confounding factors that make it hard to compare against the two model, e.g., the tokenization (the LSTM uses words, while the transformer uses BPEs). This also relates to a somewhat surprising choice by the authors (as far as I understand): to match the vocabulary sizes, which is hard to interpret given the different tokenization. 

 
To summarize, here are the concrete questions I think should be addressed in the next version:
1. What is the perplexity of the LSTM model?
2. How deep is the transform that was used in the main experiment?

In addition, I would consider reframing the story about LSTMs.

References:
[1] https://arxiv.org/abs/1805.11653

summary_of_the_review: Pros: 
A very important research question, a nice experimental setup, interesting results regarding GPT-2.

Cons:
Potentially inaccurate conclusions regarding the connection between LSTMs and transformers due to various confounding factors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Leveraging the verbatim extraction task, the authors compare LSTM and transformer architectures showing transfomers perform robust verbatim recall while LSTM tends to store a semantic gist; the question of depth, attention and how long such memory are stored are also investigated.
main_review: The paper proposes a task for pretrained language models: extract in verbatim a list of nouns occurred in previous context. The authors compare two families of models (i.e. transformers and LSTMs) using the constructed benchmark and show some fruitful results. The main conclusion seems to be that transformers can perform robust verbatim recall while LSTM tends to store a semantic gist. The other interesting results include that transformer performance on this task improves with corpus size and transformer depth, and the performance of transformers is linked to the attention patterns.

However, the paper doesn’t provide a deep dive into the topics (e.g. why pretrained transformers excel at this exact extraction task), although the authors propose some research directions in general in the conclusion. In my humble opinion, there are at least two directions that might give insights to explain such superior performance for transformers (which I think also are related works to the paper that should be appropriately discussed, the third one might be remotely related):

#Attention
Attention can be integrated into recurrent neural networks (https://arxiv.org/abs/1409.0473). If attention is the key ingredient that makes transformers perform better at this task, it will be interesting to see if integrating attention in RNNs can help with its performance.

#Copy mechanism
The copy mechanism is a standard architecture for neural autoregressive models (see pointer net https://arxiv.org/pdf/1506.03134.pdf and copy mechanism https://arxiv.org/abs/1603.06393 as examples). Different questions that can be asked such as how copy (in RNNs and transformers) can help this task; will transformers continue to benefit from having a copy mechanism? What is the difference between transformer only and transformer + copy mechanism etc.

#Linguistic structures (optional)
The robustness of transformers to intervening text might be attributed to transformers' capability of getting the linguistic instruction to copy, maybe particular for large models like GPT-2 (https://nlp.stanford.edu/pubs/hewitt2019structural.pdf). The experiments don’t seem to support or argue against such possible explanations while this might be interesting and relevant.

In conclusion, I think the constructed benchmark sheds some light on the inner workings of transformers; but these results are not well discussed with existing literature, making the novelty hard to spot and the presented research look not deep enough for ICLR.

Minor writing/clarification:
- In section 4.3 final paragraph, I suggest adding "absolute value" for the part involving GPT-2 increased 8%.
- In discussion, it is said that "overly effective short-term memory may underlie the tendency for transformers to repeat" and cite (Holtzman et al. 2020). I don't think such argument is put forward in (Holtzman et al. 2020) which mainly discusses sampling strategy to improve generation performance (which mitigates the repetition issue).
- In discussion, "Similar limitations of LSTMs in retrieving past context have been shown in past work", I think readers expect some references at the end of this sentence (some of the references afterwards can fit here).


summary_of_the_review: By using verbatim extraction as a task, the authors reveal the difference of two popular autoregressive neural architectures: LSTM and Transformers. The authors also show that the phenomenon is related to the attention pattern, and have answered different questions including model depth, training size, intervening text impacts on the results.

However, many related works do not seem to be well compared or even discussed. These include at least attention in RNNs and copy mechanism, and maybe also related to linguistic structures that transformer might capture. These missings make some important research questions unanswered for this work.
review_title: null

Meta-review: This paper explores the memorization of tokens in prior context in LSTM and Transformer based language models. While all reviewers agree this is an interesting and important direction worth studying, they raise several concerns about the validity of the experimental setup and the conclusions drawn about LSTMs. Primarily the shallow depth of the LSTM architecture seems to confound the main conclusion about their inferiority (Reviewer WHFY). Further exploration about what makes transformers better (e.g. attention) is also important to provide a more complete picture (Reviewer ax86). Other concerns include the use of synthetic data (Reviewer tpb6), a limited number of noun lists (Reviewer r2TC) and the lack of discussion about the practical significance of verbatim recall (Reviewer M2A9). Overall, while the paper takes a step towards an important insight about pretained LMs, it needs to be polished further and hopefully can be published at a future conference.
Decision: Reject