Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors define memorization as any situation in which the model predicts the correct word as maximum probability given its context. They find that larger models memorize faster under this definition. They then study the asymptotic behavior of forgetting the memorized samples, finding that larger models likely never completely forget their memorized samples. They then study memorization paper according to part of speech, suggesting that rare words tend to lead to memorized sequences.
strengths_and_weaknesses: Strengths:

The question of how memorization occurs over the course of training is an interesting one. I like that they bring in implications for privacy, though I’d like to see more explicit links to the generalization literature.

lines 227-228: As the number of parameters increases, the model forgets less. Seems obvious, but actually really nice to see this demonstrated in such a simple way. In general, some of their results seem intuitive, so when their methods are convincing and well explained, it's satisfying. This sort of work often finds people complaining that these are things that we already know, but if they aren't tested in the current empirical work, this kind of result has a lot of value. The results on forgetting curves and the lower bounds that they exhibit at various scales was convincing and presented a good framing for the phenomenon of forgetting.

Weaknesses:

A broad issue I have is I'm not convinced "memorization" is the appropriate term for the behavior discussed. 

It's not clear to me that every time the correct word is predicted as maximum probability, it should be counted as memorization rather than generalised learning. Also, how is this affected by conditions where the same context appears multiple times in the training set with different labels? If every time a certain sequence occurs in training, it always occurs with a particular word, it seems that it's not memorization to indicate that that's the highest probability word.

 What does it mean for a part of speech to be memorized? I don't agree that a part of speech is memorized just because it is predicted correctly in context. There aren't that many parts of speech, so unlike when trying to predict the missing word in a unique sentence, I don't think that it's appropriate to claim that a sequence is memorized just because the predictions made are the correct part of speech. 207 highlights this problem, as it refers to the behavior first as learning and then as memorizing.

209: If the model is just learning to predict that the token is a numeral, and not learning to predict the specific numeral, it’s not clear to me that this is actually going to have privacy implications.

In terms of the uniqueness experiments, I would want to see what the effect of word frequency is in general before considering the role of completely unique words.

174: "classical ML concepts cannot even explain such a memorization trend." Not sure what this means, you should probably talk about the concepts that you feel fail to explain the trend.

I'm not convinced by the experiments that consider memorization relationship with overfitting as though these are two separate phenomena. What if the model is overfitting on a subset of the data, but generalization is improving overall, just not on examples that might be related to that subset? Seems like overfitting by this metric might be an emergent property of having enough memorization occur.

Problems with sparse literature review:

This work is not contextualized in the existing literature on training dynamics in language models, outside of recent literature on general scaling laws. I recommend looking through a variety of work on the area: https://www.semanticscholar.org/search?q=training%20dynamics%20language%20models&sort=relevance

In general, I take issue with the lack of citations to existing work. Is label memorization extending an existing concept? You wouldn't think so, from the paper. There is a mention of spaced repetition from the cognitive science literature, but the authors do not acknowledge that it has also been applied in natural language processing training: https://aclanthology.org/D17-1255/

Many concepts are introduced without any kind of citation to the existing literature: catastrophic forgetting, machine unlearning. Terms like "basin" and "phase transition" are introduced without background, so it's not clear what type of phenomenon each of these phrases is intended to refer to in this context.

Minor:
- 132: "generally monotonically decreasing" just say that it's generally decreasing. It's not monotonically decreasing, and appending "generally" in this context just means "not monotonic".
- 51: \citep should be \citet here
questions: Why did you pick the particular reference implementations that you picked? Are these considered standard for work like this?

lines 240-242: Does this imply that the special batch is not seen once immediately when introduced, but is seen later on? Or is this implying that the spectral batch is seen repeatedly in each epoch? That was not my understanding initially, en did makes this confusing

line 202: Maybe clarify that this means memorization of which words correspond to a particular part of speech? Or are you claiming that the correct part of speech is consistently selected? Because that should definitely not be referred to as memorization.

187-189, 92-95: Could you rewrite these sentences? I find them very confusing.
limitations: I didn't feel that the authors were precise enough in their definitions, leading them to use some words, like memorization, in ways that might not reflect what we usually mean when we talk about memorization. I think being more precise in their language would help this paper greatly, as the results that they describe might not influence other notions of memorization.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper demonstrates a detailed picture of training data memorization in the process of language modeling. The authors show that larger-sized language models memorize training data faster. Besides, this memorization happens before the overfitting of language modeling. Moreover, specific part-of-speech-tagged tokens are memorized faster like nouns and numbers during training. Lastly, they use the validation set as special insertion batch to test the forgetting mechanism with different model scales.
strengths_and_weaknesses: Strength
- The experimental results are detailed with the reasonable proposed metrics in data memorization.
- The forgetting identifier experiments are well-designed.
- In general, these fine-grained findings add value to drawing the picture of how and what data are memorized in transformer-based language modeling.

Weakness:
- Some example-based analysis (e.g. the POS-wise breakdown of memorization and the forgetting experiment) would be helpful in further demonstrating the procedure.
questions: The PoS are not of comparable size (e.g. numbers can be considered relative as a closed set compared to nouns, even though they are not actually, and the size of prop. nouns are relatively small compared to the set of nouns). Therefore, the memorization of nouns demonstrated in the experiments may not be as informative as the memorization of more fine-grained categories like numbers. 
limitations: One potential related reference: https://aclanthology.org/2022.tacl-1.1.pdf
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper analyzes the phenomenon of memorization by Transformer language models in the light of model size, catastrophic forgetting and memorization of unique tokens such as part of speech. The paper finds that larger the size, more the model is prone to memorize train distribution. Larger models also tends to forget less, and they seem to memorize unique parts of speech tokens such as nouns and numbers.


strengths_and_weaknesses: - The paper is well written and easy to follow.
- The experimental protocol to measure memorization makes sense, and a good spread of model sizes have been used. It is a bit confusing to understand which model family has been used (as the authors only state they have used "Transformer language model architectures" in L114).
- While I mostly agree with the empirical results, the issue I have with them is that the majority of experiments are conducted on Wiki Text 103, which is 500MB dataset. This dataset is significantly small compared to those being used to train the large models (most of the model sizes considered in this paper are typically trained on significantly larger datasets). Thus, while undoubtedly the models somewhat favor memorization, it is unclear whether they still do the same when the input data is large enough. While I do agree such a study would be quite expensive and perhaps intractable, a disclaimer or limitation of this study should have been explicitly mentioned (on the data size front).
- The study on catastrophic forgetting is quite interesting - it is nice to see a forgetting baseline set by the models of individual sizes.
- The study on parts of speech is especially interesting, however the section lacks a bit on the explanation of the behaviour. Specifically, nouns, proper nouns and numerals being memorized noticeably faster might be due to the frequency effect (Wei, J., Garrette, D., Linzen, T., & Pavlick, E., Frequency Effects on Syntactic Rule Learning in Transformers, arXiv:2109.07020 [cs], 2021). It would be interesting/beneficial for the paper to study this angle further even using Wiki Text 103 dataset, to see if there are any correlation with memorization with the frequency of individual POS class.

*Update*: I upgraded my initial evaluation after reading the author responses.

questions: 
- In L122, the authors seem to refer the BookCorpus as "RoBERTa" dataset. Any specific reason why?
- Furthermore, BookCorpus which is used to trained RoBERTa model comes in two sizes: 16GB (which is used for RoBERTa-base) and 160GB (which is used for RoBERTa-large). I'm not sure which one is being used here.
- What is the model family used in these experiments? Are the results specific to Causal language models or Masked Language Models? Do the authors expect the results to differ based on the class of the model family being used?
- In the catastrophic forgetting experiment, the _special batch_ is taken from the validation set of Wiki 103, which is in-distribution with the training set. If I were to choose a special batch being significantly out-of-distribution with respect to the training data, would the same results hold?


limitations: The authors have not discussed the limitations of their work thoroughly. As I mentioned above, one big limitation is the usage of a significantly small dataset to explain the behaviour of large language models, those of which are typically trained on much larger datasets (for eg, The Pile or C4). This is a window of improvement for the paper, and the authors should try to clearly set the expectations early on for the reader.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper investigates the memorization and forgetting dynamics of large language models during training, as a function of model and dataset sizes, learning rate, and task (i.e. causal vs masked language modeling). The paper measures memorization as the proportion of correctly predicted labels (which corresponds to predicting the correct next/masked token) during training, and forgetting as the decline in memorization.

The main experimental results show that: (i) larger models memorize data faster and forget less; (ii) memorization occurs before overfitting; and (iii) memorization occurs for both small and large datasets, and for both causal and masked language modeling, regardless of the learning rate. The paper further investigates the memorization mechanism and shows that the presence of unique identifiers in the training data can contribute to memorization, e.g. models tend to memorize numerals, nouns and proper nouns faster than other parts-of-speech. Moreover, the paper shows that while memories from a dataset are forgotten when training with other data, the degree to which forgetting occurs has a lower bound that depends on the model size (larger models having a higher lower bound). 
strengths_and_weaknesses: 
This paper addresses an important aspect of large language models: their ability to memorize the training data without necessarily overfitting. This unexpected behavior has been observed in previous work and calls into question the classical framework of bias-variance tradeoff. However, little is understood about the dynamics of memorization and forgetting in these models. Yet, this can have problematic implications  for real-world deployments related to e.g., privacy (e.g., models can memorize personal identifiers or other sensitive information) and fairness (e.g., models can memorize negative stereotypes about groups of individuals). This paper gives insights into the memorization and forgetting dynamics in large language model training and about the scaling laws that seem to govern these models. 
Overall, the paper is well written and clear and the main claims are well supported by experimental evidence. 
questions: - Any hypotheses as to why memorization is faster when adding tokens to the dictionary without even using them?

limitations: authors have discussed the limitations
review_title: null

Meta-review: metareview: This paper studies the underlying training and memorization dynamics of very large language models. The main take aways are that larger-sized language models memorize training data faster, and that this memorization happens before the overfitting of language modeling. Tokens with certain part-of-speech tags (nouns, numerals) seem to be memorized faster during training. 

Overall, most reviewers feel positively about this paper, agreeing that it tackles an important problem and that it provides a solid contribution. The experimental results are detailed and use reasonable metrics for data memorization, including the forgetting identifier experiments. Some of the weaknesses that have been pointed out (e.g. regarding the significance of the part-of-speech tags experiment, clarifying the criteria for memorization, etc.) seem to have been well addressed during the author response. Therefore, I recommend acceptance.


confidence: Certain
Decision: Accept