Memorization Without Overfitting:  Analyzing the Training Dynamics of Large Language Models
Authors: Kushal Tirumala
Aram H. Markosyan
Luke Zettlemoyer
Armen Aghajanyan
Venue: NeurIPS 2022 Accept (2022)

Abstract: Despite their wide adoption, the underlying training and memorization dynamics of very large language models is not well understood. We empirically study exact memorization in causal and masked language modeling, across model sizes and throughout the training process. We measure the effects of dataset size, learning rate, and model size on memorization, finding that larger language models memorize training data faster across all settings. Surprisingly, we show that larger models can memorize a larger portion of the data before over-fitting and tend to forget less throughout the training process. We also analyze the memorization dynamics of different parts of speech and find that models memorize nouns and numbers first; we hypothesize and provide empirical evidence that nouns and numbers act as a unique identifier for memorizing individual training examples. Together, these findings present another piece of the broader puzzle of trying to understand what actually improves as models get bigger.

scale up Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models
Kushal Tirumala"
Aram H. Markosyan'
Luke Zettlemoyer
Armen Aghajanyan
Meta Al Research
(ktirumala, amarkos, 1sz, armenag)@fb.com com
Abstract
Despite their wide adoption, the underlying training and memorization dynamics of very large language models well understood. We empirically study exact memorization in causal and masked language modeling, across model sizes and throughout the training process. We measure the effects of dataset size, learning rate, and model size on memorization, finding that larger language models memo- rize training data faster across all settings. Surprisingly. can memorize larger portion of the data before over-fitting and tend to forget less throughout the training process. We also analyze the memorization dynamics of different parts of speech and find that models memorize nouns and numbers first; we hypothesize and provide empirical evidence that nouns and numbers act unique identifier for memorizing individual training examples. Together, these findings present another piece of the broader puzzle trying to understand what actually improves as models get bigger.
show that larger models
Introduction
The rate and extent to which model memorizes its training data are key statistics that provide evidence about how is likely to generalize to new test instances. Classical frameworks, such as bias-variance tradeoff (31), argued for fitting training without full memorization. However, recent work has established a more symbiotic relationship between memorization and generalization in deep learning [13 26 28). This paper empirically studies memorization in causal and masked language modeling, across model sizes and throughout the training process.
Much of the recent performance gains for language models have come from scale, with the most recent models reaching up to 10 parameters 122 83 Larger models are also known to memorize more training data [16), which is crucial component of their improved generalization. However, perhaps surprisingly, relatively little work has been done understanding the impact scale on the dynamics of language model memorization over training. Existing work focuses on analyzing memorization post-training [16] 88, 195). In this work, we study the memorization and forgetting dynamics in language models, with a focus better measuring how they change model size. Our primary contributions include:
1. We measure the dependence of memorization dynamics over training on model size (and other factors such dataset size, overfitting, and learning rate). We find that larger language models memorize training data faster ($ 4
2. We design controlled experiments that allow characterize the forgetting curves in language models (i.e., how language models naturally forget memories throughout training).
"Equal Contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Our empirical studies show that forgetting curves have lower bounds we coin this as the forgetting baseline and that this baseline increases with model scale, i.e. increasing model scale mitigates forgetting ($ 5)
3.
We analyze the rates of memorization of different parts of speech, finding that nouns and numbers are memorized much more quickly than other parts of speech ($ 4.4). We hypothesize this is because the set of nouns and numbers can be seen as a unique identifier for particular sample. We provide evidence this hypothesis by analyzing the rates of memorization in the setting of an existing unique identifier ($
Together, these findings present another piece of the broader puzzle of trying to understand the unique training dynamics that emerge as models grow in size.
2 Background and Related Work
Memorization in Language Models: Unintended memorization is a known challenge for language models [14 [8], which makes them open to extraction attacks [T5] [9] and membership inference
attacks [1 64 although there has been work on mitigating these vulnerabilities [5L 88) Recent work has argued that memorization is not exclusively harmful, and can be crucial for certain types of
generalization (e.g., on QA tasks) [1 46 87) while also allowing the models to encode significant
amounts of world or factual knowledge 4 35, 71 There is also a growing body of work analyzing
fundamental properties of memorization in language models [66] 47 60 95). Most related to our
work Carlini et al. [16] analyzes memorization of fully trained language models and observes dependence on model scale, training data duplication, and prompting context length. While we also study scaling behavior, our focus instead is on the memorization dynamics throughout training.
Language Model Training Dynamics: Previous work has extensively analyzed training dynamics to understand how neural models acquire information over training [T 30 34 66,774, Saphra and Lopez [80] were the first to analyze training dynamics for language modeling, focusing on the evolution of internal representations over pre-training. This inspired line of work analyzing how neural language models learn linguistic structure/world knowledge [20] 21 [33. individual words (17], and cross-lingual structure [10] over pre- training. This analysis has been extended to many downstream
tasks, including text summarization (33,, machine/speech translation [8] 86 92], and various NLP
tasks [6 61].
Forgetting in Language Models: There has also been work studying memory degradation (forgetting) in language models. Catastrophic forgetting or catastrophic interference, first reported in [59] (77.. studies how neural networks tend forget the information from previous trained tasks or training batches, when trained on new data. This provides a key challenge for continual learning (or life-long learning) 11,, where the goal is to gradually learn from a single pass over a, typically very large, stream of data. A number of mechanisms have been proposed for increasing robustness against
58 82). There is also growing body of work demonstrating that both model and dataset scale can make models more resistant to forgetting [65] 75 as well as work characterizing how forgetting naturally occurs in image classifiers [90] and how forgetting can improve training efficiency [3] Machine unlearning is technique that forces a trained model to forget a previously learned sample [12 [54), which is primarily motivated by data protection and privacy regulations (37 57 78 11). Our work is unique in its focus on measuring forgetting during training, and quantifying how it varies with scale.
catastrophic forgetting 22 18 24 49
Scaling Laws:
We have consistently seen performance gains by scaling model size 3
22
73
76,
83
and scale itself has been known to push internal model behavior away from classical -aaiiaae regimes [67]. Recent efforts have focused on trying to model the scaling laws for language models,
including data and model size 44 77., applications to transfer learning [40], routing networks 23)
and various autoregressive generative tasks (39]. While the bulk of work in scaling laws has been empirical, an interesting line of work focuses on theoretically explaining neural scaling laws (8]
Most scaling laws focus only cross-entropy loss, while study memorization (defined in $ 3)

---

3 Experimental Setup
In order to perform a large-scale study of the dynamies of memorization over training. our mem- orization metric must be reasonably easy to compute but also precise enough to tell us how much the model will actually remember from the training data. Label memorization [|2] 94 is an ideal candidate, because it has consistently provided theoretical insight into underlying properties of neural networks, remains applicable in empirical settings, and is relatively cheap to compute. We formulate our metric as analog of label memorization for self-supervised settings.
Definition 1 Let V denote the vocabulary size. Let C denote a set of contexts, which can be thought of as a list of tuples (8, y) where is an input context (incomplete block of text) and y is the index of the ground truth token in the vocabulary that completes the block of text. Let S denote the set of input contexts, and let f S + RV denote a language model. A context c = (5.y) E C is memorized if arg) y.
Note that single word can appear as the ground-truth token for multiple contexts. For given set of contexts C (i.e a given training dataset), we can then analyze the proportion of memorized contexts
M()) Eleaec 1 (argmax([(s)) v)
CI
We refer to this as exact memorization, although it can also be seen as accuracy since we measure how often the argmax of the language model matches the ground truth token. Throughout this work, when we refer to memorization, we will be referring to Definition I Junless we specify otherwise.
We define T be threshold value for M()), and denote T(N,T) the minimal number of times a language model / with N parameter needs to see each training datapoint in order to satisfy M())= 2 т. When leveraging bigger datasets, models unable to train for multiple epochs, so we instead consider memorization on a per-update basis. We introduce Mapdate (f. U) as the memorization on the batch of data on which the model performs the U'th gradient descent update, and define Tupdate [N, 7) as the minimal number of gradient descent updates a language model with N parameters needs to perform, to satisfy Mupdate (f.U) 2 т.
Previous work analyzing language modeling memorization defines memorization differently. Moti- vated by privacy concerns, both I5 and [] define memorization from training data extraction standpoint, in which a string 8 is extractable if can be produced by interacting with the language model. More specifically, [15] defines string being k-eidetic memorized if it is extractable and appears in at most l training examples. [66] defines a string as k-memorized if the language model can produce via prompting with & tokens of context from training data. This definition only works for causal language modeling because of the dependence on prompting with training data; for masked language modeling [6 uses Definition above. Note that if an example is exactly memorized, it is extractable by definition. In other words, both the set of k-eidetic memorized tokens and the set of k-memorized tokens contain the set of exactly memorized tokens (formally, different exactly memorized tokens may be contained in different sets, depending on k). Therefore, analyzing exact memorization gives a type of lower bound on the k-eidetic memorization and k- k-memorization.
In different line of work motivated by estimating the influence of individual training examples, [95] defines a training example x as memorized if the difference in expected model performance (where model performance is defined as MJ) above) over subsets of data including x: and subsets of data not including x, is sufficiently large. This definition pulls from previous work in theoretically analyzing label memorization in classification settings 227
Model Architecture: We replicate publicly available references for Transformer language model architectures [%).. We use the 125M, 355M, 1.3B, 2.7B, 6.7B, and 13B model configurations (see $ $A.4 for more architectural and training details). We study both causal and masked language models. We train using the FairSeq framework [69] with PyTorch (70) as the underlying framework. For our larger models, we use the fully sharded data-parallel implementation available in FairScale ( and use Aim experiment tracking (6]
Datasets: We use two existing datasets across all our experiments: the WIKITEXT-II 103 benchmark containing around 103 million tokens [62]. and the RoBERTa corpus [55] used to train the original
Label memorization in these prior works usually refers to perfectly fitting given set of labels
3

---

RoBERTa model, containing around 39 billion tokens (we refer to this as the ROBERTA dataset). We use both datasets in section and primarily use WIKITEXT- 103 in other sections due computational restrictions.
Larger Language Models Memorize Faster
Larger neural language models are known to be more sample efficient and require fewer optimization steps to reach the same performance [4 while also converging faster [52). where performance is usually defined as test perplexity. In this section, study T(N, r) the training set function of to answer this question.
150
T= 0.95
102
==
T(N, 0.9)
T= 0.6
T(N, T)
T= 0.5
T 0.4
10a
50
0
125M 355M
1.38
2.7B N
6.7B
13B
10a
10a
10
N
Figure We show [NN, r), which is the number of times a language model needs to see each training example before memorizing r fraction of the training data, function of model size N. Result are for causal language modeling on WIKITEXT 103, right plot is on log-log scale. Note that generally larger models memorize faster, regardless of T.
In the left plot of Figure 1 fix memorization threshold 0.9 and examine T(NN T) increase N. The larger language models need to see each training datapoint fewer times to achieve 90% exact memorization of the training set; in other words, T(N, 0.9) is monotonically decreasing in N. When we vary between 0.4 and 0.95 in the right plot of Figure we still observe that T(N, generally decreasing with N For fixed N, T(N, is increasing in т, which is expected since memorizing more of the training set requires training the model for more epochs. More interestingly, increasing 7 smoothly transitions T(N, r) from constant in N, to exponentially decreasing in N (the axes are on a log-log scale).
10a
T= 0.8
0.9
T(N, T
0.6
10a
T= 0.4
T= 0.2
10a
10a
1020
N
Figure 2: T(N, T) function of N (shown log-log scale), for various values of in masked language modeling on WIKITEXTI03. We show that larger models initially memorize training data slower, but reach high proportions of training data memorization faster.
*We fix 0.4 as the lower bound for the range because any lower value for the memorization threshold achieved within the first few epochs across all model scales (the line in Figure i essentially flat), and 0.95 as the upper bound because higher values require unreasonably long training time for smaller models.

---

4.1 Dependence on Language Modeling Task and Dataset Size
To investigate the dependence of our observations on the particular language modeling task, we repeat this analysis for the masked language modeling task on WIKITEXTIOS with mask probability 0.15. Unlike in causal language modeling, Figure [ shows that T(N, 7) is not monotonically decreasing in N for lower values of 7, and is monotonically decreasing in N for higher values of т, where the phase transition between these two regimes occurs between memorize the training data quicker initially and slower in the long run (e.g., right plot of Figure 11
0.6 and r = 0.7. Smaller models
10a
t= 0.6
F
1033
0.42
Tupdate (N, T)
Tupdate (N.
= 0.4
T= 0.5.5
== 0.35
10a
0.3
102
0.25
0.1
108
109
10
108
109
10
N
N
Figure 3: We show Tapdate (N, 7). which is the number of gradient descent updates U a language model needs to perform before memorizing fraction of the data given on the U'th update, function of model size N. Result are for causal (Left) and masked (Right) language modeling on the ROBERTA dataset, on log-log scale. We show that larger models memorize faster, regardless of T.
Language model training is heavily dependent on the dataset size 44 and therefore we expect M(f) to be similarly impacted. In Figure 3 we analyze training set memorization on the much bigger ROBERTA dataset for both masked and causal language modeling. With large datasets such as ROBERTA dataset, it becomes infeasible to perform multiple epochs and evaluate memorization on the entire training set, especially when training larger models. Consequently, we focus on smaller values of r and investigate the number of gradient descent updates it takes to reach memorization thresholds, ie.. Tupstate [N,T,, In Figure 3 we observe similar trend as Figure I where T Tupdaate (N, r) is monotonically decreasing with N for various in both masked and causal language modeling. Unlike with WIKITEXTI03, 103, masked language modeling does not have phase transition for T.
4.2 Why Do Larger Models Memorize Faster?
A natural question at this point is to ask why larger models memorize faster? Typically, memorization associated with overfitting, which offers a potentially simple explanation. order disentangle memorization from overfitting, we examine memorization before overfitting occurs, where we define overfitting occurring as the first epoch when the perplexity of the language model on a validation set increases. Surprisingly, we see in Figure as we increase the number of parameters, memorization before overfitting generally increases, indicating that overfitting by itself cannot completely explain the properties of memorization dynamics model scale increases.
The learning rate is constant across our training configurations. Intuitively, larger learning rates should lead to quicker memorization. To investigate to what extent our results can be explained by learning rate, we take a subset of the architectures available above and train on the WIKITEXTI 03 dataset across standard range of learning rates while measuring memorization, in Figure 5 Even if we fix a learning rate, larger models reach 0.9 memorization faster, suggesting that our results are not caused solely by differences in learning rates. Interestingly, sensitivity to learning rate generally decreases as we increase the model size. We also notice Figure 5 that TNN, r) goes down initially (for low LRs) and eventually rises (for high LRs), and as the long as the chosen learning rate places us near the lowest point on the curve, the memorization dynamics do not change significantly (note that axes are log-scale). This result is consistent with the growing intuition that for neural language models past a particular scale, the learning rate is not a significant hy hyperparameter
*"Pase transition" is used in physics to describe significant changes in system behavior that occurs due to varying parameter, such as temperature. In this case, the parameter
5

---

0.80
0.65
Causal LM
Masked LM
0.60
0.75
M(f)
M(1)
0.55
0.50
0.70
0.45
0.65
10
10a
1010
10
10
10
N
Figure Proportion of training data memorized Mf) before overfitting, as function of model size N (plotted on a log scale). Results are for causal (left) and masked (right) language modeling on WIKITEXTI03, Note that larger models memorize more before overfitting.
125M
1.38
6.7B
355M
2.7B
10a
T(N. 0.9)
101
10
10-3
LR
Figure 5: Examining the effect of learning rate (LR) on number of times model needs to see each training example in order to reach 0.9 proportion of training data memorization T(V, 0.9). Each line corresponds to a different model size performing causal language modeling on WIKITEXTI03, We demonstrate that larger models memorize faster for a fixed learning rate.
Exhaustively searching all such possible factors is intractable, and providing complete explanation for why larger models memorize faster is outside the scope of this work. Instead, in the following sections, present studies that hope will expand the toolkit for answering such questions.
4.3 Memorization via. Unique Identifiers
35 46 87 this subsection, question whether such architecture changes necessary. Motivated by information retrieval systems, we take a simple approach - we prepend a unique identifier to every example in fix the language
Recent work studies how to use external memory to improve performance [11
the training and examine whether memorization speed increases. Specifically, modeling task as causal language modeling on WIKITEXT 103 with the 155M parameter model, and in front of every training example, insert the string document ID <unique id> where unique_ id is unique integer, one for each training context. In order to utilize all these unique integers, must add them the dictionary of tokens, which causes a significant increase in the model size since the last layer in the language model must have an output dimension equal to the size of the dictionary. Therefore, any change in M (f) dynamics could be attributed the extra parameters we add from increasing dictionary size. To control for this, we first examine the effect of just increasing dictionary size (without using any the added tokens). Then, we utilize those added tokens to prepend every training example and observe the change in Mf) dynamics. In Figure 6 we see that increasing dictionary size does improve the speed of memorization. Even though we previously demonstrated that larger models memorize faster, this is still surprising considering that we do not increase parameter size significant way - we are effectively adding fake tokens to the dictionary. Moreover, when we leverage those added tokens to identify training examples

---

1.0
0.8
M(f)
0.6
With unique IDs Increase dictionary size Original
0.4
100
200
300
Number of Epochs
Figure The impact of adding unique identifiers to training examples on memorization M() training dynamics for causal language modeling (125M) on WIKITEXT103. The green line is the original 125M model. The orange line is the model after adding unique identifiers the dictionary (which increases model size). The blue line prepends these unique identifiers for each training example. Note that adding unique identifiers leads to faster memorization of training data.
uniquely, we yet another gain in memorization, although prompting using document ID shifts memorization dynamics away from being monotonically increasing over time.
4.4
Memorization Through the Lens of Parts Speech
1.0
1.0
0.8-
0.8
Rmm (p)
0.0
Num.
Num.
0.6.
0.6
0.6
Adj.
Adj.
0.4
Noun
0.4
Noun
Prop. Noun
Prop. Noun Verb
0.2
Verb
20 30 40
70
80
20 30 40
70 80
Number of Epochs
Number of Epochs
Figure 7: The ratios R(p) (Left) and (mem(p) (p) (Right) over training. R(p) represents proportion of POS correctly memorized (the language model outputs the right POS, but not necessarily the correct word). Rmem (p) represents the proportion of exactly memorized tokens for a particular POS Results are for causal language modeling (355M) on WIKITEXT 103. In both plots, we consider numerals, proper nouns, verbs, nouns, and adjectives as potential parts of speech (i.e., values for p). We show that nouns and numerals are memorized faster than other parts of speech.
In the previous section, we showed that a unique identifier enhances memorization. Regular text also contains strong proxies to unique identifiers in the form of numerals and proper nouns. Motivated by this, we study syntactic features of memories using part-of-speech (POS) tagging F We track the ratio R(p) of the number of positions for which the part of speech was correctly predicted to the total number tokens in the ground truth tagged with that part-of-speech p (left plot in Figure 7 In the right plot of Figure 7 we show similar ratio, denoted Rmem (p), but the numerator only considers the tokens that are also exactly memorized. The correctly predicted part of speech does not necessarily imply exact memorization, which is clearly illustrated by Figure where we see the language model memorizing parts of speech faster than the exact value of the token. While all parts of speech are eventually memorized, some parts of speech are memorized faster, which aligns with previous work (20]. However, unlike previous worl we find that nouns, proper nouns, and numerals are memorized noticeably faster than verbs and adjectives, both in terms of R(p) and Rmem (p). This has potential implications for privacy, since sensitive information is likely to be a noun/proper noun/numeral Our findings also very loosely align with work studying child language acquisition [29
'We use spaCy 42 to identify parts speech in text.
This difference could be due model family (we causal LMs while previous work uses masked LMs)
7

---

Forgetting Curves in Language Models
This section studies the dual of memorization forgetting in language models. Inspired by the forgetring curve hypothesis, according to which human memory declines over time when there is no attempt to retain [56], we are interested in understanding the dynamics of memory degradation in language models.
We first choose batch of data not available in the training set, i.e. batch of data from validation set. We refer to this batch of data as the special batch. We then take a checkpoint from model training, plug in the special batch so that the model can train on it, and resume standard training on the training set. We then evaluate how memorization degrades on the special batch and analyze the various factors the forgetting curve may depend on. We use the entire validation set as the special batch throughout this section. The special batch is only seen once when it is immediately introduced
1.0
Training Set
0.425
0.8-
Special Batch
0.400
M(1)
0.6
M(A
0.375
0.350
0.4-
0.325
Forgetting Baseline
0.300
0.20
50
100
150
10a
10
10
Number of Epochs
Figure 8: Left: forgetting curve for causal language modeling (2.7B) on WIKITEXTI03. The dashed horizontal line indicates the lowest proportion of special batch data memorized throughout training, i.e. the forgetting baseline. Right: forgetting baseline as function of model size N (plotted on log scale). We show that model scale increases, the forgetting baseline value increases.
In the left plot of Figure 8 we show the forgetting curve for the 2.7B model. Exact memorization on the special batch degrades quickly at first, but slows down exponentially as we continue training (see Figure i5 in $ A.2.2). In other words, the forgetting curve on the special batch seems to approach a baseline we refer this trend as the forgetring baseline. We approximate the forgetting baseline by looking at the lowest memorization value on the special batch throughout training.
We show the forgetting baseline as function of the model scale in the right plot of Figure We see that the numerical value for the baseline is monotonically increasing with the model scale. This implies that larger models forget less, aligning with recent work studying catastrophic forgetting on image classification tasks (75). This is beneficial because larger models can leverage more information from previous tasks; however, from privacy perspective, this is not ideal because implies larger models may be potentially retaining more sensitive information from training data.
We also investigate the sensitivity of the forgetting baseline data batch order. In Figure 9 perform the same forgetting curve analysis described above but start the analysis at different training checkpoints (we start the 14th, 39th, and 63rd epochs). This way, alter the order of the data batches given to the model (since the special batch will appear in different place in the global order of data batches given the model) without drastically changing the experimental setup. We observe that the forgetting baseline is not sensitive to data batch order
"This experimental setup is different from catastrophic forgetting, as we fix the data distribution by pulling the special batch from the same dataset as the training set. Similarly, it differs from machine unlearning since we algorithmically removing information from a language model; instead, is also different from intrinsic hallucination (3). where there is an assumption that contradicted output semantically correct (e.g.. the language model outputs a wrong date).
analyze natural forgetting.
'The average sequential difference in memorization (on the special batch) the last epochs of training at most on the order of 10-3 whereas the average sequential difference in the first epochs training consistently on the order of -0
The max difference between the numerical values for the baseline the order of 10 10-3

---

1.0
0.8
M(1)
0.6
0.4
0.2
0.0
14
39
63
100
150
Number of Epochs
Figure 9: We empirically show that the forgetting baseline does not depend on data batch ordering. We inject the special batch into the training set at the 14th, 39th, and 63rd epochs, and evaluate proportion of special batch data memorized as we continue training. Results are for causal language modeling (125M) WIKITEXT103.
Motivated by replay methods from continual learning (see 24
for a survey) and work in promoting
retention memories through repetition in both humans [55 we study the effect of repetition (left) and spaced repetition (right) on the forgetting baseline. In the left plot, inject the special batch into the training set multiple times before continuing training on the training set alone. We observe that the forgetting baseline is monotonically increasing as a function of repetition frequency (differences in the baseline value the spaced repetition, we periodically inject the held-out set into the training set, train on once, and then continue training on the training alone. We see in the right plot of Figure IO that spaced repetition incurs minimal effect on the forgetting baseline (on the order of 10 independent of the length of spacing between the repetitions.
68 84
and neural models [5], in Figure I0
the order of 10-2). To study
1.0
1.0
0.8
0.8
Inj. 1000
Inj. count
M(f)
0.6
Inj, count
0.6
In). count:
0.4
0.2
0.0 0
0.0
50
100
150
0
50
100
150
Number of Epochs
Number of Epochs
Figure 10: Effect of repeated injection (Lefft and spaced repetition (Right) special batch memo- rization. Results are for causal language modeling (125M) on WIKITEXT 103. The solid upper curve represents the training memorization. We show that repeated injection increases the forgetting baseline, whereas spaced repetition has minimal effect.
An exciting direction for future work will be to understand the structure of the baseline - for example, understanding what types of tokens (parts of speech, synonyms, facts, syntax) are memorized in the baseline and the overlap of tokens memorized in the baseline with tokens in the training set.
Conclusions and Discussion
We study the properties of memorization dynamics over language model training and demonstrate that larger models memorize faster. We also measure the properties of forgetting curves and surprisingly find that forgetting reaches baseline, which again increases with the model scale. Combined with memorization analyses that expose the unintuitive behavior of language models, we hope to motivate considering memorization critical metric when increasing language model scale.
Most work studying memorization language modeling is primarily motivated by privacy (see $ 2 While theoretically, there well-established frameworks to quantify privacy such as differential

---

privacy [25]. empirical privacy in language modeling is not well-defined - does memorizing common knowledge count as information leakage? Does outputting a synonym count as harmful memorization? As per our Definition] [ we implicitly focus information that is sensitive if outpuited verbatim (phone numbers, SSNs, addresses, medical diagnoses, etc.), rather than capturing all aspects of privacy. It is also known that text data used for training language models contain certain biases and stereotypes (e.g.. [32]); ) therefore, our work has similar implications for how long language models can train before they definitively memorize these biases from training data.
We also hope our work highlights the importance of analyzing memorization dynamics as we scale up language models, instead of only reporting cross entropy. Cross-entropp loss and memorization capture different behavior for example. in many our memory degradation experiments, even though memorization approaches a baseline, we observe that perplexity is still increasing (see Figure 14 $ A.2 for an example). This implies that the model becoming unconfident about its exact predictions, which we can only conclude because we inspect both loss and memorization. More importantly, the forgetting baseline behavior would entirely obscured if we did not inspect memorization dynamics. Similarly, there are multiple instances where uncover interesting behavior because we focus on memorization dynamics ($ cross-entropy loss.
rather than focusing only on
7
Acknowledgements
The authors would like thank Adina Williams, Chuan Guo, Alex Sablayrolles, and Pierre Stock, for helpful discussions throughout the course of this project. The authors would also like to researchers at FAIR who commented on otherwise supported this project, including Shashank Shekhar, Candace Ross, Rebecca Qian, Dieuwke Hupkes, and Gargi Ghosh.
References
[1] Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. International Conference on Learning Representations, 2018.
[2] Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. arXiv preprint 20X1v:2008.03156. 2020.
[3] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520, 2022.
[4] Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. A review on language models as knowledge bases. arXiv preprint arXiv:2204.066 2022.
[5] Hadi Amiri, Timothy Miller, and Guergana Savova. Repeat before forgetting: Spaced repetition for efficient and effective training of neural networks. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2401-2410, 2017.
[6] Gor Arakelyan, Gevorg Soghomonyan, and The Aim team. Aim, 6 2020.
URL https
7 http://ww./.m.iiiooooeeeaaaaa
[7] Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Vic- toria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint 1rXiv:2112.10632 2021.
[8] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06700 2021.
[9] Mandeep Baines, Shruti Bhosale, Vittorio Caggiano, Naman Goyal, Siddharth Goyal, Myle Ott, Benjamin Lefaudeux, Vitaliy Liptchinsky, Mike Rabbat, Sam Sheiffer, Anjali Sridhar, and Min Xu. Fairscale: general purpose modular pytorch library for high performance and large scale training. http://AAthtuoooooeeeeeeeeertrtt
2021.
10

---

[10] Terra Blevins, Hila Gonen, and Luke Zettlemoyer. Analyzing the mono-and cross-lingual pretraining dynamics of multilingual language models. arXiv preprint arXiv:2205.11758, 2022.
[11] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. arXiv preprini arXiv:21 22..v:22 2021.
[12] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium Security and Privacy (SP), pages 141-159 IEEE, 2021.
[13] Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memo- rization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Anmual ACM SIGACT Symposium Theory of Computing, pages 123-132, 2021.
[14] Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
[15] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2/),
pages 2633-2650, 2021.
[16] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski. Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 2022.
[17] Tyler A. Chang and Benjamin K. Bergen. Word acquisition in neural language mod- els. Transactions of the Association for Computational Linguistics, 10:1-16, 2022. doi: 10.1 10.1162/tac__ a a_00444. htt.ttttt///aclanttttt oorg 2022. tacl-1.1
[18] Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting, arXiv preprint arXiv:2004. 12651, 2020.
[19] Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1-207, 2018.
[20] Cheng-Han Chiang, Sung-Feng Huang, and Hung-yi Lee. Pretrained language model embryol- ogy: The birth of albert. arXiv preprint arXiv:2010.02480, 2020.
[21] Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The crummar-lcariinn trajectories of neural language models. arXiv preprint arXiv:2109. 109.:290, 2021.
[22] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmano, et al. Palm: Scaling language modeling with pathways. arXiv preprinl arXiv:2204.02311 2022.
[23] Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. arXiv preprint arXiv:2202.01 169, 2022.
[24] Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[25] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
(26) Vitaly Feldman. Does learning require memorization. A short tale about a long tail. CoRR, 0bs/1906.05271,2015

---

[27] Vitaly Feldman. Does Learning Require Memorization? A Short Tale about a Long Tail. arXiv: 1906.0527 [cs, stat], January 2021. URL http:://rxiv... httt:://axiiv.ogg/bbb/////// 1906. 05271 arXiv: 1906.05271.
[28] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:
2881 1881-288 1891, 2020.
[29] Michael Fleischman and Deb Roy. Why verbs are harder to learn than nouns: Initial insights from a computational model of intention recognition in situated word learning. In 27th Annual Meeting of the Cognitive Science Sociery, Stresa, Italy, 2005.
[30] Jonathan Frankle, David Schwab, and Ari S Morcos. training. arXiv preprint arXiv:2002.10366, 2020.
The early phase of neural network
[31] James Franklin. The elements of statistical learning: data mining, inference and prediction. The Mathematical Intelligencer, 27(2):83-85, 2005.
[32] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009. 1146,, 2020.
[33] Tanya Goyal, Jiacheng Xu, Junyi Jessy Li, and Greg Durrett. Training dynamics for text summarization models. arXiv preprint arXiv:2110.08377 2021.
[34] Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. arXiv preprint arXiv: 1812.04754 2018
[35] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. Realm: Retrieval-augmenned language model pre-training arXiv preprint 2002.0009000099999 2020.
[36] Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Investigating learning dynamics of BERT fine- tuning. In Proceedings of the 1st Conference the the Asia-Pacifie Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 87 87-92, Suzhou, China, December 2020. Association for Computational Linguistics. URL https: http://acaaaatoogyo/o org/ 2020 aacl- maneclmman..111 11
[37] Elizabeth Liz Harding, Jarno Vanto, Reece Clark, L Hannah Ji, and Sara C Ainsworth. Understanding the scope and impact of the california consumer privacy act of 2018. Journal of Data Protection & Privacy, 2(3):234-255) 2019.
[38] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08411 08415, 2016.
[39] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
[40] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2/02.0/223 2021.
[41] Sorami Hisamoto, Matt Post, and Kevin Duh. Membership inference attacks on sequence-to- sequence models. arXiv preprint 1rXtv:1904005506 2019.
[42] Matthew Honnibal and Ines Montani. spaCy 3: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear, 2022.
[43] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. arXiv preprint arXiv:2202.03663 2022.
[44] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. 200::2000008861 [cs, stat], January 2020. URL //trxvv/.vvv///////////eae/vvv www. wwrg/bbs/200 2001 08361. arXiv: 2001.08366
12

---

[45] Jeffrey D Karpicke and Henry L Roediger III. Expanding retrieval practice promotes short-tern retention, but equally spaced retrieval enhances long-term retention. Journal of experimental psychology: learning, memory, and cognition, 33(4):704, 2007.
[46] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Gen- eralization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00177 2019.
[47] Eugene Kharitonov, Marco Baroni, and Dieuwke Hupkes. How bpe affects memorization in transformers, arXiv preprint arXiv:2110.0272,, 2021.
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[49] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Srrabska-Barwinnsee et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (1((1))3521-3322) 2017.
[50] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499) 2021.
[51] Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. arXiv preprint arXiv:21 10.05000055599 2021.
[52] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gon- zalez. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In International Conference on Machine Learning, pages 5958-5968. PMLR, 2020.
[53] Leo Z Liu, Yizhong Wang, Jungo Kasai, Hannanch Hajishirzi, and Noah A Smith. Probing across time: What does roberta know and when? arXiv preprint
2021.
[54] Yang Liu, Zhuo Ma, Ximeng Liu, Jian Liu, Zhongyuan Jiang, Jianfeng Ma, Philip Yu, and Kui Ren. Learn to forget: Machine unlearning via neuron masking, arXiv preprint 1001: 2020.
[55] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints, July 2019. URL /tatps://arxiv.oor wwweeiii..rr//bass /
1907 11692v1
[56] Geoffrey R Loftus. Evaluating forgetting curves. Journal of Experimental Psychology: Learning. Memory, and Cognition, 11(2):397, 1985.
[57] Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):229-233 2013.
[58] Wojeiech Masarczyk, Kamil Deja, and Tomasz Trzcinski. On robustness of generative repre- sentations against catastrophic forgetting. In International Conference on Neural Information Processing, pages 325-333. Springer, 2021.
[59] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[60] R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty text generation using raven. arXiv preprint arXiv:2111.09500 2021.
13

---

[61] Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and lan Tenney. What happens to BERT embeddings during fine-tuning? Proceedings of the Third BlackboxNLP Workshop on
Analyzing and Interpreting Neural Networks for NLP: pages 33 33-44, Online, November 2020. Association for Computational Linguisties. doi: 10. 18.18v 10//S3/11/200/ 2lt20000.lccooooollppp. httts://acclattholt
URL
org/ org/2020. blackboxnlp-1.4
[62] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, ArXiv, abs/1609.078433 2017.
[63] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training. arXiv preprint 170.03370 2017.
[64] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Shokri. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.03929, 2022.
and Reza
[65] Seyed Iman Mirzadeh, Arslan Chaudhry, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. arXiv preprint arXiv:2110. 1552, 2021.
[66] Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, 31, 2018.
[67] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 202 ( (12):120040 2021.
[68] Shiri Oren, Charlene memory in alzheimer's s disease: A systematic review. Journal of Speech, Language and Hearing Research (Online), 57(1):247, 2014.
Willerton, and Jeff Small. Effects of spaced retrieval training on semantic
[69] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint 1rXiv:1904.01038, 2019.
[70] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8033 2019.
[71] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 1xXiv:199.001666 2019.
[72] Vinaychandran Pondenkandath, Michele Alberti, Sammer Puran, Rolf Ingold, and Marcus Liwicki. Leveraging random label memorization for unsupervised pre-training. arXiv preprint 2018
[73] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 112 2021.
[74] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
[75] Vinay Venkatesh Ramasesh, Aitor Lewkowyez, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2021.
14

---

[76] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-sho text-to-image generation. In International Conference on Machine Learning, pages
PMLR, 2021.
[77] Roger Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, pages 285-308, 1990.
[78] General Data Protection Regulation. General data protection regulation (gdpr). Intersoft Consulting, Accessed in October, 24(1), 2018.
[79] Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXiv: 1919999999777.. 2019.
[80] Naomi Saphra and Adam Lopez. Understanding learning dynamics of language models with sveca. arXiv preprint arXiv:
2018.
[81] Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. On the dynamics of gender learning in speech translation. In Proceedings of the 4th Workshop Gender Bias in Natural Language Processing (GeBNLP), pages 94-111, Seattle, Washington, org/
July 2022. Association for Computational Linguistics. URL 2022. gr122p-bbllp1.111
[82] Chenze Shao and Yang Feng. Overcoming catastrophic forgetting beyond continual learning: Balanced training for neural machine translation. arXiv preprint (rXii:2203.03910 2022.
[83] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari,, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron train megatron-turing nlg 530b, large-scale generative language model. arXiv preprint XrXiv:2201. 11900, 2022.
[84] Paul Smolen, Yili Zhang, and John H Byme. The right time to learn: mechanisms and optimization of spaced learning. Nature Reviews Neuroscience, 17(2):77-885 88, 2016.
[85] Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text rext-generation models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 106,206, 2019.
[86] Patrick Stadler, Vivien Macketanz, and Eleftherios Avramidis. Observing the learning curve of nmt systems with regard to linguistic phenomena. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 186-196, 2021.
[87] Yi Tay, Vinh Q Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, al. Transformer memory as differentiable search index. arXiv preprint 220..02200.09911,,, 2022.
[88] Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning. arXiv preprint arXiv:2006.074400 2020.
[89] Aleena Thomas, David Ifeoluwa Adelani, Ali Davody, Aditya Mogadala, and Dietrich Klakow. Investigating the impact pre-trained word embeddings on memorization in neural networks. In International Conference on Text, Speech, and Dialogue, pages 273-281 Springer, 2020.
[90] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning, arXiv preprint
2018.
[91] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 1031152676:10-55555 2017.
15

---

[92] Elena Voita, Rico Sennrich, and Ivan Titov. Analyzing the source and target contributions to predictions in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the / 1tth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1126-1144 140, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.186553/112011.acllll
URL
hwwpaooooooooogg///// acl 1onnl..ng.99
[93] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 2022.
[94] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv: 1ee111111133300 1,03530 [cs), February 2017. URL http//111..0000 03530
arXiv: 1611.03530.
[95] Chiyuan Zhang, Daphne Ippolito. Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual Memorization in Neural Language Models. OrXiv:2/12.1990 [cs], December 2021. URL http://arxiv/ wwt.ooog//aaeeee 12938
arXiv: 2112.12953
version: 1.
[96] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The main claims in both the introduction and abstract are that (1) larger models memorize faster (where memorization defined as per Definition 3) (2) larger models memorize more before overfitting, (3) larger models forget less, and (4) models memorize nouns and numbers quicker than other parts of speech. (1) and (2) supported by the beginning subsections in $ (3) is supported by [ and (4) is supported by $ 4.44 Moreover, as mentioned in the introduction and abstract of this work the scope of this work includes analyzing large language models which we accomplish by analyzing language models up to 13B parameters.
(b) Did you describe the limitations of your work? [Yes] In & 4 we discuss that while we find that larger models memorize faster, we are unable completely explain why this is the case (although we rule out certain reasons). In $5. we discuss how we are approximate the numerical value for the baseline depending however long particular model is trained for i.c. that actual numerical values for the baseline may change slightly if training for longer; however we provide evidence that the further changes to the numerical value will be relatively small in $ |A.2.22 In $ SAA.I.I] below where we define memorization, we discuss the limitations of the memorization definition.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] This work does not develop new methods / models / datasets in any way, and therefore has minimal potential negative societal impacts. However, in section 6 we discuss the implications of our analysis for privacy and ethical AL. We explain that, since work deals with memorization dynamics over training of training data, it implicitly studies how long it takes language models memorize sensitive information (privacy perspective) or bias/stereotypes (ethical Al perspective) from training data.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Ye]] The authors have read the ethies review guidelines and ensured that this work conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
16

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] Unfortunately, the exact code used to produce results is proprietary. However, all model configura- tions and training details are directly pulled from publicly available references, and described in detail in section A.4 Similarly, while for most of our experiments we use WIKI TIXTTEXTI 103 benchmark which is publicly available, some of our experiments run the ROBERTA dataset which is not publicly available, and therefore, we are unable to release the exact data to re-create those experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] For all our of experiments, use publicly available references to define model architectures and hyperparameters, which we describe in full detail in section
A.4
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Due to the scale of experiments we run (up to 13B parameter models experiments), many experiments are incredibly computationally expensive and we unable to each experiment for multiple seeds. However, since we deal with large datasets, random seed most probably has minimal effect on final model output.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In $ $A.44, we describe the type of GPUS and the amount of GPUs used to train different model sizes. We also provide estimates of the total training time across all our experiments.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? Yes] Since we use existing datasets to conduct experiments, we cite the creators in $ at in the Datasets section: similarly, we use the existing Transformer architecture (which we also cite $ 3 in the Model Architecture section); similarly we pull most of our hyperparameter configurations from existing public resources, which we cite in $ 3 in the Model Architecture section.
(b) Did you mention the license of the assets? [Yes] In A.4 mention the licenses of all assets we use.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] We create no new assets as part of this work.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] Since do not create curate any new datasets/assets part of this work, we do not discuss whether and how consent was obtained from people
whose data are using.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] In $ A.4 we mention that it is completely plausible the underlying data we use has sensitive or offensive information. However, analyzing the extent to which this is the case is outside the scope of the work, since we just aim to understand memorization dynamics of language models over training rather than analyze the underlying text in datasets
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We do not crowdsource conduct research with human subjects in this work
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. applicable? [N/A] We do not crowdsource or conduct research with human subjects in this work
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We do not crowdsource or conduct research with human subjects in this work
17


Reviews:
[{"review_id": "eMdL3Kgp2F6", "reviewer": "Reviewer_KjpN", "summary": "This paper investigates the memorization and forgetting dynamics of large language models during training, as a function of model and dataset sizes, learning rate, and task (i.e. causal vs masked language modeling). The paper measures memorization as the proportion of correctly predicted labels (which corresponds to predicting the correct next/masked token) during training, and forgetting as the decline in memorization.\n\nThe main experimental results show that: (i) larger models memorize data faster and forget less; (ii) memorization occurs before overfitting; and (iii) memorization occurs for both small and large datasets, and for both causal and masked language modeling, regardless of the learning rate. The paper further investigates the memorization mechanism and shows that the presence of unique identifiers in the training data can contribute to memorization, e.g. models tend to memorize numerals, nouns and proper nouns faster than other parts-of-speech. Moreover, the paper shows that while memories from a dataset are forgotten when training with other data, the degree to which forgetting occurs has a lower bound that depends on the model size (larger models having a higher lower bound). ", "questions": "- Any hypotheses as to why memorization is faster when adding tokens to the dictionary without even using them?\n", "limitations": "authors have discussed the limitations", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThis paper addresses an important aspect of large language models: their ability to memorize the training data without necessarily overfitting. This unexpected behavior has been observed in previous work and calls into question the classical framework of bias-variance tradeoff. However, little is understood about the dynamics of memorization and forgetting in these models. Yet, this can have problematic implications  for real-world deployments related to e.g., privacy (e.g., models can memorize personal identifiers or other sensitive information) and fairness (e.g., models can memorize negative stereotypes about groups of individuals). This paper gives insights into the memorization and forgetting dynamics in large language model training and about the scaling laws that seem to govern these models. \nOverall, the paper is well written and clear and the main claims are well supported by experimental evidence. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RvyGE5LIJxo", "reviewer": "Reviewer_xEvT", "summary": "In this paper, the authors define memorization as any situation in which the model predicts the correct word as maximum probability given its context. They find that larger models memorize faster under this definition. They then study the asymptotic behavior of forgetting the memorized samples, finding that larger models likely never completely forget their memorized samples. They then study memorization paper according to part of speech, suggesting that rare words tend to lead to memorized sequences.", "questions": "Why did you pick the particular reference implementations that you picked? Are these considered standard for work like this?\n\nlines 240-242: Does this imply that the special batch is not seen once immediately when introduced, but is seen later on? Or is this implying that the spectral batch is seen repeatedly in each epoch? That was not my understanding initially, en did makes this confusing\n\nline 202: Maybe clarify that this means memorization of which words correspond to a particular part of speech? Or are you claiming that the correct part of speech is consistently selected? Because that should definitely not be referred to as memorization.\n\n187-189, 92-95: Could you rewrite these sentences? I find them very confusing.", "limitations": "I didn't feel that the authors were precise enough in their definitions, leading them to use some words, like memorization, in ways that might not reflect what we usually mean when we talk about memorization. I think being more precise in their language would help this paper greatly, as the results that they describe might not influence other notions of memorization.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe question of how memorization occurs over the course of training is an interesting one. I like that they bring in implications for privacy, though I’d like to see more explicit links to the generalization literature.\n\nlines 227-228: As the number of parameters increases, the model forgets less. Seems obvious, but actually really nice to see this demonstrated in such a simple way. In general, some of their results seem intuitive, so when their methods are convincing and well explained, it's satisfying. This sort of work often finds people complaining that these are things that we already know, but if they aren't tested in the current empirical work, this kind of result has a lot of value. The results on forgetting curves and the lower bounds that they exhibit at various scales was convincing and presented a good framing for the phenomenon of forgetting.\n\nWeaknesses:\n\nA broad issue I have is I'm not convinced \"memorization\" is the appropriate term for the behavior discussed. \n\nIt's not clear to me that every time the correct word is predicted as maximum probability, it should be counted as memorization rather than generalised learning. Also, how is this affected by conditions where the same context appears multiple times in the training set with different labels? If every time a certain sequence occurs in training, it always occurs with a particular word, it seems that it's not memorization to indicate that that's the highest probability word.\n\n What does it mean for a part of speech to be memorized? I don't agree that a part of speech is memorized just because it is predicted correctly in context. There aren't that many parts of speech, so unlike when trying to predict the missing word in a unique sentence, I don't think that it's appropriate to claim that a sequence is memorized just because the predictions made are the correct part of speech. 207 highlights this problem, as it refers to the behavior first as learning and then as memorizing.\n\n209: If the model is just learning to predict that the token is a numeral, and not learning to predict the specific numeral, it’s not clear to me that this is actually going to have privacy implications.\n\nIn terms of the uniqueness experiments, I would want to see what the effect of word frequency is in general before considering the role of completely unique words.\n\n174: \"classical ML concepts cannot even explain such a memorization trend.\" Not sure what this means, you should probably talk about the concepts that you feel fail to explain the trend.\n\nI'm not convinced by the experiments that consider memorization relationship with overfitting as though these are two separate phenomena. What if the model is overfitting on a subset of the data, but generalization is improving overall, just not on examples that might be related to that subset? Seems like overfitting by this metric might be an emergent property of having enough memorization occur.\n\nProblems with sparse literature review:\n\nThis work is not contextualized in the existing literature on training dynamics in language models, outside of recent literature on general scaling laws. I recommend looking through a variety of work on the area: https://www.semanticscholar.org/search?q=training%20dynamics%20language%20models&sort=relevance\n\nIn general, I take issue with the lack of citations to existing work. Is label memorization extending an existing concept? You wouldn't think so, from the paper. There is a mention of spaced repetition from the cognitive science literature, but the authors do not acknowledge that it has also been applied in natural language processing training: https://aclanthology.org/D17-1255/\n\nMany concepts are introduced without any kind of citation to the existing literature: catastrophic forgetting, machine unlearning. Terms like \"basin\" and \"phase transition\" are introduced without background, so it's not clear what type of phenomenon each of these phrases is intended to refer to in this context.\n\nMinor:\n- 132: \"generally monotonically decreasing\" just say that it's generally decreasing. It's not monotonically decreasing, and appending \"generally\" in this context just means \"not monotonic\".\n- 51: \\citep should be \\citet here", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JEhui3saOMx", "reviewer": "Reviewer_YUcS", "summary": "This paper analyzes the phenomenon of memorization by Transformer language models in the light of model size, catastrophic forgetting and memorization of unique tokens such as part of speech. The paper finds that larger the size, more the model is prone to memorize train distribution. Larger models also tends to forget less, and they seem to memorize unique parts of speech tokens such as nouns and numbers.\n\n", "questions": "\n- In L122, the authors seem to refer the BookCorpus as \"RoBERTa\" dataset. Any specific reason why?\n- Furthermore, BookCorpus which is used to trained RoBERTa model comes in two sizes: 16GB (which is used for RoBERTa-base) and 160GB (which is used for RoBERTa-large). I'm not sure which one is being used here.\n- What is the model family used in these experiments? Are the results specific to Causal language models or Masked Language Models? Do the authors expect the results to differ based on the class of the model family being used?\n- In the catastrophic forgetting experiment, the _special batch_ is taken from the validation set of Wiki 103, which is in-distribution with the training set. If I were to choose a special batch being significantly out-of-distribution with respect to the training data, would the same results hold?\n\n", "limitations": "The authors have not discussed the limitations of their work thoroughly. As I mentioned above, one big limitation is the usage of a significantly small dataset to explain the behaviour of large language models, those of which are typically trained on much larger datasets (for eg, The Pile or C4). This is a window of improvement for the paper, and the authors should try to clearly set the expectations early on for the reader.\n", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper is well written and easy to follow.\n- The experimental protocol to measure memorization makes sense, and a good spread of model sizes have been used. It is a bit confusing to understand which model family has been used (as the authors only state they have used \"Transformer language model architectures\" in L114).\n- While I mostly agree with the empirical results, the issue I have with them is that the majority of experiments are conducted on Wiki Text 103, which is 500MB dataset. This dataset is significantly small compared to those being used to train the large models (most of the model sizes considered in this paper are typically trained on significantly larger datasets). Thus, while undoubtedly the models somewhat favor memorization, it is unclear whether they still do the same when the input data is large enough. While I do agree such a study would be quite expensive and perhaps intractable, a disclaimer or limitation of this study should have been explicitly mentioned (on the data size front).\n- The study on catastrophic forgetting is quite interesting - it is nice to see a forgetting baseline set by the models of individual sizes.\n- The study on parts of speech is especially interesting, however the section lacks a bit on the explanation of the behaviour. Specifically, nouns, proper nouns and numerals being memorized noticeably faster might be due to the frequency effect (Wei, J., Garrette, D., Linzen, T., & Pavlick, E., Frequency Effects on Syntactic Rule Learning in Transformers, arXiv:2109.07020 [cs], 2021). It would be interesting/beneficial for the paper to study this angle further even using Wiki Text 103 dataset, to see if there are any correlation with memorization with the frequency of individual POS class.\n\n*Update*: I upgraded my initial evaluation after reading the author responses.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "7IKWxwp0J4", "reviewer": "Reviewer_3y6g", "summary": "This paper demonstrates a detailed picture of training data memorization in the process of language modeling. The authors show that larger-sized language models memorize training data faster. Besides, this memorization happens before the overfitting of language modeling. Moreover, specific part-of-speech-tagged tokens are memorized faster like nouns and numbers during training. Lastly, they use the validation set as special insertion batch to test the forgetting mechanism with different model scales.", "questions": "The PoS are not of comparable size (e.g. numbers can be considered relative as a closed set compared to nouns, even though they are not actually, and the size of prop. nouns are relatively small compared to the set of nouns). Therefore, the memorization of nouns demonstrated in the experiments may not be as informative as the memorization of more fine-grained categories like numbers. ", "limitations": "One potential related reference: https://aclanthology.org/2022.tacl-1.1.pdf", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The experimental results are detailed with the reasonable proposed metrics in data memorization.\n- The forgetting identifier experiments are well-designed.\n- In general, these fine-grained findings add value to drawing the picture of how and what data are memorized in transformer-based language modeling.\n\nWeakness:\n- Some example-based analysis (e.g. the POS-wise breakdown of memorization and the forgetting experiment) would be helpful in further demonstrating the procedure.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept