SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Typical use-cases are to use smaller models to predict the behavior of models one to two orders of magnitude larger, see Rae et al. [2021], Scao et al. [2022b], Chowdhery et al. [2022].
during training. Thus practitioners would have an early warning signal for detecting memorization of undesired sequences. Unfortunately, we continue to find largely negative results, but hope that future research with better techniques for predicting memorization might vindicate this idea.
(a) Pythia-70M
In Figure 2, we show a correlation heatmap between which sequences are memorized by different checkpoints of the same model. We only look at memorization of the first 23 million sequences, as that is the data that our least-trained model checkpoint has seen.
(b) Pythia-1.4B
(c) Pythia-12B
Figure 2: Heat maps visualizing the correlation between which sequences are memorized by different checkpoints. Plots for other Pythia models can be found in Figure 10.
Seq Num	Precision	Recall
23·106	0.919	0.513
44·106	0.913	0.587
65·106	0.910	0.658
85·106	0.910	0.721
105·106	0.915	0.816
126·106	0.945	0.918
146·106	—	—
(a) Pythia-6.9B
Seq Num	Precision	Recall
23·106	0.918	0.500
44·106	0.915	0.575
65·106	0.913	0.641
85·106	0.911	0.711
105·106	0.916	0.809
126·106	0.943	0.916
146·106	—	—
(b) Pythia-12B
Table 2: Precision and recall for predicting which sequences would be memorized by the fully-trained model from a partially-trained checkpoint. We observe consistently high precision, but only achieve high recall after significant compute has been expended (later intermediate checkpoints).
Our results on precision and recall (Table 2) largely mirror those of Section 3 in general trends. We see that the earliest intermediate checkpoints we test do not exhibit the high recall that is desirable, for instance with the 23M checkpoint of Pythia-12B underperforming the fully-trained Pythia-6.9B in recall.
We thus observe that using intermediate checkpoints of a model run to predict memorization is not a silver bullet—it is still the case that precision remains high throughout models, but recall is low for all predictors that use significantly less compute than the final model's cost. Therefore, in this setting as well, it is easier to guarantee a sequence will be memorized through such extrapolations rather than not. Since the latter guarantee of non-memorization is more useful to engineers, our focus thus shifts to determining the compute-optimal model to train to gain a desired level of recall, in order to maximize predictive power amongst the options we explore.
5 Scaling Laws
Having established the empirical results in the previous section, we now examine our results through the lens of computational efficiency and scaling laws, where the aim is to achieve the most reliable results for the least expense. To achieve this, we examine how well models of various sizes and number of training steps predict which sequences will be memorized by the fully trained 12B
parameter model. This is in notable contrast to Section 4, where partially-trained models are only compared to fully-trained models of the same size. As a visual aid, models with the same size are colored the same.
5.1 Unusual Scaling
In the overwhelming majority of prior work on scaling laws [Brown et al., 2020, Kaplan et al., 2020, Pu et al., 2021, Mikami et al., 2021, Rae et al., 2021, Black et al., 2022, Scao et al., 2022b, Chowdhery et al., 2022], including scaling studies targeting memorization [Carlini et al., 2022, Hernandez et al., 2022, Tirumala et al., 2022a], plots of quantities of interest vs. compute are linear on a log or log-log plot. We find that this is not the case in our setup for both precision and recall.
The scaling data for precision is extremely anomalous. Not only are the plots non-linear, we find that the behavior of the 12B partially trained model is extremely out-of-line with the behavior of smaller models. The results for recall are less anomalous, lacking the divergent behavior for the 12B model, but nevertheless do not accord with what the scaling laws literature generally expects.
(a) Precision
Figure 3: Scaling curves for Pythia models.
(b) Recall
Despite the fact that there is a high-level pattern in the scaling laws curve for recall, a careful look at the data indicates unusual behavior. In the low-compute regimes, which are of most interest to engineers looking to minimize the cost of creating a prediction of the behavior of large models before they are trained, we see a consistent pattern of larger models being better than smaller models for a fixed compute budget. However, as the amount of compute expended scales, this is no longer the case. Starting at around  $1\%$  the budget of the fully trained model, equicompute models perform the same regardless of the number of parameters. Starting at around  $10\%$  the budget of the fully trained model, the smallest model trained for this compute budget becomes the best predictor of memorization in the fully trained model.
5.2 Emergent Memorization
We also see evidence of "emergent" or "semi-emergent" behavior as model scale increases. In the literature on emergent behavior in large language models [Srivastava et al., 2022, Ganguli et al., 2022, Wei et al., 2022, Caballero et al., 2022], the term refers to when a large model's performance on a task is substantially different from the extrapolated value of curves fit to the behavior of smaller models. Often, but not always, this occurs when performance goes from near-zero to meaningful. While our situation is not totally analogous, one can similarly consider "emergent memorization" to occur when data is memorized by large models which cannot be predicted based on the memorization behavior of smaller models. Since, by definition, emergent behavior implies that smaller-scale model behaviors are qualitatively different to those of larger models, this can pose challenges for traditional scaling laws or for extrapolating model behavior to models orders of magnitude larger. As a result, we suggest that this is an important area for further study, including expanding the scope of our work to models larger than 12B parameters.
5.3 Takeaways for Engineers
As discussed in Section 2.2, the primary point of interest to engineers is to predict the behavior of a large language model before it is trained. Such predictions should be grounded in low-cost regimes such as the behavior of trained "test" models that are at least an order of magnitude smaller than the target model. We find that for cases where high recall is required, our scaling law defines what size of model should be trained at a given compute budget. In compute regimes less than two orders of magnitude below the final training run's size, we find that when holding the compute budget fixed it is desirable to use the "smallest" model trained on no more the final run's total token count as possible, and to frontload the data seen by this smaller model with sequences whose memorization would be undesirable in order to predict whether these sequences would be memorized by a final model.
5.4 Takeaways for Decision-Making
The experimental procedures we present in this paper are a step toward practical risk assessment for large-scale training runs. We hope that future work both makes such assessments much cheaper and accurate, and that more practitioners adopt such measures and choose to be up-front about which mitigations they apply when training large neural models.
6 Limitations and Future Work
Our work constitutes the first steps towards developing a way to predict what data will be memorized by a large language model before that model is trained, but has several limitations and opens opportunities for exciting future work. The most important of these are:
Are we measuring the correct thing? The definition of memorization we use is derived from what is currently popular in the academic literature, but it is unclear if it is the best definition to use. We believe  $k$ -extractibility to be well-grounded in privacy concerns of language models, but other metrics such as memorization score may be more natural when studying the dynamics of memorization in training.
Does this generalize to other models? We report our experiments on the Pythia suite, because it was the only current language modeling suite suitable for such work at the time of our research. However, this leaves open many questions about whether our results generalize to models trained with different hyperparameters or different data. We validate our experiments with replications on the deduplicated Pythia models and different hyperparameters in Appendix A and B, but no other model suite is suitable for replicating this analysis. This gap points to the need for more reproducible, public dataset model releases to advance research on memorization.
What about the data contents? Our work does not take the actual content of the training data into account at any point in time: we are looking exclusively at predicting memorization based on whether other cheaper models memorize the content. Future work looking into whether there are properties of the training text that predict memorization of that text could be quite illuminating.
We propose a novel setting for forecasting model memorization prior to train-time, while minimizing the compute required to make this forecast. We present analyses on the two most natural setups for extrapolation: using fully-trained small models and partially-trained checkpoints of the final model to compare and predict memorization of the final large model. We find that using much smaller models for this task is not viable, and that partial checkpoints of an existing model are similarly ineffective predictors of final memorization behavior when adjusted for cost. We derive a scaling law to find the optimal equi-compute predictor of non-memorization and are able to provide recommendations based on this law. We hope that our focus on prediction of the memorization of specific strings will be compelling for future study, and that our analyses inform deep learning practitioners on methods to understand and reduce memorization while training large language models.
8 Corrections
Due to an error in our analysis code, an earlier draft of this paper reported a substantially higher recall in Table 2. This draft of the paper features corrected numbers in that table and has adjusted the conclusions and discussion accordingly.
This paper was made better by conversations with and feedback from many individuals not on the authorship list. Following EleutherAI's open science values [Phang et al., 2022], we shared early drafts of these results with the EleutherAI Interpretability Reading Group as well as the Discord server at large, garnering feedback from many people. We would like to acknowledge Nicholas Turner, Gurkenglas, and Amaru Cuba Gyllenste for identifying errors in our results and questioning our assumptions; Kyle O'Brien and Aviya Skowron for copy-editing; and Herbie Bradley, Nicholas Carlini, Katherine Lee, Naomi Saphra, and the EleutherAI Interpretability Reading Group for their thoughts, feedback, and advice.
We are grateful to Stability AI for providing the compute required to carry out our experiments.
Our work builds on top of the work of many teams at EleutherAI and within the broader open source community writ large. We'd especially like to recognize the GPT-NeoX [Andonian et al., 2021] team at EleutherAI whose library we used to measure memorization and the US AISI maintainers of the Hugging Face Hub whose infrastructure we used to host our data.
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
Gustaf Ahdritz, Nazim Bouatta, Sachin Kadyan, Qinghui Xia, William Gerecke, Timothy J O'Donnell, Daniel Berenberg, Ian Fisk, Niccola Zanichelli, Bo Zhang, et al. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. bioRxiv, 2022.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch, 8 2021. URL https://www.github.com/eleutherai/gpt-neox.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private bert. arXiv preprint arXiv:2108.01624, 2021.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023a.
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. arXiv preprint arXiv:2306.03819, 2023b.
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile. arXiv preprint arXiv:2201.07311, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373, 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: large scale autoregressive language modeling with mesh-tensorflow. *GitHub*, 2021. URL https://www.github.com/eleutherai/gpt-neo.
Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. What language model to train if you have one milliongpu hours? arXiv preprint arXiv:2210.15424, 2022b.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
K. N. Bharadwaj Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. *ArXiv*, abs/2205.10770, 2022a.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290, 2022b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: a 6 billion parameter autoregressive language model, 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
Frank F Xu, Uri Alon, Graham Neubig, and Vincent J Hellendoorn. A systematic evaluation of large language models of code. arXiv preprint arXiv:2202.13169, 2022.
A Robustness to Thresholding Choices
In Section 3 and Section 4, we subset the data to the sequences with a "memorization score" of 1 (i.e., sequences that are fully memorized under previous works' definition). This approach labels all sequences with more than 32 tokens memorized as equally memorized, despite the fact that in reality some will have a much longer accurately reproduced continuation than others. In this section we explore whether that effects our results.
First, we examine the shape of the distribution of memorization scores. We had originally assumed that the answer would be an (approximately) exponential distribution, under the assumption that LLMs had a constant "memorization rate" for correctly predicting each subsequent token. Our assumption was that this "memorization rate" would be based on model
Figure 4: Distribution of memorization scores for 12B parameter model. For all upcoming sections in this paper, "memorized" is defined as score = 1.
size, and that it was the primary determinant of overall memorization score distribution. This would be potentially problematic for our study, as the 32-token memorized sequences would dominate the set of memorized sequences.
However, upon examining the distribution of memorization scores for the largest Pythia models, it was immediately clear that this cannot be the case. As shown in Figure 4, there is a very evident spike in the memorization score distribution at score  $= 1$ . Exponential distributions are thin-tailed distributions, and while they would have a spike at score  $= 1$ , it is not possible for them to have such a large spike. The effect shown in Figure 4 can only occur in thick-tailed distributions, such as the power law distribution.
This is a good sign for our analysis, as it means that the typical memorized datapoint in fact has a much larger number of memorized tokens than the 32 token threshold we were worried about. We also replicate Figure 1b with the doubled threshold and find roughly the same results. We also find the same results rerunning our scaling laws plots in Figure 6.
Model Size	Precision	Recall
70M	0.949	0.140
160M	0.941	0.222
410M	0.931	0.334
1.0B	0.922	0.451
1.4B	0.918	0.497
2.8B	0.900	0.611
6.9B	0.872	0.775
12B	—	—
(a) Precision
Figure 6: Replication of Figure 3 with longer sequences.
Figure 5: Precision and Recall when using each model to predict which sequences would be memorized by the 12B parameter model. This table requires twice as many tokens to match to be considered memorized, but otherwise is a replication of Figure 1b.
(b) Recall
B Robustness to Dedduplication
In order to further confirm the validity of our analyses, we run our experiments on the Pythia (deduplicated) suite, which was trained on a deduplicated copy of the Pile [Gao et al., 2020] for 1.5 epochs. In keeping with the literature on deduplication and its connection with memorization [Lee et al., 2021, Kandpal et al., 2022], we observe that memorization is decreased for this set of models, albeit slightly (Figure 7). This may be due to the 1.5 epoch training setup we adopt offsetting the benefits of deduplicated data.
Figure 7: Fraction of all sequences memorized by both Pythia model suites. For example, Pythia-12B has memorized  $1.62\%$  of sequences. We can observe the deduplicated models memorize less of their dataset than their non-deduplicated counterparts.
(a) Pythia-12B
Figure 8: Inter-checkpoint correlations for memorization of Pythia-12B and Pythia-12B-deduped, respectively. Between the two sets of models, we observe extremely similar (though not fully identical) patterns in the correlations of their checkpoints.
(b) Pythia-12B-deduped
Model	Precision	Recall
Pythia-70M-deduped	0.952	0.218
Pythia-160M-deduped	0.943	0.304
Pythia-410M-deduped	0.939	0.422
Pythia-1.0B-deduped	0.927	0.531
Pythia-1.4B-deduped	0.924	0.535
Pythia-2.8B-deduped	0.912	0.675
Pythia-6.9B-deduped	0.891	0.807
Pythia-12B-deduped	—	—
Figure 9: Precision and recall when using each model to predict which sequences would be memorized by the 12B parameter model. Replicates Figure 1b.
Seq Num	Precision	Recall
23 · 106	0.920	0.523
44 · 106	0.917	0.595
65 · 106	0.915	0.658
85 · 106	0.915	0.724
105 · 106	0.922	0.820
126 · 106	0.949	0.920
146 · 106	—	—
Table 3: Precision and recall for predicting which sequences would be memorized by the fully-trained model from a partially-trained checkpoint, for Pythia-12B-deduped. The trends observed here match Table 2.
We replicate our analyses on the deduplicated models and find the same trends hold for our experiments on the Pythia-deduplicated models as do for the regular Pythia suite. Heatmap correlation results show the same conclusions (Figure 8), and we replicate precision and recall results from Figure 1b and Table 2 but on deduplicated models in Figure 9 and Table 3. We therefore believe our results to be reasonably robust across hyperparameters and engineer train-time choices, but hope that future work may replicate some of our findings on entirely distinct corpora.
(a) 70M Parameter Model
(b) 160M Parameter Model
(c) 410M Parameter Model
(d) 1.0B Parameter Model
(e) 1.4B Parameter Model
(f) 2.8B Parameter Model
(g) 6.9B Parameter Model
(h) 12B Parameter Model
Figure 10: Heat maps visualizing the correlations between which sequences are memorized by different checkpoints.
C Author Contributions
Stella Biderman Conceived, organized, and lead the project, and wrote the paper.
USVSN Sai Prashanth Implemented and carried out the evaluation of memorization of pretraining strings.
Lintang Sutawika Analyzed and interpreted the precision and recall results and plotted data.
Hailey Schoelkopf Carried out the evaluation of memorization of pretraining strings, performed the robustness evaluation, found and fixed several bugs in our code, and wrote the paper.
Quentin Anthony Analyzed and interpreted the results and wrote the paper.
Shivanshu Purohit Optimized the implementation and assisted with carrying out the evaluation of memorization of pretraining strings.
Edward Raff Designed the experiments, interpreted the results and wrote the paper.