Emergent and Predictable Memorization in Large Language Models
Authors: Stella Biderman
USVSN Sai Prashanth
Lintang Sutawika
Hailey Schoelkopf
Quentin Gregory Anthony
Shivanshu Purohit
Edward Raff
Venue: NeurIPS 2023 poster (2023)
Keywords: large language model
emergent properties
memorization
TL;DR: We introduce and attack the problem of predicting which sequences will be memorized by a LLM

Abstract: Memorization, or the tendency of large language models (LLMs) to output entire sequences from their training data verbatim, is a key concern for deploying language models. In particular, it is vital to minimize a model's memorization of sensitive datapoints such as those containing personal identifiable information (PII). The prevalence of such undesirable memorization can pose issues for model trainers, and may even require discarding an otherwise functional model. We therefore seek to predict which sequences will be memorized before a large model's full train-time by extrapolating the memorization behavior of lower-compute trial runs. We measure memorization in the Pythia model suite and plot scaling laws for forecasting memorization, allowing us to provide equi-compute recommendations to maximize the reliability (recall) of such predictions. We additionally provide further novel discoveries on the distribution of memorization scores across models and data. We release all code and data necessary to reproduce the results in this paper at https://github.com/EleutherAI/pythia.

Emergent and Predictable Memorization in Large Language Models
Stella Biderman 12, USVSN Sai Prashanth', Lintang Sutawika', Hailey Schoelkopf Quentin
Anthony24, Shivanshu Purohit', and Edward Raffle
Booz Allen Hamilton, BleutherAA, Yale University, *Ohio State University, SStability AI, University of Maryland, Baltimore County
Abstract
Memorization, or the tendency of large language models (LLMs) to output entire sequences from their training data verbatim, a key concern for deploying language models. In particular, it is vital to minimize model's 's memorization of sensitive datapoints such as those containing personal identifiable information (PII). The prevalence of such undesirable memorization can pose issues for model trainers, and may even require discarding an otherwise functional model. We therefore seek to predict which sequences will be memorized before large model's full train-time by extrapolating the memorization behavior of lower-compute trial runs. We measure memorization in the Pythia model suite and plot scaling laws for forecasting memorization, allowing us to provide equi-compute recommendations to maximize the reliability (recall) of such predictions. We additionally provide further novel discoveries on the distribution of memorization scores across models and data. We release all code and data necessary to reproduce the results in this paper at http:///juuub.coo///lettere///ae
1 Introduction
Recent natural language processing (NLP) research in generative tasks has largely been driven by two findings: (1) the transformer architecture performs well [Vaswani al. 2017 Devlin et al. 2018 Radford et al., 2019]; and (2) increasing the scale of transformer architectures leads to improved performance [Brown al., 2020 Chowdhery et al. 2022]. In addition these benefits, transformers
are general and multipurpose architecture that have achieved saae-oof-the-ar results outside of
NLP on diverse tasks such as text-to-image synthesis [Ramesh et al.) 2022 Crowson al. 2022
Rombach et al. [0022]. code generation [Chen et al. 2021, Xu et al. 2022] et al. [022 and
protein modeling (Jmpper et al., 2021 Ahdritz et al., 2022]] Despite their widespread success and increasing use, the learning dynamics of transformer models are poorly understood and research into how a given model learns and internally represents data has the potential to affect a broad range of high-impact applications. As these models become increasingly adopted, it is essential that we develop better and more reliable tools for measuring and controlling their behaviors.
1.1 Memorization in Large Language Models
In particular, the demonstrated capacity and ability of these large language models to memorize data has become significant concern [Carlini al., 2019 2021 Hu al.) 2022]. The most obvious ramification is personal information or otherwise sensitive data being leaked to the public at large and extracted by a bad actor. This has motivated extensive research into mitigating memorization
Correspondence stellabieii aeellaabdderaa@@mall.comm
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

by decreasing the total quantity of memorized text [Lee et al. 2021 Kandpal et al. 2022] Carlini
ctt al. 2022 Hernandez al., 2022]. Although this is an admirable goal, we ultimately do not view it as an appropriate solution only some memorization is bad. What's more, some types of memorization are good: we want large language models to memorize factual events and details to avoid "hallucinating" plausiblessouning but errant facts to unsuspecting users [Power et al. 2022
Cao et al. 2022 Tirumala et al. 2022b].
Despite the extensive literature on memorization in trained models [Carlini et al.
2019 202
Hu
al. 2022]. , there are few tools to help practitioners either prevent memorization or detect early in model training. Before the advent of transformer mansformer-based large language models, using differential
privacy was popular (Abadi et al. 2016 McMahan et al. 2017 PPpov al., 2017]. However, such methods have been observed to hurt performance during pretraining [Amil et al., [02T], and are therefore not popular among people who train large language models. recent years, the bulk of interventionist work has focused on how removing duplicated samples from the training dataset can decrease memorization [Lee et al., 202T Kandpal et al., [022 Carlini et al., 2022 Hernandez et al.) 2022]. Importantly, these works focus memorization average and cannot be relied on to prevent memorization of specific training examples. Additionally, Biderman et al. [2023] shows that even when models are trained on deduplicated data they can still memorize substantial amounts of their training corpus.
Another approach to constraining model behavior is to intervene at inference time instead of at training time. Ippolito et al. (2022) introduce an interference-time intervention that has a 100% success rate at preventing verbatim memorization, but they note both that their methodology is easily subverted, and does not fulfill the intention behind the term "memorization." " which is the tendency of models to learn entire samples during training without understanding their underlying meaning. We view test-time intervention as a promising avenue for future research, especially due to its success in other domains (Haghighatkhah et al. |021 Ravfogel et al. 2022, Belrose et al. 2023a b). but it is insufficent as a solution because some the concern regarding PIl and copyrighted text concerns instrinsic properties of the model and not merely its output text. Although at first glance these techniques may seem more viable as downstream users can apply them to API models, most models offered via an API do not give sufficient access to the user to apply them. A missing component of the memorization literature is an investigation of which specific data points are memorized in large language models.
1.2 Scaling Laws
Due to the substantial cost of training large language models, it is highly desirable to be able to make predictions about model characteristics before they are actually trained. The literature scaling laws
Kaplan et al., 2020) Henighan et al.) 2020) Hernandez et al. 2021 Mikami et al. [021] Hoffmann et al. 2022] has been successfully used to inform the pocciiionmmaingg of a variety of researchers model training-time
by allowing them to generalize the decisions made while investigating smaller models to inform the design of larger (sometimes by many orders of magnitude) models [Rac et al.)
2021 Black et al. 2022 Scao et al., 2022b Chowdhery et al. 2022]. While this work on scaling laws does extend to memorization [Carlini et al.) [022] Hernandez et al. 2022). how memorization evolves during a model's 's training process across a variety of scales has not been studied.
Scaling law studies are typically used to find a model with the best loss trainable within given computational budget. However, the intuitions and experimental design around cheaply estimating the qualities of a final model can be applied to more than just training loss-we would like practitioners to be able to extrapolate other more complex properties of their final trained model prior to performing the costly training run.
In the case of memorization, because some data points are far more undesirable for a model to memorize, such as PII, it would be desirable for engineers to be able to predict whether model will successfully avoid memorizing such harmful data, and make informed risk assessments and decisions about training prior large compute expenses.
In our work, we study the creation of tools to predict the memorization of specific data points prior to model training, rather than the macro-level corpus-wide statistics considered by prior work. We take a first step in this direction by proposing two strategies: 1) making predictions from a smaller model to larger model; and 2) making predictions from partially trained model of a given size to the fully
2

---

trained model. Using smaller or partial model training runs to inform large model training runs is critical, because these runs provide a cheap method inform training behavior for given corpus. rather than training an entire large model from scratch. We find that the efficacy of our proposed methods varies with their downstream intended use, depending on whether precision (we want to confirm something was memorized) or recall (we want something "forgotten") is most desired.
1.3 Our Contribution
In this paper, introduce the question of extrapolating model's memorization behavior for specific training data points based on evaluations set relatively low-cost training regimes. These low-cost regimes enable
abort models without wasting significant compute resources in the event of undesirable hehavior. This includes the typical setting, where we extrapolate the qualities of large models based small models, as well novel setting where extrapolate the behavior of fully-tred bully-trained models based on partially-traine models. As far as we are aware, we are the first paper study forecasting model behavior in this novel setting.
Our primary contributions are:
1. Introducing the problem of forecasting whether or not model memorizes a specific training datum.
2. The discovery that the memorization of a specific training string by large language model is not reliably predicted by either studying smaller language models or partially trained checkpoints, unless sizable fraction of the pretraining compute of the target model is used.
3. A preliminary analysis of scaling laws for forecasting memorization, and recommendations for maximizing forecast reliability given set compute budget to make this prediction.
1.4 Roadmap
The rest of this paper organized follows: in Section present relevant facets of our methodology, including definitions of metrics (Sections 2. and 2.3 the threat model (Section (2.2). and choice of pretrained models (Section 2.4).. In Section 3 explore the feasibility of predicting the memorization behavior of large models based on small models. Further, in Section 4 we explore the feasibility of predicting the memorization behavior of the fully-trained model based on intermediate checkpoints. We then analyze these two kinds of predictors head-to-head and plot scaling behavior in Section 5 We perform ablations confirm our method is robust thresholding choices in Appendix and to deduplication in Appendix B
2 Methodology
2.1
Measuring Memorization
The paient muse
Jan: Doe und she lives in the United Staoe.
John
Doc
und
Ix:
lines
the
Uniual
Kinpdam
0.7
Pi deiacl io:
the calso of the nidas of's cielk is
dostmall
dac.
neve
cunern
repeating
polcm
The cen defendent is
Billy Bobs They are on rial for tas fraudl
Billy
Boli
An:
dley
noally
I8a
For
bx
0.5
The case defendazz is
Billy Bob. They are tñaa for fraud
Bulby
Bob
They
fraud
Table 1: Examples of memorization score calculation with different prompts. Note that these are provided for illustrative purposes and are not from the actual training data. The final example demonstrates a 4-extractible string.
"Memorization" is typically cast as a form of overfitting or failure to generalize outside the training data distribution, distinet from "good learning" in some senses. However, formalizing this intuition presents challenges. In this paper, we consider the framework introduced by Carlini et al. 2021 grounded in k-extractibility:
Definition 2.1. A string is said to be k-extractible if it (a) exists in the training data, and (b) is generated by the language model by prompting with prior tokens.
To demonstrate, the training data sequence "Their email address is me@alice.com is 3-extractible (memorized) if the prompt "Their email address" yields "is me @ @alice.com" com"-thus thus producing an
3

---

exact copy of the training data sequence. We tern the accuracy of tokens in the continuation as the memorization score of the sequence and call a sequence (k- )memorized or (k-)exttractable if the memorization score is 1. Illustrative examples are shown in Table
N Σ 185+i =
score(M,
In addition to k-extractabiility, we evaluate the memorization score, defined as the number of ordered matching tokens between the model's greedily generated sequence G32.64 and the dataset's true continuation S32.64 of a sequence E D on given prompt. See Equation ① for the formal equation, where N is the length of the true continuation and greedily generated sequence (32 in our case), and M is the length of the prompt (also 32 in our case). A memorized or extractable sequence has memorization score of
Doing a forward pass on large transformer is relatively expensive, costing about one third the cost of full gradient update step. Consequently, feeding the full training data through the model for forward pass would cost approximately one third the amount of compute that training the model did, and doing the full seven checkpoints that do would come to larger compute budget than training the models themselves.
To ensure computational feasibility in our experiments, we choose k 32 and evaluate the first 64 tokens from each sequence (we verify the robustness of this choice in Appendix (A). Each sequence is of 2049 tokens, sampled from shuffled documents. These sequences the input data points to the model during training.
2.2 Threat Model
Throughout this paper, we assume that an engineer is looking to train a large language model with billions of parameters on dataset, and that there is a small subset of the dataset that would be undesirable to have the model memorize. The engineer wishes to accurately predict whether this subset of the training data will be memorized by the fully-trained model, and would like to do so cheaply by expending relatively small amount of compute on testing prior to the full training run.
Following the literature on scaling laws (Kaplan et al. 2020 Hoffmann et al. 2022 the cost in FLOPs of training model is approximately
C 6 x [# Params] x [# Tokens]
and we assume the engineer has computing budget that allows them to perform substantial testing before performing the full model training run. Because the engineer is the model trainer, We assume that the engineer has full access to the final and test models' training runs, including viewing the ordering and content of data points seen by their model and saving intermediate checkpoints as desired.
We focus on the prediction of memorization, rather than simply filtering all data points that contain PII or are otherwise undesirable to memorize, for several reasons. First, although PII can be filtered from L.LM training datasets via heuristic filters, or more advanced methods like classifier- approaches, these filtering methods are not perfect, and may fail to remove certain instances of PII from the dataset [i ct al. 2023].
Additionally, there are also non-PII data points that are undesirable to memorize but desirable to train on, including copyrighted material. Some works suggest such data is essential achieving high quality performance in certain domains [Longpre et al., 2023 Min et al. [202)]. While the ethical and legal intricacies of permissible training data are outside the scope of this paper, the memorization of such content is widely contested issue
2.3 Predicting Memorization
We treat smaller model's memorization of a sequence, lack thereof, predictor for the memorization behavior of a larger model. Whether the interested model did memorize the sequence is the ground truth label, and the smaller model's behavior is the prediction.
For instance, Github Copilot's memorization of training data.

---

For example, if a smaller model memorized a sequence and the larger model did not, we can think of this case as a false positive. Likewise, if both models memorized the sequence. then the smaller model's prediction was a true positive. Models not memorizing the target sequence are negative cases.
This "prediction" by the smaller model compared against the ground truth allows us to calculate classification metrics such as precision and recall. In this case, precision tells us how many of the sequences memorized by the smaller model are also memorized by the larger model. Recall conveys the percentage of sequences memorized by the larger model that are also memorized by the smaller model. The same framing can also be applied when analyzing across time-where we compare the memorized sequences a certain intermediate checkpoint, and wish to predict which sequences will be memorized by the completed model.
As the engineer's sole concern is to avoid memorization false negatives and false positives in predicting memorization have very different impacts on their workflow: false positive (i.e. incorrectly predicting that model will memorize the undesirable subset) results in throwing away a cheap model that could have been fruitfully continued to train the final model, while a false negative (i.e. incorrectly predicting that model will not memorize the undesirable subset) results in the costly training of a full model that could leak sensitive samples from the training dataset. We are therefore primarily interested in assessing the recall of the predictors and will tolerate low precision if it comes with high recall. We explore the tradeoffs in these costs in Section 3
undesirable subset (see Section 2.2)
2.4 Choice of Models and Datasets
At the time of writing, the only publicly-vvailale pretrained LLM scaling suites trained on fully
public training data are EleutherAl's GPT-Neo [Black al. 202 Wang and Komatsuzaki 202
2022] and Pythia models [Biderman et al. 2023 and Cerebras systems' Cerebras-GPT Black et al.
[Dyy et al.
2023 All of these suites were trained on the Pile [ao et al.] 2020 Biderman al.
2022
Additionally, we were able to obtain access to the ROOTS dataset
SeeMillan-Majon et al.
Laurençon et al., 2022] that the BigScience Workshop's BLOOM (Scao et al., [002aa] model was trained on. Of these model suites, we choose to use Pythia because (a): All Pythia models saw data samples in the exact same order, and that order is publicly available, (b): the training data differs slightly across the GPT-Neo models, (c): some BLOOM models only have three partially- trained checkpoints, and (d): Cerebras-Gii GPT models don't provide partially-traines checkpoints. The OpenLlama, RedPajama, and StarCoder [Geng and Liu 2023 Computer [223] Li et al.) 2023) models have additionally been trained on public data, but were released after the completion this work and do not release intermediate checkpoints.
2022
We note that this limitation of eligible open models not a drawback according to our threat model- we intend to create a novel procedure and tool for risk assessments usable by model creators, who have full control and access to information about their model training.
The computational cost of many of the experiments we run is quite large, Consequently, we are unable to evaluate every partially-trane traiined model checkpoint in the Pythia suite For most of our experiments, we choose to evaluate seven checkpoints spaced evenly throughout training. Specifically, we evaluate on checkpoints trained for (23 10%), (44 10%), (65 10%). (85- 10%), (105 10%). (126 10%) and (146- 10%) sequences respectively, where these checkpoints approximately correspond to 7 checkpoints evenly spaced throughout training. We use the GPT-NeoX library (Andonian al. 2021
that trained Pythia to efficiently implement our evaluation protocol.
3 Memorization Across Scales
By far, the most common type of scaling law to study (and indeed, the origin of the term itself) is looking at how performance for very large models can be predicted based on performance of much smaller models. Fully-trained smaller model variants are independently useful as artifacts and can be applied in resource-consee constrained environments in place of larger models. Therefore, when projecting the characteristics of higher-compute model runs via scaling studies, training smaller model variants for this purpose is an actively desirable by- propdutt in contrast to the alternative of producing many
"The cost of doing so would be comparable to the cost of training the models in the first place.
5

---

shorter- shorter-training duration checkpoints of the same single large architecture to extrapolate properties of final full run. Therefore, the first question we seek answer is: can am LLM's memorization behavior be predicted across model scales?
Model
Precision
Recall
C44
6.39
0.13
0.m
1.90
Pythia-70M.
0.956
0.197
0.11
6m
Pythia-160M
0.948
0.289
Pythia-4
100M
0.940
0.401
0.13
0.00
6.71
0.11
0.77
1,00
0.13
0.10
Pythia-
0.931
0.512
Pythia-1.4B
0.926
0.554
6.30
+.07
0.71
0.31
1,00
0.71
0.71
0.10
Pythia-2.88 0.909
0.658
Pythia-6.9E 0.884
0.795
6.30
6.
0.70
1,00
0.013
0.50
0.33
Pythia-12B
1,00
0./13
0.00
(h) Precision and Recall when using each model to predict which sequences would be memorized by the 12B parameter model. For example, 95.6% of the sequences mem- orized by the 70M model were also memo- rized by the 12B model, but those only ac- counted for 19.7% of the sequences that the 12B model memorized.
1008
6.50
1.80
5.71
0.12
0.7
0.00
0.37
0.00
1.00
0.10
6.31
0312
14.0
(a) A Hinton diagram for visualizing the correlation between fully-memorized sequences by different model sizes. All models are fully trained.
Figure A look at memorization across scales
To evaluate how productive training small models can be for the purpose of predicting which data- points will be memorized by large models, we subset our data to the sequences with a memorization score of 1 (meaning all 32 target tokens were produced accurately by the smaller model). Then, we look at the correlations between each pair of fully-trained model sizes for which sequences are memorized. The results are shown in Figure
aa
We see a sharp decline in correlation between which sequences are memorized by smaller models and the 12B model as the gap between the model sizes increases. Unfortunately, we find that these low correlation scores cause the set of sequences memorized by small models to have very poor predictive power terms of what sequences will be memorized by larger model. We also measure precision and recall of fully-memorized sequences using each smaller model to predict the memorization of the 12B model as shown Figure lb] Although the precision is high for all models (see Section 2.2). we are more interested in achieving a high recall than a high precision. The recall is incredibly low across the board, with even the 1.4B parameter model only achieving a recall of 0.554 when trying to forecast the behavior of model an order of magnitude larger
Our findings suggest that using smaller model runs to forecast the memorization of larger models is not accurate. Due to the low recall, practitioners cannot use a small model's 's lack of memorization of a given sequence as a strong guarantee that their larger model will not memorize that same sequence. We therefore do not recommend using smaller model runs for this task, and seek to provide a setup that grants practitioners more assurances and better compute tradcoff.
Memorization Within Training
The second question we seek answer is: can an LLM's memorization behavior be predicted ahead of time within a training run? We wish to determine if, by testing memorization behavior after partially completing a training run, an engineer can achieve reliable signal about whether undesirable portions of the training data are memorized and if so to abort a training run early.
Our analysis in this section is motivated by the finding ofBiderman et al. [2033 that location within the training data does not impact whether a particular sequence is memorized. Therefore, we hypothesize that those concerned about the memorization of particular strings could move them early
'Typical use-cases are to use smaller models to predict the behavior of models one to two orders of magnitude larger, see Rae et al. [2021) Scao et al. 20226 Chowdhery al. 2022].

---

during training. Thus practitioners would have an early warning signal for detecting memorization of undesired sequences. Unfortunately, we continue to find largely negative results, but hope that future research with better techniques for predicting memorization might vindicate this idea.
640
410
PYthia-70M
(b) Pythia--1444
(c) Pythia- 12B
Figure 2: Heat maps visualizing the correlation between which sequences are memorized by different checkpoints. Plots for other Pythia models can be found in Figure TO
In Figure 2 show correlation heatmap between which sequences memorized by different checkpoints of the same model. We only look at memorization of the first 23 million sequences, as that is the data that our least-trained model checkpoint has seen.
Seq Num
Precision
Recall
Seq Num
Precision
Recall
23 106
0.919
0.513
23. 106
0.918
0.500
44 106
0.913
0.587
44 106
0.915
0.575
65 106
0.910
0.658
65 106
0.913
0.641
85 100
0.910
0.721
85 10
0.911
0.711
105 106
0.915
0.816
105 106
0.916
0.809
126 106
0.945
0.918
126 106
0.943
0.916
146 106
146 106
(a) Pythia-6.9B
(b) Pythia-12B
Table 2: Precision and recall for predicting which sequences would be memorized by the fully-trained model from a partially-trainel checkpoint. We observe consistently high precision, but only achieve high recall after significant compute has been expended (later intermediate checkpoints).
Our results on precision and recall (Table 2) largely mirror those of Section 3 in general trends. We see that the earliest intermediate checkpoints we test do not exhibit the high recall that is desirable, for instance with the 23M checkpoint of Pythia- in recall.
12B 12B underperforming the fully-trained Pythia-6.9B
We thus observe that using intermediate checkpoints model run to predict memorization is not a silver bullet-it is still the case that precision remains high throughout models, but recall is low for all predictors that use significantly less compute than the final model's cost. Therefore, this setting well, it is easier to guarantee sequence will be memorized through such extrapolations rather than not. Since the latter guarantee non non-memorization is more useful to engineers, our focus thus shifts to determining the compute-optimal model to train to gain desired level of recall, in order to maximize predictive power amongst the options we explore.
Scaling Laws
Having established empirical results in the previous section, we now examine our results through
the lens of computational efficiency and scaling laws, where the aim is to achieve the most reliable results for the least expense. To achieve this, we examine how well models of various sizes and number of training steps predict which sequences will be memorized by the fully trained 12B

---

parameter model. This is in notable contrast to Section 4 where purtially-traings models are only compared fully-trained models of the same size. As visual aid, models with the same size are colored the same.
5.1 Unusual Scaling
In the overwhelming majority of prior work on scaling laws Brown et al. 2020 Kaplan et al. 2020
Pu et al. [021] Mikami al. [021] Rae al. 2021 Black et [d.][202)] Scao et al.
2022b Chowdhery
al. 2022]. including scaling studies targeting memorization [Carlini al. [022] Hernandez et al.
2022 Tirumala et al. 2022a], plots of quantities interest vs. compute are linear on log or log-log
plot. We find that this is not the case in setup for both precision and recall.
The scaling data for precision is extremely anomalous. Not only are the plots non- non-linear, we find that the behavior of the 12B partially trained model is extremely out-of-line with the behavior of smaller models. The results for recall are less anomalous, lacking the divergent behavior for the 12B model, but nevertheless do not accord with what the scaling laws literature generally expects.
19'
10a
70 M
160 M
90
410 94
9.6x10-1
6.9
Ireckton
12
10-1
9'x10
1.4
2.3
8.9
x10-
10
0%
14
10%
100%
016
1%
30%
100%
Compute 1% of Finll 128 Model
Compute 0% Final 128 Model
(a) Precision
(b) Recall
Figure 3: Scaling curves for Pythia models.
Despite the fact that there is high-level pattern in the scaling laws curve for recall, careful look at the data indicates unusual behavior. In the low-compute regimes, which are of most interest to engineers looking to minimize the cost of creating prediction of the behavior of large models before they are trained, we see consistent pattern of larger models being better than smaller models for a fixed compute budget. However, the amount of compute expended scales, this is no longer the case. Starting at around 1% the budget of the fully trained model, equicompute models perform the same regardless of the number of parameters. Starting around 10% the budget of the fully trained model, the smallest model trained for this compute budget becomes the best predictor of memorization in the fully trained model.
5.2 Emergent Memorization
We also evidence of "emergent" "semi-emergent behavior model scale increases. In the literature on emergent behavior in large language models [Srivastava et al. 2022 Ganguli et al., 2022 Wei al. 2022 Caballero et al. 2022], the term refers to when a large model's performance on a task is substantially different from the extrapolated value of curves fit to the behavior of smaller models. Often, but not always, this occurs when performance goes from near-zero to meaningful. While our situation is not totally analogous, one can similarly consider "emergent memorization" to occur when data is memorized by large models which cannot predicted based on the memorization behavior of smaller models. Since, by definition, emergent behavior implies that smaller-scale model behaviors are qualitatively different to those larger models, this can pose challenges for traditional scaling laws or for extrapolating model behavior to models orders of magnitude larger. As result, we suggest that this is important area for further study, including expanding the scope of our work to models larger than 12B parameters.
8

---

5.3 Takeaways for Engineers
As discussed in Section 2.2 the primary point of interest to engineers is to predict the behavior of large language model before it is trained. Such predictions should be grounded in low-cost regimes such as the behavior of trained "test" models that are at least an order of magnitude smaller than the target model. We find that for cases where high recall is required, our scaling law defines what size of model should be trained at a given compute budget. magnitude below the final training run's size, find that when holding the compute budget fixed it is desirable to use the "smallest" model trained on no more the final run's 's total token count as possible, and to frontload the data seen by this smaller model with sequences whose memorization would be undesirable in order to predict whether these sequences would be memorized by a final model.
compute regimes less than two orders of
5.4 Takeaways for Decision-Making
The experimental procedures we present in this paper are a step toward practical risk assessment for large-scale training runs. We hope that future work both makes such assessments much cheaper and accurate, and that more practitioners adopt such measures and choose to be up-front about which mitigations they apply when training large neural models.
Limitations and Future Work
Our work constitutes the first steps towards developing way to predict what data will be memorized by a large language model before that model is trained, but has several limitations and opens opportunities for exciting future work. The most important these are:
Are measuring the correct thing? The definition of memorization is currently popular in the academic literature, but it is unclear if it is the best definition to use. We believe k-extractibility to be well-grounded in privacy concerns of language models, but other metrics such as memorization score may be more natural when studying the dynamics of memorization in training.
is derived from what
Does this generalize to other models? We report our experiments the Pythia suite, because was the only current language modeling suite suitable for such work at the time of our research. However, this leaves open many questions about whether our results generalize to models trained with different hyperparameters or different data. We validate our experiments with replications on the deduplicated Pythia models and different hyperparameters: in Appendix AA and B but other model suite is suitable for replicating this analysis. This gap points to the need for more reproducible, public dataset model releases to advance research memorization.
What about the data contents? Our work does not take the actual content of the training data into account at any point in time: we are looking exclusively at predicting memorization based on whether other cheaper models memorize the content. Future work looking into whether there are properties the training text that predict memorization of that text could be quite illuminating.
7 Conclusion
We propose a novel setting for forecasting model memorization prior to train-time, while minimizing the compute required to make this forecast. We present analyses on the two most natural setups for extrapolation: using fully-trained small models and partially-trained checkpoints of the final model compare and predict memorization of the final large model. We find that using much smaller models for this task is not viable, and that partial checkpoints of an existing model are similarly ineffective predictors of final memorization behavior when adjusted for cost. We derive scaling law to find the optimal equi-compute predictor of non- non-mmmorization and are able to provide recommendations based on this law. We hope that our focus on prediction of the memorization of specific strings will be compelling for future study, and that our analyses inform deep learning practitioners on methods to understand and reduce memorization while training large language models.

---

8
Corrections
Due to an error in our analysis code, an earlier draft of this paper reported a substantially higher recall in Table This draft of the paper features corrected numbers in that table and has adjusted the conclusions and discussion accordingly.
Acknowledgments and Disclosure of Funding
This paper was made better by conversations with and feedback from many individuals not on the authorship list. Following EleutherAI's open science values [Phang al. 2022], we shared early
drafts of these results with the EleutherAI Interpretability Reading Group as well as the Discord server large, garnering feedback from many people. We would like to acknowledge Nicholas Turner, Gurkenglas, and Amaru Cuba Gyllenste for identifying errors in our results and questioning our assumptions; Kyle O'Brien and Aviya Skowron for copy-editing; and Herbie Bradley, Nicholas Carlini, Katherine Lee, Naomi Saphra, and the EleutherAl Interpretability Reading Group for their thoughts, feedback, and advice.
We are grateful to Stability AI for providing the compute required to carry out our experiments.
Our work builds top of the work of many teams at EleutherAl and within the broader open source community writ large. We'd d especially like to recognize the GPT-NeoX [Andonian et al. 2021] team EleutherAI whose library used to measure memorization and the US AISI maintainers of the Hugging Face Hub whose infrastructure we used to host our data.
References
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
Gustaf Ahdritz, Nazim Bouatta, Sachin Kadyan, Qinghui Xia, William Gerecke, Timothy O Donnelll, Daniel Berenberg, Ian Fisk, Niccola Zanichelli, Bo Zhang, et al. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. bioRxiv, 2022.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hal- lahan, Josh Kramer,, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Wang Phil, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregres- sive Language Modeling in PyTorch, 8 2021. URL pettps://wwxe github..hh...... com/ eleutherai/ gpt
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private bert. arXiv preprint arXiv:2 108.::24080022,, 2021.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.088 12, 2023a.
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. arXiv preprint arXiv:2306.03819. 2023b.
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the pile. arXiv preprint arXiv:2201.07311, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXivv2304.01373 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: large scale autoregressive language modeling with mesh-tensorflow. GitHub, 2021. URL hattps://wwt github. comeub..oomeeee wtt. wwwttheraiigpttnee..e
10

---

Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-2001 An open-source autoregressive language model. In Proceedings of BigScience Episode Workshop on Challenges & Perspectives in Creating Large Language Models, pages 95-136, 2022.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 200:: 2020. URL htttts://aaartt org/ abs/ 2005. 14165)
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. arXiv preprint arXiv:2210.1443: 2022.
Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convo- lutional neural networks. Advances in neural information processing systems, 35:25237-25255 2022.
Nicholas Carlini, Chang Liu, Úlfar Brlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint 1.Xiv:2202.076466. 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.0333 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023: 2022.
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, April com www.///ogeeeeerrrooopttt////alap
2023. utt.stttss///// htapttsss/ggt..b..m/mg/gggerrrrp
ama Datta
Katherine Crowson, Stella Biderman, Daniel Korii,, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. arXiv preprint 2tt.::220400858833 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/18 1o0./880,,4 2018. URL 04805
http:://arxiv.orgg///
Nolan Dey, Gurpreet Gosal, Zhiming, Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestness. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprini arXiv:2204.055: 05999, 2022.
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747-1764, 2022.
11

---

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Amish Thite. Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: an 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. URL.
/uttps//aaxxxv.or//aa/////1//0
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023. URL https
//github. www/ www..pwww/occec/./.o reseerrcppeseeeccpppppee..o _1ama
Pantea Haghighatkhah, Wouter Meulemans, Bettina Speckman, Jérôme Urhausen, and Kevin Verbeek. Obstructing classification via projection. arXiv preprint arXiv:2105.0900 2021.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam Mc- Candlish. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.1470) 2020. URL http://ww..rg/bb///.....1. 14701
Danny Hemandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021. URL htttt:t/tas:iiiii http//aawww....raaa/a////////22/ 01293
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodke Tom Henighan, Tristan Hume, al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203, 15556, 2022.
Hongsheng Hu, Zoran Saleic, Lichao Sun, Gillian Dobbie, Philip S Yu, and Xuyun Zhang. Member- ship inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(1ss): 1-37, 2022.
Daphne Ippolito. Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Chon and Nicholas Carlini. Preventing verbatim memorization in language models gives false sense of privacy. arXiv preprint arXiv:22 /7546, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakcol, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583--55) 2021.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint (:XXi:20011088661 2020. URL httpps:/s///rrtt hwt.://.rggabb///s a2rgaabs/000..... 2001. 08361
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Seao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv: 107vv:2107.06499 2021.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas,
12

---

Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf. Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes. Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!. 2023.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne lppolito. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity, 2023.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv: /0. 171000663, 2017.
Angelina McMillan-Major, Zaid Alyafeai, Stella Biderman, Kimbo Chen, Francesco De Toni, Gérard Dupont, Hady Elsahar, Chris Emezue, Alham Fikri Aji, Suzana Ilié, et al. Documenting geographically and contextually diverse data sources: The bigscience catalogue of language data and resources. arXiv preprint arXiv:2201. 10066, 2022.
Hiroaki Mikami, Kenji Fukumizu, Shogo Murai, Shuji Suzuki, Yuta Kikuchi, Taiji Suzuki, Shin-ichi Maeda, and Kohei Hayashi. A scaling law for -onteelic-to-real transfer: How much is your pre-training effective? arXiv preprint arXiv:2108. 11018, 2021. URL http:tttt////rrr 2108. 11018
org/abs/
Sewon Min, Suchin Gururangan, Eric Wallace, Hannaneh Hajishirzi, Noah A. Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk nonparametric datastore, 2023.
Jason Phang. Herbie Bradley, Leo Gao, Louis J Castricato, and Stella Biderman. Eleutherai: Going beyond" open science" to" science in the open". In NeurlPS Workshop on Broadening Research Collaborations, 2022.
Vadim Popov, Mikhail Kudinov, Irina Piontkovskaya, Petr Vytovtov, and Alex Nevidomsky, Differ- entially private distributed learning for language modeling tasks. arXiv preprint arXiv: 1712v:171..07473. 2017.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra, Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02.177, 2022.
Jie Pu, Yuguang Yang, Ruirui Li, Oguz Elibol, and Jasha Droppo. Scaling effect of self-supervised speech models. Proc. Interspeech 2021, pages 1084-1088, 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 12. 1446, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint XXXi:2224400625 2022.
Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 0034-6055. 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.051000 2022a.
13

---

Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, al. What language model train you have one million gpu hours? arXiv preprint arX 15424, 2022h.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 04661, 2022.
K. N. Bharadwaj Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memo- rization without overfitting: Analyzing the training dynamics of large language models. ArXiv, abs/2205. 10770, 2022a.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-382985 2022b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: 6 billion parameter autoregressive language model, 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0768) 2022.
Frank F Xu, Uri Alon, Graham Neubig, and Vincent J Hellendoom. A systematic evaluation of large language models of code, arXiv preprint arXiv:2202.1311 13169, 2022.
14

---

A Robustness to Thresholding Choices
In Section B and Section 4
subset the data to the sequences with a "memorization score" of 1 (i.e., sequences that are fully memorized der previous works' definition). This approach labels all sequences with more than 32 tokens memorized as equally memorized, despite the fact that reality some will have a much longer accurately reproduced continuation than others. In this section we explore whether that effects our results.
Recertor of beqparocco
First, examine the shape of the distribution of memorization scores. We had originally as- sumed that the answer would be an (approxi- mately) exponential distribution, under the as- sumption that LLMs had constant "memo- rization rate" for correctly predicting each sub- sequent token. Our assumption was that this "memorization rate" would be based model size, and that it was the primary determinant overall memorization score distribution. This would be potentially problematic for our study, the 32-token memorized sequences would dominate the set of memorized sequences.
30
Figure 4: Distribution of memorization scores for 12B parameter model. For all upcoming sections in this paper, "memorized" is defined as score =
However, upon examining the distribution of memorization scores for the largest Pythia mod- els, was immediately clear that this cannot be the case. As shown in Figure 4 there is a very evident spike in the memorization score distribu- tion score Exponential distributions are thin-tailed distributions, and while they would have spike score 1, it is not possible for them to have such large spike. The effect shown Figure I can only occur in thick-tailed distributions, such as the power law distribution.
Model Size
Precision
Recall
70M
0.949
0.140
160M
0.941
0.222
410M
0.931
0.334
1.0B
0.922
0.451
1,4B
0.918
0.497
2.8B
0.900
0.611
6.9B
0.872
0.775
12B
This is good sign for our analysis, as means that the typical memorized datapoint in fact has a much larger number of memorized tokens than the 32 token threshold we were worried about. We also replicate Figure Tb with the doubled threshold and find roughly the same results. We also find the same results rerunning our scaling laws plots in Figure
Figure 5: Precision and Recall when using each model to predict which sequences would be mem- orized by the 12B parameter model. This table requires twice as many tokens to match to be con- sidered memorized, but otherwise is replication of Figure Tb
36418-1
2.808
70
430N
2x 10-
1.0%
32.10-
280
630
efiiiall Modil)
Naze:
(a) Precision
(b) Recall
Figure 6: Replication of Figure with longer seqeunces.
15

---

B Robustness to Deduplication
In order to further confirm the validity of our analyses, we run our experiments on the Pythia (deduplicated) suite, which was trained on deduplicated copy of the Pile [Gao et al. 2020 for 1.5 epochs. keeping with the literature on deduplication and connection with memorization [Lee et al. [2021 Kandpal et al. 2022], we observe that memorization is decreased for this set of models, albeit slightly (Figure 7.. This may be due the 1.5 epoch training setup we adopt offsetting the benefits of deduplicated data.
Pythia (standard)
0.016
Pythia (deduped)
0.014
Fraction memorized
0.012
0.010
0.008
0.006
0.004
0.002
70M
160M
410M
1.4B Models
2.8B
6.9B
12B
Figure Fraction of all sequences memorized by both Pythia model suites. For example, Pythia -2B has memorized 1.62% of sequences. We can observe the deduplicated models memorize less of their dataset than their non-deduplicated counterparts.
3404
0.112
5.111
c.m
6.90
0.44
3.09
S4H
0.19
5.14
6.10
6.14
6.4
0.44
6.4
6AM
6.8
6:44
6.40
648
6481
30M
6714
6.0
A08
1080
9000
0/31
6.0
2073
1080
Festoan's correnatio
Fesman commantioo
(a) P2thia-12B
(b) Pyytha--1B-deddaee
Figure 8: Inter-checkpoint correlations for memorization of Pythia- 12B and Pythia- Pythia-12B-aa Mafiia--2BBeeduonn deddudde respectively. Between the two of models. we observe extremely similar (though not fully identical) patterns in the correlations of their checkpoints.
16

---

Model
Precision
Recall
Pythia- 7ytiia-70M-deduped
0.952
0.218
Pythia- 100-dddped
0.943
0.304
Pythia-410M-dddupped
0.939
0.422
Pythia- d.d-dddu
0.927
0.531
Pythia- 1.4--deduped
0.924
0.535
Pythia- 2ittia-..8Bddduuper
0.912
0.675
Pythia-6.9P BB-ddduped
0.891
0.807
Pythia 12B-deduped
Figure 9: Precision and recall when using each model to predict which sequences would be memorized
by the 12B parameter model. Replicates Figure lb
Seq Num
Precision
Recall
23 106
0.920
0.523
44 106
0.917
0.595
65 106
0.915
0.658
85 10
0.915
0.724
105 10
0.922
0.820
126 106
0.949
0.920
146 106
Table 3: Precision and recall for predicting which sequences would be memorized by the fully-trained model from partially-trained checkpoint, for eythia-12-ddedpee The trends observed here match Table
We replicate our analyses on the deduplicated models and find the same trends hold for our exper- iments on the Pythia-deduplicated models as do for the regular Pythia suite. Heatmap correlation results show the same conclusions (Figure 8 and we replicate precision and recall results from Figure and Table but on deduplicated models Figure P and Table 3 We therefore believe our results to be reasonably robust across hyperparameters and engineer train-time choices, but hope that future work may replicate some of our findings entirely distinct corpora.
17

---

60
...
(a) 70M Paramoter Model
(b) 160M Parameter Model
+.11
6411
(c) 410M Parameter Model
(d) 1.0B Parameter Model
111
(e) 1.4B Parameter Model
(f) 2.8B Parameter Model
254
km
1AN
600
000
6.9B Parameter Model
(h) 12B Parameter Model
Figure 10: Heat maps visualizing the correlations between which sequences are memorized by different checkpoints.
18

---

C Author Contributions
Stella Biderman Conceived, organized, and lead the project, and wrote the paper.
USVSN Sai Prashanth strings.
Implemented and carried out the evaluation of memorization of pretraining
Lintang Sutawika
Analyzed and interpreted the precision and recall results and plotted data.
Hailey Schoelkopf Carried the evaluation of memorization of pretraining strings, performed the robustness evaluation, found and fixed several bugs in our code, and wrote the paper.
Quentin Anthony Analyzed and interpreted the results and wrote the paper.
Shivanshu Purohit Optimized the implementation and assisted with carrying out the evaluation of memorization of pretraining strings.
Edward Raff' Designed the experiments, interpreted the results and wrote the paper.
19


Reviews:
[{"review_id": "2bAMGx6lMt", "reviewer": "Reviewer_c2gw", "summary": "This work empirically predict the memorization behaviors as measured by k-extractible and memorization scores. They tested on two settings 1) use small model to predict the memorization behavior or larger model, 2) use partially trained model to predict the fully trained model. The main finding is that 2) is has much higher correlation than 1). Also included scaling laws and discussions on emergent memorization.", "questions": "The heatmap is figure 3 is a wash, some variant of the Hinton diagram might show such correlation more clearly than color scales.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Clearly motivated and clean experiments with clear conclusions\n* Well written and flows well\n* interesting scaling law for emergent memorization\n\nA clean paper with a clear message, addressing an important enough problem should be accepted.", "weaknesses": "* Not too obvious if the methods are good enough for practitioners to predict undesirable memorization due to lack of absolute metrics, in particular with respect to PII or something else people care about. Seems like random sequences were used in the experiments, which may have different behavior than PII\n* Only tested on Pythia model suite and limited range of data (partly addressed in limitations). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We are grateful that you feel our problem setting is important and that the paper contains “clearly motivated and clean experiments with clear conclusions”.\n\n>  The heatmap is figure 3 is a wash, some variant of the Hinton diagram might show such correlation more clearly than color scales.\n\nWe appreciate the feedback regarding how to communicate our results more clearly! We will regenerate the heatmaps as Hinton diagrams.\n\n> Not too obvious if the methods are good enough for practitioners to predict undesirable memorization due to lack of absolute metrics, in particular with respect to PII or something else people care about. Seems like random sequences were used in the experiments, which may have different behavior than PII\n\nYou are correct that we studied randomly distributed sequences. This was a deliberate decision to limit the scope of our investigation because we were tackling a problem that has never been studied before. While PII is a motivating example, in this paper we are not studying memorization of PII specifically. We already have follow-up projects in the works studying how taking semantic information into account changes this analysis, but view this paper as an essential first step and an interesting self-contained result. \n\n> Only tested on Pythia model suite and limited range of data (partly addressed in limitations).\n\nWe have discussed this further in G3 and G4 of our general response!\n\nWe thank you for improving our visualization of our results and believe this will improve the usefulness of the paper to a reader, and your questions have been a helpful opportunity to assess which points are not adequately discussed in the paper body or limitation section. We hope that these further points will help answer your questions and clarify the motivation of the experiments we chose to perform in our work.\n\n\n"}, {"review_id": "8TEbYCJUGf", "reviewer": "Reviewer_kL1W", "summary": "The paper focuses on predicting memorization of specific data point prior to training the model. The main approach of the paper relies on scaling law, where the memorization property of a large model is predicted using a smaller model, or a partially trained model. \nThe findings are (1) small models are not good at predicting the memorization patterns of large LMs, causing very low recall score. \n(2) using intermediate checkpoint to predict also attains low recall. (3) the prediction of memorization recall and precision does not follow the scaling law. ", "questions": "see above. ", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The setting of predicting whether a specific data is memorized is novel and interesting. The problem is well-motivated. \n2. The approached tried in the paper are reasonable: predicting the memorization behavior of a large model using a smaller LM, using a intermediate checkpoint, and fit a scaling law curve. \n\n", "weaknesses": "1.  none of the proposed approaches produce the desirable recall that's needed for this approach to be viable. Even though there is a scaling law pattern, the best recall still seem to be not good enough. \n\n2. If the developer know which data is sensitive, can they just remove the data from training distribution, and how much would that hurt performance? If they can just remove these data without harming performance, the basic motivation of this problem might be questioned. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback on our work! We are grateful that you find the target of our work well-motivated and novel, and agree that it is an important target for future work to improve beyond our performance on the task we have defined.\n\n> If the developer know which data is sensitive, can they just remove the data from training distribution, and how much would that hurt performance? If they can just remove these data without harming performance, the basic motivation of this problem might be questioned.\n\nWe understand that there may be more clarification needed for how we envision our experiments’ practical application and connection to our described threat model. We believe our method can also prove helpful in the setting where all known PII cases are filtered, but not every occurrence will be successfully caught: for more information, please see our general response, sections G1 and G2 for clarification regarding this threat model and assumed problem setting.\n\n> none of the proposed approaches produce the desirable recall that's needed for this approach to be viable. Even though there is a scaling law pattern, the best recall still seem to be not good enough.\n\nWe agree that the recall scores, even at high compute cost, are below a comfortable threshold for use by a practitioner or model trainer. We believe this is still a valuable result to report because\nIt demonstrates that the problem of memorization forecasting is a difficult one, with much room for progress in absolute and equi-cost terms from the community. It appears to be quite a bit more difficult than the standard loss-prediction target of scaling laws, due to the intended prediction of fine-grained behavior.\nWe believe that our simple approaches are the natural first steps, as they are such that practitioners might already produce lower-cost artifacts (smaller or partial model checkpoints) that can be applied via the same methods that are tested in our paper. It was not clear a priori that these natural approaches would be unsuccessful.\nOur work does not consider more advanced predictors, such as, for instance, training a small model for many epochs on a subset of the to-be-trained large model’s dataset. A limitation of using small models’ memorized sequences as predictors is that small models simply memorize less of their data than larger ones, which repeatedly training on the harmful-to-memorize subset might circumvent by intentionally overfitting to that data, memorizing similar quantities to a large model.\n\nWe hope that this clears up some of the questions that you have had in reading our paper, and would be more than happy to discuss further with you.\n"}, {"review_id": "dUiHe0gGTN", "reviewer": "Reviewer_S833", "summary": "This paper introduces the difﬁculties of both forecasting a future model’s memorization of speciﬁc training data and forecasting the memorization behavior of fully-trained LLMs from partially-trained checkpoints. Scaling laws proposed for forecasting memorization, and recommendations for maximizing forecast reliability given a set compute budget to make this prediction.", "questions": "Why not utilize LLaMA (Alpaca/Vicuna) or Falcon or other popular LLMs for experiments?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、The core idea of  of  scaling laws for forecasting memorization makes sense to me, especially for larger models.\n\n2、Proposed method is quite simple and works very efficient.\n\n3、The paper is clearly motivated and written in a smooth and easy-to-read manner. For example, describes in detail the connections and differences with related work.\n", "weaknesses": "1、The essence of scaling laws is just the method that the author borrowed and tried instead of his own innovation.\n\n2、The model used in the experiment in this paper is relatively uncommon and not convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, and we are glad you feel the paper is well-written and effectively communicated the takeaways we intended. \n\n> The essence of scaling laws is just the method that the author borrowed and tried instead of his own innovation.\n\nWe believe that our application of scaling laws to the memorization forecasting problem is a natural application and valuable to the community because:\nScaling laws formalize the tradeoff between (compute) cost and performance on our novel problem setting. This is actively desirable, as it allows us to track progress on our defined forecasting task, where success is defined as strong predictive accuracy for a lower cost. They also allow practitioners to select freely along this cost-accuracy tradeoff.\nFollowing existing scaling law setups from the literature enables us to explore how a practitioner could *repurpose the small models they have already trained in a scaling law-fitting sweep* for this alternative purpose. \nThis amortized cost is our target, as we hope that driving down the cost of memorization forecasts and countermeasures will encourage model trainers to place memorization and other security or privacy considerations as a first-class concern and a design decision on par in importance with model loss and downstream performance.\n\nAdditionally, we hope that our application of scaling laws to this task will inspire other practitioners to consider how their typical scaling law studies can be designed such that *the compute cost of running the study/hyperparameter sweep* (the cost of making an extrapolated prediction) can be minimized while trading off a desired quality of extrapolation, as this angle is not often considered in the literature.\n\n> Why not utilize LLaMA (Alpaca/Vicuna) or Falcon or other popular LLMs for experiments?\n\nPlease see section G3 of our general response for more detail on why we were unable to use other models for our experiments.\n\n> The model used in the experiment in this paper is relatively uncommon and not convincing.\n\nWe would be happy to answer any questions about why we chose the prediction setups we did, and why we cast the memorization problem as binary classification in the paper. \nIf this refers to our use of the Pythia models, we have responded to this in G3 and G4 of our general response, and hope that this may clear some things up.\n\nWe are grateful that you found our paper “clearly motivated” and “written in a smooth and easy-to-read manner”. We hope that our work in this area will spur further study on forecasting and detecting memorization, and we hope our response will additionally convince the reviewer that the usage of “the essence of scaling laws” here is novel, and will help the community rally around this problem setting with a quantitative measure of success in future.\n"}, {"review_id": "P62lkB4nUI", "reviewer": "Reviewer_5bNj", "summary": "This paper discusses the issue of memorization in large language models, which can lead to the output of sensitive data. The authors propose to predict which sequences will be memorized by partially-trained checkpoints. They use the Pythia model in experiments and plot scaling laws for forecasting memorization, allowing them to provide equi-compute recommendations to maximize the reliability of such predictions. ", "questions": "In practical scenarios, how can we know in advance which data should be memorized well (such as factual knowledge) and which data should not be remembered (such as private personal data)? If we know in advance, the proposed method will no longer be needed; if we do not know, how can we improve the reliability of the model specifically after forecasting the distribution of the memorized data points?", "limitations": "The authors adequately discussed the limitations in the last section.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well structured and addresses an important problem\n\n2.The memorization score and extractibility are easy and effective\n", "weaknesses": "1.The authors set k=32 and evaluate the first 64 tokens from each sequence because of the computational costs, \nwhich deviates greatly from the actual situation\n\n2.They only use the Pythia models for evaluation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, and we are glad you find our posed memorization behavior forecasting problem important. \n\n> in practical scenarios, how can we know in advance which data should be memorized well (such as factual knowledge) and which data should not be remembered (such as private personal data)? If we know in advance, the proposed method will no longer be needed; if we do not know, how can we improve the reliability of the model specifically after forecasting the distribution of the memorized data points?\n\nWe have received several requests for clarification on the interaction between our threat model and such filtering techniques. We apologize for the confusion our paper has caused, and we have responded to these questions in our general response.\n\n> 1.The authors set k=32 and evaluate the first 64 tokens from each sequence because of the computational costs, which deviates greatly from the actual situation\n\nWe evaluate additionally on settings in which we require 64 tokens to be predicted accurately following a prompt of the first 32 tokens in a sequence (Supplementary Material, Appendix A), and find that all our results+conclusions in the paper are maintained. This gives us reassurance that our findings are not overly determined by our bounding the length of memorized generations. However, we respectfully disagree that this “deviates greatly” from the real world. Researchers in the real world are also resource constrained, and we believe that memorizing only 64 tokens will be problematic in some situations.\n\nWe believe that the narrative and description of what experiments we reran can be improved, so we intend to clarify the experiments that we ran (namely, rerunning all our experiments to determine which of the first 96-token sequences (rather than 64-token sequences) in a context window are 32-extractible).\n\n> 2.They only use the Pythia models for evaluation\n\nPlease see Q3 and Q4 in our general response for more detail on why we were forced to accept this limitation, and the steps we took to assess whether our results may transfer to very different datasets.\n\nWe thank the reviewer for engaging with and improving our paper, and hope that our response will assure them as to the usefulness and robustness of our experiments as run in the paper. \n\n"}, {"review_id": "XH9aTpyvse", "reviewer": "Reviewer_bdyA", "summary": "This work presents an empirical study on predicting the memorization of examples in large language models. Specifically, using the Pythia model, they test two ideas for memorization prediction: (1) prediction via smaller models, and (2) prediction via partially trained models. Through extensive experiments on models with different scales, this work shows that while both method achieves non-trivial results, neither of them can reliably predict memorization with high recall. Therefore, this paper also includes experiments on selecting the best method w.r.t recall and discovers interesting trends.", "questions": "1.\tWhy is there a 6 in Equation 2?\n2.\tFor Figure 1 and Figure 3, are the correlations computed using memorization scores or are they computed using 0/1 values? I assume it’s the former, but I don’t think it has been clarified in the paper.\n3.\tAre the precisions and recalls computed at the token level or at the example level?\n4.\tI’m wondering what is the upper bound of the prediction is. In Figure 1, the values at the diagonal are all 1, which makes sense as they are the same model. However, what will that value be if the initialization seed or the data order are different?\n", "limitations": "The authors have included a great limitations section in Sec. 6 in the main paper.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThis work is an extensive empirical study on an important and timely topic.\n2.\tThe analysis conducted in this paper are highly connected to real use cases and the findings also lead to practical suggestions.\n3.\tThis is a very well-written paper with clear motivation, clean organization, and an honest, insightful limitations/future work section.\n4.\tI find the additional analysis in the appendix to be very interesting and can add significant value to this work. I would encourage the authors to move them to the main paper especially considering the current main paper is not very long.\n", "weaknesses": "The limitations section in the main paper is insightful and honest. I agree with everything written there and do not think any of the points is a severe flaw. \n\n1.\tAs the authors acknowledged in the limitations, the use of k-extractible and the memorization score simplifies the problem. In practice, the attacker may not need to extract exactly the same information as the training data.\n2.\tAlso mentioned in the limitations, the analysis in this work does not touch data properties or data contents, which are important future works.\n3.\tAlso mentioned in the limitation, despite the large amount of experiments done in this work, it is still unsure how the findings will generalize to other models and data.\n4.\tMinor presentation issue: I find the takeaway in LINE 308-310 hard to understand.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their very thorough and positive comments on our work, and are excited that they find our work may be of immediate empirical usefulness!\n\n> As the authors acknowledged in the limitations, the use of k-extractible and the memorization score simplifies the problem. In practice, the attacker may not need to extract exactly the same information as the training data.\n\nWe agree that such an extension would be very valuable and insightful! For our work, we limit ourselves to the k-extractible setting (+ continuous memorization score) as the majority of memorization literature focuses on this setting, but a more thorough understanding of what sorts of memorization are harmful versus benign would provide excellent direction to future work. We are currently following up this work by investigating the taxonomy of what sorts of data points are memorized.\n\nWe are grateful you believe the robustness analyses in the appendix add value to the work, and if we are able to present them within the length limit, we will move these to the main body of the text.\n\nThe derivation of Equation 2 can be found in [Kaplan et al. 2020, Section 2.1] (https://arxiv.org/pdf/2001.08361.pdf). In brief, there are two operations per parameter per datum in a single forward pass and twice as many computations required to do the backward pass (as you operate on both the weights and the gradients)..\n\nPrecision and recall are computed at the example level, and the correlation heatmaps are computed using the data subsetted to sequences that are fully memorized, though we see much the same results regardless of subsetting choices when computing these correlations. We will strive to improve the exposition and clarity of such captions and measurement details in the updated paper!\n\n>  In Figure 1, the values at the diagonal are all 1, which makes sense as they are the same model. However, what will that value be if the initialization seed or the data order are different?\n\nAt the time of writing this paper, we did not have access to any models trained on different initialization or data seed that were otherwise the same. However, as Pythia-160m models trained on different data orderings + with different initializations were since released, we would be happy to rerun this analysis on such a model, though the time required to perform this experiment will not permit us to provide these results prior to the end of the review period.\n\n> Minor presentation issue: I find the takeaway in LINE 308-310 hard to understand.\n\nWe will improve the presentation of this takeaway in an update to the paper! Our scaling law experiments calculate the memorization statistics *only on the first 1/7 of the corpus seen by every predictor we compare*, therefore assuming that if a practitioner knows of a subset of data they would like to predict memorization on, they should place that subset within the first 1/7 of the training data seen, so that predictions on it can be best measured. Else, measuring memorization on data appearing at the end of training would require a fully-trained predictor model to assess memorization under that predictor.\n\nWe hope that these clarifications help increase your understanding of the paper, and intend to incorporate the feedback on presentation and exposition in an updated version of the paper if accepted. We appreciate your thorough feedback and believe it will lead the paper to become stronger as a result.\n"}]

Author rebuttal:
We thank all reviewers for their valuable feedback on our work. We have worked to revise the presentation of the paper where recommended, and hope to provide this updated version in the final camera-ready submission. We would like to clarify some details regarding the application of our prediction method in practice where either PII can be filtered from the dataset prior to training, or a set of “known-harmful to memorize” sequences are not already known, and our threat model in those conditions: 

G1: Why can’t we simply remove all harmful-to-memorize / PII-containing data from training datasets beforehand?

PII can be filtered from LLM training datasets via heuristic filters, or more advanced methods like [classifier-based approaches](https://arxiv.org/abs/2305.06161). However, these filtering methods are not perfect, and may fail to detect certain instances of PII that end up not filtered from the dataset.

Additionally, there are also non-PII data that are undesirable to memorize but desirable to train on. Copyrighted books form a substantial portion of the training corpus of a wide variety of models, including GPT-NeoX, Pythia, LLaMA, LLaMA 2, GPT-3, GPT-3, and many more. The current dominant thinking is that this is essential to achieving high quality performance (Longpre et al., 2023), but there are not nearly as many license-compliant books available as is standard for training models (Brown et al., 2020; Gao et al., 2020; Touvron et al., 2023). While the ethical and legal intricacies of permissible training data is outside the scope of this paper, we can confidently say descriptively that many organizations do in fact train on copyrighted text and [are currently being sued for it](https://www.theverge.com/2023/7/9/23788741/sarah-silverman-openai-meta-chatgpt-llama-copyright-infringement-chatbots-artificial-intelligence-ai). In particular, the lawsuit about [Copilot](https://www.theverge.com/2022/11/8/23446821/microsoft-openai-github-copilot-class-action-lawsuit-ai-copyright-violation-training-data) where the exact text of the model generations are explicitly the key issue.

Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901.
Gao, Leo, et al. "The pile: An 800gb dataset of diverse text for language modeling." arXiv preprint arXiv:2101.00027 (2020).
Longpre, Shayne, et al. "A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity." arXiv preprint arXiv:2305.13169 (2023).
Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).

G2: How can our method (forecasting memorization with a small model) be applied in the absence of a known-bad-to-memorize data subset ahead of time?

One adaptation of our threat model (which takes into account the separation of intended and unintended memorization) is to perform a prediction of what data will be memorized, and *examine that set of predicted-to-be-memorized examples* for cases of PII by hand, as it is much smaller than the entire dataset. This extra PII-detection is a scenario in which high precision is valuable, so that any datapoints surfaced by the predictor are likely to be memorized by the final model and so this can serve as an early warning of undesirable memorization behavior. 

We thus believe that even in the absence of a subset known to be harmful to memorize, our method can be applied as a “quality check” that confirms the absence of unintended memorization/overfitting/misgeneralization in a smaller preliminary model. Our strong results on precision do seem to indicate some quantity of data that is memorized across many different models. We would like to note that we are currently exploring the topic of the characteristics of the sequences memorized by our models, to assess the rate of occurrence of PII in memorization, what other properties are commonly held by memorized sequences, and other characteristics.

G3: Why did you not also use [Falcon/LLaMA/other non-Pythia model suites]?

We would be ecstatic to be able to run our analyses on LLaMA, Falcon, or other LLMs trained on other datasets. Unfortunately, the authors of those models do not release sufficient data to study questions relating to memorization of training data. Our analysis requires having access to the pretraining dataset, which we do not have for LLaMA, LLaMA 2, or Falcon. Additionally, some of our experiments require having the data *in the same order it was seen by the model*, or partially trained model checkpoints  The Pythia models are the only models known to us with a fully reproducible training data loader and possess far more partially trained checkpoints than any comparable model suite.

Finally, the Falcon models were impossible for us to compare to as they came out after the submission deadline for this conference.


G4: Do you expect these results to transfer to models trained on other datasets?

We hope so. In an effort to reduce doubts that our results would not transfer across different model training corpora, we replicated our experiments on the Pythia-deduplicated models trained on a deduplicated variant of the Pile and found the same conclusions. We hope that further work  either compare or allow us to subsequently compare results on models with highly differing training corpora, and note that the nonexistence of model suites outside of Pythia fitting our experimental requirements of:
Publicly available training data
Known, consistent data ordering
Varied model sizing
prevented us from evaluating any other non-Pythia models in this work.

We hope that these clarifications, along with other improvements to its presentation and organization, will increase the quality and usefulness to the community of our work, and would be happy to answer any further questions.


Meta-review: The work empirically studies how to predict the memorization behavior of LLMs. It shows that using a partially trained model to predict memorization is more effective than using a small model. 

Reviewers recognize the importance of the problem as well the empirical study and insights provided by the work.

Most of the concerns were addressed by the author in the rebuttal. The final scores are unanimous accept (5, 6, 6, 6, 8).

Please add the additional discussions, especially the ones regarding generalization of the method to other models and datasets in the final version.
Decision: Accept (poster)