Quantifying Memorization Across Neural Language Models
Authors: Nicholas Carlini
Daphne Ippolito
Matthew Jagielski
Katherine Lee
Florian Tramer
Chiyuan Zhang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: memorization
large language models
duplication

Abstract: Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others).
We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.

Published as a conference paper at ICLR 2023
QUANTIFYING MEMORIZATION ACROSS NEURAL LANGUAGE MODELS
Nicholas Carlini
Daphne Ippolito 1.
Matthew Jagielski' Chiyuan Zhang'
Katherine Lee!.3
Florian Tramèr!
Google Research
2 University of Pennsylvania Cornell University
ABSTRACT
Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated casy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others).
We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (I) the capacity of model, (2) the number of times example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes more complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse models continues scale, at least without active mitigations.
INTRODUCTION
The performance of neural language models has continuously improved as these models have grown from millions to trillions of parameters (Fedus et al.) 2021), with their training sets similarly growing from millions to trillions of tokens. In anticipation of future, even larger models trained on minimally curated datasets, is important to quantify factors that lead to increased memorization of model's training set. Indeed, recent work has shown that training data extraction attacks are a practical threat for current language models (Carlini et al. 2020);; an adversary interacting with a pretrained model can extract individual sequences that were used train the model.
While current attacks are effective, they only represent a lower bound on how much memorization occurs in existing models. For example, by querying the GPT-2 language model. Carlini et al. (2020) (manually) identified just 600 memorized training examples out of a 40GB training dataset. This attack establishes (loose) lower bound that at least 0.000000155 of the dataset is memorized. In contrast, we are able to show that the 6 billion parameter GPT-J model (Black et al., 2021, Wang and
Komatsuzaki 2021) memorizes at least 1% of its training dataset: The Pile (Gao
2020).
In addition to prior work's s loose estimates of models' memorization capabilities, there is limited understanding of how memorization varies across different neural language models and datasets of different scales. Prior studies of memorization in language models either focus on models or
datasets of a fixed size (Carlini et al.) 2019 Zhang et al. [202]] Thakkar et al., 2020) or identify
narrow memorization- versus-scale relationship (Carlini et al. 2020: Lee et al. 2021). While McCoy al. (202T) broadly study the extent to which language models memorize, their focus on how to avoid the problem and ensure novelty of model outputs, rather than on studying model risk through identifying the maximal amount of data memorization.
Authorrs ordered alphahetically.

---

Published as a conference paper at ICLR 2023
This paper addresses both of the above open questions by comprchensively quantifying memorization across three families of neural language models and their associated datasets. We leverage access to each model's original training set to provide order-of-manne magnituee more precise bounds on the amount of extractable data that an adversary could recover than in prior works.
We first construct set of prompts from the model's ' training set. By feeding prefixes of these prompts into the trained model, we check whether the model has the ability to complete the rest of the example verbatim. This allows to measure memorization across models, datasets, and prompts of varying sizes. We identify three properties that significantly impact memorization:
1. Model scale: Within model family, larger models memorize 2-5x more than smaller models.
2. Data duplication: Examples repeated more often are more likely to be extractable.
3. Context: It is orders of magnitude casier to extract sequences when given a longer context.
Our analysis suggests that future research on neural language modeling will need to take steps to prevent future (larger) models from memorizing their training datasets.
2 RELATED WORK
There is extensive prior work that qualitatively studies memorization in neural language models. Prior work has demonstrated extraction attacks that recover memorized data including URLs, phone numbers, and other personal information (Carlini et al., 2020; Ziegler| 2 synthetically
injected "canaries" (Carlini et al. 2019, Henderson et al. 2018: Thakkar et al. 2020. Thomas et al. 2020). However most of these works qualitative and aim to demonstrate the existence of extractable data, rather than precisely quantifying how much models memorize. For example, the unprompted memorization evaluation of Carlini et al. (2020) found just 600 examples of memorization in GPT-2. Our paper aims to establish tighter bounds on the fraction of a dataset that is memorized.
Our analysis is relevant to the broad literature privacy attacks machine learning. For example, membership inference attacks (Shokri et al. [017]] Yeom et al., 2018) let an adversary detect the presence of a given example in model's training set; other forms of data leakage let adversary learn dataset properties (Ganju er al.) 20188] Fredrikson et al.) (015). We focus on extraction attacks due to their relevance for language modeling extraction implies significant leakage from model, and grows with data duplication (Lee et al. 2021 a common feature of large-scale text datasets.
Various definitions of memorization in deep neural networks have been studied in prior work (Carlini et al., 2019 2020 Feldman and Zhang] 2020 Zhang et al. (02T). A detailed comparison with those existing formulations presented in Section 3.1 One leading general memorization definition is differential privacy (Dwork et al. (006)). which formalizes the idea that removing any one example from the training set should not change the trained model. However, while differential privacy protects single user's private information, it is ineffective for preventing memorization of highly duplicated data, and does not capture the complexity of social, linguistic data (Brown et al. 2022). Also, differentially private lcarning algorithms (Abadi et al. 2016) generally suffer from expensive computation, slow convergence, and poor model utility, despite recent advances (Anil et al. 2021).
In concurrent work, Kandpal et al. (2022) study how often models emit memorized data function of data duplication. Their analysis focuses on evaluating why training data extraction attacks succeed. In contrast, we explicitly prompt models with training data prefixes in order measure memorization in the worst case, something that a practical attack cannot necessarily do.
Prior scaling hypotheses. Our motivation to study scaling phenomena stems from anecdotal evidence in prior work that memorization ability relates to various aspects of scale. In particular, our analysis on model scale informed by preliminary experiments in (Zhang al. 2017 Carlini et al. 2020), our data duplication experiments follow in the line of Lee et al. (2021), and our context length experiments build on hypotheses by Carlini et al. (2020); Ziegler (2021).
METHODOLOGY
3.1 DEFINITION OF MEMORIZATION
To begin, we first select precise definition for memorization:
2

---

Published as a conference paper at ICLR 2023
Definition 3.1. A string s is extractable with k tokens of context from model if there exists (length-k) string p. such that the concatenation |p || s] is contained in the training data for f, and f produces when prompted with p using greedy decoding.
For example, if a model's training dataset contains the sequence "My phone number is 555-6789", and given the length & = 4 prefix "My phone number is", the most likely output is "555-6789", then this sequence is extractable (with 4 words of context). We focus on greedy sampling in this paper. and verify in Section 4. that our choice of decoding strategy does not significantly impact our results.
While prior work proposed other definitions, we prefer ours in this paper as it is more actionable. Some memorization definitions, including dower-bounde on differential privacy (Dwork et al. 2006
Jagielski et al. 2020 Nasr al., 202T) or counterfactual memorization (Feldman and Zhang 2020
Zhang et al.) 2021), require training hundreds or thousands of models, which is impractical for large language models. Alternatively, computing exposure (Carlini et al.] 2019) requires thousands of generations per sequence, and is only designed for carefully crafted training examples.Finally k-eidetic memorization (Carlini et al. 2020)) is a useful definition for unprompted memorization, but less useful for tightly bounding memorization by prompting with training data (as we will do). Future work might explore how our three scaling observations apply to other definitions of memorization.
3.2 SELECTION OF EVALUATION DATA
Having chosen a definition, we next describe our evaluation procedure. Ideally, we would consider every sequence x [p || s] in the model's training dataset (where has been split into length-k prefix and a suffix 8). For each sequence, we would report if the model exactly reproduces & when prompted with p. following Definition 3.1 Unfortunately, performing this test on every sequence in the training data would be prohibitively expensive. For example, the largest 6 billion parameter GPT-Neo model has throughput of roughly one 100-token generation per second on V100 GPU. Extrapolating to the 800GB training dataset, this would require over 30 GPU-years of compute.
Instead, query on smaller subset of the training data, that still produces statistically confident estimates. In this paper we randomly choose subsets of roughly 50,000 sequences, allowing us to efficiently run inference in just few hours. The primary criteria when choosing subset of the training data is to obtain a representative sample that allows us to draw meaningful conclusions from the data. We consider two approaches to constructing a subset of the data.
Our first subset is a uniformly random sample of 50,000 sequences, drawn from the training dataset without repetition. While a uniform sample is useful to estimate the absolute amount of memorization in model, it is poorly suited for studying how memorization scales with data properties that are not uniformly represented in the training set. For example, prior work has identified that data duplication (i.e., how often the same sequence is repeated either exactly or approximately) is an important factor for memorization. Yet, because the frequency of training data duplication decays extremely quickly (Lee et al., 2021). uniformly random sample of 50,000 sequences (accounting for < 0.02% of the dataset) is unlikely to contain any signal that would allow us to accurately measure the tail of this repeated data distribution. A similar concern arises for measuring how memorization scales with prompt length, since very long sentences account for only a small fraction of the training set.
Therefore, second subset is random sample normalized by both sequence lengths and du- plication counts, which allows us to accurately measure memorization of large language mod- els in the worst-case, on highly duplicated data with long prompts. For each sequence length l E (50, 100, 150, 500). and integer n, we select 1, 1,000 sequences of length E that are contained in the training dataset between 2n/4 and 21/+ times. We do this until we reach an n for which 1,000 sequences are not available. This gives us 1,000 sequences that repeat between 6 and 8 times (~= 1/4 and x 22/44 and also 1,000 sequences that repeat between 724 and 861 times (*= 238/4 and x 239/4). This biased sampling allows to more accurately measure memorization of a sample's 's duplication factor and prompt length, without querying the entire dataset. Note that constructing this duplicate-normalll duplicate substrings in an S00GB training dataset is computationally challenging. We make use of the suffix array construction from Lee et al. (202T (see Appendix).
function
data subset requires some work, efficiently identifying
For each length from 50 to 500 tokens, we collect 50,000 examples duplicated varying numbers of times, totaling roughly 500,000 sequences. For each sequence of length f, we prompt the model with
3

---

Published as a conference paper at ICLR 2023
GPT-Nee Baseline
68
+
68
Fraction extractable
0.8
2.7B
2.7B
1.38
125M
125M
0.6
Baseline
Baseline
0.4
120M
345M
762M 1.58 2.78
68
10a
10a
50
100
200
500
Model Size
repetitions in training data
Prompt length
(b)
Figure 1: We prompt various sizes of GPT-Neo models (green) with data sampled from their training set set-The Pile, and normalized by sequence lengths and duplication counts. As a baseline (yellow), we also prompt the GPT-2 family of models with the same Pile-derived prompts, even though these models were trained on WebText, different training dataset. (a) I Larger models memorize larger fraction their training dataset, following log-linear relationship. This not just a result of better generalization, shown by the lack of growth for the GPT-2 baseline models. (b) Examples that repeated more often in the training set are more likely to be extractable, again following log-linear trend (baseline is GPT-2 XL). (c) As the number of tokens of context available increases, does ability to extract memorized text (baseine is GPT-2 XL).
the first L 50 tokens and report the sequence "extractable" if the model exactly emits the next 50 token suffix this sequence. Fifty tokens corresponds to an average 127 characters or 25 wordsin the GPT-Neo training set, well over the length of typical English sentence. Finally, compute the average probability that sequence extractable by averaging over all lengths L.
EXPERIMENTS
We primarily study the GPT-Neo model family (Black et al. 2021 Wang and Komatsuzaki 2021) trained on the Pile dataset (Gao et al., 2020). The GPT-Neo models are causal language models trained with the objective of predicting the next token in sequence given the previous ones. They come in four sizes: 125 million, 1.3 billion, 2.7 billion and 6 billion parameters The Pile a dataset of 825GB of text collected from various sources (e.g., books, Web scrapes, open source code). Prior to the recent release of OPT (Zhang et al., (022), the GPT-Neo models were the largest language models available for public download, and The Pile the largest public text dataset available.
4.1 BIGGER MODELS MEMORIZE MORE
We begin by considering the impact of model size memorization. expanding prior studies which qualitatively established a relationship between the size of GPT-2 models and their ability to memorize <30 URLs (Carlini al. 2020). In contrast, we study million model generations in order to describe how model scale relates to memorization.
Results. We first study biased random data sample normalized by duplication count and sequence lengths. The results of this experiment are given in Figure Tal The y-axis reports the fraction of generations which exactly reproduce the suffix for their prompt, averaged over all prompt and sequence lengths in our evaluation set. Because our biased sampling over-represents duplicated strings, the absolute degree of memorization in Figure Ta Lalis not particularly important rateer,ther, we are interested in how memorization varies with scale We find that larger models memorize significantly more than smaller models do, with a near-per pear-perfect log-linear fit (R² of 99.89 99.8%) a ten fold increase in model size corresponds to an increase in memorization of 19 percentage points.
1 of February 2022, there also a 20 billion parameter variant. Unfortunately this model uses a different training setup and tokenizer making it difficult to apply here. We repeat this experiment for uniformly random subset of the data in Figure 2a
4

---

Published as a conference paper at ICLR 2023
To confirm that larger models are indeed memorizing more data, and not simply generalizing better, we repeat the analysis with the GPT-2 model family as a baseline. The GPT-2 models are similarly sized, and also trained on anternet-scrapdd data. If our "larger models memorize more" result was due to the predictive strength of larger models, and not the memorization of specific training data, would expect similar relationship between comparably sized GPT-2 models trained on similar data. Put differently, this baseline allows to establish what fraction of the training data is sufficiently "easy" that any language model can correctly predict the 50-token suffix, even if the example has not been seen during training. For example, language model trained on multiple examples of number sequences can likely correctly complete some other unseen number sequences. We find that GPT-2 correctly completes approximately 6% of the examples in our evaluation set, compared to 40% for the similarly sized 1.3B parameter GPT-Neo model. A qualitative analysis (see examples in Appendix Figure 15) suggests that examples "memorized" by GPT-2 are largely uninteresting sequences (e.g., number sequences, repetitions of the same few tokens, or common phrases). Therefore, we conclude that larger models have higher fraction of extractable training data because they have actually memorized the data; it is not simply that the larger models are more accurate.
4.2 REPEATED STRINGS ARE MEMORIZED MORE
Prior work provides preliminary evidence that memorization in language models increases with the number of times sequences are repeated in the training set (Carlini et al. 2020 Lee al.. (021). We expand on this observation and quantitatively measure the effect of data duplication on memorization. Using our duplication-normaliae extractable, for buckets of sequences duplicated between 2 and 900 times. Each bucket consists of 1,000 distinct sentences, and we compute the average amount of memorization for each bucket.
data sample, we measure the fraction of sequences which are
Results. Figure lb shows our results, aggregated over all sequence lengths. We observe a clear log-linear trend in memorization. While models rarely regurgitate strings that are repeated only few times, this probability increases severely for highly duplicated strings. The small memorization values at low numbers of repetitions corroborates the positive impact of training dataset deduplication on memorization observed by Lee et al.] (202T). However, we find that memorization does still happen, deduplication will not perfectly prevent leakage. While this
even with just a few duplicates relationship is perhaps obvious, and has been corroborated for specific training examples in prior work (Carlini et al. 2019 2020), our results show that it holds across the entire training set.
4.3 LONGER CONTEXT DISCOVERS MORE MEMORIZATION
The previous two questions evaluated how data collection and model training decisions impact the leakage of model's 's training data when it is provided fixed number of tokens from sequence as context. As result, those experiments suggest particular actions that could be taken to mitigate memorization (by reducing model size, or limiting the number of duplicate examples).
However, even when the model is fixed, it is possible to vary the amount of extractable training data by controlling the length of the prefix passed the model. By studying how the number of tokens of context impacts extractability, we demonstrate the difficulty of discovering memorizationn-laaguaee models may only exhibit their memorization under favorable conditions.
Results. In Figure Ic we observe that the fraction of extractable sequences increases log linearly with the number tokens of context. For example, 33% of training sequences in our evaluation set are extractable from the 6B model at 50 tokens of context, compared to 65% with 450 tokens of context. We call this the discoverability phenomenon: some memorization only becomes apparent under certain conditions, such as when the model is prompted with a sufficiently long context.
The discoverability phenomenon may seem natural: conditioning a model on 100 tokens of context more specific than conditioning the model on 50 tokens of context, and it is natural that the model would estimate the probability of the training data as higher in this situation. However, the result is that some strings "hidden" in the model and require more knowledge than others to be extractable.
From one point of view, it is good that some memorization is difficult to discover. This makes harder for attackers to perform training data extraction attacks (Carlini et al. 2020). otherwise exploit memorization. Indeed, an exact 100 token prompt is required to make the model output a given string, then, in practice, adversary will likely be unable to perform the attack. The difficulty
5

---

Published as a conference paper at ICLR 2023
0.07
0.07
GPT-Neo Random GPT2
Random
Froction extractable
0.06
Fraction extractable
0.06
Frcctin extractable
0.8
0.05
1.38
0.04
0.04
125M
Baseline
0.03
0.4
0.02
0.01
0.0
120M
762M 1.58 2.78
100
200
500
1000
10"
204
102
Size
Sequence length
# repetitions in cata
(a)
(b)
(c)
Figure 2: (a) Fraction of sequences extracted as a function of model scale where we sample uniformly vary the length of the prompt. For each sequence length n, n-50 tokens are used as the prefix, and we check for extraction the remaining 50 tokens. (c-left) Using beam search with b=100 slightly increases the data extracted. (c-right) We observe considerably more memorization when checking whether the generated sequence occurs anywhere in the entire training (Section C). However, this approach is very computationally expensive so we do not use it for our other experiments.
from the training set. (b) Fraction of sequences extracted
in discovering memorization also reduces the likelihood of non-adversarial training data regurgitation. For example, the GitHuu Copilot model (Chen et al.; 202T) reportedly rarely emits memorized code in benign situations, and most memorization occurs only when the model has been prompted with long code excerpts that are very similar to the training data (Ziegler [021]) Practitioners building language generation APIs could (until stronger attacks are developed) significantly reduce extraction risk by restricting the maximum prompt length available to users.
Viewed differently, however, the difficulty of discovering memorization can also harm our ability audit privacy in machine learning models. Because provably-correet approaches for privacy- preserving training of machine learning models are applied only rarely in practice (Abadi et al. 2016 Thakkar et al.] [2020) Ramaswamy et al., 2020). auditing (Jayaraman and Evans | Jagielski al. 2020; Nasr et al. [021). lb. Our results suggest that correctly auditing large language models likely requires prompting the model with training data, there known techniques to identify the tail of memorized data without conditioning the model with a large context. Improving upon this limitation is an interesting problem for future work.
common to attempt post-hoe privacy
4.4 ALTERNATE EXPERIMENTAL SETTINGS
In this section, we briefly review other strategies that we could have used to quantify memorization.
Random dataset sampling. The majority of this paper uses subsets of the training data that were explicitly sampled according to training data duplication frequency. Now, we consider how our results would differ if chose truly random subset of the training data, where each sequence is sampled uniformly. instead of sampling duplicate-normaiin sample 100,000 000 sequences of varying lengths from The Pile dataset, then prompt the model and test for memorization as before (more details in Appendix C).
dataset. Specifically, we randomly
Figure 2a and Figure 2b present the results. We observe similar qualitative trends with model scale and context length as in Figure I Larger models memorize more training examples than smaller models and much more than the GPT-2 models that were not trained on The Pile. Similarly, providing more context to model increases the likelihood we discover memorization. We can extract the last 50 tokens of length-1000 sequence with 7% probability for the largest GPT-J 6B model, compared to 4% probability for the smallest 125M GPT-Nco model. (And both of these much larger than the 2% probability of extraction for the 1.5B parameter GPT2-XL model.) These results, taken together, allow to estimate lower bound that there is at least 1% of The Pile dataset that is extractable by the 6B GPT-J model, but not by GPT-2 XL.
Alternate decoding strategies. We have defined memorization as model's ability to generate the true continuation when choosing the most likely token at every step of decoding. Yet, this greedy decoding strategy does produce the overall most likely sequence. Many language model

---

Published as a conference paper at ICLR 2023
Prompt
Continuation (= 68)
27B
138
125M
Gallery "Toogh defensive violence will always nocessity in eyes of men principle, it would still more unfortanate if wronglsers should dominate just men. Augistine 't. new
conderined ridiciloss, and then dismissed trival, until
condemned ridicilous, and theni dismissed
tried the mind of a childl then
and forenicst man is to
before boing accepted the most important thing in the wortett George Bernard
finally becomes what everybody knows." - Villiam james "This is reassert
mind of man:" thomas
a man of princiale and to a man of
Paine "Th only
the task before our commitment nation lase higher than our owr.
Shaw The sltimate measure man
redessary for the triumphi
principle and be mal aiman ppinciple, and be
in moments comfort and converience, But where
not he
of nothing- Ldmund Burlee
good do
principle
_oPulorptoo unrejiter Largg crypt, registze templatefstruct crypto, template "umpli struct cryoto_ template int en -EDIST:
down, writew bcrypoo alg semix list_ coccecch_eeee enrtecchentttiaa Scrypto template, litt, (if) impl)
fs sectecttteee entrylj
e ods ie
struct coypto demsllee "Emp
Scrypto_plc_ plc list) (mmll
GHiDesseee Ji (llg))
crypto wttp:atammmpteiiieeeae new(tmel)
>name && impl-nnmee xelgoe, _emee
gotolout a eeae unpl-
jiferri returjerr tmpl-
>elizl -bese
-unot (cbmol--tmpp--
Figure 3: Text examples that are memorized by the 6B model, but not by smaller models. Green highlighted text matches the ground truth continuation, while red text indicates incorrect generation.
applications use other decoding strategies, such beam search to find the generation with highest likelihood. To understand how our choice of decoding strategy affects the amount of memorization measure, we compare greedy decoding with beam search in Figure Ce) We find that using beam search with 100 beams results in marginally more extracted memorization. The difference in extractable memorization is just under 2 percentage points on average, with a maximum of 5.6%. Interestingly, beam search and greedy decoding generated the same output 45% of the time.
The most common decoding strategy employed by modern LMs is random sampling, where the next token is selected at random according to a probability distribution derived from the model's predictions. McCoy et al. (202I) found that random sampling resulted in generated text with a greater number of novel n--ramss Since the goal of our study is to maximize discoverability-an antithetical goal to maximizing linguistic novelty- we do not present experiments that use random sampling.
we
Alternate definition of extractability. Our main experiments report sequence as "extractable" if the model's generation is identical to the true suffix of the considered training example. However is possible this suffix is still present (elsewhere) in the dataset. We now consider a loose lower bound on memorization that considers a sequence memorized if the generation [pl[[(p) from a prompt p is contained anywhere in the training dataset. Searching within the entire dataset finds more memorized content than comparing with the ground truth (Figure (c).. For examples at 100 repetitions, 32.6% of outputs are contained somewhere in the dataset but just 15.8% match the ground truth continuation.
4.5 QUALITATIVE EXAMPLES OF MEMORIZATION
we present qualitative examples that are only memorized by the largest (6B) model, but not the smaller ones. We highlight some interesting patterns in these sequences: while the generations from the smaller models do not match the training data, they generally Deematically-reevvaa and locally consistent. However, a closer inspection reveals that those generations are only syntactically sound, but semantically incorrect. Appendix Figure g shows further examples of sequences that are memorized by all the models. We found most of these universally-memoree "unconventional" texts such as code snippets or highly duplicated texts such as open source licenses. Figure 13 shows sequences which are memorized by the 6B parameter model despite being infrequent in the training set. These tend to be easily completed text- Figure 14 shows sequences which are repeated thousands of times but are surprisingly not memorized by the 6B parameter model. Many of these are mostly correctly completed, only differing on semantically unimportant characters.
In Figure 3
sequences to be
5 REPLICATION STUDY
The above analysis provides evidence that memorization scales log-linearly with model size, data duplicates, and context length. We now replicate this analysis for other language models trained with different datasets and training objectives, namely: (1) the T5 family of models trained on the C4 dataset (Raffel et al. 2020), (2) models from Lee et al. (2021), trained on a deduplicated version of C4, and (3) the OPT family of models (Zhang et al. 2022). also trained on the Pile. We expected our results to cleanly generalize across settings, and this is indeed true for model scale. Yet, the situation is more complicated when considering data duplication, due to training set idiosyncrasies.
7

---

Published as a conference paper at ICLR 2023
0.10
T5 Base
Original
Fraction extrastable
0.045
Fraction extractable
15
Fnction extractable
0.5
NearDup
0.08
T5
ExaciSubtr
0.035
0.06
0.4
0.025
0.04
0.02
0.015
0.00
0.0
220M
770M
2,88
10²
10"
10a
10a
10"
Model Size
# repetitions in training data
repetitiios in original C4
(a)
(b)
(c)
Figure 4: (a) Masked language model objective: Larger models have higher fraction of sequences
extractable on T5. (b) Masked language model objective: Relationship between number of repetitions and extractable tokens on T5. (c) Causal language model objective: Relationship between number of repetitions and memorization on language models trained with deduplicated data.
5.1 T5 MASKED LANGUAGE MODELING
Model and dataset. The T5 v1.1 models masked encoder-decoder models trained reproduce randomly deleted spans from an input sequence. The models vary in size from 77M to IIB billion parameters, and are trained C4-a 806 GB curated version of English web pages from the Common Crawl. The largest T5 model (11B parameters) is the largest publicly available masked language model. T5 models thus good candidates for studying how memorization scales with model size.
We must first define what is meant by "extractable data" for the masked language modeling task. T5 models trained by removing random 15% of tokens from each training sequence (i.i.d,, and the model must then "fill in the blanks" to restore the tokens that were dropped from the input. As result of this different training objective, Definition 3.1 is not directly applicable: the model does not operate on prefix and output a suffix. We instead call sequence memorized if the model perfectly solves the masked language modeling task on that sequence. For example, we call 200-token sequence memorized if the model can use the 170 (= 200 0.85)) tokens of context to perfectly predict the remaining 30 tokens (= 200 0.15). Because this coken-dropping procedure stochastic, it is possible that one set of dropped tokens might yield an output of "memorized" and another might not. For simplicity, we inspect only one set of masked tokens per sequence; because we are already averaging over 50,000 sequences this additional randomness does not harm the results of our analysis.
Results. In Figure 4a we reproduce the model scaling effect (from Figure Ta) for TS models. Larger models similarly have increased ability to perfectly solve the masked prediction task. Surprisingly, while a scaling trend does hold here as well, the absolute memorization in masked models is an order of magnitude lower than for comparably sized causal language models. For example, the 3B parameter T5-XL model memorizes 3.5% of sequences repeated 100 times, whereas the GPT-Neo 2.7B model memorizes 53.6% of sequences repeated 100 times (with 150 tokens of context).
Next, we turn to reproducing the analysis of how memorization scales with data duplication. The situation here becomes significantly less clear. As shown in Figure 4b) sequences duplicated more often tend to be easier to memorize, but there is no monotonic scaling relationship. Compared to the case of the GPT-Neo models trained on The Pile. the relation between data duplication counts and memorization for T5 models trained on C4 exhibits large variance. This variance is statistically significant: sequences repeated 159 196 times are memorized with probability less than 5.1% with 99.7% confidence (three standard deviations from the mean), however sequences repeated 138 to 158 times (that is, less often) are memorized with probability least 6.2% (also with 99.7% confidence). That is, for some reason, sequences that occur ~ 140 times more likely be memorized, despite occurring less often, even we assume a three-sigma error both measurements simultancously.
In order to explain this counter-intuitive phenomenon, we qualitatively study each of these two buckets of examples to understand this difference. We find that most the duplicate examples repeated 158 times consist mainly of whitespace tokens. These sequences thus much easier to predict correctly than other sequences, even if they are repeated more often. This effect, to a lesser extent, can be found in other buckets which contain many approximately duplicates.
8

---

Published as a conference paper at ICLR 2023
5.2 LANGUAGE MODELS TRAINED ON DEDUPLICATED DATA
Model and dataset. The models used in Lee et al. 2021) are 1.5B parameter causal language models. This model family consists of one model trained on C4 (the same dataset as T5), one model trained on a version of C4 that was deduplicated by removing all documents which were near-duplicates of other documents, and one model trained on a version of C4 that was deduplicated by deleting any string of length-50 tokens that occurred more than once. [ee et al. (2021) found that both types of deduplication reduced the likelihood of memorization.
Results. We were most interested in whether models trained on deduplicated data would still exhibit increased memorization of examples which were repeated frequently in the original, non-deduplicated C4 dataset (e.g., because the deduplication missed some near-duplicates). Figure 4c plots the fraction of sequences memorized by these three models. We draw two interesting conclusions from this data.
First, we confirm that models trained on deduplicated datasets memorize less data than models trained without deduplication. For example, for sequences repeated below 35 times, the exact deduplicated model memorizes average of 1.2% of sequences, compared to 3.6% without deduplication, statistically significant (p < 10- 15) decrease by a factor of 3 xx. Second, while deduplication does help for sequences repeated up to ~100 times, it does not help for sequences repeated more often! The extractability of examples repeated at least 408 times is statistically significantly higher than any other number of repeats before this. We hypothesize that this is due to the fact that any deduplication strategy is necessarily imperfect in order to efficiently scale to hundreds of gigabytes of training data. Thus, while may be possible to remove most instances of duplicate data, different and valid definitions of duplicates can mean deduplication is not exhaustive.
5.3 LANGUAGE MODELS TRAINED ON A MODIFIED VERSION OF THE PIL.E
Model and dataset. We finally study the OPT family of models Zhang et al. 2022). that vary from 125 million to 175 billion parameters These models were trained on a SOUGB dataset that overlaps with The Pile but is not identical and contains data from many new sources, while also removing some data from the Pile. This dataset was also deduplicated prior to training, and so we do not expect to see duplicate sequences memorized (much) more than sequences repeated only a few times.
Results. Overall, we find that while there are nearly identical scaling trends to those we found for GPT-Neo's model family, the effect size is orders-of- ofddr--o-mmaanitude smaller (figure 7 Even the 66 billion parameter model memorizes smaller fraction of The Pile than the smallest 125 million parameter GPT Neo model. This suggests two possible conclusions: (a) careful data curation and training can mitigate memorization, or (b) even slight shifts in data distribution can significantly alter what content gets memorized. Without direct access to the original training dataset, we can not distinguish between these two conclusions and hope future work will be able to resolve this question.
CONCLUSION
Our paper presents the first comprehensive quantitative analysis of memorization in large language models, by re- re-processing the training set to find memorized data. Our work has two broad conclusions.
For the study of generalization, we have shown that while current LMs do accurately model the statistics of their training data, this need not imply that they faithfully model the desired underlying data distribution. In particular, when the training data distribution skewed (e.g., by containing many duplicates of some sequences) larger models are likely to learn these unintended dataset peculiarities. is therefore important to carefully analyze the datasets used to train ever larger models, as future (larger) models are likely to remember even more training details than current (smaller) models.
For the study of privacy, our work indicates that current large language models memorize a significant fraction of their training datasets. Memorization scales log- lineer with model size-by by doubling the number of parameters in a model we can extract a significantly larger fraction of the dataset. Given that current state-of-the-a -ot---- models contain more than 200x as many parameters as the largest 6B parameter model we analyze, it is likely that these even larger models memorize many sequences
We were unable access the 175 billion parameter model; we OPT models up to 66 billion parameters.

---

Published as a conference paper at ICLR 2023
that are repeated just handful of times. At the same time, we have shown that this memorization is often hard to discover, and for an attack to actually extract this data it will be necessary to develop qualitatively new attack strategies. Fortunately, it appears that (for the comparatively small models we study) training data inserted just once is rarely memorized, and so deduplicating training datasets (Lee) et al. 202T) is likely practical technique mitigate the harms of memorization.
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private BERT, arXiv preprint
2021.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autore- gressive Language Modeling with Fess--Tensrflow, March 2021. URL http:: /wwi//i... org/ www.//22e1ece/ 5297715 If you use this software, please cite it using these metadata.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does mean for a language model to preserve privacy?, 2022.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemnj Kos, and Dawn Song. The secret sharer: Evalu- ating and testing unintended memorization in neural networks. In USENIX Security Symposium, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, al. Extracting training data from large language models. arXiv preprint arXiv:2012. 20X2:22012.07855, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint wrXiv:2107.03374 2021.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Advances in Neural Information Processing Systems, 2020.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-1333, 2015.
Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, pages 619-633, 2015 8.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00m 100.00077000 2020.
Peter Henderson, Koustuv Sinha, Nicolas Angelard Angtlerd-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, pages 123-129, 2018.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? arXiv preprint arXiv:2006.07709, 2020.
10

---

Published as a conference paper at ICLR 2023
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In 28th {USENIX} Security Symposium (f {USENIX) Security 19). pages 1895-1912, 2019.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv preprint wrXiv:2202.065333 2022.
Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. CoRR, abs/2107. abs/2107.06499, 2021. Jttes::///aax www.vvoorr/aaaaa.ooeaooeo abt
06499
R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? Evaluating linguistiic novelty in text generation us- ing RAVEN. CoRR, 11.02999 2021. URL htttp::///eaa http://axxvvv.rro wttp://gggaabaa.aag.o/// abaaaabs///1a 0tt.//1111.09999999 09509
Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlini. Adver- sary instantiation: Lower bounds for differentially private machine learning. arXiv preprint arXiv:2101.0453: 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
vv21/222222222/00 074.html
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing data, arXiv preprint arXiv:2009, 100331, 2020.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18. IEEF, 2017.
Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning, 2020.
Aleena Thomas, David Ifeoluwa Adelani, Ali Davody, Aditya Mogadala, and Dietrich Klakow, Investigating the impact of pre-trained word embeddings on memorization in neural networks. In International Conference on Text, Speech, and Dialogue, pages 273-281 Springer, 2020.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https: com/ wwkkkoinlolooooo wowz /eehh tesssht-ranse transformee jax May 2021.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection overfitting. In 2018 IEEE 3 Computer Security Foundations Symposium (CSF), pages 268-282. IEEE, 2018.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01 01068, 2022.
Albert Ziegler. GitHub Copilot: Parrot or crow? http://dd...sshhhooooa///iiiiipp recitation, 2021.
11

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS FOR DATASET CREATION
Intuitively speaking, it is straightforward to construct a dataset containing specifiable proportions of documents various frequencies. We need only enumerate all sequences repeated various numbers of times, and then sample uniformly at random from each of these subsets. However in practice this difficult to do, given the scale of these datasets: even asking the question "how many times is this sequence present in the training dataser" requires linear work for each query, and so repeating this thousands times for an 800GB dataset would be infeasible.
To do this efficiently, we build on the work Lef et al. (202I) and construct a suffix array over the training dataset. Such a data structure allows efficient queries to enumerate all sequences of length k that are repeated between N and M times for any N, ... This can be accomplished by linear scan the suffix array. As notation, write as the pointer into the dataset at a certain position j of the suffix array (i.e., A[j] 2), i' as the index at position j + N (so that Ali N i'), and z but appear between N and M times in the dataset. As result, we can scan linearly through the suffix array and enumerate all values of j j to efficiently find all potential sequences repeated between N and M times. From here, we then randomly sample 1,000 indices within these buckets to construct all of our sequences.
as the index at position j + M (so that Ali + M] = i" Then, if Dli i + ki Dii'=
D(i i A:] * D(E" 8 + k]. the sequence D[k: + k)] is guaranteed
B LONGER DOCUMENTS ARE NOT EASIER TO MEMORIZE THAN SHORTER DOCUMENTS
Fraction extractabn
0.6
Seq len
0.5
100
200
300
0.4
400
500
50
100
200
500
Prompt length
Figure 5: Longer sequences are not easier to extract. We compute the probability that an adversary extract sequence function of the number of tokens of context available, when varying the length the sequences. All sequences are repeated the same number of times, and evaluated with the same 6B parameter model. Each line represents the fraction extractable in sequences of increasing lengths. Because all lines nearly perfectly overlap, longer sequences are not fundamentally "easier" extract than shorter sequences.
Intuitively, one might think that longer sequences are more likely in the tail of the distribution, and the model is trained to a low perplexity, then the tail of the distribution may be more likely be memorized. This could lead our context length results to be exaggerated (as it would be difficult to untangle the tail effect of memorization from the context length effect). To check sequence length plays role in the amount of memorization we can extract with this method, we generated the next tokens after the prompt for various sequence lengths and various prompt lengths. Figure shows the fraction of extractable tokens in the next 50 tokens after the prompt. Each line on the figure represents a set of sequences with sequence lengths between 100 and 500 tokens. For each sequence length, we looked at prompt lengths from 50 tokens to (sequence length 50) tokens. We do not significant differences between the fraction of extractable tokens with varying prompt lengths across various sequence lengths.
12

---

Published as a conference paper at ICLR 2023
Prompt
Continuation (c: 68)
27B
138
125M
Gallery "Toouhh defensive vialence recorsity in the eyes men of principle,
condemned as ridiculous, and theni dismiesed trivil,, fnaily becomes what everybody knows:" William lames This is the real task before reassert our commitment
condemned ridicilous, and theni dismissed trivial, before boing
tried the mind of a childl then
and foremictt man
alwaya
is to
accepted the most important thing in the wortett George Bernard
mind of man:" thomas
a man of princiale and
would be still more unfortunate if wrongloers should dominate just ment- Aagustine "A first
Paine "The only
te a man of
Shaw"" The sltimate measure nan
redessary for the trumphh
prin dile and be mal principle and aiman ppincipte, and be
nation law higher than our owrs
not he comfort and converience, But where
in moments
of
good da
nothing- Ldmund Burlee
_GPulorypoo unrejiter Laggtteeellliiee crypta, registze, Jemplatefstruct crypte, template "umpli sstruct cryoto template int err -EESIST
down, wwiteecccppoooooooo semt pachh entrylal
ISt eacceecccee entryly
e kcal od' ig GHijeeseee (f
struct coypto lem "Emp
Lcrypto_ol__ise lis, list) (mmp
crypto wttp:atammmpmeiiiieeae newdtmel)
Scrypto templese lis. 160) lifig impi)
>name && impl--namenng >elgooa nemee
gotolout q malt unpl-
jif(err retumjerr tmpl-
>elizl -bese
-bnol qptnpl--tmpp--00
ions ( content "[fsebr; ..fe-
la dividebefore Lonneent: V1539 dizzcbefore contents (19562t fa-dnazbeffor
[: "JJ.a-dcccbefore content:
diechefore content:
=] fefediccssbeforr
ofscontibeoree content:
"[Jt;:) dochubbbeeooee content: "#as0; l.fa-occkerrbee beffrre
[(Bect; dollar-
content: */[s*":
"(Eser;) fo discoursebeefore
diebbeoore ( contenti
drupal:bofors contenel 1SBa9Sho fe-.rruplller didssdd
[ [content 1230
"I bot fo distittr
new users as an exploration tour and getting started guide, with evencises at the end each chapter. more advanced trainees it can a desitop reference, and a collection of the base lnowledge needed proceed with system and administration
Book contairs many real life esamples
bookis designed to give the reader
is good place to start for
aigertt way to pet Started
derived from the author's experience
frm istandang of the technologies needed to install and manage Linux sisters, using the sarous availabble tools and techniques for the tast. the book begins with
new user Atl would
with a new project At weuldlouggest
Linux
and network
recommend
administrator, trainer consullanti
book &qugetimu
use allgwing Create new project Create new udjr Createla flear user Create new Create
They hope these examples will help
orking by David 5 iu very good book for besliiness A //would recommeldd
you to get better understanding of
the Linux system and that feel
rapié-fre introduction to the basic. principtes Lingg operating
encouraged try things
Figure 6: Text examples that are memorized by the 6B model, but not by smaller models. Text highlighted in green matches the ground truth continuation, while text in red indicates incorrect (novel) generation.
C ALTERNATE EXPERIMENTAL SETTINGS
In this section, we study other strategies that we could have used to quantify memorization.
Random dataset sampling. In Section 4.4, we explored what would happen if we instead chose a truly random subset of the training data, where each sequence is sampled uniformly. Specifically, we randomly sample 100,000 sequences from The Pile dataset of length 100, 200, 500, and 1,000; prompt the model with the first N 50 tokens; and then test for memorization by verifying if the model can emit the remaining 50 tokens perfectly. In our analysis in Figures 22 and 2bb we vary the size of the trained model and the context length we provide it to understand how these factors impact memorization but this time through prompting the models with randomly sampled training sequences. As expected, the absolute probability of memorization is much lower than in Figure I where we prompted models with training data from the sampled applicationnnoraaiizee subset.
We observe similar trends with model scale and context length as in our other results. Larger models memorize more training examples than smaller models and much more than the baseline GPT-2 model that was not trained on The Pile. Similarly, providing more context to a model increases the likelihood we can discover memorization. In Figure 2b, we prompt models with: prompt length- = sequence length- - 50. We see that the longer prompts are casier to predict correctly than shorter prompts. The baseline GPT-2 model is nearly twice as accurate on sequences of length 1,000 (prompt length = 950) compared to sequences of length 100 (prompt length 50).
Alternate definition of extractability. Our main experiments report a sequence as "extractable" if the model's 's generated continuation is identical to the true suffix within that training example. This method is a loose lower bound on memorization. Consider two sequences 21, .2 both contained in the training dataset. Suppose these two sequences share the same prefix, and differ only in the final suffix; that is, T1 = [pl [ss] and 222 = [p] [p]] When we select x1 and prompt the model on the prefix p. will report "success" only if the output equals $1, but not if the output is s2, even though this is also a form of memorization.
We now consider how our results would change if we instead checked that the generation (p|| [p)] from a prompt p was contained anywhere in the training dataset. This gives a strictly larger mea- surement of memorization. By comparing these two methods (checking for memorization within the ground truth continuation, and within the entire dataset), we can understand how the choice of measurement affects the results in our experiments.
13

---

Published as a conference paper at ICLR 2023
Searching within the entire dataset finds more memorized content than comparing with the ground truth (Figure 2c) For examples at 100 repetitions 32.6% of outputs are contained somewhere in the dataset but just 15.8% match the ground truth continuation. This difference becomes pronounced as the number repetitions increases. The maximum difference between these approaches is 28.4%, 2,200 repetitions.
We refrain from using this approach for our main experiments, because this definition requires vastly larger computation resources; requires querying whether hundreds of thousands of sequences contained in an 800GB training dataset. Therefore, to promote reproducability, the remainder of this paper continues with testing the generated suffix against the single expected training suffix.
D TEXT MEMORIZED BY ONLY SOME MODELS
Table I: The number of sequences memorized by one model, and not memorized by another.
Not Memorized By
Model
Memorized
125M
1.3B
2.7B
6B
125M
4,812
328
295
293
1.3B
10,391
5,907
1,205
1,001
2.7B
12,148
7,631
2,962
1,426
6B
14,792
10,273
5,402
4,070
Table 1 shows the total number of sequences that are memorized by one model but not another. Larger models have uniquely memorized sequences, although every model has some memorization shared by any other model. (Even the 125M model memorizes few sequences that the 6B model does not.)
E MEMORIZATION OPT MODELS
0.25
0.25
66B
Fraction extractable
0.20
Fraction extractable
0.20
308
0.15
0.15
350M
125M
0.10
0.10
0.05
0.00
0.00
30B
10a
10a
10a
Model Size
# repetitions in training data
(a)
(b)
Figure 7: We prompt OPT models with data sampled from their training set. We use prompt length of 100 here. (a) Fraction of sequences extracted as function of model scale. (b) Fraction of sequences extracted as the number of repetitions of that sequence in the training set increases.
F EXAMPLES OF MEMORIZED TEXTS
show examples texts that are memorized by different models. We consider the case of 50-token prompts and 50-token generation. We sample texts with various number of repetitions in the training data. is impossible to inspect all the generated examples, so we random sample examples satisfying certain criterion and show few interesting memorized by models of all sizes, in the sense that the 50-token generations match the groundtruth continuations of the prompts.
in the paper. Figure g lists examples that
14

---

Published as a conference paper at ICLR 2023
Prompt
Continuation
125M)
this except in compliance with License. may obtain copy of the License at httez//
distributed under the License is distributed on an "AS is" BASIS, "WIHHUU WARRANTIES OR CONDITIONS ANY KIND, either express or implied see the License for the specific language
writing.
www.apache.org org/ /licenses/ 2.0 Unless required by applicable law agreed
This program is free software: you can redistribute and/or modify" under the terms of the 6NU General Public License version and only version 2 as published the Software Foundation.
This program is distributed the hope that it will usefull WITHOUT ANY WARRANTY, without even the implied warranty of MERCHANTABILITY or FITNESS FOR PURPOSE,
Privacy & Cookies Policy Privacy Ovenview This website uses cookies to improve your experience while you navigate through the website. Out these cookies, the cookies that are categorized as necessary are stored on your browser they essential for the working
of basic functionalities of the website. We also use third: party cookies that help analyze and understand how you this website. These cookies will stored browser only with your consent. You also have the option to opt
lend(document3s in front of $1 di 12pt) (minimal)
lusepackage wasyssyy lusepackagelamstonts
Len int for shift: uint shift - shift >- 64 return ErrintoverllowRaft
if return io. io.ErrunexpectedE0f b dAtal
</object> enil key="sourcelll"/> <int kny="maxlD*=18 118///n/-
[BDocument Classeess sobject "SMuttalrrrrr
</object> sobject caass "BClaasDeesrribrr key="
key- sege:-/eeeeencccartttllssleeerss key= "rrraddeediiiillooooorrrrss/o/o
Figure 8: Text examples that are memorized by all the models: given 50-token prompts on the left, the next 50 tokens generated by all the models match the groundtruth continuation.
Figure lists examples that are memorized by the 6B model but not by smaller ones. Specifically, the 50-token generations of the 6B model match the groundtruth continuations exactly, but the generations from the smaller models match neither the groundtruth continuations of the prompted examples any other training examples with the same prompts. We find that when smaller models do not get the groundtruth continuation right, they are generally still able to stick to similar topics. However, in many cases, the texts generated by the smaller models are only syntactically sound, but semantically incorrect. Figure T0 and Figure IT show more examples.
In Figure 12 show examples that are only memorized by the smallest model, using similar criterion as when we filter examples that are only memorized by the largest model. There are significantly fewer number of examples that are only memorized by the smallest model (35) than that of the largest model (2860). One of those examples (the first row of Figure 12 is particularly interesting: the groundtruth continuation contains a typo due formatting cutoff. While the smallest model memorized the typo, larger models try to fix the typo.
In Figure 13 and Figure 1
we show examples that are memorized but not heavily duplicated in the training set, and examples that are heavily duplicated but not memorized, respectively. Finally, we show examples that are memorized by GPT2-XL Figure
15

---

Published as a conference paper at ICLR 2023
Prongt
Continuation (+r 68)
2.7
1330
W2MM
Gallery "Tough defensize vollnnce will always he 2 sad
candenred ridicaous, and then dismissed eriial, until finally what everybodl knows. William james
sonderned as nd and then dismissed as trival, before boing accepted
tried in the mind of a @hiidl and ten in tried the
and foreniouta nan manSicury to his follow man. it is
necessity
important
Thomas
principle, would he unfortunate wrrrgdzres should domiinate just nan"" St. Augustine new idea first
thing the
George Berand Show" utimate missurre of is where he stands moments of comfort conseniacce Jist whtree
Paine tnigtting recessery for triumshi
ploddple be
before reessert
comm nation to hisher than our oami
crael for good mon to do nothing" Edmund Burlie
prine Sviit bel maij principle, and be aimant of pénciple,nns jad 18 be
Comment: smaller modelss is get quotes wrong (note George
Shaw and Thomas occurs commontyy list quotes).
before content:
3 befodd contentt
N dio contentt
3 dicebbeoore content
fa-dscurbbefoor
"/Seb''s 1 defooed
1VS39-1 da dizysthforra concent)
UPecct doldlla
content: (e
content: "(B322) J fi--
[VSe"- Lardiaabefore
conteni: ([a55) Fir
slordefore contenti
drepet I content)
discurse:befora content: 1(893
Boceerbefors I
11155 2: faadiggin
(ESS Lo drupal discait
Comment: the smaller
generate of correct "tormat" la table tont awesome icons), but with mix-ano matched contents that which is actually fn-ckokker. The 1:38 and 2.78 models both fa-digg. but one
incomeet. For example, 125M model says fa-drupalli as 188c, and the other as 1281. The actual unicode value for fe-digg in fort tone-awee is fta8.
exploration and gelling started gride exercises at end each chaptet moss advanced trainees it can be deskoa reference, and e collection of the base knosledge reeded to procedd with sjstem and network adminisrazion.
book contains many
book designed give understanding
good place
gaigeaat way Startädd
esamples derived
aither's
uses would
new projec)
experience Linux system
technologjes needed irstall and manage Linus ojotem, using the samus aotilable aod tee teclniques for the lukk, The book begins with rapid fire
recommend
rgquts supgedt
networks administator trainer and
book de with
to the fillowing Create
consuliant. They hope thesa) eeamples will help you to get
anking by Dovid S. but very good book for) beginners A: (soould recommend
new project Cwaat new sher Creatiela fjew usef Create new user Create
better understanding of the Linux)
gystem and that you feel
introductinn the bosk prine pies the Lingg oparating
encouraged
things
Commentt: the book is Intrudvvatio to Linuxe, by Machtell Garreess Note the prompt do not have the word "Lirux". Smaller models do not memorize the verbati passage, but generate texts related to Linux and book (except for the 125M modell.
Figure 9: Text examples that are memorized by the 6B model (according to true-continuation match), but not memorized by smaller models (the generated texts do not match the true continuation, nor any other training examples). The first column shows the prompt. The second column shows the prediction from the 6B model, which matches the groundtruth continuation exactly. The remaining columns shows predictions from smaller models.
Prompt
Continuation
disabted, ccm.com
age, roli soc, socual
5ge, roligion, sex soual
ags, religion, sot, socual retigion, ses, national origin
better for you if you esable
orientation lon, gender identity
orientation, gender identlly
enlentation, gender identity
gunder identiy, souss!
joodeript ipt saitch to
expression, national origin,
eopression, national origit,
eopression, notional origis,
ee erlertation, gender
lenetirigi medioripttan supported browset
protected veteren stalus, ofl
protected veteren status, of
protected veteren status, ary
identity exgression, protected
Bestor Consutingg Group an Opportunity Employer. All
any other chanstrristic
other charactristie
ether charasteriztle
veterar status, or any other
protected under federal, state
protected under federal
protected last BCG improve
characteristi protected by
qualitte appilcants
local where applicable.
where uses cooliesto imfrove your
applicable
Consuiting Group is full-servie, Equell Opportunity Emplooeer All
consideration empllgyment
and
crininal
suplicable
functionally performance,
without regard roce, color,
histories will considered iil
and etfectiveness our
nanner consistent with
experience. Well
communications. Detailed
qualitted applicants lill
Commant: smaller
get dietall right.
hin: "That should pickyou tears" "But
sine: "WeaR time you have your intervies"" 'Don't warry"
ling" "curtains" "Cartains"
geniuz? "You genius" "Voure genius."
joz good man"" "You
"Curtains"" "Curtains" "Cutaiiss" "Cutaine""
good man." good man.!
doctos cays. "I am
gat plenty
gerlus.
"ooure good man"" "Wouhe good man"" "oo're good men.!
Pagliecci "Good jolee"
time. "Youn finger's licking
"Curtains." "Cutains."
gerias! Yoo're genius." "ouure genius" "louto genius"
"Exerybody laugh." "Boll on snare crum. "Cart
bottery." "God, jon, stop!)
"urtains: "Cutt
"hourre good man."
"ihhat are you deingen
Comment:: only the largest modell generates masningful continuation.
experience The most important places to visit in Rome are: The Colossours, Trei Foustain, The Pontheon, Hazze di Spegna. Vatican City. This video offers tis help you plan the perfact
vacation. you want saell
trip Rome
trip Reme) The bost Way
Ramettig Colesseuri The
tine and money, the most important floe travil tis is
pub mullom
got Rome by train.
Coleszeum is Romar tountainth
siewect tob (ougon 201
The is lijcct way
at loratjd in lif Ensessum, the
compare before booking! hotal room a nlght. Youl
viewec pubdishedion
sel Pariee he train's the fastect way
Ruman Fotur. it Roman raüntain that located in Colesseum, the Roman forum.
2010 vieessi
Romee The thin the best way
this free atta natunterrzon,
pál iatied of 2017 views
tearches Uhrowati
Commentt: the largest model memorices the details, while smaller models deviate to talking about sverryy relevant stuff.
Figure 10: Continuation of Figure 9
16

---

Published as a conference paper at ICLR 2023
Prompt
Continuation 68)
277
1.38
125M
Gittropt unagister aigt crypto, jegiterr berpuats(ttuut orypto, template "trpll struct crypto template "q. intterr orr FFXST:
writelrropii for roch entryle, Scopli tesplatee list, I60 la sn trpll
crysto "Sinal Erypto, semplato pewtmmolk
Sccypte Iimpl-mmare && bege-mnare qnalg.cra ramre)
Gin) KERNEL Eint gonalout, 4:
hild himpl-eeleee eileeeee
ferr) reture ler; Lnaal-trppi qctmpi--tmpl-b
Commant:: smaller models do not generate axact coce, but the variables notor names in the prompt.
ssuch Google,
WOZ, najor
M02 are comples
RCCE examples of
search engine used
Yahool, use coadeees
shich closed in
2it?)
directoriess this
divectories. The search engises
fied pages algoriteei soorch results. The Google search engine is search engine that
pages for their algorithmic search results. Reges that are linked fiom other search not be submittted because theyy The tahoo! Directory and
respcctiolly y, both requised
tichnique. The Google
are ajsn used to find pages that
manuel subimisiion and hunani
Webmusler Tool tool is free
are not indleed the search
toriiiaa restems]29) Google
tool that alogg you to check
ongines, These pages are
uses Google's search engine te aed jages for aigurttiilc
ergine indreed
offers Google Searth Consolk)
the status your site and bo
refurred as Rqjinnarphnnes""
which Sitomap
submit jour Googlo.
pages,
results)
autoometically.
subritted
The Google Wefresster foods) tool
are abso used find pages Uhit
engine sees the Geogle
free ensure that
Comment:: nt: smaller models about relevant
not coherent 68 model (which memorizes the training data ver verbatimm)
purchese this Addiboonal Shipping tess/essticcions apply cannot ship any Next Day or Secord Day Express delivery methods. Shipping restrictions apply to this product under some
be
jarisficiioee Knows state local restrict cnticicios
juriseictiore Knows age, state
juriuiitions Please check with
jurisictiinsl This item cannol
local shipping
your local dealer retailer specific shipping restrictions this prnduct. This cannat ship Caneda, Ihis item
ship Secord Day Fagress delberr methods, Shipping rsstrictios apply to this product under some jurisd ctions. This itom cannot ship via any our Nest Bay or Sacond Day Express dajleeery
Nest
applled checklout maj
apalied checklour and may result changas an ander. PLEASE NOTE: Wascat guarantte that this item will ho de dialededd to any states we support. This llett cannot ship
changes onder.
PLEASE NOTE We do not sussill returns Arnmunition for safityy seasons, NPFFRR use any Arnmunition unless is specificllly designed freamm
cannot ship internation orally This han cannot be g tooid) shispes.
Comment: nt: smaller models s talk vaguety about shipping restrictions (125M model simply repeat the prompts)..
Figure 11: Continuation of Figure 9
Prompt
(= (=: Continustonn
2018
transport sustaicabli dites and regions'' will Leke place or 22 and November in Manchester ster United did
in tronsport arsainable diies and region' will take place
transport for sustairabla and regions will lake place un 22, and November Marchester
transport sustiratte
Conference Innovation in transport for sustainable
and reglors' wil. take place on
22 and 23 November in Manchester Unrtted Did Traffand
and 23 Nevember in Manchester
citiss and regions' will
lnited old Trafford Stadiari, Manchester,, United Kington., The
listed Die traffard Stadium, Manchecter, tinited Kisgro... Thai 2918 knmual Polls Confirrencel
take place 22 and November Manchesier
Trafford Stadiem, Manchessgg (
Stadium, Mandhester, Unbtd
Uniled
Kinglln, The canierence and planning policy makerss academics, practitioners
2018 Annual Pois Conference Innovation transport for sustainable cities and regional
United old Traffordd Manchester,
singdont The conference will
bning together
Innovatio
bring together 1,00
sustainable cities resione'
United kingto.. The 2015
delagates across
will
Armual Conference
world to dias the the lives
and stadents to
tImovation
dejitoopments transport and
The 68 modals
Comment:
memorized the typo Kingdo...'; the 1.38
the typo but (semantically) wrong word. 2.7B and
consetly.
Scale: De 1. Offiee 0. 0)
m. Testure
Tovtuure
Testural
FL Texturn (Mil 0 Scale De Offietsls
(neell 0030 Seide
(fie:: 2000000 (e5565 Batesdbaadbillllllleee Hyp
Iilel: www.oooooggiioooooooooo sawasatteddiiisieeeenoo
MainTes
Comment: the smallest model memorized the code, while the larger models try generate some GUIDs,
NOT LIITED TO, PROCUREMENT OF
USED AND OM AWY TIEORYY Of LIABLITY, WHETHER
USED AND ON ANY THEORY Of LIGAUUTY, WHETHER
USED AND ON WMY THIORY OF LABILITY ITY, WHETHER
USED AND ON AMY THEORY of
LABILITY WHETHER CONTRANE STRUCT LIABLIIY TORT (NNCLIOING NEGLIGENCE OTHERWISE)
SUBSIITUTE GOODS OR
CONTRACE STRICT LIBILITYY MEGLIGENCE
IN CONTRACE STRICT LSBBUIT TORT ONCLUGIG OTHERNISEE A885
CONTRACE STR ONCLLIOG
SERVICES, LOSS USF,
(IMCLUOING)
PRORTSS 15:
MEGLIGENCE
BUSIMESS (ERRUUPTIOO)) "NOWEVER CA
OTHERWISEE ARR
OTHERWISE) ARS
Commant: larger modals generate sight variation the licanse Our array search did exact match this fraining set.
Figure 12: Text examples that memorized by the 125M model (according to cuuecontinuation match), but not memorized by larger models (the generated texts do not match the true continuation, nor any other training examples). The first column shows the prompt. The last column shows the prediction from the 125M model, which matches the groundtruth continuation exactly.
17

---

Published as a conference paper at ICLR 2023
Frequency Prompt
Continuation (==
L_OONG LONG STL_TEMPLATE NULL struct
constructor; typedef type trivial copp constructor; typedef true [ype has triviall_ssignnnnnn assignment operator;
lype traitssflooat typedef true_type
has ariivil_deffult,,
"yroupbys_mapppe "grouphys_ "groupbyl_nosken" _nokew", "groupby5;
"groupby5_ map"; "groupby5 map skew,, "groupby5 noskew" "groupby6"
pair<KIz>{_12 (2), fusion:make_nne (aatraaKk___iir111_11))
namespace result of template typename KO, typename KI, typename K
fusion:make wttpa/eeeeaaaiaa_____a fasion:make_,pee p
(variant & (Lexer:Variant GLSL_115100000 400)) QLatin
(000) list + QUatinStrrin("
1Strinng' "dmat3";; (variant [Lexer:Variant, GLSe:: GLSL (ISL)400) list Strinn" "drann""mmma""""" (variant &.
disabled. BCG.com will work better for you you enable JavaScript switch JavaScript supported browser. Boston Consulting Group is Équal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color,
age, religion, sex, sexual orientation, gender identity expression, national origin, protected veteran status, or any other characteristic protected under federal, state local where applicable, and those with criminal histories will he considered in a manner consistent with
Text examples that are memorized but are not heavily duplicated in the training set. Many of these have a simple sequential structure (the middle three), may be boilerplate code (the first), or starts out with unique text, and completes with frequently repeated text (the last example). Overall, these are easily completed sequences.
Figure 13:
Frequency Prompt
Continuation
5526 contributors, All rights reserved. /l If Licensed under the Apache License, Version 2.0 (the "I iccnse') If you may not use this file except compliance with License. You may obtain a copy of the
i/ apache.cccoo
License If i/ htto:///ww..paat. www.h..oooe licenses)
Icenses/LLCCEN LICENSE-2 Unless required by appiicable agreed writing. software distributed on an "AS
LKCNSE-20 H Unless required by applicable lam
agreed to writing, softuam distributed under
License
License
5652
ators are Standing By Our Dumpster Specialists wailing the phone help you started, However, information we need before we can give price quate delivory time, Some things need to know
What size dumpster you will need What you on putting your dumpster When you need to drop your roll container off How lang you plan on keeping your roll- City
include What size dumpster do you need? What waste do you How much you
important
delfvery time?
What ts dakuupiimm
4323
copy of the License a htip:!! apac-p ore/ lecensss//ucccecc /UCENSE-22
or agreed to in writing, sottware distributed under the License distributed on 'AS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY express
agreed to writing software distributed under
the License is distributed an "AS is*
required by applicable
BASIS, WITHOUT WABRANTIES CONDITIONS ANY KIND, either espress or implied.
date default timerone satÚ function, you used any of those methods and you still gotting this warning, you most likely nisspelled timezone ddettiher, selected the timezone UTC for now, but please
select your timezone. homm//rlyprooppubli himl/naturaaldaaaan persobanner center, on line 17
date.
himezone to nptel/public, htm/ wnc-inncuddstuueee /tnudde/tunntionsp.ol on line 3810 waining: date!) function datel is safe
timezone. in /home]
Deprecated: Function eragi_ replace
3920
(AnbsppAnbsp: 1 12 28 t3 FEI Itttons! com bwakker/quides (wik)) anbsay&
nbsspp&nbssp:anssp: [AA 32 (5)
lhttps: //ghts///hhhhhu lofus breaker webback wwwp:paaaaiiiiiientttte interpmete) 8ibbpp (P
ibsp: [R]
fulips//ggithh comiggge brw.em/gggweebbbeeeeeeeoo wnw.eeeno---nniiiii &nbsp:&nn
3675
"date": "1755u20 "1ate"""175u200 11851,, "displayOrder":
Tumerr, 'gendar': "Male', 'Id": 558, 'mda": "Tumer, Jaseph Mallord William", Jeseph Mallord william!,
"gender" "Male" 604 "'mdaa "Turner
"c": "Josphhhhh Mallord William
Figure 14: Text examples that are heavily replicated in the training set but not memorized. We find many examples which have slight differences with semantic (English) meaning. This includes comment characters in code, non- English characters, template values, error messages, and meaningless symbols. We also surprisingly find large number of slightly different but heavily repeated documents about dumpsters.
18

---

Published as a conference paper at ICLR 2023
Prompt
Continuation (= GPT2-XL)
165.00 $175.00 $180.00 $185.00 $190.00 $195.00 $200.00
00 $270.00 $280.00 $290.00 $300.00 $310.00 $320.00 $330.00 $340.00 $350.00 $360.00 $370.00 $380.00
$210.00 $220.00 $230.00 $240.00 $250.00 $260.
ISRC3DEC2,
BBAE:/22MMX_NNUUUL11111111 SOURR,), ARIZONA_MUX SUUUUSSSSSSSCCEECC,, ARIZONA_
ISRC3DEC3MIX_INPO SNPUT_1_SURRCES SOUURC)), ARIZONA_ MM___MUUEEEEEEEEE__ ENUMS(SSCCDDEC,, ARIZONA ISRC3DEC4MIX INPUT
the councils of government, we must guard against the acquisition of unwarranted influence, whether sought unsought, by the militaryindustrial complex. The potential the disastrous of misplaced power exists and persist. We must never let the weight
of this combination endanger our liberties or democratic processes. We should take nothing for granted. Only an alert and knowledgeable citizenry compel the proper meshing the huge industrial and military machinery defense with our peaceful methods and goals, so that security and
full spreadsheet NEW! Buy our book Facts are Sacred: the power of data (on Kindle) More open data Data journalism and data visualisations from the Guardian World government data Search
world's 'ssss government data with our gateway Development and aid data Search the world's global development data with our gateway Can you do something with this data? Flickr Please post your visualisations
mash ups on
Original press release Get ahead of the crowd by signing for 420 Investor, the largest most comprehensive premium subscription service for cannabis traders and investors since 2013. Published by NCV Newswire The NCV Newswire
by New Cannabis Ventures aims to curate high quality content and information about leading cannabis compan companies to help our readers filter out the noise and stay on top the most important cannabis business news. The NCV Newswire hand-curated
of sole responsibility of the author(s). The Centre for Research on Globalization will not be responsible for any inaccurate incorrect statement in this article. The Centre of Research on Globalization grants permission cross- post Global Research articles on community internet sites
as long the source and copyright are acknowledged together with a hyperlink to the original Global Research article. For publication of Global Research articles in print or other forms including commercial internet sites, contact: [email protected) www.globalresearch.ca
Figure Text examples that are from The Pile and memorized by GPT2-XL. The first two examples have natural sequential structure, while the others appear to represent overlap in GPT2-XL's training set and The Pile.
19


Reviews:
[{"review_id": "yDrjElUUT-", "reviewer": "Reviewer_vuN8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple method to measure how well LLMs memorize training data. The authors found that memorization increases when 1) the model size is increased 2) the sample size is increased 3) the context of the example is increased. \nConsidering the level of memorization of T5 and comparing it with causal language models of similar size is the most interesting part of this paper for me. ", "summary_of_the_review": "This is an important topic, and the paper is interesting. Although the idea is not that novel, I think this is a good addition to the conference. ", "strength_and_weaknesses": "Strength:It's an interesting and intuitive idea in the paperI found the paper to be easy to read and understand. Very interesting results and analysis.\n\nWeaknesses:\n- Limited novelty", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XHJ74Ybfi_", "reviewer": "Reviewer_G66g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies memorization in large language models and tries to quantify memorization rates depending on prefix length, duplication rate in training data and model size. The authors define an example memorised (given prefix of some length) if greedy decoding conditioned on this prefix reproduces the training example verbatim. This allows to estimate memorization rate in a more thorough manner differently from e.g. extraction attacks done in previous work. Experiments are conducted for different datasets (Pile, C4), models (GPT, T5, etc) and settings (e.g., with or without deduplication). The authors find that memorization rate is log-linear with respect to model size, prefix length and duplication rate in training.", "summary_of_the_review": "The paper measures memorization for NLMs by looking at how often output of greedy decoding given prefix reproduces training examples verbatim. The experiments show log-linear memorization rate with respect to model size, prefix length and duplication rate in training. The results are not surprising, but given solid methodology and thorough experiments I believe overall the paper is of good quality and can give a solid ground for future work on memorisation.", "strength_and_weaknesses": "Strengths:\n\n1) The problem is important\n\n2) The methodology is solid, the experiments are thorough\n\n3) The definition of memorization is reasonable and makes a lot of sense when dealing with language models: the paper considers generated examples and not e.g. token-level accuracy or loss\n\nWeaknesses:\n\nThe results are expected and some of the trends in one way or another were reported in previous work. \nHowever, I believe that overall the paper gives a solid ground for future work on memorization.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LjntxbGxna_", "reviewer": "Reviewer_CQAU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extensively studies the memorization of training data in neural language models. First, it defines memorization as \"given a prefix, the model can generation a continuation that exists in the training data\". Then, it focuses on three aspects that affect a model's memorization rate: duplication times, length of prefix and model scales. It draws three main conclusions from a series experiments: (1) bigger models memorize more; (2) data that has more duplicates are memorized more; (3) longer prefixes can more easily extract memorized text. Furthermore, authors extend their experiments to other types of language models and models that are trained on deduplicated training data.", "summary_of_the_review": "The empirical results presents in this paper provide insights to pre-training language models: model can memorize even when using deduplicated training data. But it lacks guidance or insights on how to reduce memorization and the explicit complications brought by memorization. Thus I feel that this paper has marginal contributions.", "strength_and_weaknesses": "**Strength**\n1. Paper is very well written and easy to understand.\n2. The experiments conducted in this paper are quite extensive and results are convincing.\n3. I appreciate that authors perform experiments on recently released models, e.g. OPT, although its training data is not fully transparent to the public.\n\n**Weakness**\n1. Duplication is a very important factor in the paper when measuring memorization, but I didn't find how a sequence of length l is obtained /extracted from the training data. Did you consider all the substrings of length l in the training data? Did the segmentation you used to obtain substrings match that of those deduplication methods used for pre-training (section 5.2)? Are there overlap between sequences of length 100 and sequences of length 200?\n2. As discussed in the paper, the biased sampling method normalized by length and duplication times is more biased towards duplicated sequences, although there are experiments on uniformly sampled subsets, I think doing experiments on a subsets without any duplications is necessary. Again, duplication is dependent on how the substring is defined, which is worth discussing.\n3. The definition of memorization is pretty strict in this paper, but in reality a paraphrased continuation which might not exactly exist in training corpus can still leak data. I think studying the generation in a relaxed setting is more interesting, e.g. consider using paraphrased prefixes or perturb the prefixes, as considering the paraphrased continuation can be hard to quantify the extractability.\n4. Although authors provide many qualitative memorization examples, it's still hard for me to have a clear idea what types of sequences are more easily extracted, I think a dedicated section", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0bxJHIbQybg", "reviewer": "Reviewer_6k5t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the reproduction of training examples from pre-trained language models.\nThe amount of memorization is measured across 1) different model sizes, 2) the number of times an example is duplicated in the training data, 3) the length of prompt given as input.\nIt is concluded that larger models, more duplication and longer prompt all increase the likelihood of reproducing the training example.\n", "summary_of_the_review": "The investigated area is important and the experiments are more thorough than in previous work.\nHowever, the novelty of the experiments and the findings is limited and it is not clear that all the experiments show what they are claiming to show.", "strength_and_weaknesses": "Strengths:\n\n* Investigation of the foundational language models is an important topic.\n* The experiments are systematic and more thorough than previous work.\n\nWeaknesses: \n\n* The same properties of pre-trained language models have all been investigated and demonstrated by previous work. There don't seem to be any novel hypotheses or findings.\n* Some of the experiments have questionable setups (more details below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%