Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
Authors: USVSN Sai Prashanth
Alvin Deng
Kyle O'Brien
Jyothir S V
Mohammad Aflah Khan
Jaydeep Borkar
Christopher A. Choquette-Choo
Jacob Ray Fuehne
Stella Biderman
Tracy Ke
Katherine Lee
Naomi Saphra
Venue: ICLR 2025 Poster (2025)
Keywords: memorization
ontologies
language modelling
Primary area: foundation or frontier models, including LLMs
TL;DR: We break memorization down into a taxonomy: recitation of highly duplicated sequences, reconstruction of inherently predictable sequences, and recollection of sequences that are neither.

Abstract: Memorization in language models is typically treated as a homogenous phenomenon, neglecting the specifics of the memorized data. We instead model memorization as the effect of a set of complex factors that describe each sample and relate it to the model and corpus. To build intuition around these factors, we break memorization down into a taxonomy: recitation of highly duplicated sequences, reconstruction of inherently predictable sequences, and recollection of sequences that are neither. We demonstrate the usefulness of our taxonomy by using it to construct a predictive model for memorization. By analyzing dependencies and inspecting the weights of the predictive model, we find that different factors have different influences on the likelihood of memorization depending on the taxonomic category.

Published as a conference paper at ICLR 2025
RECITE, RECONSTRUCT, RECOLLECT:
MEMORIZATION IN LMS AS A MULTIFACETED PHE- NOMENON
USVSN Sai Prashanth
Alvin Deng" 1.4
Kyle O'Brien *.12 Jyothir V*.1,3
Mohammad Aflah Khan Jaydeep Borkar
Christopher A. Choquette-Choo"
Jacob Ray F Stella Biderman' Tracy Ket.00
Katherine Lee Naomi Saphra 11011 110,11
EleutherAA Microsoft New York University *DatologyAI Northeesternn University
MMPI-SWS "IIIT Delhi *Google DeepMind Uniivrrity of Illinois Urbana- Chaampaign
Harvard University 11 Kempner Institute
Correspondence: katherinelee @ google.com com and nsaphra fas harvard.edu
ABSTRACT
Memorization in language models is typically treated as homogenous phe- nomenon, neglecting the specifics of the memorized data. We instead model memorization the effect of a set of complex factors that describe each sample and relate it to the model and corpus. To build intuition around these factors, we break memorization down into taxonomy: recitation of highly duplicated sequences, reconstruction of inherently predictable sequences, and recollection of sequences that neither. We demonstrate the usefulness of our taxonomy by ing it to construct predictive model for memorization. By analyzing dependencies and inspecting the weights of the predictive model, we find that different factors influence the likelihood of memorization differently depending on the taxonomic category.
INTRODUCTION
The existing literature on Language Model (LM) memorization' -the tendency to generate exact copies of training samples at test time-varies widely in stated motivation. Papers might focus on copyright (Shi et al., 2023; Karamolegkou et al., 2023; Meeus et al., 2024), privacy (Carlini et al., 2018; 2022b; Brown et al., 2022; Mireshghallah et al., 2022), or scientifically understanding how interpolation (Mallinar al., 2022) leads generalization (Feldman, 2021; Tirumala et al., 2022; Henighan et al., 2023a). Although these objectives share commonalities, they also drive distinct and sometimes contradictory notions of memorization. To disentangle these motivations and articulate the factors that determine or signal memorization, we propose a taxonomy inspired by colloquial distinctions of memorization behavior humans.
Our taxonomy, illustrated in Fig. 1, defines three types of LM memorization based on colloquial descriptions of human memorization. Humans recite direct quotes that they commit memory through repeated exposure, so LMs recite highly duplicated sequences. Humans reconstruct passage by remembering a general pattern and filling in the gaps, so LMs reconstruct inherently predictable boilerplate templates. Humans sporadically recollect an episodic memory or fragment after a single exposure, so LMs recollect other sequences seen rarely during training.
We use our taxonomy in a variety of experiments that highlight the multifaceted nature of memoriza- tion. summary:
We introduce an intuitive taxonomy and heuristics for categorizing memorized data.
By comparing memorized and unmemorized distributions, we assess how variety of corpus- wide statistics, datum-level metrics, and representational differences influence the
Equal contribution
1 Equal contribution
As defined by www. www.aw.org/ghaaeeee www.w/ssaggggg.......... html.

---

Published as a conference paper at ICLR 2025
sequence 32-extractable?
Is the continuation duplicated times?
Is the sequence continuation solely boilerplate templute: cither incrementing numerals or substring repetition?
x
Not Memorized
Recitation
Reconstruction
Recollection
Figure 1: Our intuitive memorization taxonomy has three categories determined by simple heuristics.
likelihood given sequence being memorized. Our dependency tests confirm existing findings that low perplexity is strongly associated with memorization- though not equally for all memorized examples. This fact guides our heuristic for partitioning memorized data into recitation category.
We study scaling factors in memorization by monitoring each taxonomic category over the course of training and across model sizes. The number of memorized sequences increases with training time and model size, regardless of taxonomic category. Recollection, however, sees the fastest increase-ant and this outsize growth cannot be attributed solely to repeated exposures to sequences or random memorization.
To demonstrate the value of our taxonomy, train logistic regressions to predict the likelihood of memorization for candidate sequences from each memorization category. This predictive model outperforms both simple baseline with taxonomy and model that uses a taxonomy optimized by searching for the best set of mediating factors. These experiments show that the intuitions behind our taxonomy
improve more generic approaches.
We highlight differences between categories by exploring statistical dependencies, finding recitation is enabled by low-perplexity prompts and recollection is constrained by the presence of rare tokens.
EXPERIMENTS
In this section, detail the definitions and data analyze varying factors in memorization.
Defining Memorization There are multiple competing definitions for memorization (Zhang al., 2021; Ippolito et al., 2022). Because our experiments employ memorization data released by Biderman et al. (2023a), we use their preferred definition of k-extractable memorization (Carlini al., 2022a) with 32. A sample k-extractable if the LM, when prompted with the first tokens, generates the following l tokens verbatim.
Language Models We study memorization across model scale and training timing using the dedu- plicated Pythia models (Biderman al., 2023b), which range in size from 70M to 12B parameters trained on deduped version of The Pile (Gao et al., 2020). Data order is fixed across runs, enabling causal claims about the effect of model scale on memorization.
Datasets Our memorized sample is public list of sequences memorized by Pythia, released by Biderman al. (2023a). Unlike other works that estimate whether a generation from model's training set using predictive techniques (Carlini et al., 2020; Shi al., 2023; Yang et al., 2024), this dataset contains all 32-extractable samples from the Pile, verified by referencing the training data (Gao et al., 2020). We also collect representative sample by taking random 3% subset of The Pile, retaining the first 64 tokens of each sequence. Some analysis also considers an unmemorized distribution estimated by subtracting the memorized data distribution from the entire Pile, inferred from the representative sample.
Excluding the 160M parameter model, as its memorization dataset exhibits outlier behavior that could be either buggy data artifact real phenomenon, hut is regardless outside of the scope of our work.
2

---

Published as a conference paper at ICLR 2025
Poroaatgge
801
4.%
30%
80%
815
12%
856
40%5
4%
10a
36"
Dapáoates
senantic Matchos
Testral
Max Freq.
Moan Freq.
Freq.
Modian
Pareenlago
12%
30%
8%
8%
1.%
19%
2%
356
109 3
Token Fiuu.
P75 Tokoon Phoq.
Huffman Langth
Prompt
Continmation PPL
Siquestc
Not Memorized
Memorized
Figure 2: Histogram of various properties of interest (described in Section 3) for memorized and unmemorized (estimated by assuming the representative dataset's statistics hold for the Pile) samples.
POTENTIAL FACTORS IN MEMORIZATION
We consider a number of possible factors in whether given sequence is memorized. These factors are based on corpus statistics, datum statistics intrinsic to that sample, or model perplexity. Features may be computed over the first 32 tokens (the prompt); the last 32 tokens (the continuation); and the full sequence of 64 tokens subsampled from the training data. Implementation details are provided in Appendix A.
Many of these properties have different distributions for memorized and unmemorized data. Fig. illustrates these differences, highlighting that for some properties, the memorized distribution is more concentrated. Other properties particular perplexity and number of duplicates in the training havp memorized and unmemorized distributions with visibly different medians. Where the distributions differ, the property in question is likely influence memorization, an assumption which we employ predictively in Section 6.
3.1 CORPUS STATISTICS
Some factors relate given sequence to the entire training corpus. Overall, the following features illustrate how memorization is influenced by various types of duplication.
Duplicates For each 32-token window in any 2049-token sequence seen during training, we count the number of duplicates in the Pile.
Semantic Matches assess the prevalence of semantically similar samples training, we generate document embeddings for each full sequence using SBERT and count the number of sequences with cosine similarity 2 0.8, out of all 64-token sequences in The Pile. These sequences are semantically similar but may not be exact token-level duplicates.
Textual Matches We filter the set of semantic matches for a given target sequence to identify those with a low Levenshtein edit distance in their prompts (Levenshtein et al., 1966) from the target sequence. These matches flag slight variations on boilerplate prompts. We compute edit distance at the character level, thereby accounting for different tokenizations of identical sequences.
Token frequency We also compute summary statistics about the corpus-wide frequency of individual tokens in the sequence: mean, median, maximum, minimum, and 25th 75th percentile counts.
3.2 SEQUENCE PROPERTIES
Because some sequences are inherently easier to encode, we also consider factors determined by intrinsic metrics on the sample itself.
Templating A sample is classified as templating if it follows predictable pattern. We do not comprchensively consider all possible templates, but focus on two common patterns defined by handcrafted heuristics:

---

Published as a conference paper at ICLR 2025
Repeating: Consisting only of short repeating sequence of tokens, e.g.. "Go Go Go Zhang et al. (2021) previously discussed repetitive templates as a common feature of apparently memorized data which was not classified counterfactually memorized.
Consisting of incrementing numerical sequences. For example, consider the sequence "23: Oxfl, 24: Oxf2, 25: Oxf3", a set of interspersed numerical sequences with repeating separators.
Incrementing:
Compressibility We use Huffman Coding (Huffman, 1952) length to measure how easily sequence is compressed. Compressibility generalizes repeating templates to cases where minor variations on repeating patterns must be memorized. The connection between learning, memorization, and compression is drawn from the existing literature: Carlini et al. (2020) attempts to filter out sequences that are "easy" to produce by comparing zlib compression with perplexity identify memorized training data.
3.3 PERPLEXITY
We compute average perplexity across tokens the prompt, continuation, and full sequence. The importance of perplexity is one of the most reproduced results memorization research (Zhang et al., 2021; Carlini et al., 2018) and we confirm that low perplexity sequences far more likely to be memorized than high perplexity sequences (Fig. 2). Perplexity is the only factor we consider that relates to model behavior, rather than being intrinsic to the data.
MEMORIZATION TAXONOMY
To analyze the fundamental causes of k- extracted memorization, subdivide memo- rized samples into three types. The following rules categorize sample
Dourgsnoe
0.2
candidate for recitation, reconstruction, or recollection; candi- dates memorized by the model are therefore re- spectively recited, reconstructed, or recollected.
0.1
0.0
10a
10a
10a
10a
Duplicatos
Figure 3: KL divergence between generation per- plexity of memorized and non-memorized exam- ples for Pythia 12B with bootstrapped confidence intervals. Non-memorized samples are treated as the reference distribution. Divergence is highest for sequences with 6 duplicates, while highly du- plicated sequences have near-identical memorized and unmemorized distributions.
4.1 RECITATION
The existing memorization literature agrees that the duplication sequence across the training corpus is strongly correlated with its memoriza- tion (Lee et al., 2021; Kandpal et al., 2022). For example, LMs produce verbatim copies of bible quotes or software licenses that are commonly duplicated.
We consider a sample to a recitation candidate
it is highly duplicated in the training corpus. Model perplexity is good predictor of memorization on rare sequences because the perplexity distributions more different memorized and unmemorized data with few duplicates (Fig. 3). For highly duplicated sequences, however, perplexity is no longer a good predictor of whether the sequence is memorized not. We therefore define recitation candidate as a sequence with least 6 duplicates because the three-way relationship between perplexity, memorization, and duplicate count differs hefore and after that maximum divergence point. We test this threshold against others in Appendix G and find that it matches or beats examples of other small thresholds.
LMs memorize wide variety of highly duplicated texts, as shown in the example of Appendix F. Recited natural language text largely comprises webpage boilerplate text, liturgy, and software licenses or other legalese. Table 3, which includes random samples of natural language recitation, includes all of these common cases. Recited code text, development (HTML, CSS, JavaScript, etc.) boilerplate that describes common elements or derives from popular webpage templates.
in Appendix 4, is largely web
4

---

Published as a conference paper at ICLR 2025
4.2 RECONSTRUCTION
Are all perfectly reproduced sequences truly "memorized"? We consider cases that may be spuriously classified by definitions like k-extraction. Rather than encoding the entire sequence, the model learns templates and then reconstructs the sample based on these more broadly applicable patterns. A sequence can thus be perfectly reproduced even if it never appeared during training.
We consider a few templates- sequence patterns with a single logical continuation- define reconstruction candidates. These templates are not intended to be comprehensive, any stereotyped pattern may permit reconstruction. Our reconstruction candidates are sequences classified as incrementing or repeating by the heuristics deseribed in Section 3.2. As seen in Appendix E, code is more likely to be reconstructed than natural language text. When natural language text is reconstructed, as seen Appendix often takes the form of a chapter index and it is more likely than code to contain cases of phrase repetition rather than arithmetic sequences.
4.3 RECOLLECTION
After excluding highly duplicated recitations and template-based reconstructions, what remains memorized? Despite only seeing sample small number of time, the model might still be able to recollect a given sample, although the factors that lead to instant memorization are poorly understood. We consider sample be a recollection candidate if reconstruction.
candidate for neither recitation nor
Recollected code, seen in Table 4, largely made up of templating patterns that are not strictly the combination of incrementing and repetition that we use to define templates. The examples of natural language recollection in Table 3 might likewise at first appear to be misclassified recitation cases. Natural language recollection frequently comprises legal or liturgical texts, which would be expected to appear frequently throughout the corpus.
One might conjecture that these sequences are cases of retokenization, i.e., the particular token sequence is rare but the same string is heavily duplicated in the corpus under different tokenizations. However, the dependency tests in Appendix B contradict this hypothesis: the correlation between textual match count and memorization consistently neutral or negative for recollection candidates. In other words, tokenization of a common string. We instead conjecture that the model appears to memorize slight differences in translation (liturgical text) indexing (legal) between each variation sequence.
token sequence is less likely be memorized, not more, if it is different
(b)
100%
1.0%
10c
10a
Memories 10:
907
90%
101
80%
80%
109
700
100
20%
60%
100%
80% 100%
Parameters
Parameters
Training Time
Training Time
Recitation
Recoastruction
Recollection
Figure 4: The quantity of memorized data categorized by taxonomy across parameter size and training time. For fully trained models of varying parameter sizes, we give (a) total counts and (b) proportion of memorized samples by category. For the 12B parameter model, we consider intermediate checkpoints during training, also providing for each checkpoint the (c) total memorized counts and (d) proportion of memorized samples by category. Note that the proportional plots are truncated at 80%, as recitation is consistently majority of the overall memorized data.
5 DISTRIBUTION ACROSS SCALE AND TIME
Larger models memorize more data (Biderman et al., 2023a; Carlini et al., 2023; Tirumala et al., 2022), likely because they have more parameters with which recreate those sequences. Recent
5

---

Published as a conference paper at ICLR 2025
work deduplication (Sorscher et al., 2022) has argued that larger models more distorted by duplication, potentially because heavily duplicated sequences are more likely to be memorized (Lee et al., 2021).
Likewise, models memorize more data as training progresses (Tirumala et al., 2022), but it is not known whether the accumulation of memorized examples is caused solely by increased exposure to heavily duplicated samples or whether other factors eventually cause memorization of rare sequences. In this section, study the impact of training time and model size on each category of memorization.
5.1 MODEL SIZE
Fig. 4(a) reports the number of examples memorized by each fully trained model, confirming that memorization increases with parameter count. While all types of memorization increase with model size, some increase faster than others. Recollection grows the most (Fig. 4(b)) from 4.49% of examples memorized in the 70M model, to 11.34% in the 12B model. This disproportionate growth suggests larger models tend to memorize rarer sequences that cannot be trivially reconstructed. Meanwhile, reconstruction barely increases, indicating the smallest models have learned to extrapolate repeating and incrementing templates almost effectively as the largest.
5.2 TIME
Over the course of training. LMs are known to memorize an increasing pool of the training data (Tiru- mala et al., 2022). However, is the cumulative effect due solely exposure more memorizable sequences? Due to repeated exposure to the same heavily duplicated data? Or is some structural property the later model more amenable to exact memorization? To understand why memorization accumulates throughout training, we measure each taxonomic category in intermediate checkpoints for the 12B parameter Pythia model. We find that accumulated memorization cannot be ascribed solely to the number of available samples to memorize or to repeated exposure to highly duplicated samples.
First, in Figure 4(c), see that models do not simply accumulate memorized samples with uniform probability through training since memorization increases sub-linearly. Second, if memorization accumulates solely due to repeated exposure to cach duplicated sample, recitation of these highly duplicated samples would be the main source increasing memorization. Instead, the proportion of recitation decreases relative to the amount of memorization (Fig. 4(d)). Therefore, the additional memorization cannot be due to repeated exposure to recitation candidates. Instead, again the largest proportional increase among all categories is in the recollection category. This trend holds until approximately 86% of total training time, which sees sudden increase in reconstruction. We conjecture that this increase represents a breakthrough in generalizing more complex templates but leave further investigation to future work.
Having considered and rejected both exclusive explanations, we must presume that memorization continues to occur late in training through a combination of repeated exposure, opportunities for memorizing new sequences, and other unexplored factors that may be the focus of future work.
PREDICTING MEMORIZATION
What makes taxonomy useful, or a reflection of natural kinds? Our position is that categories should differ in the dependencies between features of interest. The most obvious example of validated natural kinds is the case of Simpson's Paradox (Simpson, 1951), statistical phenomenon in which pair of variables are correlated across population, but the direction of correlation reverses when considering each subpopulation category separately. Simpson's 's Paradox only the most obvious evidence for natural kinds, but large changes in correlation may support categorical differences even that correlation does not change direction.
We measure a number of categorical differences in dependencies, including sign and significance differences, in Appendix B. If our intuitive taxonomy did not reflect meaningful differences with respect to the factors in Section 6.3, their dependencies would not differ significantly. We instead find significant differences through statistical tests, suggesting the taxonomy expresses some natural kinds.

---

Published as a conference paper at ICLR 2025
Model Performance Evals
Recorstruction
Aggregate
0.0
0.5
1.0 6.0
0.5
1.0 0.0
0.5
1.0 0.00
0.05
0.10
0.15
0.0
0.5
1.0
Under
Score
Optimally Partitioned
Figure 5: Performance of baseline, proposed taxonomy and optimally partitioned models against various metrics on subsets of test dataset. Confidence interval is standard deviation computed by bootstrapping.
Not only do these differences support our taxonomy as ontology, but they suggest our taxonomy can help predict memorization from dependent factors. We therefore test the applicability of our taxonomy by creating predictive model based on the intuitive taxonomic model. Our predictive model divides sequences according to which memorization category they are candidate sequences for, and then uses that category information when predicting the likelihood that the given sequence is memorized. We compare it with a generie baseline model lacking a taxonomy and with a model that automatically selected optimal partition, finding that our taxonomic model supports more accurate predictions.
6.1 MODELS
Each model is logistic regression trained with L2 regularization, a bias parameter, and balanced class weights. We split the representative sample into and train sets. We then combine the train set with the full memorized sample, reserving portion as validation set. For each set, continuous features are normalized zero mean and unit variance.
Generic baseline model The generic baseline logistic regression model trained to predict whether a sample is memorized given the features from section 3. It is trained on the training split of the entire memorized dataset and the entire representative Pile sample.
Intuitive taxonomic model (Ours) The predictive model based our intuitive taxonomy made up of a set of three binary logistic regression models. We divide samples into taxonomic groups before training a separate regression on each taxonomic category.
Optimally partitioned model To demonstrate that our intuitive taxonomic model simply benefiting from having more degrees of freedom, we devise an equally complex- is, with the same architecture of three binary logistic regressions-aateraatti taxonomic model. To provide strong baseline, we search for partition based on a set of possible feature-threshole threshold combinations. We train predictive models with the same three-regression architecture as our intuitive taxonomic model, but partitioning based on each feature-threshold combination. The optimal partition is that which supports the best predictive model, which we find categorizes samples based on Huffman coding length followed by sequence duplicate count.
For given feature, we consider the 25th, 50th and 75th percentiles the value distribution distri- bution as potential thresholds. Each feature-threshold pair provides possible partition split; we select the optimal three-category partition based on Fl score on the aggregate representative test set. Note that our "optimal" partition may not explore intuitive taxonomy the threshold search limited to each feature's quartile values. Our intuitive taxonomy may- and does- therefore outperform the optimal partition.
option because

---

Published as a conference paper at ICLR 2025
6.2 How GOOD IS OUR TAXONOMY?
To test intuitions, compare proposed taxonomy the homogeneous baseline and to optimal partition. As seen in Fig. 5, the greedy-optimal partition outperforms the aggregate baseline slightly on most metrics, but our intuitive taxonomy better calibrated and more accurate except on the recollection set, where it has low precision. We conclude that our intuition has guided us to better taxonomy than searching possible data partitions.
6.3 CATEGORICAL DIFFERENCES
Having confirmed the benefits separately con- sidering these three taxonomic categories, Fig. 6 shows how they differ through the feature weights from our regression models.
Duplicates
templating
Recollection candidates- that is. rare sequences-aue are more likely be memo- rized if they have no rare tokens. We posit that there is more resistance memorizing rare tokens within a sequence, as their prior probability low.
Textual Mutches
Sementic Metches
Max Toksen. Freq
Meanwhile, the more duplicates recollec- tion candidate has, the more likely it is to memorized, whereas recitation candidates are hardly affected by duplicate count. These results suggest that beyond the 5-duplicate threshold, greater exposure hardly leads to memorization.
Mean Taken Prest
Min Teken Preq
Median Tokenn Freq
Another notable difference is in the effect of perplexity: while predictable continuations are strongly associated with memorization across all categories, unpredictable prompts are strongly associated with memorization except for cases of reconstruction. The clear explanation is that hegh-perpeexity prompts often only occur as prelude to the same continuation, providing unique index for the memorized sequence, but that low-perples porpeeppyyxtty prompt may also initiate common template, enabling reconstruction.
P23 Takenn Freqq
P75 Token Preq
Huffman Length
Continuation PPL
Prompt PPL
Sequeace PPL
-10
-100
Peature Weights
7 DISCUSSION AND FUTURE WORK
Raseline
Recomstruction
Recollection
We have established that intuitive taxonomy can be used to improve understanding of mem- orization. We now relate our methods to the existing literature memorization and to possi- ble future directions.
Figure 6: Feature weights from predictive models trained the homogeneous aggregate baseline and the intuitive taxonomy categories.
7.1 ONTOLOGIES OF MEMORIZATION
Our work is strongly related to several recent efforts to develop an ontology of memorization. Dankers et al. (2023), studying machine translation, focus primarily on the influence sequence during training rather than on the semantics properties of individual sequence. They investigate the factors that influence counterfactual memorization, a type of memorization likely to dominate "recollection" cases. They find that rare tokens, long sequence lengths, and high BPE segmentation rate are correlated with counterfactual memorization (Zhang et al., 2021); of these, we only consider rare tokens, which confirm predict recollection in particular.
8

---

Published as a conference paper at ICLR 2025
Hartmann et al. (2023) consider what facets of memorization are likely to be relevant to different targets, just as we discuss the differences between motivations grounded in copyright infringement and privacy. Bansal et al. (2023) consider two different kinds of memorization: heuristic memorization, iee, shortcut learning, and example memorization. Our work focuses on what they call example memorization, further decomposing that category. We do not test their result that high-entropy features can indicate example memorization, but like us, they use this factor to differentiate between their memorization categories.
7.2 MEMORIZATION AND TRAINING TIME
Our work fits into existing literature on how time and scale affect memorization. Biderman al. (2023b) find that the position of sequence in training does not affect its likelihood of being memorized, and that smaller models fail to memorize even when repeatedly exposed to term. Tirumala et al. (2022) find that larger models memorize more training data and forget less during training. They also observe that models memorize nouns and numbers first, using these entities as unique identifiers for individual samples. Our work further expands our understanding of scale in memorization by highlighting that sequences compose the fastest-growing category of memorization.
7.3
WHICH CATEGORIES DO WE CARE ABOUT?
The relevance of each category depends on our motivation for studying memorization.
1. Intellectual property violations:
The content most relevant to concerns about intellectual property may be highly duplicated data, such as frequently excerpted passages from a popular book. However, some rare sequences may also be memorized, making recollection potentially relevant to issues of copyright infringement.
2. Privacy: the primary motivation is preventing the memorization personally identifying information, we may focus on recollection, as issues may arise if model generates such information even after even a small number of exposures.
3. Scientific understanding of generalization: Work like Henighan et al. (2023b) and Bartlett et al. (2020) points to eventual generalization as a result memorization dynamics. A deeper understanding of these phenomena might focus on reconstruction, which exposes direct link between apparent overfitting and general pattern recognition.
7.4 ONTOLOGIES AND STATISTICS
This taxonomy may serve as an example for future methods of interpreting complex phenomena, in deep learning and elsewhere. We have, in particular, quantified the validity and usefulness of such a taxonomy by comparing predictive models which treat memorization in aggregate to models which treat memorization as multifaceted phenomenon with our taxonomy. We provide evidence for the taxonomic model by measuring the improvement in predictive judgments when reflecting the dependent and nonlinear thresholded relationship between memorization and the properties that define each taxonomic category.
In future work, we hope that interpretable and useful ontologies can be validated by similar approach. Our proposal for what makes good taxonomic model is not only applicable to memorization or even deep learning phenomena. Instead, by studying interactions and nonlinearities in arbitrary settings, researchers may find complex dependencies and artifacts like Simpson's 's paradox.
LIMITATIONS
Our primary goal to intuitively describe the memorization behavior with a taxonomy and con- sequently use that taxonomy to investigate how several dominant factors in memorization interact with each other. A secondary goal to provide an example how an ontology can be constructed and tested in general, as tested with our predictive models. However, these predictive models not measurements of statistical dependency in general, instead only focusing on linear dependence. Although more general statistical dependencies studied in the supplementary experiments of

---

Published as a conference paper at ICLR 2025
Appendix B, the experiments in the main body of the paper assume linear dependence and so the interacting factors should be evaluated in the context of our supplementary dependency experiments. We believe ontological work inspired by approach could improve on work by incorporating more general dependencies.
Another limitation is definition of memorization. The choice of 32-elicitation has number of disadvantages, one of them being that we lose a notion of fuzzy or partial memorization, which is considered important in some contexts. Arguably, under counterfactual memorization definition, we may not see substantial patterns of either recitation or reconstruction. The measurement of memorization is large area research with many possible definitions choose from (Carlini et al., 2022a; Tirumala et al., 2022; Kandpal et al., 2022; Zhang et al., 2021; Zhao et al., 2022; Stock et al., 2022; Schwarzschild al., 2024).²
ACKNOWLEDGMENTS
This work was enabled in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. We would like to thank EleutherAl and CoreWeave for providing the computing resources used this paper.
We thank Demba Ba for discussion that informed this work.
AUTHOR CONTRIBUTIONS
USVSN Sai Prashanth built sampling infrastructure for the memorized and representative datasets; designed and trained the classifier that determined whether a given sequence was code or natural language; engineered the predictive features for duplicate count, token frequency, and templating; helped optimize perplexity measurement for efficiency; debugged, expanded, and refactored the predictive model training pipeline; generated and modified number of plots; and helped write.
Alvin Deng built the data processing pipeline, designed the predictive model training and evaluation pipelines, analyzed classifier performance on code vs. natural language, generated exploratory visualizations, and helped write.
Kyle O'Brien conducted the initial literature review, implemented the pipeline for calculating perplex- ity, designed several figures including the explanatory diagram, and helped coordinate and manage the project.
Jyothir conceived the initial idea while conducting early experiments to investigate various memorization patterns and their characteristics. He also engineered the compressibility. semantic match, and textual match features.
Mohammad Aflah Khan helped engineer the templating features, visualized and analyzed data, and helped write.
Jaydeep Borkar conceived, coded, and visualized potential predictive features for memorization Christopher A. Choquette-Chine Chhett ran early experiments that shaped the project.
Jacob Ray Fuehne assisted in exploring features by labelling data and feature engineering.
Stella Biderman offered high-level guidance, material resources, and help with writing.
Tracy Ke supervised and advised statistical testing, especially the dependency tests. Katherine Lee supervised and advised the project and helped with paper writing.
Naomi Saphra supervised and advised the project and led paper writing.
REFERENCES
Rachit Bansal, Danish Pruthi, and Yonatan Belinkov. Measures of information reflect memorization patterns, 2023. URL httt:://axxiii httt:://arxi..org/bb/2222 0. 09404. 2.9444444...0.4....
org/abs/221
For discussion of definitions of memorization, https://gemlaw.orgg hwtpn//wwwwwggggo//aaasg f html# memorization.
10

---

Published as a conference paper at ICLR 2025
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30063 2020.
Stella Rose Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin G. Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. ArXiv, abs/2304.111555 2023a.
Stella Rose Biderman, Hailey Schoelkopf, Quentin G. Anthony, Herbie Bradley, Kyle O' O'Brien, Fric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. ArXiv, abs/2304.01373, 2023b.
Hannah Brown, Katherine Lee, FatemehSadat Mireshghallah, R. Shokri, and Florian Tramèr. What does it mean for language model to preserve privacy? Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022. URL https: /appi. semanticscholar. org/CorpusID::2 246823897.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Xiaodong Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposimm, 2018.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song. Ülfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2020.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. ArXiv, abs/2202.07666, 2022a.
Nicholas Carlini, Matthew Jagielski, Nicolas Papernot, A. Terzis, Florian Tramèr, and Chiyuan Zhang. The privacy onion effect: Memorization is relative. ArXiv, abs/2206. 10469, 2022b.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models, March 2023. URL http: www.xiv.org/bb..../ 2202. 07646. arXiv:2202.07646 [cs].
Verna Dankers, Ivan Titov, and Dieuwke Hupkes. Memorisation Cartography: Mapping out the MennoriationnGGGneralisttooo Continuum in Neural Machine Translation, November 2023. URL [cs].
9.57......//////11......... arXiv:231 Ttt::2311.053777 1.05379
Vitaly Feldman. Docs Learning Require Memorization? A Short Tale about a Long Tail, January http::////aaiii..rg/////////// 0969999669 05271. arXiv:1906.0527 [cs, stat].
2021. URL http:http///xii/i www. abs/
Leo Gao, Stella Rose Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang. Horace Hc, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. ArXiv, abs/210) .00027, 2020.
Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. SoK: Memorization in general-purposs large language models, 2023, URL /ttp:///rxiv.. org/abs/2310. 18362.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 2023a.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, Memorization, and Double Descent, 2023b. URL tttass//ttrasfoomeeriirriitt
wereescceet/nnee.hheeee htmm.
David A Huffman. A method for the construction of minimum-reduminn codes. Proceedings of the IRE, 1098)100981111 1952.
11

---

Published as a conference paper at ICLR 2025
Daphne Ippolito. Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette-Chovr and Nicholas Carlini. Preventing verbatim memorization in language models gives false sense of privacy. arXiv preprint arXiv:2210.1556., 2022.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. ArXiv, abs/2202.006399 2022. URL https://api. semanticscholar.omm
CorpusID: 246823128.
Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Sogaard. Copyright violations and large language models. ArXiv, abs/2310.13771, 2023. URL httpss//apis semanticscholar. org/
CorpusID:264423000
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Annual Meeting of the Association for Computational Linguistics, 2021. URL https://api. semanticscholar.c wwwpoolaarrrggr//poo///e wwwp/oopgssIoaoa-o.ooooo 235829052.
Vladimir Levenshtein et al. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, volume 10, pp. 707-710. Soviet Union, 1966.
Neil Mallinar, James B. Simon, Amirhesam Abedsoltan, Parthe Pandit, Mikhail Belkin, and Prectum Nakkiran. Benign, tempered, catastrophic: A taxonomy of overfitting, 2022.
Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alcxandre de Montjoye. Copyright traps for large language models. ArXiv, abs/2402.09363, 2024. URL httpps://api semanticscholar, org/ CBCCorpuIDD:277557777
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg- Kirkpatrick. An empirical analysis of memorization in fine-tuned autoregressive language models. In Conference Empirical Methods in Natural Language Processing, 2022. URL https://api. wwwanti wwaplaa.orgg/ooppe orgg/CorpsSID::5555 256461 422.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter, 2020.
Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. Rethinking Ilm memorization through the lens of adversarial compression, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. ArXiv, abs/2310.16789, 2023. URL httht: //api. semanticscholar
Edward H Simpson. The interpretation of interaction contingency tables. Journal of the Royal Statistical Society: Series B (Methoddological) 13(2):238-22 241, 1951.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 15:19953-19536, 2022.
Pierre Stock, Igor Shilov, Ilya Mironov, and Alexandre Sablayrolles. Defending against reconstruction attacks with rényi differential privacy, 2022.
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Mem- orization Without Overfitting: Analyzing the Training Dynamics Large Language Mod- els. arXiv:2205. 10770 [cs], May 2022. URL http://arxiv.. org/ 10770. arXiv: 2205.10770.
Zhou Yang, Zhipeng Zhao, Chenyu Wang. Jieke Shi, Dongsun Kim, Donggyun Han, and David Lo. Unveiling memorization in code models. In Proceedings of the IEEE/ACM 46th International Conference Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400702174, doi: 10.11457557500333699744 URL https: //doi. www.orgg///../ 1145//597500.... 3639074.
12

---

Published as a conference paper at ICLR 2025
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. ArXiv, abs/2112.12935, 2021. URL https://api semanti wrg/CorpusID:225502255
Xuandong Zhao, Lei Li, and Yu-Xiang Wang. Provably confidential language modelling. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 943-955, Seattle, United States, July 2022. Association for Computational Linguisties. doi: 10. 10.11865//1/2022.mmg www.naaaal.ai9999 main.69, URL https: //ac wac.lannoooog..rrgg// g/222...2..2222e maacl-main.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
A IMPLEMENTATION OF METRICS
We now provide details of the implementation for metrics considered potential factors for memo- rization.
A.1 NUMBER OF EXACT DUPLICATE SAMPLES
compute the number of exact duplicates as 3-step process:
For every 32-gram window in the data point (comprised of 2049 tokens) we compute rolling hash and store it, along with the window's 's index and offset in parallelized set of data frames.
2. For every index position, we compute the same hash for all S[32: 64] (sequence continua- tions) and store all (index, offset, hash) tuples their hash is one of the computed sequences continuation hashes.
3. Now, for every sequence continuation, we look at all 32-gram windows with the hash and compute their number of duplicates as the number of equivalent (same set of tokens in the same order) samples.
The hash function used is similar to Sabin-Karp's rolling hash algorithm. Specifically, consider token sequence of 32 tokens.
S | C3 03,...C32)
Let us define two primes = 60013 and MOD = 18 + 3. We define their hash function to be
H(S) (cr
02 * P p2 ....+e 032 p31 JMOD
A.2
TOKEN FREQUENCY
Token frequencies are calculated across the Pile. For every sequence continuation, we consider the maximum, minimum, median and quartile frequencies of tokens.
A.3 COMPRESSIBILITY
We use Huffman Coding length to measure how easily sequence can compressed. Compressibility provides rough generalization of internal repetition, where only few exceptions to some simple repetition pattern might need to be memorized. However, unlike straightforward repetition templates, compressible sequences may not be considered to be reconstructed by the model. Instead, include compressibility as filter to evaluate whether LLMs memorize samples that are easier to compress into their parameters.
13

---

Published as a conference paper at ICLR 2025
A.4 INCREMENTING AND REPEATING TEMPLATES
A.4.1 INCREMENTING TEMPLATES
To check for an incrementing sequence, we perform the following steps:
Split the text by whitespace and convert any splits which are numerals in non-decimal bases (e.g., hexadecimal) into base 10.
Remove escape sequences.
Within each string, separate contiguous numeric characters from anything else. two contiguous numeric characters separated by period, combine them into their floating point representations.
Discard if there are fewer than potential numerals in the sequence.
Check if the sequences incrementing repeating.
A.4.2 REPEATING TEMPLATES
We perform the following steps to check for repeating sequences:
Obtain sequence by splitting the text by character.
Check if the sequences incrementing repeating.
We perform the following steps to determine if a sequence generated from either of the above steps is incrementing or repeating.
For every templating length, defined to be less than half length of splits, and for every position less than templating length, we iterate through splits with start position as position and step size set templating length. We then determine if the current iteration is repeating or incrementing.
For example, if position is and templating length is 5, we iterate through positions [1, 6, 11, 16.... if our input splits length is 10, iterate for all templating lengths through 5 and for all positions less than current templating length
Within each iteration, we check:
If the current iteration has both texts and numerals, it is neither incrementing repeating.
If the current iteration has only texts, we consider the current iteration to be repeating all elements in the iteration are the same.
the current iteration has only numerals, we consider current iteration be increment- ing if all the numerals are in arithmetic progression. If the difference in AP is O, we consider it to be repeating instead.
Input splits are considered as repeating if all iterations for given templating length are repeating.
Input splits are considered as incrementing if atleast one of the iterations for a given templating length are incrementing and others, for the same templating length, are either incrementing (or) repeating.
For all templating lengths, any length of them has been found to be incrementing or repeating, we return True (corresponding to the fact that the text is indeed a template) and diff.
Note that, in the case of sequences generated while checking for repeating template, do not have any numerals.
B DEPENDENCY TESTS FOR INFLUENCE OF FEATURES ON MEMORIZATION
This section contains visualizations of various dependency tests between memorization likelihood and our target features. We look at dependencies code (Fig. 9), natural language (Fig. 8), and both
14

---

Published as a conference paper at ICLR 2025
Baadiine
Recoretructin
1144 k801
x414
Medan
P75
Proggt
"tatus Macres
Hufrran Lengrs
Pearson
Duplicate
Hean Taaen Fres
Median Taken Froq
PTS Taae hog
PravE
"kmtal Hoschic
"eecuttng
1.0
-10
Velue
Vilue
Ville
Jeselie
Recoretruction
Recollction
Rectbll on
Nax iulo
Min 3ann
I25 Iplen lres
Continudonn
Haftmen Lengh
-1.0
-1.0
-1.8
Figure 7: Dependency measurements between influence factors and memorization.
(Fig. 7). These tests more general and have stronger guarantees than simply looking regression weights, which have a number flaws. example, we see Fig. that regression can reallocate bias terms to features that take on consistent value, giving them spurious weight.
C TABLES FOR SCALING EXPERIMENTS
Figure 4 visualizes the count and proportion of memorized samples category time and scale. We present the raw statistics for each taxonomic category across model size in Table and training time in Table 2.
Recitation
Reconstruction
Recollection
Model
Count
Percent
Count
Percent
Count
Percent
70m
362,550.0
88.12%
30,430.0
7.40%
18,468.0
4.49%
410m
690,726.0
85.17%
46,076.0
5.68%
74,238.0
9.15%
lb
878,456.0
85.05%
49,253,0
4.77%
105,163.0
10.18%
1.4b
887,549.0
84.68%
49,435.0
4.72%
111,120.0
10.60%
2.8b
1,141,180.0
84.21%
52,416.0
3.87%
161,620.0
11.93%
6.9b
1,416,014.0
84.27%
53,968.0
3.21%
210,314.0
12.52%
12b
1,566,369.0
84.56%
55,114.0
4.10%
249.733.0
11.34%
Table The number of memorized samples for each taxonomic category across model size. These results visualized in Figure 4(a) and 4(b).
15

---

Published as a conference paper at ICLR 2025
Natural Language
Daseline
Racsestruction
Recallection
Rec tation
Duplcetes
1ax
F4: 1628
205 1aan Fim
Corthuaiinn F0)
Secvarce PTL
berparing
0.0
22
044
0.5
3.0
00
00.2
2.4
09
1.0
612
04.
2.6
08
3.0
0.0
022
C.4
10
Natural Language Pearson
Seasline
Recoratration
Nacollastion
Recitaioo
Daplicatec
Max
Takan
Medlan Takan
IT5 Takn
pronce
"eoual Horcee
HMlmeo Lengt
Value
Velue
160
Natural Language Spearmnn
Tacal
Recoretration
RecoBaccion
Secibition
Noan Nacc
Medin lakan king
Continuation
Poatk
Semsartv Hertes
-1.
-0.5
2.5
1.5
-1.0
-0.5
6/1
0.5
1.0
-1.0
-2.5
0.5
1.0
-1.0
-0.5
0.00
0.5
Figure 8: Dependency measurements between influence factors and memorization for natural lan- guage samples.
Code Xi
Saseline
Recondtrution
Recollection
1ax 3441 Trc4
191 3aan Pltt
H25 1801 110)
Contruation
Secuerce
"katua Matchs
Hulhman Lengin
Code Pearson
Hean Taaen
Mediun Taken
PTS Taeen from
"aeaa Harches
kmicktingg
Volue
Value
Vilee
V0
Code
Teseline
Recoratruction
Recollaction
Rec on
Nax iuknn
Min 24m1
F25 Taken free
Contruudon
Sequetion
(uftmen Lengt
0.5
-1.0
-1.0
-3.0
Figure 9: Dependency measurements between influence factors and memorization for code samples.
16

---

Published as a conference paper at ICLR 2025
Recitation
Reconstruction
Recollection
Checkpoint
Count
Percent
Count
Percent
Count
Percent
16%
137,681.0
84.25%
7,960.0
4.87%
17,777.0
10.88%
30%
301,146.0
83.92%
15,379.0
4.29%
42,338.0
11.80%
44%
489,629.0
83.69%
23,333.0
3.99%
72,105.0
12.32%
58%
710,823.0
83.42%
31,260.0
3.67%
109,985.0
12.91%
72%
999,867.0
83.63%
38,999.0
3.26%
156,712.0
13.11%
86%
1,308,538.0
83.66%
47,145.0
3.01%
208,372.0
13.32%
100%
1,566,369.0
84.56%
55,114.0
4.10%
249,733.0
11.34%
Table 2: The number of memorized samples for each taxonomic category across training time for Pythia 12b. 14,000 is the final checkpoint. These results visualized in Figure 4(c) and 4(d).
0.6
False Positive Rate
0.5
0.4
0.3
0.2
0.1
0.0
0.40
0.45
0.50
0.55
Model Threshold
Natural Language
Code
Figure 10: False positive rates across various thresholds randomly sampled sequences of Pile. We choose S 0.4 as threshold for determining code samples and threshold of 2 0.525 for determining natural language
D CLASSIFYING EXAMPLES AS NATURAL LANGUAGE OR CODE
To train Natural Language Code classifier, fine-tune DistilBert (Sanh et al., 2020) uniformly random sampled Bookcorpus (Zhu et al., 2015) and github-code datasets. We train with learning rate of 10-7 and batch size of 256 for total of 1000 steps and observe validation fl score of 0.9950 on a held evaluation set.
To select optimal threshold for this classifier memories dataset, randomly sample 500 sequences and manually label them. To make sure that precision is high for our models, we choose < 0.4 as threshold for determining code samples and threshold of 0.525 for determining natural language samples, based on the points marked in Figure 10, which mark points of near 100% precision for classifying each category.
E LIKELIHOOD OF MEMORIZATION FOR CODE AND NATURAL LANGUAGE
We study the likelihood that a sample that has been confidently classifier as code or NL (Appendix D) is memorized across time and scale. For example, for all samples confidently classified as code Figure, what the proportion of samples which are memorized?
Figure 11 shows that code samples more likely to be memorized than NL across categories. This trend suggests that certain intrinsic factors about code make more susceptible to memorization, for recollection samples where memorization cannot be attributed to obvious patterns and
17

---

Published as a conference paper at ICLR 2025
Code
Code
P(Memorized
P[Mernorizd)
13+3
10-1
30-1
10-1
1004
10-
10-1
10-
10.0
10
30
10
10
0.25
5.30
0.76
1.00
0.25
3.30
0.75
3.00
10:1
10
30.1
10.
P(Recitation)
P(Recitation)
10-3
10-1
20-1
10-1
10->
10-1
10-1
10-1
10"
10P
10t
100
3000
0.25
1.50
9.20
6.75
1.20
P(Reconstruction)
PRReconstruction
bunz
10-1
10-4
10-1
10-3
10-5
10-1
P(Recollection)
P(Recallection)
30
30
10-3
10*2
30
12.
10
10-1
30-1
10
107
10t
10P
10t
0.25 Training Time
0.50
6.75
1.50
0.25
3.50
6.75
3,00
Parameters
Parameters
Training Time
Figure We study how likely models are to memorize samples confidently classified as code or NL. We calculate the likelihood for each distribution (code vs. NI.) separately. Figures include probability across a model scale and training time. Models memorize a greater proportion of code samples than NL across all categories, model scale, and training time.
high duplication. Both code and NL become more likely be memorized across scale, except for reconstruction samples, which remain comparatively unchanged.
F EXAMPLES OF MEMORIZED CONTINUATION SEQUENCES
Table 3 provides examples of memorized natural language text in each memorization category and Table provides examples of memorized code. Samples are classified using the methodology in Appendix D.
G ALTERNATIVE RECITATION THRESHOLDS
We have selected >> as our duplication threshold for categorizing sequence a recitation candidate, based on the analysis in Fig. While we have shown that our resulting taxonomy outperforms those based on possible quartile cutoffs, we have not compared it to other small thresholds. In Fig. 12, we perform this comparison and find that the threshold we selected based on intuitions is at least as good similar but smaller (>1) or larger (>10) threshold.
18

---

Published as a conference paper at ICLR 2025
Category
Text
Count
-11 NASB
Or do you not know that the unrighteous will not inherit the kingdom of God? Do not be deccived; neither fornicators, nor idolaters, nor adulterers, nor effeminate, nor homosexuals, thieves, the covetous, nor drunk
Recitation
175
2d 1234, 1238 (8th Cir. 1990). On the other hand, the Federal Rules of Civil Pro- cedure have authorized for nearly 60 years "motions for summary judgment upon proper showings of the lack genuine, triable issue of material fact." Celotex Corp. V. Catrett
Recitation
86
view of life, food, cocktails, fitness, and fun.
This blog is just regular guy's 's view of life, food, cocktails, fitness, and fun. My opinions, musings, observations, rantings, ravings, foodie adventures, and overall humorous pontification of
Recitation
000,
from the tribe of Simeon 12,000,
from the tribe of Levi 12.000,
Recitation
from the tribe of Issachar 12,000.
from the tribe of Zebulun 12,000,
from the tribe of Joseph 12,000,
THEIR EAGLE ONAND THE DIRTY BIRDS READY WELL DOWN AND GET THEIR EAGLE ONAND DIRTY BIRDS READY WELL DOWN AND GET THEIR EAGLE ONAND THE DIRTY
Reconstruction
181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202
Reconstruction
1970-7 71 Turkish Third Football League season Promotion and relegation:
1971-72 Turkish Third Foothall League season
Reconstruction
Promotion and relegation:
1972-73 Turkish Third Foothall League scason
Promotion and relegation:
1973-74 Turkish Third Football
28-63, 28-64, 28-65, 28-66. 28-67. 28-68, 28-69, 28-70, 28-71, 28-72, 28-73, 28-74, 28-75, 28-76, 28-77, 28-78
Reconstruction
affect the child;
'700 (b) The wishes of the child, as expressed directly by the child or through the child's guardian ad litem, with due regard for the maturity of the child;
Recollection
471 (c) The custodial history of the child,
will be too late!
24 "Strive to enter through the narrow door. For many, tell you, will seek enter and will not be able, 25 When once the master of the house has risen and shut the door, and you begin to stand outside and to knock at the door,
Recollection
freedom. Revava Revava O. is an Orthodox Jewish Israeli settlement in the West Bank. Located between Barkan and Karnei Shomron, it falls under the jurisdiction of Shomron Regional Council. In it had population of.
Recollection
The international community considers Israeli settlements in
2254(d)).
A state-court decision is considered "contrary to... clearly established Federal law" if the two are "diametrically different, opposite in character or nature, or mutually opposed." Williams Taylor, 529 U.S. 362, 405
Recollection
Table 3: Examples of natural language (classified per Appendix D) from each memorization category.
19

---

Published as a conference paper at ICLR 2025
Category
Text
Count
>-task <a class= nww-grropp-task--ink href=" hnd"" </li> <li class= "uass="navgrooppttst""""" <a claas="nav--ooop-ttsk--iiii htw.==../Extennaeee ../Extensions/
Recitation
5310
<widget classt id="cmtClcanBut") <property name="label" translat- able="yes">... >............................... :/strrong></p>> <ub> <li> <p>Must be one of: <code>tmec/codes, <code>false</codes, <code>1</code; <code>A</code>.</pp </li> </ul>
Recitation
3689
null, Tokeellation cancellation defaaut(Canccellatton Token) if Subscrriiio null) throw new ValidationExxep- SiliidattonRulle.CCCmoBBBBBe Cwt.iieentt.ubccppiioooddd
Recitation
Object</h3>> <code>equals, getClass, hashCode, notify, notifyAll, wait, wait, wail</code>><<ll>> </ul> </li> </ub> </li> </ul> </div> <div class="details"
Recitation
18963
FileEntry""hasee/ddill"" fe. age): fe.name "file3": fi >> updateFileEnn-
Reconstruction
(/("/aacl/dirr//ffe ago); fe.name = "file4"; fi -> updaatciicEEnty((basclll
="time2[]" value="5" ></d> <td><input type="checkbox" name="time2[]] value="6" ></d>> <td><input type="checkhort "chockoox" name= "time="time2["" valuc="7" ></d>>
Reconstruction
XMM3 (TULL 28) #deffiee DBG_CTX CTT_EE_PART__LALCECCCECCCC
CTX_ WART_FLAG_AA XMM4 (IULL 29)
Reconstruction
#define
PLRT_FFLGGXMMNN XMM5 (IULL 30) #define DBG
<Compile Include= MeaeeeMMeeeeeMMNNMMMMM M06.cs" />
<Compile In-
Reconstruction
clude=" clude= MessagelMFNN "M <Compile Include= Include="Messapee MMssaaeeLMM8..c
<Compile Include="Messoe "MesaadeEeMM........
OFFSET (2
FPREG SIZE)) #define PROBE CPU Q3 OFFSET
Recollection
(PROBE FIRST FPREG OFFSET
FPREG SIZE)) #define
PROBE CPU Q4 OFFSET (PROBE FIRST FPREG OFFSET
): uint16 WS2812FX: custom 50 return customModes(510: uint16 WS2812FX custsmuutttoooo return customModes(6100 uint16
Recollection
XMMM128,
OxDC
NORMAL(" "ORMAL("padduus"" paddusb",
MM XMM,
Recollection
MMM64 B111644XMMM228,. / OxDD */ NORMAL(" paddusw", MM MM_XMM, MMM64_X
DBF3B 1 i Bronl.png */: J: 651A5W/V11/AEEED0003BBBB soon2.png in Re- sources */ (isa = PBXBuildFile; fileRef = = (51A55A7C177AE220000D0
Recollection
Table 4: Random examples of code (as classified per Appendix D) from each memorization category.
20

---

Published as a conference paper at ICLR 2025
Performance Evals (Comparing Thresholds)
Precision
Recall
BCE
Becitation
Receastruction
Aggrogate
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50 Score
0.75
1.00
0.00
0.25
0.50 Score
0.75
1.00
0.00
0.05
0.10
0.15
Area Unier Curve
Hrror
Tanonorate Model Thresholc
Tasonomic Model Threshold 10
Tamonorado Model Threshold
Figure 12: Comparison of memorization predictor performance, similar to those trained in Section 6. Thresholds or do not generally outperform our selected threshold of 5.
21


Reviews:
[{"review_id": "hp8mgs6D2L", "reviewer": "Reviewer_5vzG", "summary": "The paper proposes a taxonomy for model memorization and classifies model memorization into three categories, namely recitation, reconstruction and recollection. The authors identify several data-related or model-related features and test their correlation with model memorization. To verify the effectiveness of the proposed taxonomy, the authors trained three linear regression models for three categories respectively and found that group the regression models attain better performance than the predictors trained on other features.", "questions": "+ The taxonomy of memorization is purely established based on the property of data, i.e., the number of duplications in the pre-training corpus and the implicit template within the data. However, memorization is also a concept and phenomenon related to model behavior and model behaviour can also be included into the taxonomy as an evidence to classify different types of memorization.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ A new taxonomy for understanding and analyzing the model memorization. \n+ Interesting findings on the dynamics of memorization during the scaling-up of data and model size.\n+ An empirical evaluation of the utility of the taxonomy based on predictability.", "weaknesses": "+ A perplexing part of the taxonomy is the classification of repetitive or incremental sequences following a specific pattern. If the sequence duplicates more than five times, how do we know whether it is truly \"memorized\" or it is reproduced simply because the LLM learns its pattern? \n+ Why do we use more than five times duplication as the decision boundary for recitation and non-recitation. How is the hyper-parameter decided? Using a single threshold actually assumes an equality in difficulty for reciting every sequence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W7qHXyuxwZ", "reviewer": "Reviewer_bXcZ", "summary": "This paper introduces a novel taxonomy for memorization in language models, categorizing it into three types: recitation (highly duplicated sequences), reconstruction (predictable templates), and recollection (other memorized sequences). The authors validate their taxonomy through predictive modeling and analysis across model scales and training time, demonstrating how different factors influence each category distinctly. The work provides valuable insights into understanding memorization as a multifaceted phenomenon rather than a uniform behavior.", "questions": "What insights from previous work on memorization mechanisms support or conflict with these findings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed memorization taxonomy is intuitive and interesting, drawing parallels with human memorization. This taxonomy is particularly valuable as it provides a structured approach to analyzing what has typically been treated as a uniform phenomenon. \n\nThe analysis methodology is another strong point, featuring a thorough examination of dependencies between features and memorization categories, supported by effective predictive modeling to validate the taxonomy.", "weaknesses": "The main weakness of this paper boils down to two key issues. First, while the idea of categorizing memorization into three types sounds cool, the paper doesn't dig deep enough to tell us why we should care. Sure, they show that code gets memorized more than text across all categories - but why? And what does this mean for how these models actually work? How different types of memorization contribute to model capabilities. These are the kind of insights that would make the taxonomy actually useful, but they're missing.\n\nIn addition, the experimental setup is not convincing. For example, the experiments are conducted solely on Pythia models without validation of other popular models. And some of the key choices seem pretty arbitrary like picking $k=32$ for their memorization tests or saying \"more than 5 duplicates\" counts as recitation. Why those numbers? What happens if you change them?\n\nOverall, I think the paper lacks insights and the experiments are not very solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TGWhBHdNGu", "reviewer": "Reviewer_FK6z", "summary": "This comprehensive paper presents a novel taxonomic analysis of memorization in LLMs, breaking it down into three distinct categories: recitation (highly duplicated sequences), reconstruction (inherently predictable sequences), and recollection (neither duplicated nor predictable). Through extensive experimentation with the Pythia models ranging from 70M to 12B parameters, the authors demonstrate that different types of memorization exhibit distinct patterns and dependencies on factors like sequence duplication, model size, and training time. They validate their taxonomy by showing its effectiveness in predicting memorization likelihood and reveal that recollection grows disproportionately with model size and training time. The work provides valuable insights into how different factors influence memorization depending on the taxonomic category.", "questions": "1) Can you provide empirical justification for this specific cutoff? How sensitive are your results to this choice?\n2) Could you include statistical significance tests for the reported trends across model sizes?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper exhibits several notable strengths that demonstrate its potential value to the field. The proposed taxonomy of memorization provides an intuitive and practical framework for understanding different types of memorization in LLMs. The extensive experimental validation across model scales and training time points offers valuable insights into how memorization behavior evolves. The authors' approach to validating their taxonomy through predictive modeling and dependency analysis shows methodological rigor and provides empirical support for their theoretical framework.", "weaknesses": "1) The template detection approach appears oversimplified. For instance, only basic patterns of \"repeating\" and \"incrementing\" sequences are considered, potentially missing more complex templates. The duplication counting relies on exact matches without accounting for semantic similarity or near-duplicates (e.g. slightly modified code or text passages).\n2) The paper insufficiently compares its taxonomy against existing memorization frameworks. For example, the relationship between these categories and counterfactual memorization, which is mentioned but not analyzed, deserves exploration. The advantages of this taxonomy over other approaches to studying memorization are not quantitatively demonstrated.\n3) The exact procedure for computing KL divergence in Fig 3 is unclear, and the methodology for computing perplexity scores used throughout the analysis lacks essential details. The robustness of results to different tokenization choices is not evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l5BKAC9HC6", "reviewer": "Reviewer_p6LS", "summary": "This paper proposes a taxonomy of memorization in language models, breaking it down into three categories: recitation of highly duplicated sequences, reconstruction of inherently predictable patterns, and recollection of rare sequences. The authors validate their taxonomy through statistical analysis and predictive modeling, showing that different factors influence memorization differently across categories. \n\nThey analyze how memorization patterns evolve with model scale and training time, finding that recollection grows disproportionately with model size. The paper illustrates the practical value of this taxonomy by showing how different categories matter for different applications (privacy, copyright, scientific understanding) and by achieving better predictive performance compared to both a baseline without taxonomy and an optimally partitioned model.", "questions": "1. How would the predictive model perform without continuation perplexity as a feature? This would help assess how much signal the other identified factors provide to predict memorization.\n2. How novel is the introduction of statistically validated taxonomy? Do similar case studies exist in other fields? Beyond the reference to the Simpson paradox, the paper doesn't include references in this area.\n3. Are corpus statistics computed on the prompt, continuation or the full sequence? \n4. What are the characteristics of sequences with high duplicate counts (>100) that don't get memorized? Understanding these cases might provide insight into factors that inhibit memorization despite repeated exposure.\n5. How sensitive are the semantic similarity measurements to the length mismatch between the 64 (or 32)-token samples and the 2049-token sequences they're compared against? \n6. (Less important) Could the sudden increase in reconstruction at 86% of training be related to the \"induction bump\" phenomenon described in the mechanistic interpretability literature? Again, see \"In-context Learning and Induction Heads\", by Olsson et al. for the introduction of the concept.\n\n\nMinor points:\n\nThe Figure 6 would benefit from being vertically compressed without deforming the text.\n\nLikely typo lines 140-142: \"we generate document embeddings for each full sequence using SBERT and count the number of sequences with cosine similarity ≤ 0.8. These sequences are semantically similar but may not be exact token-level duplicates.\" -> I guess it should be \"cosine similarity ≥ 0.8\" instead.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **The paper provides a methodologically sound approach to developing and validating taxonomies in ML research (and beyond).** By grounding qualitative distinctions in statistical analysis (e.g., following the example of Simpson's paradox), it offers a template for studying complex phenomena in ML beyond memorization that could be of interest for the ICLR community. \n- **The taxonomy enables discoveries about memorization dynamics.** For instance, the finding that duplicate count beyond 5 barely affects recitation likelihood challenges simple assumptions about exposure and memorization. The categorical distinctions also help align research directions with specific applications (e.g., privacy vs. copyright concerns). \n- **Analysis of the semantic properties of the sequence.** This type of statistics provides valuable insights into how models can perfectly reproduce sequences through pattern completion rather than pure memorization for the reconstruction category. This distinction, while simple in hindsight, is important for understanding the relationship between memorization and generalization, and the limit of the k-extractability definition of memorization.", "weaknesses": "- **The predictive model's performance is relatively poor** (precision ~0.5, PR AUC ~0.5), despite including continuation perplexity as an input feature. This raises questions about the practical utility of the identified factors in predicting memorization. The heavy reliance on continuation perplexity for every category (see Figure 5.), which is closely related to the definition of k-extractability, makes it difficult to assess the independent predictive power of other factors. \n- **No clear progress in understanding the memorisation of rare sequence.** While the paper identifies recollection of rare sequences as an important phenomenon, particularly as models scale, it provides limited insight into the underlying mechanisms. This gap is particularly notable given the paper's emphasis on understanding different types of memorization.\n- **The presentation lacks clarity at times.** \n\t- When introducing the taxonomy, early concrete examples of each category would significantly improve understanding. \n\t- The paper should also better highlight the distinction between intuitive notions of memorization and the technical definition of k-extractability used in the study. This could help the reader understand why the reconstruction phenomenon (where sequence outside of the training set could be predicted perfectly) fall in the scope of the study of memorization. \n\t- The study could benefit of including reference to a broader set of references such as the study of mechanistic interpretability and training providing more insights on how and when models become able to predict simple sequences. See for instance \"In-context Learning and Induction Heads\", by Olsson et al.\n- **Methodological limitation in the computation of the corpus statistics.** \n\t- The corpus statistics are not broken down into prompt/continuation/full sequence. This could enable the isolation of sequences with a frequent prompt but infrequent continuation, or the opposite for instance. The paper doesn't clearly state which one of the three is used for the corpus statistics.\n\t- If I understood correctly, the semantic similarity measurements are made between sequences of length 64 (or 32) tokens (the memorized/non-memorized samples), and the 2049-token sequences from the Pile. This length mismatch could introduce heavy distortion as even if the small sequence is included in the large sequence, it is not clear that the cosine similarity of their embedding would be similar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)