Uncovering Latent Memories in Large Language Models
Authors: Sunny Duan
Mikail Khona
Abhiram Iyer
Rylan Schaeffer
Ila R Fiete
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Memorization
Empirical Study
Data Leakage
Privacy
LLMs
Dynamics
Interpretability
Mechanistic
Primary area: foundation or frontier models, including LLMs
TL;DR: We study "latent memorization" in AI models, showing that complex sequences can persist and be revealed later, even after a single encounter, and study how these latent memories can be recovered.

Abstract: Frontier AI systems are making transformative impacts across society, but such benefits are not without costs: models trained on web-scale datasets containing personal and private data raise profound concerns about data privacy and security. Language models are trained on extensive corpora including potentially sensitive or proprietary information, and the risk of data leakage, where the model response reveals pieces of such information, remains inadequately understood. Prior work has investigated that sequence complexity and the number of repetitions are the primary drivers of memorization. In this work, we examine the most vulnerable class of data: highly complex sequences that are presented only once during training. These sequences often contain the most sensitive information and pose considerable risk if memorized. By analyzing the progression of memorization for these sequences throughout training, we uncover a striking observation: many memorized sequences persist in the model's memory, exhibiting resistance to catastrophic forgetting even after just one encounter. Surprisingly, these sequences may not appear memorized immediately after their first exposure but can later be “uncovered” during training, even in the absence of subsequent exposures - a phenomenon we call "latent memorization." Latent memorization presents a serious challenge for data privacy, as sequences that seem hidden at the final checkpoint of a model may still be easily recoverable. We demonstrate how these hidden sequences can be revealed through random weight perturbations, and we introduce a diagnostic test based on cross-entropy loss to accurately identify latent memorized sequences.

Published as a conference paper at ICLR 2025
UNCOVERING LATENT MEMORIES IN LARGE LAN- GUAGE MODELS
Sunny Duan
Mikail Khona
Abhiram Iyer EECS MIT
Brain and Cognitive Sciences MIT sunnydemit.edu
Physics
MIT
mi Seailmitt.du
abi abiyer@mit.ede
Rylan Schaeffer
lla Rani Fiete
Computer Science
Brain and Cognitive Sciences
Stanford University
MIT
rschae fecseffcc... stanford.edi
Trete@mit.eedi
ABSTRACT
Frontier Al systems are making transformative impacts across society, but such benefits are not without costs: models trained on web-scale datasets containing personal and private data raise profound concerns about data privacy and misuse. Language models trained extensive corpora including potentially sensitive or proprietary information, and the risk of data leakage, where the model response reveals picces of such information, remains inadequately understood. Prior work has demonstrated that sequence complexity and the number of repetitions are the primary drivers of memorization. In this work, we examine the most vulnerable class of data: highly complex sequences that are presented only once during training. These sequences, often containing the most sensitive information, pose considerable risk if memorized. By analyzing the progression of memorization for these sequences throughout training, we uncover a striking observation: many memorized sequences persist in the model's 's memory, exhibiting resistance catastrophic forgetting even after just one encounter. Surprisingly, these sequences may not appear memorized immediately after their first exposure but later be "uncovered" during training, even in the absence of subsequent exposures phenomenon we call "latent memorization." Latent memorization presents serious challenge for data privacy, as sequences that seem hidden at the final checkpoint of model may still be easily recoverable. We demonstrate how these hidden sequences can be revealed through random weight perturbations, and we introduce a diagnostic test based on cross-entropy loss to accurately identify latent memorized sequences.
INTRODUCTION
Frontier AI models are trained on vast web-scale datasets (Touvron et al.
2023 Gemini Team et al.
2023
OpenAl al. 2023 Brown al. (020). The sizes these pretraining corpora enable fluency, knowledge about various domains (Alkhamissi et al. 2022 Guu et al. 2020). and the ability to
perform in-context learning (Brown al. (222).. However, these datasets often include proprietary.
copyrighted, or otherwise private information (Smith et al. 2023 Karamolegkou et al. 2023 Bordi
al. 2024 Duan et al. 2024] Staab al. | (hhi et al.) 2023) lang et al. 2023]
al. 2019), which is problematic because LLMs have been shown possess vast capacity for detailed memorization. Specifically, with appropriate prompting, LLMs can regurgitate verbatim text
from their training corpora.
Prior work has found that even sequences encountered early in training can be extracted from the model, long after they have been encountered (Biderman et al. (023b). One possible cause of this is that memorized sequences appear multiple times within the corpus, allowing the network to reinforce and store this data in its weights. Our findings confirm that repeated sequences constitute the majority

---

Published as a conference paper at ICLR 2025
of the memorized content. However, we also find many sequences which are encountered only once during training but are memorized by the model and persist in the model's 's memory throughout the training process. Many of these memories may are later recalled without additional exposure, indicating they remain encoded in the model's weights. These 'latent' memories present significant challenges, they not easily detected by current memorization metrics, raising the question of how to effectively identify and quantify memorized training data in large language models.
forgotten during certain stages of training but
1.1 CONTRIBUTIONS
This work provides significant insights into the dynamics and mechanics of memorization in language models during pretraining, contributing to the broader understanding of data privacy and security within machine learning. Our primary contributions are as follows:
Quantification of Memorization Susceptibility: We systematically evaluate how the statistical characteristics of training data, specifically sequence complexity and repetition, influence the likelihood of memorization in language models. Our findings demonstrate that the probability of memorizing a sequence scales logarithmically with its repetition in the training data well the complexity of the sequence under consideration. These results extends prior work characterizing which sequences become memorized (Prashanth et al. 2024 Tirumala et al. 2022).
Stationarity of Memorized Sequences: By analyzing how memorization changes through- out training, we discover that the memorization status of sequences remains largely stationary after initial exposure, despite not being re-encountered. We find that for many sequences, memorized sequences may disappear and re-appear in the model's output without repeated exposure. This indicates a fundamentally persistent property of the memory, revealing how the state of memorized sequences is preserved and how subsequent training only modifies the model output.
Latent Memorization and Recovery:
We identify the presence of "latent" memorized sequences, which are not evident at certain checkpoints but can be uncovered later in training or through controlled perturbations. Our experimental results show that adding random Gaussian noise to model parameters can recover these latent memorized sequences, supporting the hypothesis that further training acts random additive noise rather than fundamentally altering the memorization state.
Development of Diagnostic Test: memorized sequences by analyzing their cross-entropy loss. This test provides practical tool for detecting and mitigating potential data leakage in deployed language models.
We propose novel diagnostic test for uncovering latent
Our study underscores the risks associated with data leakage in language models, emphasizing the need for more robust mechanisms to ensure data privacy. The persistence of memorized sequences poses a challenge for the prevention of data leakage. By characterizing the nature of memorization as well as the nature of these latent memorized sequences, we elucidate possible mechanisms of how sequences become memorized and offer practical solutions for mitigating data privacy risks, and developing safer and more trustworthy models.
2
METHODOLOGY
2.1 PROPERTIES OF PRETRAINING DATA RELEVANT TO MEMORIZATION, REPEATS AND COMPLEXITY
Previous studies have identified that the number of repeats of sequence affects whether it will be memorized, with more frequently occurring strings being more likely to be memorized (Carlini et al. 2020: Razeghi et al. 2022 Biderman al. 2023a). Consequently, starting property to measure is the number of repeats of specific strings in the pretraining corpus.
In our work, we also consider second and newer property: the complexity of specific strings. Our decision to do so is motivated by previous studies (Carlini et al.) 2020) which identified a prevalent class of easily memorized data: simple sequences composed of repeated patterns, numbers, or other
2

---

Published as a conference paper at ICLR 2025
60
80
Levenshtein Distance
60
Amba-7b
800
Levenshtein distannc
-90
10
10 0
60
10
6
Lavenshtein
-6
18
LMWY
Pytrr- 1b
10
101
106
10²
104
Repests
z-complexity
Repeats
Training data
Lovensteain Dicamra
Sipt 108
10002
100302
134
10u
20000
114
poomppall
Figure 1: Data statistics and the probability of memorization Plot of average kI-LD function of the number times the sequence is repeated in the dataset for Pythia- and Amber-7b b. Average kl-LD function of the Z-complexity of the sequence. Relationship between kI-LD and repeats for different complexity levels. c. Schematic of pipeline for analyzing memorization over time. All samples were selected from early on in training. Various model checkpoints are selected and evaluated to determine if early training sequences are still memorized, The changes in k-LD over time are used in our analysis evaluate how the memorization of these sequences changes throughout training. d Distribution of z-complexity over all of the data.
straightforward patterns. While models readily learn these samples, they often lack substantive content and are unlikely to represent sensitive information. Thus, it is important distinguish between memorization of these trivial sequences from more complex and informative sequences.
In order to quantitatively measure the complexity of specific strings, we use modern compression algorithms to determine the extent to which sequences have smaller description than the original sequence. To calculate the complexity of sequence we define a metric, z-complexity, which the ratio between the compressed sequence length to the original sequence length. This metric contains values from 0 to 1 and is efficiently computable using the z ib package in Python.
2.2 QUANTIFYING MEMORIZATION
Several different definitions have been put forward quantify memorization in language models. Intuitively, a memorized sequence is a training sequence which can be reproduced given the right context (Carlini al. (2022) Schwarzschild et al. 2024). One popular definition of memorization is fl-memorization (Carlini et al.) 2022). al-memorization is evaluated by considering sequence of length k + L. The first k tokens are presented to the model as context. The model is used generate a continuation of length via greedy (i.e., temperature = O decoding). The model's continuation is compared to the "true" continuation, and sequence said be kl memorized if the model's output exactly matches the true continuation.
While undoubtedly useful, we introduce and utilize a new metric for measuring memorization. kl-memorization is an overly strict such that even a single-token deviation from the true continuation may cause us to misclassify a sequence as forgotten;
many cases, the model may make small errors such inserting or modifying single token. We identified and provide several examples in Table []] order to to more robust to small changes in the learned sequence, we propose modification of kl-memorization by introducing kl-Levenshtein distance (kl-LD).
Definition 2.1 (kl-LD distance), Let S = ($), $2.... ) be sequence of tokens. We denote the first k tokens as the context C (81...., 8k) and the last tokens as the target T (8k+ 1 8k+1). The
3

---

Published as a conference paper at ICLR 2025
model is provided these context tokens and produces predicted continuation of of (m1.-... ................................ mi)-
We define the kI-LD distance as the Levenshtein distance (Levenshtein) 1965) between sequences and 7 where the Levenshtein distance is the minimum number of (token) insertions, deletions or substitutions that must be performed on to obtain f.
We find that this natural measure of memorization which also provides range of values to provide more granular insight into the strength of the model's memory, akin to how continuous metrics have been used to improve evaluations of language models (Schaeffer et al. [023 2024).
Throughout this study, we set k = 32 and compare the continuation of the model with the original sequence by computing the Levenshtein distance between the next 64 tokens.
Table 1: Model continuations at various stages in training for a few selected sequences which were complex and encountered only once during training. Minimum edits are highlighted such that character edits are highlighted in orange, deletions are highlighted in red and new characters are highlighted in green.
Continuation cubotiu> Goestl 1992: En la mayocAxada de cannles Ubuntu glASs. Para ayuda EspaAmol, por favor entre los canales fuuuntu es o Akubuntu <BluesKajo/es
Checkpoint
Checkpoini
Checkpoint
9925 por laver ayw donne para instaar DBDesigner <him> hay <BBlueKaajoles 7995
<ubotru> Guest//952 En
<ubonus Guestl La la mayord de canales de Ubuntu habla sAAto glARs.
cubotta>> Guest 1992: Li In mayork dellil canales sãloo de Ubuntu, en inglÄEs. Si busca syuda en espaAzolaumea
la mayarA de canales Ubuntu
Blss
inglåOs busea ayuda
ayuda espaAtol a charlar catra ea el canal vubuntu es. Escribe "fjoin tubuniu ex"
Espa Azoloper fovaremet soveammes fubuniueesa Mkubumm <Bilestiajgt
Rnmurmenn canal dubunce expescriba
dubunca-
es
findings, beliefs, or et- perienzes on those products.
claim, statistic, quote or other representation about prud- uci
claim, statistic. quote other representation about product
chim,, statistic, quote other represemation about
claim, statistic. quete
other representation about
should verified with mamifa--
product service should
product
should be
and opinions expressed un Cate CaeTheeOayycom are purely own. Any product
the marufae- fae- tuee, provider or party in question. CaiTTheOkay.com contain any consent which might present a conflict of inserest.
verifled with manufac-
verified with
turenee
turewur provide pons
turense provider parpan
Eline question.
TheeThokkyyooo
pharmacns, reptstered
leucher and
a endoesed bssuut spunsores
diatmnan uninot
mucher Stnttaa
byrnter Chorseaate Compang
and she sondent
CuretheOkas
dientim registered
She
altitined endorsed
2.2.1 ANALYZING REPEATED SAMPLES
In this study, we seek to understand both how repeated encounters of a sequence during training drives memorization and also how sequences which are encountered only once are retained by the model. To this end, training. In our study, we focus on the / portion of the sequence. For this study. we fixed to 64 tokens. Given a target sequence, we compare the target sequence with all of the training sequences which were presented to the model during the period of training under consideration. We compute the largest subsequence match between the target and every individual training example and call training example a "repeat" if there was a sub-sequence match of length 30 or longer.
analyze where training sequences
repeated throughout the course of
2.3 LANGUAGE MODEL.S
In this study, we largely focused on the Pythia IB language model (Biderman et al. 2023a), which was trained on 300B tokens from the Pile (Gao et al. 2020 For selected experiments, to ensure results hold on other language models, we reproduced our results using a second model, Amber-7B (Liu et al. 2023). We selected these two models as they were large, high performing models complete with fully reproducible data sequences and frequent checkpoints. As in previous works (Biderman) al. 2023a), all experiments were run with the models run with half precision (f (float16 and temperature O.
2.4 DATASETS
In this work, we use the deduplicated versions of the Pile (Gao et al. 2020) as well as the Amber dataset which is combination of the RedPajama V1, RefinedWeb and CtarCoderData datasets (Computer) 2023 Li et al.) 2023) Penedo et al., (003), all of which employ deduplication. In the early part of our work also employ the standard Pile which did not use de-duplication in order to
4

---

Published as a conference paper at ICLR 2025
observe the effects of repeated exposure memorization. In the latter part of our work we focus on the deduplicated versions of the datasets in order to eliminate the influence of repeated exposure on our analysis of memorization.
2.4.1 LANGUAGE MODEL CHECKPOINTS
In our analysis, we used checkpoints from every 3k training steps between from step 10k-43k in Pythia- 1B and every 10 checkpoints of Amber-7B, corresponding to roughly 1.7 million training examples between revision 100 to 350. These selections were checkpoints from each model which represented a sizable portion of training. These were chosen to be offset from the beginning of training to avoid artifacts initial transients from random initialization, learning rate warmup and other peculiarities from initial phases of training.
3 EXPERIMENTAL RESULTS
3.1 DATA STATISTICS PREDICT MEMORIZATION
We analyze two primary drivers of memorization during training: sequence complexity, and the number of repetitions. Previous work showed that the probability a training string can be extracted from model is related to the model size and number of repetitions (Carlini et al. (2020); find that this relationship is true in the models we analyzed as well (Figure (b.. Additionally, we found that the complexity of the string itself was strong predictor of whether would be memorized (Figure [p): strings with smaller z-complexity had smaller kl- kl-Levenshtein distance (kI-LD), meaning simpler strings are more easily memorized. Interestingly, recent work showed that pretraining language models on data with lower z-complexity causes the training losses to decrease more rapidly (Pandey 2024); our results here suggest an explanatory mechanism: with more compressible data, the model can memorize the data more quickly. Furthermore, we found that for strings of different complexity exhibited different memorization curves (Figure (.). whereby lower complexity strings were more casily memorized with fewer repeats. Both of these factors influenced the memorization probability with a log-linear relationship. Since highly sensitive information is likely contained in complex and rare training sequences, focus our efforts these sequences. In the rest of this work, we restrict our analys to sequences which are presented once and have high complexity (> 0.8).
3.2 DYNAMICS OF MEMORIZATION
In order to produce a more complete picture of how successive training affects the state of memorized sequences within our model, we analyze how the kl-LD changes throughout the course of training for individual sequences. We select sequences early on in training and evaluate how the memorization status of these sequences evolves throughout training (Figure [a). In this section we utilize the deduplicated version of the Pile dataset (Gao et al., 2020) as well as the Amber LLM360 dataset which also uses deduplication in order to remove the effects of repeated exposures. We filter these sequences so that only sequences which have a z-complexity of 0.8 or higher are included in our analysis. Additionally, we employed our own duplication detection scheme which eliminated sequences which had a sub-sequence match of length 30 or longer and rerun our analysis using these sequences (Figure $(3).
Surprisingly, we find that the memorization status of a sequence is largely stationary throughout training. After the initial checkpoint, the kI-LD of the sequences fluctuate (Figure 3 but do in a way which is stationary across training (Figure 2 This is consistent across both Pythia-Ib and Amber-7b models. This is reflected in the individual trajectories, and also in the overall mean of the population which shows no clear trend as training progresses. Furthermore, unlike random walk, we see that the variance of the does not grow over time, but remains fixed. We can quantify this by running a variance ratio test A.2 where the variance of random walk is expected to grow linearly. We can reject the null hypothesis that our data is generated from a random walk with p < 10- for the samples drawn from Pythia-lb and Amber-7b. This is indicative of mean reversion tendency of the dynamics and demonstrate the stability of the memories within the model weights. Additionally, symmetric (Figure
observe that the changes in the kl-LD between consecutive checkpoints 2b) and roughly follow a laplace distribution (Figure 33. This again confirms the counter-intuite property of sequences to become memorized as often they forgotten. Notably, the model is able
5

---

Published as a conference paper at ICLR 2025
Sample Trajectories (Pythia-1b)
TW:m1
Training
Memorized Sequence
Un-memorized Sequence
1000
Chackpoint
2903
Sample Trajectories (Amber-7b)
Levenshtein distance from true continuation
Checkpaaee 13000
16
40
00
Checkpoint 19300
Checkpoint number
Figure 2: a. Visualization of individual samples and the change in the memorized length during training. Grey lines are subsampled single sequence trajectories throughout training. Each sequence was normalized such that the distribution of memorization lengths was mean O and variance 1. The red line denotes the mean and shaded area denotes region of standard deviations of the ki-LD of all sequences at single point in time. Notably, the distribution at each timestep is the same for all checkpoints. This is in contrast to both the expected exponential decay behavior exhibited by models which experience catastrophic forgetting as well as the linear growth of variance which is expected of processes exhibiting random walk behavior. b. Conceptual schematic of how memorized sequences may be stabilized during training in order resist the interference from weight changes caused by subsequent training. c. Joint distribution kI-LD during checkpoints 10k and 13k. Color is the log of the number of sequences in each bin. The vast majority of sequences are not memorized in either checkpoint.
to recall memories which, at one point time, appeared to be forgotten, despite never encountering that sequence again.
The stationarity of the memorization status of these sequences indicates that the memorized sequence is stable throughout time, but this is in conflict with the fact that the model weights are constantly evolving. This stability in the presence of noise indicative the encoding of the sequence memory is preserved by some restorative process illustrated in Figure 2b where the memorized sequence becomes a stable fixed point in the weight space of the model under training dynamics. Since this is true of all sequences, but only the few which exhibit this persistent memorization, it may point to a phase transition that occurs when the sequence is first encountered.
stabilizing mechanism by which
3.3 LATENT MEMORIZATION AND RECOVERY OF LATENT MEMORIES
Since some sequences exhibited seemingly random variations in their memorization state across different checkpoints, we hypothesize that these sequences remain memorized but not be visible at a given checkpoint and are "latent" memorized. Indeed, we found many sequences which were not memorized at the initial checkpoint (10k) but exhibited memorization by checkpoint 19k (Table I..
For these sequences, the nature of the random changes shown in Figure 2 indicate the form of random walk. We hypothesize that the process of training in frontier Al models acts as random noise on the weights with respect to the memory of the sequence. Thus, simply perturbing the weights with random noise should produce similar effects training.

---

Published as a conference paper at ICLR 2025
Pythia-1B
10k to 13k
13k to 16k
16k to 19k
19k 22k
22k to 25k
25k to 28k
28k to 31k
10*
10a
10a
10a
10a
103
10a
Count
31k to 34k
34k to 37k
37k to 40k
40k to 43k
10
10
10
10a
Levenshtein Distance
Amber-7B
100 to 110
110 to 120
120 to 130
130 to 140
140 to 150
150 to 160
160 to 170
10
10a
10
109
10*
10%
50
-50
170 to 180
180 to 190
190 to 200
200 210
210 220
220 230
230 to 240
10
Count
10a
107
10
10a4
10'
10a-
60
50
-50
.50
0
50
to 250
250 to
260
270
290
to 310
10
10
10a
10
10a
10a
10a
-50
0
50
-50
50
-50
50
-50
50
500
50
-50
50
-50
0
50
310
320
320
340
350
10
10a
-50
50
-50
50
-50
50
-80
0
50
Levenshtein Distance
Figure 3: Visualization of how memorization status, measured by kI-LD, changes between consecu- tive checkpoints for both Pythia-IB IB and Amber-7B.
We find that this prediction is true. We randomly perturb the model weights by adding small amount of random gaussian noise (of magnitude 2 x 10-3) to each of the weight parameters. We repeat this process 200 times and find the perturbation which yields the lowest ki-LD (h). Notably, in the high dimensional weight space. it is difficult to reproduce arbitrary sequences using random weight perturbations, thus the recovery of memorized sequences must be due intrinsic factors of how the memory is encoded in the weights.
We find that sequences which were "latent" memorized able be recovered using random perturbation (Figure 4pc). In contrast, sequences which were not memorized during the period of consideration could not be recovered. As control, we also selected sequences which were not presented to the model yet, and observed that their distributions closely matched those which were encountered by not memorized by the model (Figure 4D). Furthermore, we found that the perturbations yiclded memorization patterns which closely matched that of the model at later point in training. These observations support the view that with respect to a memorized sequence, subsequent training acts similar to random noise perturbations the model weights.
As an additional control, we attempted to recover the latent memorized sequences by sampling from the model different temperatures. Since these sequences stored in the model weights, it may be the case that the model would simply reproduce the target sequences prompted enough times. We tried four different temperatures and sampled 200 different sequence continuations from each
7

---

Published as a conference paper at ICLR 2025
Schematic of perturbation process
ki-LD after perlurbation
Model weights
condent
true continuttion
Computtc méninu
Generate eroemble of perturbed models
Add gauasion nokme
sample continuatona
Eacower ranii
Perturbations of 10k and 19k
dhockpolnt 19: best parturt
Count
30
chockpolit 10x best perturb 120k)
10
Levensttaii
20
16
60
60
Levenshsaln Distance
Figure 4: a. Schematic how model weights are perturbed order to extract memories from the model. The same context is given 200 perturbed modesl and the best continuation is chosen. b. Comparison of the distribution of best achievable kI-LD by perturbing the model weights. Data points were selected such that they were un-memorized (kl-LD 50) at 10k but (kI-LD < 10) at some point during the next 10k training steps. Top panel is the histogram of the perturbations of the checkpoint at 19k and bottom 10k. Notably, the perturbations cause the 10k model distances to match the distribution of the 19k model, and perturbing the 19k model does have significant effect. This is indicative of how model training mimics random noise with respect to the memorization status of the sequences. Visualization of the Levenshtein distances from the target for various weight perturbations. Each row single sequence, and the heights of the bars correspond to the number of perturbations which resulted in Levenshtein distance of that corresponding bin.
memorized
of the temperatures. We find that the this method fails to recover the latent memories that weight perturbation was able to produce (Figure(5h) 5p).
"Latent" memorized sequences pose significant risk for leakage since they are not easily detectable from evaluating kl-memorization of those sequences. To this end, we discovered that these "latent" memorized sequences had significantly lower cross entropy loss when evaluated by the model (Figure 5b), thus simply evaluating the likelihood those sequences using the trained model is a natural diagnostic for detecting these "latent" memorized sequences.
3.4 RELATED WORK
Extracting memorized sequences from language models an area of high interest. Early work established that it was possible to extract sensitive data including phone numbers. URLs and personal information from trained language models (Carlini et al. 2020). Other studies injected canaries to determine what aspects of the training process contributed to whether sequence is extractable
(Henderson et al. 2017) 7 Thakkar et al. 2020). More recent work have extended this to investigate how these properties scale with model size and data statistics CCrlinii et al.) 2022). This has motivated the use of deduplication, which in addition to reducing the chance of data leakage (Kandpal et al. 2022). also has been shown to improve sample efficiency and improve evaluation (Lee al. 2021
8

---

Published as a conference paper at ICLR 2025
8
Sequence type comparison
Sampling baseline
Cross-entropy diagnostic
snleri-maatorrree sequences
Original
125
Load flowsood
Temperature
80
Temperature - 0.5
100
Uh-nomolbaee soquonoes
60
Temperature
Temperature 1.0
76
20
Unsson secuarncea
20
23
0-
10
20
30
40
50
Cross Entnpy Loss.
Figure 5: a. Comparison of using perturbations to evoke a target sequence for three different classes of sequences. In the top panel, we examine the sequences which are "latent" memorized. In the middle panel, we find sequences which weren't 't memorized during training and in the bottom panel, we analyze sequences which were encountered later in training but were not encountered by the model. We not that perturbing the weights is only able to evoke sequences which "latent" memorized. b. Attempts at evoking latent memories using 200 samples at various temperatures. None of the temperatures were able reliably recover the latent memories. Comparison of the cross entropy losses of sequences separated into the three different classes of sequences analyzed in b. The cross entropy losses of "latent" memorized sequences are much lower.
The definition of memorization is also still debated and various approaches to quantifying memoriza- tion have been made (Zhang et al. 2021 Feldman & Zhang 2020). A variety of attacks have been
designed to extract memorized sequences using designed prompts (Thakaar et al. 2020) and model
activation perturbations (Kassem et al., 2024).
More generally, the notion of membership inference has been studied as a way to determine whether given training example was part of the corpus (Shokri et al., 2016 Mireshghallah et al., 2022 Hisamoto et al., 2019), and these approaches have been applied to language models as well Duan al. (024)).
Forgetting has also been studied extensively in neural networks, typically in the context of preventing forgetting. (Kirkpatrick al. 20177 Zenke al. 2017 Chen et al. 2020). Studies have also shown that forgetting decreases with model size (Tirumala et al.) [022]] Mirzadeh et al.) [202T). This work has also been examined in the context of understanding what aspects about model and the data
contribute to forgetting (Toneva et al. 2018)
Finally, there has also been work studying how the training process affects the status of memorization (Tirumala et al. 20222] Prashanth et al. 2024). This work focuses on how parameters of training and size the model affect the dynamics of training. They find that scaling the model generally leads to less forgetting. In our work, we focus on sequences which counter-intuitively do not obey the forgetting laws presented in this work and expanding on the implications of these persistent "episodic" memories.
CONCLUSION AND LIMITATIONS
We study how memorization changes throughout training and focused on sequences which occurred only once throughout training. Under these conditions, we find that rather than forgetting these sequences, the model retains them for the duration of training. This stationarity indicates stability of the memorized sequence in weight space since the training process necessarily modifies the weights which encode the memorized sequences. We test this mechanistic view of how the training process interacts with the memorized sequence by using random weight perturbations to the model weights. These perturbations confirm that sequences which appeared to be forgotten at one point during training, may still be memorized by the model and able to be uncovered with small amount of random noise. We concluded by demonstrating a simple diagnostic to distinguish between "latent" memorized sequences and un-memorized sequences.

---

Published as a conference paper at ICLR 2025
This study highlights one surprising behavior of frontier Al models and begins to uncover what mechanisms are present in the memorization behavior of these models. Our work suggest possible mechanism of how memorized strings are sustained throughout training and further experiments needed to confirm the underlying mechanism. Notably, further testing is required across other frontier Al models which were not considered here. Finally, propose mechanistic explanation for this phenomenon which requires further study to explain the cause of these persistent memories.
5 REPRODUCIBILITY STATEMENT
All code used for this project is available at Estent_menoorization
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. A review on language models knowledge bases. arXiv CL), April 2022.
Stella Biderman, Usvsn Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. arXiv [cs. (I.) April 2023a.
Stella Biderman, Usvsn Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin G Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. Adv. Neural Inf. Process. Syst., wwwp/230..111555 April 2023b.
Sebastian Bordt, Harsha Nori, Vanessa Rodrigues, Besmira Nushi, and Rich Caruana. Elephants never forget: Memorization and learning of tabular data in large language models. arXiv [cs.LG], April 2024.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter. Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv [cs.CL], May 2020.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Vlossert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. arXiv [cs. CRJ, December 2020.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv [cs. February 2022.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. arXiv [cs. (cs.CL), April 2020.
Together Computer. RedPajama: an open dataset for training large language models, October 2023.
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi. Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? arXiv [cs. CL.] February 2024.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. arXiv [cs. August 2020.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800GB dataset of diverse text for language modeling. ArXiv, abs/2101.00027. December 2020.
Gemini Team. Rohan Anil, Sebastian Borgeaud, Jean- Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Jack Krawezyk, Cosmo Du, Ed Chi, Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manaal Faruqui, Aliaksei Severyn, Hanzhao Lin, Yaguang Li, Yong Cheng, Abe Ittycheriah, Mahdis Mahdich, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orban, Fabian Güra, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, Hyunjeong Choe, Ágoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd AI Merey, Martin Baeuml, Zhifeng Chen, Laurent El Shafey. Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas
11

---

Published as a conference paper at ICLR 2025
Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalic Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iñaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Xavier Garcia, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq lqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Ravi Addanki, Antoine Miech, Annie Louis, Denis Teplyashin, Geoff Brown, Elliot Catt, Jan Balaguer. Jackie Xiang. Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wos Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M R Amold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontack Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz. Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery. Dipanjan Das, Dominika Rogozińska, Vitaliy Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo. Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando. Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang. Shuo- Yiin Chang, Paul Komarek, Ross Mcllroy, Mario Lučié, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lene, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuchan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko linuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma. Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo. Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang. Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussemot, Livio Baldini Soares, Kate Baumli, Michael B Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye. Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Victor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojeiech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Salch, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran
12

---

Published as a conference paper at ICLR 2025
Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kern Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao. Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gautam Vasudevan, Chenxi Liu, Mainak Chain, Nivedita Melinkeri, Aaron Cohen, Venus Wang. Kristie Seymore, Sergey Zubkov, Rahul Goel, Summer Yue, Sai Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey. Jonah Joughin, Egor Filonov, Tomasz Kępa, Yomna Eldawy, Jiawern I. Lim, Rahul Rishi, Shirin Badiezadegan, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Subha Puttagunta, Kalpesh Krishma, Leslic Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurzrok, Shuntong Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragagnolo, Tej Toor, Tina Chen, Valentin Anklin, Feiran Wang. Richie Feng. Milad Gholami, Kevin Ling, Lijuan Liu, Jules Walter, Hamid Moghaddam, Arun Kishore. Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafeezul Rahman Mohammad, Josip Matak, Yadi Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Oana David, Ankur Garg. Yifan He, Oleksii Duzhyi, Anton Algmyr, Timothée Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhabrata Das, Zihang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Maurya, Luyan Chi, Sebastian Krause, Khalid Salama, Pam G Rabinovitch, Kumar Reddy M Pavan, Aarush Selvan, Mikhail Dektiarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akerlund, François-Xavier Aubet, Teryy Huang, Chen Zhu, Eric Zhu, Elico Teixiraa, Matthew Fritze, Francesco Bertolini, Liana-Eleonora Marinescu, Martin Bölle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejezuk, Dominik Rabiej, Vipul Ranjan, Krzysztof Styre, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Mirylenka, Chen Zhou, Obaid Sarvana, Abhimanyu Goyal, Samuel Andermatt, Patrick Siegler, Ben Horn, Assaf Israel, Francesco Pongetti, Chih-Wei "louis" Chen, Marco Selvatici, Pedro Silva, Kathie Wang, Jackson Tolins, Kelvin Guu, Roey Yogev, Xiaochen Cai, Alessandro Agostini, Maulik Shah, Hung Nguyen, Noah Ó Donnaile, Sébastien Pereira, Linda Friso, Adam Stambler, Adam Kurzrok. Chenkai Kuang. Yan Romanikhin, Mark Geller, Z Yan, Kane Jang, CCeng-Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzhi Shi, Jasjot Singh, Chris Hidey, Niharika Ahuja, Pranab Saxena, Dan Dooley, Srividya Pranavi Potharaju, Eileen O'llill, Anand Gokulchandran, Ryan Foley, Kai Zhao, Mike Dusenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safranek-Shradee Andrew Goodman, Joshua Kessinger, Eran Globen, Prateck Kolhar, Chris Gorgolewski, Ali Ibrahim. Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preethi Lahoti, Cip Bactu, Ali Ghorbani, Charles Chen, Andy Crawford, Shalini Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-l Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccold Dal Santo, Siddharth Goyal, Jitesh Punjabi, Karthik Kappaganthu, Chester Kwak, L V Pallavi, Sarmishta Velury, Himadri Choudhury, Jamie Hall, Premal Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chintu Kumar, Thomas Jurdi, Sharat Chikkerur, Yenai Ma, Adams Yu, Soo Kwak, Victor Åhdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanjie Zhou, Mehrdad Khatir, Charles Sutton, Wojeiech Rzadkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung. Kai Yang. Nihal Balani, Arthur Bražinskas, Andrei Sozanschi. Matthew Hayes, Héctor Fernández Alcalde, Peter Makarov, Will Chen, Antonio Stella. Lisclotte Snijders, Michael Mandl, Ante Kärrman, Pawel Nowak, Xinyi WU, Alex Dyck, Krishnan Vaidyanathan, R Raghavender, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushil Mittal, Akhil Udathu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaleldin Elsayed, Elnaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cideron, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Dylan Scandinaro, Heinrich Jiang, Jasper Snoek, Mukund Sundararajan,
13

---

Published as a conference paper at ICLR 2025
Xuezhi Wang, Zack Ontiveros, Itay Karo, Jeremy Cole, Vinu Rajashekhar. Lara Tumeh, Eyal Ben- David, Rishub Jain, Jonathan Uesato, Romina Datta, Oskar Bunyan. Shimu Wu, John Zhang. Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez. Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong. Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving. Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikolaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Lc, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez. Andrey Khorlin, Albert Cui, L N Tian, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Perng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang. Jeff Piper. Nathan Ie, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng. Katherine Lee, Shuyuan Zhang, Somer Greene. Duc Dung Nguyen, Paula Kurylowicz. Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal. Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller. David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulanova, Rémi Leblond, Shirley Chung, Harry Askham, Luis C Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Daniel J Mankowitz, Alex Polozov, Victoria Krakovna, Sasha Brown, Mohammadhosseinn Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Matthieu Geist, Ser Tan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Kathy Wu, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao. Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sammiya, Evgeny Gladchenko, Neje Trdin, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue. Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie Cosmo Du Chloe Thortoo,, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Yeongil Ko, Laura Knight, Amélic Héliou, Ning Niu, Shane Gu, Chenxi Pang. Yeqing Li, Nir Levine, Ariel Stolovich, Rebeca Aitammria-Fermandeer
Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Charlic Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice Olivier Bachem, Sho Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A Choquette-Chnee Choo, Yunjie Li. T Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins. Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre- Chimoto, Hanna Siimccaa--Plciisste
David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahil Dua, Dong Li,, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty. Yi Sun. Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas. Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver
14

---

Published as a conference paper at ICLR 2025
Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang. Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, Xianghai Sheng, Emily Xue, Sherjil Ozair, Christof Angermueller, Xiaowei Li, Anoop Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, M K Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Ken Durden, Harsh Mchta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hechtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Mikulik, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Aroma Mahendru, Andrea Hu, Joshua Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kaku, Jigar Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Tejvi M., Xinyun Chen, Aida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Amruta Muthal, Senaka Buthpitiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avigail Dabush, Nan-Jiang Jiang. Harshal Godhia, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang. Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vít Listík, Mathias Carlen, Jan van de Kerkhof, Marcin Pikus, Krunoslav Zaher, Paul Müller, Sasha Zykova, Richard Stefanec, Vitaly Gatsko, Christoph Hirnschal,, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Dhandhamia. Manish Katyal, Akshay Gupta, Atharva Parulekar, Divya Pitta, Jing Zhao, Vivaan Bhatia, Yashodha Bhavnani. Omar Alhadlag. Xiaolin Li, Peter Danenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipso Ghosh, Ben Limonchik, Bhargava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Ianna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majmundar, Michael Alverson, Michael Kucharski, Mohak Patel, Mudit Jain, Maksim Zabelin, Paolo Pelagatti, Rohan Kohli, Saurabh Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandruni, Xiangkai Zeng. Ben Bariach, Laura Weidinger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Sheleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hassabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models. arXiv CL.] December 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming- Wei Chang. REALM: Retrieval- augmented language model pre- pre-training. arXiv [cs. CLJ, February 2020.
Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontin Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. Ethical challenges in data-driven dialogue systems. arXiv [cs. [cs.CL], November 2017.
Sorami Hisamoto, Matt Post, and Kevin Duh. Membership inference attacks sequence-to-sequenc models: Is my data in your machine translation system? arXiv [cs.LG], April 2019.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. arXiv CR), February 2022.
Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Sogaard. Copyright violations and large language models. arXiv, October 2023.
Aly M. Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, and Santu Rana. Alpaca against vicuna: Using LLMs to uncover memorization of LLMs. arXiv CL). March 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabska-Barwinska Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. U. S. A., 114(13):521-3326 March 2017.
Katherine Lee, Daphne Tppolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv [cs.CL] July 2021.
Vladimir Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Sovier physics. Doklady, 10:707-710, 1965.
15

---

Published as a conference paper at ICLR 2025
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zehaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang. Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf. Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dovan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder: may the source be with you! arXiv [cs. (cs,CL), May 2023.
Zhengzhong I. Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang. Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei Hc, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric Xing. LLM360: Towards fully transparent open-source LLMs. arXiv [cs.CL], December 2023.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Kirkkaikka,,,,,, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. arXiv [cs.LG], March 2022.
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. arXiv [cs. October 2021.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine. Gabriel Bermadett-
Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage. Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Alty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez. Ryan Lowe, Patricia Lue, Anna Makanju. Kim Malfacini. Sam Manning, Todor Markov, Yaniv Markovski. Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reicchro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H Pong, Tolly Powell, Alethea Power, Boris Power,
16

---

Published as a conference paper at ICLR 2025
Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond. Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong. Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. arXiv CLJ, March 2023.
Rohan Pandey. gzip predicts data-dependent scaling laws, 2024.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for falcon LLM: Outperforming curated corpora with web data, and web data only. arXiv [cs. (cs.CL), June 2023.
Usvsn Sai Prashanth, Alvin Deng, Kyle O'Brien, Jyothir S, V, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A Choquette-Choo Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, Katherine Lee, and Naomi Saphra. Recite, reconstruct, recollect: Memorization in LMs as multifaceted phenomenon. June 2024.
Yasaman Razeghi, Robert L Logan V au2, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies few-shot reasoning, 2022.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In A Oh, T Naumann, A Globerson, K Saenko, M Hardt, and $ Levine (eds.), in Neural Information Processing Systems, volume 36, pp. 55565-55581. Curran Associates, Inc., 2023.
Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier Al models with scale remained elusive?, 2024.
Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C Lipton, and J Zico Kolter. Rethinking IIm memorization through the lens of adversarial compression. arXiv preprint arXiv:2404. 15146, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv [cs.CL], October 2023.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. arXiv (cs. CRs. October 2016.
Victoria Smith, Ali Shahin Shamsabadi, Carolyn Ashurst, and Adrian Weller. Identifying and mitigating privacy risks stemming from language models: A survey. arXiv [cs. (Cs, September 2023.
Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev. Beyond memorization: Violating privacy via inference with large language models. arXiv [cs.Al], October 2023.
Ruixiang Tang, Gord Lueck, Rodolfo Quispe, Huseyin A Inan, Janardhan Kulkamn,, and Xia Hu. Assessing privacy risks in language models: A case study summarization tasks. arXiv [cs. CLJ, October 2023.
Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unintended memorization in federated learning. arXiv [es.LG], June 2020.
17

---

Published as a conference paper at ICLR 2025
Kushal Tirumala, Aram H Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics large language models. arXiv (cs.CL), May 2022.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. arXiv [Cs, December 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. ArXiv, abs/2302.13971. February 2023.
Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. arXiv [cs.LG], December 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proc Mach Learn Res, 70:3987-3995, 2017.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv [cs.CL], December 2021.
A APPENDIX / SUPPLEMENTAL MATERIAL
A.1 COMPUTE DETAILS
All experiments were run on cluster with access to 16 concurrent a100 GPUs. All of the language models were run using a single GPU and multiple GPUs were used parallelize the experiments order to speed up progress. Searching for repeats within the dataset was performed using the library dask, using 64 CPUs distributed in cluster, each with 32Gb of R.A.M
A.2 VARIANCE RATIO TEST
Random walks have hallmark property of linearly increasing variance over time. We can demonstrate statistically that the sequence of memorization lengths does not follow random walk by conducting variance ratio test. Given sample {{X,j) sism)) hisisn of sequences of length we can calculate the F-statistic by taking the ratio of the variances
ΣΣ m
(Xin Xin)²
Σm (X, X,1)²
which, for a random walk has an F-distribution with and m degrees of freedom.
A.3 LICENSES
This project used code from the Pythia project Biderman al. (2023a) released by EleutherAl under the Apache license version 2.0. We also used the Pile dataset Gao et al. (2020) which released under the MIT license. The Amber model was produced by LLM360, and the code and dataset are both released under Apache 2.0.
A. ADDITIONAL FIGURES
We include figures which were omitted from the main paper. These provide additional details that were not central to the claims made in the paper.
18

---

Published as a conference paper at ICLR 2025
Density edit distance samples
50
40
longest match
30
20
0
3
log(repeats)
Figure 6: Histogram of the repeats the edit distance Hue is log density.
Figure 7: Histogram of the repeats the edit distance split by complexity Hue is log density.
19

---

Published as a conference paper at ICLR 2025
18.0433 0.0099
10075 0 0000
-00
1012 0.19
10207 02224
10:307 6.3241
102 0:397
1004 0:017
10-22 0-68)
(0354 0.501
10332, 1.55
10. 10.007. 0 224
(0.674 6.00
10.307 6 7244
10
(074. 0.707
ahe mard tn
Sos
Cumdative
Variance
0200
(0.0403.
(3 (3.0736 0 0.08631
1750
10.107,
10,207. 0.2241
(0.174 6.19
1500
(0.4. 0.707
10 10.007,
complett
(0 0.777
17.776,
1000
(0.A) 0.6571
(D.T7) D.40
750
(0.574 0.50)
10,8C7, 0.524
10.727, 0.724
(0.74, 0.7571
Cumulative repcats
Figure 8: Average of the kI-LD metric kI-LD values are binned by number of repeats and complexity and the mean and variance of the samples in those bins are computed and colored.
20

---

Published as a conference paper at ICLR 2025
18.0433 0.0099
10075 0 0000
-00
1012 0119
10207 02224
10:307 6.3241
102 0:397
1004 0:017
10-22 0-68)
(0354 0.501
10332, 1.55
10. 10.007. 0 224)
(0.674 6.00
10.307 6 7244
10
(074. 0.707
d Commene ae aore ahe mard
Sos
Cumdative
Variance
0200
(0.0403.
(3 (3.0736 0 0.08631
1750
10.107,
10,207. 0.2241
(0.174 6.19
1500
(0.4. 0.707
10 10.007,
complett
(0 0.777
17.776,
1000
(0.A) 0.6571
(D.T7) D.40
750
(0.574 0.50)
10,8C7, 0.524
10.727, 0.724
(0.74, 0.7571
Cumulative repcats
Figure 9: Average of the kI-LD metric kI-LD values are binned by number of repeats and complexity and the mean and variance of the samples in those bins are computed and colored.
21

---

Published as a conference paper at ICLR 2025
Shaap Condinaation
Madel Costivvation
ectindern lcotin
paite
(ile Rassen attroberyy drett retu 4o4
worde prtteet oe pest Lene
The dil The pettoiue
cogges wneee day
lare tie bo
Tereee die de
pitto Sua cisaer)
you mounng
gette and Seeee 1 beste denterre
5.5
Igotior
COODEC ONTINICI
Voadrh 1oa orll Tee
sinquac drows
580 600
Figure 10: Examples of strings which were seen once during training. Top left plot shows the kl-LD over for different trajectories and bottom left plot is histogram of when the examples were repeated and at what length with the time on the x axis and the length of the repeat on the axis. The text of the context, true continuation and model continuation are shown as well.
22

---

Published as a conference paper at ICLR 2025
-0.012
11.00
1.011
5011
03
12000
13000
0.004
DUTTA
14000
6:41
15000
0.006
0.
18000
6241
CCB17
19000
6:31
COMI
00000
11000
13000
14000
16000
3,7000
30400
33000
Checkpoint
Figure 11: Distribution of weight changes of the model throughout training. Computed as the L2 distance between the flattened model weights at two different times during training
Model weight distribution (Ckpt 10000)
105
10a
10a
Count
102
101
10a
10
10
10
10
10a
Weight magnitude
Figure 12: Distribution weight magnitudes of the trained model at checkpoint 10k
14a
Figure 13: Dynamics of sequences with high complexity (>0.7) filtered by using maximum substring match of 30
23


Reviews:
[{"review_id": "t6c4GkKQTM", "reviewer": "Reviewer_qyKz", "summary": "The paper studies the mechanics of memorization (as defined currently in LLM literature) in depth along two axes: number of repeats of a string during pretraining and complexity of string and largely focuses on high complexity strings that are exposed only once and thus, based on existing evidence, are less likely to be memorized. Based on this investigation the paper finds that (1) memorization of such strings is rather stable during training (ie there's no clear trend of catastrophic forgetting) and that (2) there is evidence of \"latent\" memorization, ie, sequences which the model has seen only once and seem to not be memorized, can be surfaced using simple modifications such as adding random noise to model weights. Since such latent memorization can go unnoticed by existing measures, the paper also shows that a simple evaluation of cross entropy loss on such sequences clearly shows lower loss for latent memorized sequences and could be a simple diagnostic to measure latent memorization.", "questions": "- [Data statistics and memorization] I am a little confused by Section 3.1. You say \"simpler strings are more easily memorized\" -- but this could very well be an artifact of the checkpoints you use. Since you pick checkpoints not exactly at the beginning, I would suspect simpler kinds of in-context learning to be at play here. If strings are highly compressible (ie simpler on your z-complexity scale) I would imagine based on the \"k\" part of the prompt one can infer the \"l\" part without necessarily memorization. Secondly, your paper later on shows that this kl-LD definition of memorization can miss other kinds of memorization like latent memorization. So taking latent memorization into account, is the statement that \"simpler strings are more easily memorized\" still true?\n\n - [Stationarity of memorization] Maybe I missed this, but could you say how many tokens were seen in the window you consider for the stationarity analysis in eg Fig 2a? I wonder if stationarity is a function of how many tokens it sees in the considered window -- because surely a once-encountered string in training should have some limited shelf life? How many tokens does it take for the capacity in the model to run out to retain this latently memorized string? \n\n - [Mechanics of memorization] The intuition that the somewhat strange mechanics of memorization are related to weights is really intriguing to me. However, I wonder if the statement \"but this is in conflict with the fact that the model weights are constantly evolving\" is super convincing. Let's run with your hypothesis that strings are encoded in model weights, then to memorize a single 64 character string you'd need 32 parameters (assuming again the float16 setting in the paper and no redundancy in encoding the string in weights) out of more than a billion parameters overall. The chances that you are changing such a small fraction of weights significantly is, I think, rather small. Not absolutely necessary for this work but maybe future work should look at how weights change when a string goes from memorized to non-memorized (as per the kl-LD definition).\n\n - [Mechanics of memorization] since the memorized sequence is only encountered once, I would suspect when it's encountered also plays a role in these dynamics. You consider checkpoints that are still fairly early on in training (compared to more recent models like llama/mistral which train for much longer), what would happen if this encounter happened much later on when training has matured somewhat? If we are to run with your hypothesis that this memorization is encoded in the model weights, then I would suspect as you overtrain models, the capacity to keep such rare and hard strings latently memorized would decrease. Again perhaps something to look at in future work.\n\n - [Recovering latent memorized strings] I really liked the idea of weight perturbations, however I wonder if you could also pull another lever here, which is the \"k\" part of the context given to the model. That is, it would be very cool to know if perturbing the weights is the only way to get the model to surface up these latent memorized strings and whether some kind of prompt hacking couldn't get you the desired outcome? If indeed no amount of prompt hacking can get you to the \"l\" part of the string then I think this will be much stronger evidence for your hypothesis of these strings being encoded in the weights.\n\n - [General comment] Your proposed diagnostic of latent memorization based on CE loss could also be extended for \"standard\" memorization, right? Is there any reason you would see to prefer the kl-LD definition?\n\n - [General comment] An overall comment I have about the paper is that you *could* call this entire paper a critique of existing defintions of memorization. Ultimately \"latent\" memorization is also memorization and perhaps we should expand our current definition of memorization to also take into account things one could get the model to output given *any* modifications (eg the weight modifications you propose). Any reason where you think the kl-LD definition for memorization is still the \"better\" definition?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-  The paper studies the very important problem of memorization, is well written and adds a fresh new perspective to this space. \n -  I appreciate that the authors use carefully constructed metrics to study this rather nuanced phenomenon. I appreciate the relaxation of Carlini et al's definition using Levenshtein distance since such a measure allows for careful analysis of what factors contribute and to what degree towards memorization.\n - It was also quite nice to see complexity of string being considered a a factor. There's always the question with compressible strings whether reconstructing them given context is memorization eg: given context \"aabbaabb\" if one were to logically extrapolate, \"aa\" would likely be the next two characters. At that point I'm not sure if calling it memorization is the right thing. So I appreciate that the authors focused on strings where such extrapolations are almost impossible.\n - The result with \"latent\" memorization is really neat and I really liked the mechanistic approach to arrive at the method of adding noise to weights to surface the latent memorized strings. This result also has many practical implications, eg, someone taking an off the shelf base model and then finetuning the model is running the risk of having a model that can regurgitate information that was exposed during pretraining -- something the entity finetuning has likely no control over.\n - The diagnostic for latent memorization is also very neat and intuitive, I wonder if cross entropy loss should be used for measuring all kinds of memorization, since ultimately \"latent\" memorization is also memorization.\n - Overall I really enjoyed reading this paper. I think it provides a fresh perspective on memorization and poses many interesting questions for the field.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0vC8qRtGpK", "reviewer": "Reviewer_74QL", "summary": "he paper investigates the phenomenon of memorization in large language models (LLMs), specifically focusing on complex sequences encountered only once during training. The authors highlight that certain sequences, termed \"latent memories,\" can be memorized by the model and retained persistently throughout training, even without further exposure.", "questions": "- Are there training modifications or strategies that could preemptively minimize or control latent memorization?\n- How does the latent memorization phenomenon interact with fine-tuning? Would fine-tuning mitigate the presence of these latent memories?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The concept of \"latent memorization\" adds a significant new layer to understanding how LLMs retain and recall data.\n- Providing a diagnostic for detecting latent memorization.", "weaknesses": "- The study primarily focuses on Pythia-1B and Amber-7B, limiting the generalizability to other, potentially larger, LLMs not tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8y3xGy4z50", "reviewer": "Reviewer_psZH", "summary": "This paper investigates memorization dynamics in large language models, focusing on \"latent memorization\" – a phenomenon where models retain complex sequences in memory even after minimal exposure. The authors demonstrate that these sequences, initially not retrieved verbatim, can later re-emerge in model outputs through weight perturbations, raising concerns about data privacy. The paper introduces a novel diagnostic test for identifying latent memorized sequences based on cross-entropy loss and suggests potential mitigations to enhance data privacy in model training. This work adds to ongoing conversations about data leakage and highlights the emerging challenges of memory persistence in AI models.", "questions": "1. Have the authors considered how latent memorization might vary with larger or smaller model sizes?\n2. Could the proposed kl-Levenshtein distance metric be generalized for other memory-intensive tasks or model architectures?\n3. Are there any specific limitations to the cross-entropy diagnostic when scaling up to larger models or applying it to real-world settings?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Originality**: The concept of latent memorization and the critique of kl-memorization provide new perspectives on memorization in LLMs, advancing our understanding of potential data privacy risks. TO further substantiate the novelty, authors could consider comparing with the recent findings related to verbatim memorization mechanisms, for example,  Huang et al.  (https://arxiv.org/pdf/2407.17817)\n- **Quality**: The methodology is sound, with experiments designed to isolate memorization effects. The use of weight perturbations to reveal latent memories is innovative and strengthens the study's claims.\n- **Clarity**: The writing and visualizations are clear, making complex ideas accessible and well-supported by empirical evidence.\n- **Significance**: The findings are relevant to privacy and security in LLMs, addressing a gap in our understanding of how sequences can be memorized and later recalled. Huang et al.'s framework on verbatim memory could provide an anchor point for discussing how latent memorization behaviors extend verbatim patterns.", "weaknesses": "- **Scope of Evaluation**: The experiments are limited to specific language model architectures, raising questions about generalizability to other architectures and model sizes.\n- **Mechanistic Explanation**: The authors suggest that latent memorization reflects stability in weight space, a hypothesis that could benefit from further clarification, particularly in light of Huang et al.’s findings that verbatim memorization relies on distributed states rather than specific model weights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6sh83tt4UA", "reviewer": "Reviewer_TM5T", "summary": "This paper discusses memorization in language models (LM) through training focusing on \"latent\" memorization when a model has _soft_ memorization strings during the training. The experiments measure memorization as a function of the dataset (complexity and repetitions, Figure 1), and training steps (using intermediate checkpoints, Figure 2). The paper also details a method to identify sequences that are \"latent\" memorized using a guess-and-check optimizer via adding random perturbations to weights and measuring memorization (Figure 4). Finally, the paper showcases that sequences with lower cross-entropy loss exhibit lower (latent) memorization.", "questions": "1. Is the deduplication discussed in Section 3.2 applied to isolate the repetitions affecting memorization and focus only complexity of samples (as measured by zlib)? If so, is the same deduplication applied to training data as well (specifically for additional deduping discussed in line 244)?\n1. What are the x-axis and y-axis in Figure 3a?\n1. What does \"the kl-LD of the sequences fluctuate (Figure 3) but do so in a way which is stationary across training\" mean exactly in line 248?\n1. Is latent memorization a general case of memorization (as measured by any chosen metric)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "- The paper discusses a fresh perspective on LM memorizing sequences with _latent_ memorization. This type of memorization feeds into the memorization at the checkpoint, which is finally considered to be deployed.", "weaknesses": "- The main claim of the paper that memorization is stationary throughout training (Section 3.2) is largely unsubstantiated in experimental results (Figure 3). There is no trend on the sample level to make an inference. It is unclear to me what the mean line entails here. Moreover, it is difficult to parse the experiment since axes are unlabelled and not clearly discussed in the Figure 3 caption.\n- The variance of memorization comparison with that of random walk is not (directly) measured experimentally and analysis in Section 3.2 is unsupported.\n- It is unclear how the claim that sequences are \"memorized as often as they are forgotten\" (line 255 in Section 3.2) is made. \n- The assumption that the sequences with low duplication and high complexity (measured using zlib heuristic) are more likely to contain private/sensitive information makes sense, but experiments only cover latent memorization on highly complex sequences (zilb>0.8). Why not ablate on different complexity levels (like Figure 1) and see how results vary?\n- The paper only focuses on syntactical memorization (Definition 2.1) with edit distance used as the only metric. It would be interesting to see the results for semantic memorization using soft embedding similar to methods like BERTScore [1].\n- Adding random noise and checking if the edit distance is lower is a guess-and-check zeroth order optimization formulation. This is applied on the training/chosen dataset and thus this noise is not \"random.\" This strongly opposes the hypothesis made that simply perturbing the weights with random noise produces similar effects as training.  \n- The latent memorization is poorly defined (also not explicitly) and oftentimes read as a circular definition. For example, samples recovered from an intermediate checkpoint are latent memorized (Section 3.3) but also the other way around.\n- Limited experiments in terms of model sizes. All Pythia [2] models can be evaluated to see the trends across different sizes.\n- Largely, throughout the paper, the results are unsubstantiated (as mentioned above) or anecdotal (Table 2 and Line 305). The writing is unclear and difficult to parse for the reader. The aim or main conclusion of the paper is thus unsupported and unclear.\n---\n[1] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, & Yoav Artzi. (2020). BERTScore: Evaluating Text Generation with BERT.\n\n[2] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A Suite for Analyzing Large Language Models Across Training and\nScaling. In International Conference on Learning Representations, 2023b", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)