For Better or for Worse, Transformers Seek Patterns for Memorization
Authors: Madhur Panwar
Gail Weiss
Navin Goyal
Antoine Bosselut
Venue: NeurIPS 2025 poster ()
Keywords: memorization
training dynamics
interpretability
generalization
language models
Primary area: deep_learning
TL;DR: Memorization in transformer LMs is tied to pattern acquisition. It is non-trivial and happens in bursts according to shared patterns. Intriguingly, the relative memorization speed of larger and smaller models can change based on the pattern type.

Abstract: Memorization in language models is a critical yet poorly understood phenomenon. In this work, we investigate memorization in transformer-based language models by analyzing their memorization dynamics during training over multiple epochs. We find that memorization is neither a constant accumulation of sequences nor simply dictated by the recency of exposure to these sequences. Instead, much like generalization, memorization appears to be driven by pattern recognition. Tracking memorization dynamics in mixed datasets, we observe that models memorize different sub-datasets in distinct bursts, suggesting that each subset is associated with unique underlying patterns, and that the model prefers to learn these patterns in a consistent order. We also find that easily learnable patterns tend to support generalization on unseen data, while more complex patterns do not. Furthermore, in datasets with weak or absent patterns, larger models may delay memorization relative to smaller ones, a behavior we term $\textit{overthinking}$. Our results show that the subset of sequences memorized by a model over time is not arbitrary, and give insights into the internal processes a model goes through during training. Our code is available at: https://github.com/mdrpanwar/memorization-patterns.

For Better or for Worse, Transformers Seek Patterns for Memorization
Madhur Panwar' Gail Weiss'
Navin Goyal?
Antoine Bosselut'
EPFL
Microsoft Research India
madhur, www.:rpppwaaaaacacccomm
Abstract
Memorization in language models a critical yet poorly understood phenomenon. In this work, we investigate memorization in transformer-based language models by analyzing their memorization dynamics during training over multiple epochs. We find that memorization neither constant accumulation of sequences nor simply dictated by the recency of exposure to these sequences. Instead, much like generalization, memorization appears be driven by pattern recognition. Tracking memorization dynamics in mixed datasets, we observe that models memorize different sub-datasets in distinct bursts, suggesting that each subset associated with unique underlying patterns, and that the model prefers to learn these patterns in consistent order. We also find that easily learnable patterns tend to support generalization on unseen data, while more complex patterns do not. Furthermore, in datasets with weak or absent patterns, larger models may delay memorization relative to smaller ones, behavior we term overthinking. Our results show that the subset of sequences memorized by model over time is not arbitrary, and give sights into the internal processes model goes through during training. Our code is available at: https: /top::///iitan thtbbbbt.ttttg/tg//ggg// wwwccoommmmaeeeeee www.//mrpawwar/////ooooooa.aaaae memor ization iettiioon-ppttttee
Introduction
Memorization, particularly in the sense of outputting entire training sequences verbatim, has been observed in many large transformer models and is increasingly recognized as both critical capability and potential liability [Hartmann et al.) 2023 In particular, prior work has shown that certain train- ing examples-inchdeee and later extracted from model's output CCrrlini et al. 2023b]. and the phenomenon is seen as serious privacy concern YYang al. 2024).
those containing personal identifiers unique texts-can be memorized
This risk has motivated efforts to characterize and mitigate memorization in language models. For example, Carlini al. [2023b] demonstrated that large language models often memorize parts of their training data verbatim, and that such memorization tends to grow with model size and training data duplication, meaning some sequences memorized over others. Other studies and defenses consider memorization in a fully-trained model, defining it in various ways (Zhang et af.) 2023 Carlini et al. 2023c Nasr al.) 20236). However, the mechanisms of the phenomenon itself, and in particular how it emerges during training, remain underexplored.
So how does memorization happen? The taxonomy of memorization in Prashanth et al. 2025 suggests that memorization can come from either recitation (memorizing a sequence with no regard for its structural similarity to other data), reconstruction (using general patterns to fill in the gaps), or recollection (memorization of rare sequences). In this work, we investigate both this distinction and other mechanisms underlying memorization by tracking the memorization dynamics of different models during training.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

By tracking when and what a model memorizes during training, we gain insight into its internal learning process. We conduct experiments on data from both real-world corpus (WikiText) and set of synthetic datasets where we control the presence and "difficulty" of patterns in the data. The mixed dataset setting allows us to observe how a model handles easy-to- aas-tto-generaliee data (reconstruction) versus data that is more likely to be memorized as is (recitation). We find that memorization often occurs in distinct bursts, where each burst tends to correspond to a subset of data with its own characteristic pattern. This suggests that the model is effectively segmenting the training data by patterns and fitting those subsets one at a time (reminiscent of a curriculum).
Overall, we find that memorization is far from being an arbitrary accumulation of sequences and is fundamentally tied to how models learn. Commonly, memorization in a language model is seen as the opposite extreme to that of generalization: the ability to extract and utilize patterns from the training data to handle unseen examples. Yet, as we find in this work, memorization and generalization may utilize the same pattern-seeking mechanisms. Transformers are highly adept at finding statistical and linguistic regularities ("patterns") in data. If meaningful patterns exist, the model will leverage them to minimize loss (this yields generalization). In fact, deep networks are known to prioritize learning simple patterns first, and only later fit noise or idiosyncratic data [Arptt et al. 2017a].
Ultimately, by understanding memorization from training-dynamics perspective, we aim to guide future work in balancing memorization and generalization for safer and more effective language models. In particular, we suggest designing interventions targeting how models detect patterns, so that they neither overfit noisy data nor overlook subtle structures.
Our main insight is that memorization is not trivial but rather tied to the availability of underlying patterns in the data. It is suggested by the following key observations:
Non-triviality of Memorization Dynamics (4.1)
We demonstrate that rather than steadily accumulating examples, transformer models repeatedly forget them (non nonnmmontoonicity) and the examples a model chooses to memorize at a given time are not simply those it saw most recently (non-recency).
Pattern-Depender Memorization ( ($4.2) We show that the propensity to memorize is highly dependent on the presence and types of patterns in the data. This reveals that what a model memorizes is tightly linked to the data's inherent patterns (or lack thereof).
Generalization and Pattern Acquisition ($43)): : We find datasets where the model both memorizes and generalizes on exact sequence completion, and others where the model only memorizes despite the presence of patterns in the dataset. For the latter, we find some cases where patterns in the datasets have nevertheless been picked up by the memorizing model.
"Overthinking" in Larger Models ($4.4): We identify a behavior larger transformer models wherein they postpone memorization in settings with less-frequent patterns relative to smaller models. We hypothesize that the increased capacity can amplify the model's 's bias toward -eekking... leading to late memorization of examples with fewer patterns.
2 Terminology
We provide definitions for some of our terminology in this section.
Memorization There are many definitions of memorization [Zhang al., 2023 Carlini al. 2023c Nasr et al.) 2023b Hartmann el al. 2023 Broadly speaking, the difference lies in whether we have access the training set not. If we do, we can prompt with prefixes from training sequences and check for matching completions (discoverable). If we do not, we must construct prompts that can elicit completions corresponding to parts of training data (extractable).
Extractable memorization is relevant in the study of attacks that extract training data, which is beyond the scope of this study. Hence we focus on discoverable memorization:
Discoverable Memorization (Nasr al. (003b)). For a model Gen and an example [p || x)] from the training set X, we say that x is discoverably memorized if Gen(p) x.
Patterns By pattern, we refer to any quality of dataset that could be exploited by sufficiently expressive language model to more efficiently capture the data. This could range from simple

---

qualities such as containing a high frequency of specific trigrams to more intangible qualities such as containing only grammatically correct English sentences.
Pattern Acquisition We say that model has acquired or learned pattern in our dataset if its predictions on new data adhere to that pattern, even if not necessarily being the overall correct prediction for that data. For example, uppercase, and are lexicographically ordered (e.g., aaabbdfff, MMOSSST), model that continues cc with a has possibly acquired the case rule, but certainly not the lexicographic rule.
dataset where all sequences are either uniformly lower
Generalization Classically, model is said to have generalized when it obtains setter-than-ramee performance on unseen data drawn from the same distribution as the training set. This is tracked by testing performance on held-out validation and testing sets. For this work, consider stricter inter- pretation of generalization that requires successful predictions of multi-token sequence completions (similar how measure memorization). This allows to align the two measurements.
We generally see good generalization as an indication of successful pattern acquisition, but don't necessarily expect the converse to be true.
3
Experimental Setup
In this section, we detail the data, models, and evaluation strategies for study of memorization.
3.1 Datasets and Tokenization
We represent natural language with the help of samples from the WikiText-103 dataset [Merity et al. (0166) which we tokenize with the GPT-2 tokenizer. We build "natural-language" datasets of different sizes as subsets of this dataset, filtering always to samples with lengths from 100-300 tokens.
Natural Data
Example: Troops are divided into five classes:
Scouts, Engineers,
Synthetic Data We also the following synthetic datasets. For these, maintain the same distribution of sequence lengths (100-300 tokens) and use a character-level tokenizer.
1. In-Context Mapping (ICM): Have fixed mapping of 10 letters to sequences of 3 digits. For each example, letters (L) are selected. The example then digit mappings of L, random sequence of letters in L, and digit mapping for the random letter sequence. Example: A: 1e:1:3>B::46<5555 7B8<5>>BBASS44666661111
2. Monotonic Increasing (MI): Next digit is previous 1 mod 10. Example: 7890123.
3. Monotonic Increasing with Jumps (MIJ): Monotonic increasing with 15% chance of random jump. Example: 234590123...
4. Monotonic Increasing with Jumps and Letters (MIJL): Similar to MIJ, but each jump is followed by letter; letters also follow monotonic increasing sequence. Example: 90126K78902L...
5. Shuffled WikiText: classes are Troops Engineers five Scouts,
Shuffle words in each Wiki'eet sequence. Example: into divided
6. Random Digits: Randomly occurring digits. Example: 82671903...
7. Repeat Tokens Ago: Random sequences with 10% chance of repeating a token from
places ago. Example: 762130653...
8. Every 7th Token Same: Either random sequence, or (with 10% chance) a sequence where one of the first tokens repeats every 7th token. Example: 1863054396...
9. Random Seed Lookup: Each next digit is obtained by lookup into the initial seed of length (s). Lookup index the addition modulo s of last digits. We use = 30, An example for s 5, = 2: 13820821...

---

3.2 Selecting model configurations
We checked the model configurations of modern models and found that their architectures follow an approximately fixed model width (E) to model depth (L) ratio across scales. The GPT-2 family Radford et al. 2019] follows an E I. ratio between 30 and 70; Llama 3 [Grattafiori et al. 2024] follows E7 L ratio between 100 and 130. Following this insight, for given parameter count, select the models with an E/L ratio in the range [100, 130].
Moreover, scaling law papers either include the embedding layer in parameter count (Hoffmann et al. [2022] or present the laws for both (including and excluding embedding layer) settings Kaplan et al. 2020 The embedding layer constitutes minute fraction of the total model capacity at real-wort scales. We study much smaller models, where the embedding layer (which has size corresponding to the input vocabulary) constitutes 50-95% capacity of the entire model. Hence, we choose to exclude them from the parameter count when dealing with WikiText data. Our synthetic datasets use much smaller vocabulary (less than 50 unique tokens) and the embedding layer need not be excluded. Therefore, the parameter counts henceforth refers to non-embedding parameters in the model.
3.3 Models and training
We use a decoder-only GPT-2 [Radford et al. 2019] architecture for our models. The model sizes studied are up to 12M non-embedding parameters; however, we do use GPT-2 small (85M non-embedding parameters) for a few experiments. The number of layers and the model's s hidden dimension are determined as indicated above. The number of attention heads (H) is kept as 1 As stated, GPT-2 embeddings can be very large fraction of model's capacity at small scales. Since we exclude them from parameter count, we must limit their contribution to model capacity while training. Hence, for experiments involving WikiText, we initialize the embedding layer of our models with that of the nearest (in parameter count) model from GPT-2 family and freeze it for the training. Principal Component Analysis (PCA) is used to reduce the dimensionality to the desired size. For experiments with synthetic data, the embedding layer is trained from scratch. Models are trained on datasets with varying number of examples (called "ex" for short), using the language modeling weext-token-predictinn
loss for 50k training steps with batch size of 64. Adam optimizer is used with a learning rate of le 3. We train our models on NVIDIA A100 80 GB GPUs, and depending on the model and dataset sizes, the runs take from 2 to 20 hours.
3.4 Evaluation
To evaluate memorization, we generate 30-token completions given a 50-token prompt from each sequence in the training data. We also measure performance on a held-out validation set when relevant (discussed $4.3). For the completion, we track the following metrics over the course of training: (1)
Exact match (EM), (2) Individual and Cumulative BLEU scores (Papineni et al.) 2002], (3) METEOR
[anerjee and Lavie 2005], (4) 13-gram Jaccard Similarity, (5) Training and Validation loss on the generated sequence. We note that all the metrics follow same trend, therefore, we choose EM as our primary metric and only show its trend for brevity. (See Appendices $ and for further clarification these choices.) When EM over the training set is I, the model has memorized its training data (up to our completion prompt length). For model selection purposes, we take an EM 2 0.95 to denote memorization of training data. The smallest model that achieves this is said to be the minimal model for that dataset size. As an aside, we find that for full memorization, the minimal model size scales linearly with dataset size (see Appendix C
Experiments and Results
In this section, we present the main experiments from our study and the corresponding results on various properties of memorization. We present our findings in the order stated in $ We first show that memorization is a non-trivial process, in that the set of memorized sequences is neither
We validate results on large models (Appendix A.5 and with more attention heads (Appendix AA66

---

Texning los
Foott March
10
12
0.5
Trating
Eoart
0.2
CE0 2M
0.00
410 1280
1000
3000
3000
2003
2000
(003
2000
1800
2000
3800
1000
5000
6000
7000
Epoct
Epcch
Figure Rate of memorization is not constant over training. The rate of memorization as measured by exact match is shown for models of size 2M and 12M trained on a dataset of 400 examples. Memorization increases a non-linen non-linear rate over training and the training loss decreases. (Error averaged over independent runs. 95% confidence intervals over 1000 bootstrap trials.)
200 ex; 1M 400 ex; 2M 800 ex; 4M 4000 ex; 12M
29.4
20.5
19.3
39.4
Table I: Memorization of individual sequences is dynamic. We track the memorization state of training sequences: memorized (EM = 1) forgotten (EM = 0). Average number of times sequence forgotten for various model configurations shown. Note that each sequence is memorized and forgotten many times.
constantly accumulated nor a cache recently trained ones. Next, we show that memorization exhibits differentiation along sub-datasets with different patterns. We then verify that our models have generalized and not just memorized from some our datasets, showing that this happens even when simple memorization is option. Finally, for certain tasks, observe delay in memorization for larger models relative to small ones. We suspect that this phenomenon stems from failed attempts at pattern acquisition before memorization begins, and thus term it overthinking.
4.1 Temporal Non-Monntonicity and Non-Recency
Result
Memorization is neither constant accumulation of sequences nor recency-biased.
In Fig. [] we show the training loss and EM for models trained on WikiText for 400 examples and two model sizes. We note that the rate of memorization denoted by EM is non-linear. When the model has sufficient capacity for the dataset that it is trained on, most of the dataset is memorized early during training (an EM score of 0.8 reached in under 50% of the training time required to reach an EM of l). The sequences memorized at the end (or ones that were never memorized) tend to sometimes contain numbers (e.g., years). Note that do not always observe such difference; yet, it suggests that the order of memorization may be related to the perceived difficulty or the amount of randomness in the sequence (which we touch upon in $4.2)
While the number of sequences memorized seem to increase over time (Fig. Tp. the memorization of individual sequences is dynamic. To study the memorization of individual sequences, we track whether a sequence is memorized (EM 1) or forgotten (EM = 0) by the model every 100 training steps during training. We find that individual sequences continue to be forgotten and re-memorized throughout training (Table IT. Using a 'softer' measure of forgetting (whether the sequence is among the top generations when sampling many times) also leads to the same conclusion (See Appendix $D.2p. The above two insights denoting the non-linear rate and dynamism allow us to conclude that the memorization is not constant accumulation of sequences (Result IT]
Since the model not accumulating sequences, another plausible hypothesis to be governed by the recency of sequences seen. In Table we show the intersection between the number of newly memorized sequences and the recent batch as a fraction of both. We find that both of these are low across all scales, suggesting that the sequences model chooses memorize
memorization
5

---

200 1M
400
2M
800 ex, 4M
1000
4M
4000 ex, 12M
% new mem. last batch % of last batch in new mem.
3.7
14.7
7.9
5.5
1.488
6.54
4.37
4.2
10.0
8.32
Table Memorization is not governed by the recency of sequences. We track the percentage of newly memorized sequences (whose EM changed from O to in the last train step) from the last batch and vice versa. Both of these metries are low across different scales.
100 ex; 0.5M
200 ex; 1M
400 ex; 2M
800 ex; 4M
2000 ex; 8M
Baseline Shuffled 0.68
0.96
0.96
0.96
0.98
0.95
0.63
0.70
0.69
0.72
Table 3: Presence of patterns increases the memorization rate. (Training on shuffled word order data reduces exact match.) We train different models and dataset sizes on both WikiText and Shuffled WikiText (where words in each sequence are randomly shufiled) for same amount of training time. A drop in EM of 30% is observed for the latter, signifying that the linguistic patterns increase the rate of memorization.
at a particular training step are not necessarily the ones it has most recently seen in the last batch. Therefore, recency of sequences does not govern memorization.
4.2 Differentiation
Result 2 Models exhibit different memorization profiles for different datasets, both when trained independently single dataset and on data mixtures.
Next we ask: Is all memorization alike (across datasets)? We find the answer is No. We start by constructing a simple variant of the WikiText dataset: for each sequence, we shuffle the order of words randomly, leading to a new shuffled sequence. We find that models trained on this dataset for the same amount of time as original WikiText show drop in EM by about 30% (Table We also try other shuffle variants (Appendix şB.1) and have consistent findings. This implies that (linguistic) patterns affect the memorization rate.
To further investigate this finding, we design various datasets with different amounts of randomness and patterns (3..I). We train models of size 4M on these datasets of size 500 examples. The EM curves for the models are plotted in Fig. 2a We note that different datasets have different memorization profiles: with "Random Digits" " on the far right and MI on far left. We also observe two distinct type of curves: sudden jump and gradual rise, which is another characteristic feature of the data the model trained on. Both of these observations inform us that patterns beyond unigram statistics are taken into consideration for memorization.
In Appendix B.2 we present more experiments examine the effects of changing different hyperparameters in our synthetic datasets and make consistent observations.
Further, check whether the same insight holds for models trained data mixtures. We train models on two data mixtures: (a) mixture of monotonically increasing sequences with different amounts of randomness and random digits sequences, (b) WikiText mixed with random digits sequences. In both cases, we train three models each and plot the average metrics. The datasets have equal proportion of all component sub-datasets.
For (a), we train 4M- AM-parameter models on dataset of 500 examples. We plot in Fig. 2b number of epochs needed to achieve different levels of exact match for all component sub-datasets. We find that the model memorizes them in the order of gradual increase in the randomness from MI to "Random Digits". (For unbounded sequences, the probability of a jump in both MIJ and MIJL sequences would be the same. However, sequences in both of these datasets have fixed lengths and follow same length distribution. Therefore, MIJL is less random the interleaved sequence of letters is deterministic after the first letter, making a larger portion of these sequences deterministic.)
For (b), we train 8M-parameter models on dataset with 2000 examples and plot the EM over time for both sub-datasets Fig. (Left). We observe that memorization happens in phases: almost all of Wixi is memorized before the memorization of random sequences starts. This agrees with

---

300
Dight
Nenste Hoom Leter
InContien
Moretaee hocninng Jamm
Staflic WalTe
100
200
1091 1380
380
0.2
Exact Match
(a) Trained individual datasets
(b) Trained mixture of datasets
Figure 2: Models exhibit different memorization profiles and the order of memorization for different datasets when trained them independently (left) or simultaneously (right). Left:: The exact match for various datasets for models of size 4M trained on 500 examples. Right: Number of epochs required achieve different exact match values for each sub-dataset size 4M trained on 500 examples equally distributed across the four different datasets (MI, MIJ, MIJI, and Random Digits). Note that the datasets with more randomness take longer be memorized.
shown for model of
the insight in $4 where noted sequences of numbers preferentially being memorized the end. Moreover, in Fig. 3 (Right), we plot overall (combined) EM and training loss for an 8M model trained 2000 examples for different mixture compositions of WikiText and random digits datasets. The phase changes in loss and EM are evident. Note the position at which the phase corresponding to memorization of random digits starts: the less the amount of random digits data, the later their memorization starts. These phase shifts are reminiscent of disjoint, yet qualitatively similar results in Chen et [2025] where they note sudden drops in loss for syntax acquisition.
These observations suggest that the order and speed of memorization of different datasets are related to the extent and type of patterns contained in them, leading different memorization profiees.
4.3 Generalization and Pattern Acquisition
Result Models generalize fixed mapping-basee based patterns on unseen data.
Do models generalize these patterns to an unseen validation set? For this, we test our models on datasets with deterministic patterns such MI and ICM on disjoint validation set. Since there only 10 unique monotonically increasing sequences (one each with a different starting digit). to test generalization, we held out the sequence starting with digit '7'. Models of different sizes trained on different data sizes were able to successfully complete sequences starting with '7'. For ICM dataset, the "500 ex; 4M" model when evaluated on an unseen validation The underlying letter-to-digits mapping is same across all sequences in train and validation
of 500 examples, achieves an
1. 1. sets. Note that both of these datasets have sequences with static lookup patterns. In MI, the next digit is deterministic based on the previous digit (bigram lookup), while the ICM sequences require lookup from a fixed letter-to-digits mapping by definition. This implies that the models can generalize to datasets with static lookups. The patterns in these datasets can therefore be acquired, and hence, the model uses reconstruction to memorize them.
EM
To test the extent of generalization of lookup-basse based patterns, we test on "Random Seed Lookup" dataset. We also design dynamic version of the ICM dataset where the letter-to-digits mapping is dynamic across sequences. We train 4M -M--prrmeter models on 500 examples each from these datasets. While the models memorize these datasets perfectly (EM 1), EM on the validation set remains identically zero. Therefore, we find that for more complicated and dynamic lookups, the models fail to generalize. So, we suspect that this pattern was not acquired by the model, and the performance on training set corresponded to the recitation of those examples.
7

---

1.0
10.8
Exact Matah
0.0
0.5
Training
2-
010
0.0
1000 1500
Epoch
Figure 3: Phase shifts corresponding to the memorization of mixture of WikiText and Random Digits data. Left: EM for the two sub-datasets for an 8M model trained on 2000 examples (an equal mixture of WikiText and random digits). Memorization of random sequences start after most of WikiText portion has been memorized. Right: EM for all training sequences (all sub-datasets combined) for "2000 ex; 8M" models with different fractions of random data. The length of each phase is proportional to the amount of the corresponding sub-data.
But what about generalization datasets that partially contain deterministic patterns with some amount of randomness? To answer this question, we check how often do the models trained on MIJ dataset follow the monotonically increasing pattern on an unseen validation set. Given 50-token prompts from sequences in this validation set, we generated 30-token completions for 4M and 8M models (both trained on 500 sequences). For every bigram (I,,1, ] in completions, we measured how often is it monotonically increasing (i.e., the two models respectively. This is much larger than the random chance (10%)
(#t-1 1)%10). We find this number to be
97.3% and 98.1 of generating monotonically increasing next digit. Therefore, despite there being randomness (15% random jumps), the model acquires the monotonically increasing pattern.
These results demonstrate models' affinity toward patterns for memorization, and the general- ization of static lookup-based patterns to unseen sequences.
4.4 Overthinking
Result 4 (Overthinking): On datasets with weak or absent patterns, large models memorize slower than smaller models.
While investigating our central thesis (pattern-seeking behavior in memorization), we made an interesting observation: on certain datasets without patterns or with less-frequent patterns, the memorization in larger models can be delayed relative to small models. We believe it caused by the models' inherent affinity to patterns, which potentially backfires when patterns cannot be found.
In Fig. we plot the EM for small and a larger model trained on 500 examples each from different datasets. The model sizes are mostly 4M and 8M respectively, with the exception of dataset with weak patterns (repeat some previous token with low probability), where the larger model is GPT-2 small We note that for dataset with abundant patterns (linguistic patterns such as in WikiText, or static mapping-based ones for ICM and MI family), the larger model either better or on par with small model. However, for models with large amounts of randomness (weak pattern datasets, "Random Digits"), or complex dynamic lookups ("Random Seed Lookup"), larger models have much worse memorization rate. Therefore, a larger model is not always better or faster. We call this behavior overthinking, and hypothesize its cause being model's pettern-seeking behavior which may lead to delay when patterns missing not present in abundance.
For weak patterns, the effect only evident higher model scale.

---

WikiText
In- ConCoct Mapping
Monotonic Increasing
4M
Monotonic Increasing with Jumps
41d
Exact Match
Shufiled WikiText
atdl
Random Digits
4tM
Repeat tokens ago (10% chance)
4M
Every 7th token (10% sequences)
BSM (GPT-2 small)
Random Seed Lookup
4M
SM
500
1000
1500
2000
Epoch
Figure 4: Overthinking, EM for small and larger model trained dataset of size 500 examples is shown across many datasets. Larger model is slower for data with absent (Random Digits) and weak (repeating previous token with low probability) patterns, and for complex dynamic lookups.
5 Related Work
Memorization has been central topic in machine learning since the earliest neural networks. We organize prior work into six themes: (1) classical and associative memory models, (2) memorization in modern language models, (3) privacy and data extraction, (4) memorization and pattern-secking behavior, (5) interpretability and training dynamics, and (6) related learning phenomena such forgetting and curricula.
generalization
Classical and Associative Memory Models with Hopfield networks Hopfield [982]] which showed that recurrent networks could store discrete patterns as stable attractors. These systems, along with bidirectional associative memories Kosko [1988] and sparse distributed memory Kanerva 1988), were designed to explicitly encode and retrieve stored patterns from the network's 's weights. Recent surveys emphasize that modern LLMs behave differently from these classical models, exhibiting implicit memory through distributed representations, attention mechanisms, and in-context learning rather than explicit associative storage Zhang et al. [2024). This shift motivates a fresh look at memorization in the transformer era.
The study of neural associative memories began
Memorization in Transformer Language Models Large-scale studies demonstrate that transform- ers memorize exact sequences from their training data, starting with early extraction experiments on GPT-2 Carlini and Trumpler (2021). Subsequent work confirmed that unique duplicated examples
can be reproduced verbatim Carlini et al. [2023b], Nasr al. [2023a]. Dagielski al. [2022] ,, [ppolito] et al. 2022]. Empirical evidence indicates that memorization grows with model scale, dataset du-

---

plication, and training length; strategies such as deduplicating training data mitigate this tendency
Lee al.] [02T]. Alternative framings include counterfactual memorization Zhang et al. [021] and adversarial compression [Schwarzschild et al. [2024]. Memorization is also observable across
modalities, including diffusion models Carlini et al. [2023a] code models Yang et al. [2023 and
vision models Lukasik et al. 2023].
Privacy and Data Extraction Memorization poses privacy and IP risks: it can lead to unintended data leakage, including PII and copyrighted text Carlini and Trumpler (202T), |Carlini et al. [2022]
2023b Nasr al. [2023a], [[ppolito et al. [2022] Large-scale attacks show thousands of training
examples can be extracted cheaply Nast et al. [2023a] and model-stcaling setups remain vulnerable
even with API-only access Carlini al. [2024]]]
Memorization vs. Generalization and Pattern-Seeking Behavior While memorization is often viewed as the opposite of generalization, research shows that networks can memorize random labels Zhang et al. [2017], yet tend to learn simpler patterns first before memorizing irregularities AArit al. 2017b]. The inductive bias capture structure first is evident in language models through continua Dankers)
frequency-driven rule acquisition WWii et al. [2021), memorization-geneallzeee Taxonomies such as
al. (2023). and long-tail memorization dynamics |McCoy et al. [2020]
recitation vs. reconstruction highlight multiple modes of memorization Prashanth et al. (2024). Grokking shows sudden generalization after overfitting Power and et al. [0022] notably, memory can serve as a form of compression Schwarzschild et al. [2024] Our results align with these dynamics.
Interpretability and Training Dynamies Braining-dynamice studies highlight phase transitions where models abruptly acquire structural knowledge Chen et al. [2023). Representation analyses reveal that linguistic features emerge in progressive layers Hewitt and Manning (2019). Beyond training, methods for updating models with knowledge explore the interplay between memorization
and reasoning post hoc Li and Goyal 22255.
Related Learning Phenomena: Forgetting and Curricula Early work on catastrophic forgetting
documents how models overwrite prior knowledge McCloskey and Cohen 1989] Ratcliff (1900].
This persists in modern models, where fine-tuning can cause forgetting of memorized examples Jagielski et al. (2022). Continual memorization research aims to avoid this trade-off Chen al. 2024]. Curriculum learning shows that ordering training from easy to hard promotes generalization over rote memorization Bengio et al. [2009). Our observations suggest natural analog: transformers tend to memorize in ordered bursts aligned with data patterns.
Limitations
Our study is conducted on relatively small models under a multi-epoch training regime, which does not directly reflect the large-scale, single-epoch training used for modern foundation models, but does provide relevant insights. Additionally, while synthetic datasets can be designed with the intent of diversity in type and "difficulty" of patterns in mind, identifying and quantifying patterns in real-world language modeling data (e.g., WikiText) remains challenging. Finally, our evaluation of memorization is based on 30-token completions, which may not capture subtler forms of memorization or longer-span dependencies.
7 Conclusion
In this work, we studied how memorization arises transformer language models through the lens of training dynamics. We showed that memorization is not a monotonic or -ececcy-baeed accumulation, but pattern-driven process: models memorize in bursts, generalize when possible, and delay memorization when patterns are ambiguous. Larger models often "overthink" low-pattern data, suggesting a strong inductive bias toward structure. These findings suggest that memorization and generalization emerge from shared mechanism of pattern- seatern-seeking.. Several questions remain open. What kinds of patterns are most conducive to memorization? Can we impose hierarchy over them, and understand how the model prioritizes one over another during training? Another direction to understand the mechanistic underpinnings of overthinking, and whether this behavior can be encouraged mitigated based on downstream goals.
10

---

8 Acknowledgements
We are grateful to Deniz Bayazit and Badr AlKhamissi for their helpful discussions during this project, and for their feedback on our manuscript. We also gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29) the EPFL Center for Imaging, Sony Group Corporation, and Meta LLM Evaluation Research Grant.
References
D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj. A. Fischer, A. Courville, Y. Bengio, and Lacoste-Julien, A closer look at memorization in deep networks, 2017a. https://///aaxt http://////axxivoo/a/oee/e/// 1706. 05394
D. Arpit al. A closer look at memorization in deep networks. arXiv preprint, www.v:100.10339....... 2017b. doi: 00.48550/arxivv17066555
URL httphtttt/sxxiii wwgg/abs/7706... 05394
S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Goldstein, A. Lavie, C.-Y. Lin, and Voss, editors, Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 63-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.
org/005-0999/
Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, pages 41-48, 2009, doi: 10.1145/1553374, 1553380.
N. Carlini and Trumpler. Extracting training data from large language models. arXiv preprint. vrXiv:2012.078000 2021. doi: 10445500axX..2122205555 URL http::/s:://aatt org/ aba/ 2012.07805
N. Carlini, M. Jagielski, C. Zhang, N. Papernot, A. Terzis, and F. Tramer. The privacy onion effect: Memorization is relative. arXiv preprint, arXiv:2206. Ntt.:2206.10469,, 2022. doi: 10.48550/arXiv.22066 10469. URL htttt:t//ts:xi:> wwassiivorrg//aae /twg/b//////// /206. 10469
N. Carlini, J. Hayes, M. Nasr, M. Jagielski, V. Sehwag, F. Tramer, B. Balle, D. Ippolito, and E. Wallace. Extracting training data from diffusion models. arXiv preprint, arXiv:2301. 2023a. doi:
10.48550karXiv.2301.B5
|tttssss://aaxxx
httpttpp://aaxi..orr/bbb//////// wtwpi//.rrrabbb//////// 13188
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memoriza- tion across neural language models. In The Eleventh International Conference on Learning Representations, 2023b. (tRLhttps::/ pttttps://openrevieet www.errreeeeenne/ooru////// wett:/orromiideTTatR......
N. Carlini, D. Ippolito, M. Jagielski, Lee, Tramer, and Zhang. Quantifying Memorization Across Neural Language Models, Mar. 2023c. URL http:://rxxxa org/abs///////oooooooooooooe 2ttps//220.0075 arXiv:2202.07646 [cs].
N. Carlini, D. Paleka, K. Dj Dvijotham, T. Steinke, J. Hayase, A. F. Cooper, K. Lee, M. Jagielski, M. Nasr, A. Conmy, E. Wallace, D. Rolnick, and F. Tramer. Stealing part of production language model. arXiv preprint, arXiv:2403.066: 2024. doi: (0.48550/aXXi..203.66633 URL https://arxiv. org abs/2403. 06634
A. Chen, R. Shwartz-Ziv, K. Cho, I., Leavitt, and N. Saphra. Sudden drops the loss: Syntax acquisition, phase transitions, and simplicity bias in mims. arXiv preprint, arXiv:2309.077 01,11, 2023. doi: (0.48550/arXiv.2309.00m 1. URL http:::/tta/ttt
A. Chen, R. Shwartz-Ziv, K. Cho, M. L. Leavitt, and N. Saphra. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in mlms, 2025. URL httph::/tt//att org/ abs/2309.07311
H. Chen, J. Geng, A. Bhaskar, D. Friedman, and D. Chen. Continual memorization of factoids in language models. arXiv preprint, arXiv:2411.0m 11,0175,, 2024, doi: 10.48550/anXiv:4411.0m URL. https::/arxtee wttp:///.ww///bb//...000..

---

V. Dankers, I. Titov, and D. Hupkes. Memorisation cartography: Mapping out the memorisation- generalisation continuum in neural machine translation. arXiv preprinz, arXiv:231 L.003797, 2023. doi: 10.48350WaXii.2311.. 1.05379. URL
www.oorgg//bbaaaeeeee 053779
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang. A. Mitra, A. Sra- vankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia- Orrcio,OOano,... D. Perino, D. Hup- kes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic. F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang. G. Cu- curell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, Evtimov, J. Zhang. Copet, J. Lee, Geffert, J. Vranes,
Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong. J. Lee. J. Fu, J. Chi, J. Huang, J. Liu, J. Wang. J. Yu, J. Bitton, Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plaw- iak, K. Li, K. Heafield, K. Stone. K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yearr L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kar- das, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi. P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Kr- ishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa. S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do. V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia. X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaci. Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Fein- stein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisemman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, Civin, D. Beaty, D. Kreymer, D. Li. D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil. E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, Guangyi, Zhang, G. Lakshminarayanan H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj. I. Molybog, I. Tufanov, I. Leontiadis, I.-E. Veliche, Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg,
C. Zhou, C. Hu, C.-H. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao,
J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. 7and, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg. L. A.
L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. San- thanam, N. Parks, N. White, N. Bawa, N. Singhal. N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mchta, S. Siby, S. J. Bondu,
12

---

S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto,
S. Ramaswamy. S. Lindsay, S. Lindsay, Feng. S. Lin, S. C. Zha. S. Patil, S. Shankar, Zhang,
S. Zhang, S. Wang. S. Agarwal, S. Sajuyighe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng. S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman,
T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang. Y. Zhang, Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. Vetio, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma. The llama herd of models, 2024.
URL https: http://axivvvoo wttpvvv.rrggab//2011010 21783
V. Hartmann, A. Suri, V. Bindschaedler, D. Evans, S. Tople, and R. West. Sok: Memorization in general-purpose large language models. ArXiv, abs/2310.18362, 2023. WtLLhttps://ee semanticscholi wwg//og/ggpoooooo Trgg/CorpusID::. 264590727
J. Hewitt and C. D. Manning. Understanding learning dynamics of language models with sveca. arXiv preprint, wtXiv:1811.002220 2019. doi: 10.48550/arXiv.188 www.iii..1111.000 1.00225. URL https://arxiv btggabs//8111.0 00225
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoe, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L.. Sifre. Training compute-optimal large language models, 2022. URL https: http::://aaxiv.orr////// /200 15556
J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 79(8):2554 2552225888 1982. doi: 10. 10.1073/pnas.77
8.2554. httpss::::/ //doi. org/ 10. 1073//mas pma.. . 79.8. 2554
D. Ippolito, F. Tramer, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo and N. Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint, arXiv:2210, 17546, 2022. doi: 10:48550\axxw..220..7555 2210 17546]
URL https: / //arxiv..vvvv/// org/ /bs/
M. Jagielski, O. Thakkar, F. Tramer, D. Ippolito, K. Lee, N. Carlini, E. Wallace, S. Song, A. Thakurta, N. Papernot, and C. Zhang. Measuring forgetting of memorized training examples. arXiv preprint. Mikiv:22070000990 2022. doi: 10448550/arXi.22207.00000 000099. URL Stlhst::: //arxiv org/ abs/ 2207
Kanerva. Sparse Distributed Memory. MIT Press, 1988.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020. URL httpss://axxiii /rrg//or//vvoo//////// abs Abs/2001.08333
B. Kosko. Bidirectional associative memories. IEEE Transactions on Systems, Man, and Cybernetics, 18(1):49-60, 1988. doi: 10.1109/21.8705 URL sttt::://ai..org/10.111000 1ttp:/.1..11992..111 87054
K. Lee, D. Ippolito, Nystrom, C. Zhang. D. Eck, KallissonBuuchh and N. Carlini. Deduplicating training data makes language models better. arXiv preprint, w1t.:::2007006999,, 2021.
doi:
10.48550/axxiv22107..66 0.49999 URL https:t //apav///////v////aeeaaeeeeaaa wwwprvv.rrg/bb//210... 0649999997709999
A. O. Li and T. Goyal. Memorization vs. reasoning: Updating llms with new knowledge. arXiv preprint, 1xXx:25504.12233 2025. doi: 10.485500/axxx.2504.15533 URL http://xxxxvvvoooee org/
abs/2504. 12523
M. Lukasik, S. Kumar, et al. What do larger image classifiers memorize? arXiv preprint, arXiv:2310.05337, 2023. doi: 10.485500/xxi.2310005333
abs/
2310.05337
M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequen- tial learning problem. In Psychology of Learning and Motivation, volume 24, pages 109-165. Academic Press, 1989, doi: 10.10166500799777210000033333333
13

---

T. McCoy et al. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguis- lics, 8:28 45, 2020. doi: 10.1 a aapptaccll__066,..._ URL. btttps://do:: 1tt../r.//...11//ac.. 005677
S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models, 2016. URL https://arxiv, wwwooorgga.......oo...o.......ee 01669900844
M. Nasr, N. Carlini, J. Hayase, M. Jagiclski, A. F. Cooper. D. Ippolito, C. Choquette-Choo E. Wallace, F. Tramer, and K. Lee. Scalable extraction of training data from (production) language models. arXiv preprint, arXiv:2311.17035, 2023a. doi: 10485500axXx.2211..7753 URL
https: //arxv///.ivv....a org/ abs/ 2311 17035
M. Nasr, N. Carlini, Hayase, M. Jagielski, A. Cooper, D. Ippolito, C. A. Choquette-Choo, E. Wallace, F. Tramèr, and K. Lee. Scalable Extraction of Training Data from (Production) Language Models, Nov. 2023b. URL/http://artt //xxiv.// www.oog///////ooooooooo.ooeo. ab/////s//111.11 17035, arXiv:2311.17035 [cs].
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: method for automatic evaluation of machine translation. In P. Isabelle. Charniak, and Lin, editors, Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. July 2002. Association for Computational Linguistics. doi: 1033115/1073388.1073333.. URL
wpg/p02-- 1040/ 1040/
A. Power and al. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint, arXiv:2201.021 120177, 2022. doi: 10.48550//nXi..2201..222 abs/2201.0211
Utthttss:///art
U.S. S. Prashanth, A. Deng, K. O'Brien, and N.... Saphra. Recite, reconstruct, recollect: Memorization in Ims as a multifaceted phenomenon. arXiv preprint, arXiv:2406. 2024. doi: 10.48550/ arXiv.2406. 17746. URL URL. 17746
U. S. Prashanth, A. Deng, K. O'Brien, J. S. V, M. A. Khan, Borkar, C. A. Choquette-Choo, J. R. Fuehne, S. Biderman, Ke, Lee, and N. Saphra. Recite, reconstruct, recollect: Memorization in lms multifaceted phenomenon, 2025, URL httttpp:aaax.v.ogg//bb////////// abrgg//b/2240.... 17746
A. Radford, J. Wa, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
R. Rateliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97(2):285-300 1990. doi: 0137770033-25XX.977 285.
A. Schwarzschild, Feng, P. Maini, Z. C. Lipton, and J. Z. Kolter. Rethinking I1m memorization through the lens of adversarial compression. arXiv preprint, mxXvv:2404.11156 2024. doi: 10. 104485550/axXxn2244455555 15146.
15146
Wei, Garrette, T. Linzen, and E. Pavlick. Frequency effects on syntactic rule learning in transformers. arXiv preprint, arXiv:2109.0700000 2021. doi: 10.488550/arXi1.11000000 URL
https://arxiv. org/ /ttp//ab/////0000.
X. Yang, Z. Wen, W. Qu, Z. Chen, Z. Xiang, B. Chen, and H. Yao. Memorization and privacy risks in domain-specifie large language models. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models, 2024. Uttp:
Z. Yang, D. Lo, et al. Unveiling memorization in code models. arXiv preprint, preprin, arXiv:2308 099332, 2023. doi: 10.485550/rXiw.22388000 099332, URL http://peerrr....oooo/ooooommtta
org/ htwrg/bbs/2000 09932
C. Zhang, D. Ippolito, K. Lee, M. Jagielski, F. Tramer, and N. Carlini. Counterfactual memorization in neural language models. arXiv preprint, 12, 12938, 2021, doi: 10.485500aaxi22211222233
URL https: :/t:://aaxxxe /atxiviivv.. org.vi.orrg/eeo httgg/abs/2112.. 12938
14

---

C. Zhang, D. Ippolito, K. Lee. M. Jagielski, F. Tramèr, and N. Carlini. Counterfactual Memo rization in Neural Language Models, Oct. 2023. http://aawxiv..ogbb//////s/ 12938 1xXiv:211.11993 [es].
C. Zhang et al. Understanding deep learning requires rethinking generalization. arXiv preprint, 1rX0v:1611.03530, 2017. doi: 10485500aXXx.111..0355. URL BRLLhhts:///are org/ abs/
1611. 03530
Z. Zhang, X. Bo, C. Ma, R. Li, X. Chen, Q. Dai, J. Zhu, Z. Dong, and J.-R. Wen. A survey the memory mechanism of large language model based agents. arXiv preprini, arXiv:2404.13500 2024. doi: 10.48550/arXiv 2404 13501. URL //t::///aviv/////a www.aaiwwww.aaaa...... otwzaii..rr//ab///40...1. 13501
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We ensured that all claims made in abstract and introduction are well supported by our results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
We have a "Limitations" section, where we discuss limitations of this work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper includes theorems other non-empirical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Besides explaining all experimental details, we also provide the code.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided the code with the paper. We either used open-source data or synthetic data. Code for generating the latter is also included.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/publcc/eee.c wttpg//ddde/ooeetuiiieeeeeeeeeee ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We specify all relevant details, and also include the code for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report error bars across multiple training runs for each result.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We state compute requirements under "Experimental Setup" section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps:/neeurpps ottte
Answer: [Yes]
Justification:
The research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: impact.
This is foundational research on neural networks, and we anticipate no societal
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do any risk of misuse of data models from research.
Guidelines:
20

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use the GPT-2 Small model and WikiText- 103 dataset, both of which we cite. We use no other existing models or data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code for running experiments and generating data are properly documented. No other assets have been created.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
21

---

Answer: [NA]
Justification:
The paper involves no crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper involves no crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core methods of this research do not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non otanndaddrdd components.
Please refer to our LLM policy (https: pattps://neuriie / for what should or should not be described.
22

---

Contents
Introduction
Terminology
3
Experimental Setup
Datasets and Tokenization
3.2
Selecting model configurations
3.3
Models and training
Evaluation
4
Ex Experiments and Results
Temporal Non-Monotonicity and Kon-Recency
4.2
Difierentiation
4.3
Generalization and Pattern Acquisition
Overthinking
5
Related Work
6
Limitations
10
7
Conclusion
10
Acknowledgements
11
Experimental Setup Details
24
Examples from WikiText and Sequence Length Distribution
24
Selecting Model Configurations
25
A.3
Choice of Prompt and Generation Lengths
25
Other Metrics for Memorization
26
A.5
5
Model Sizes
26
A.6
Attention Heads
27
Dataset ablations
27
B.1
Controlling the Degree of Shuffed-ness
27
B.2 Further Studies with Synthetic Datasets
27
Note on Scaling
29
D Further Dynamics of Memorization
29
D.I First- Memorized and First-Forgotten Statistics
29
D.2 Using Softer Metrics to Measure the Dynamic Nature of Memorization
29
D.3 Sequences Memorized across Different Epochs
30
23

---

D.4 Metrics for Non-Memorized Sequences
31
A Experimental Setup Details
A.1 Examples from WikiText and Sequence Length Distribution
Some examples from the WikiText dataset are shown in Table 4 dataset based on the length of sequences The resulting distribution of lengths is plotted in Fig. 5 For consistency, maintain the same sequence length distribution across synthetic datasets.
As mentioned in $3.1 we filter the
Example
Text
Cicely Mary Barker 28 June 1895 16 February 1973 was an English illustrator best known for series fantasy illustrations depicting fairies and flowers Barker 's art education began in girlhood with correspondence courses and instruction at the Croydon School of Art Her earliest professional work included greeting cards and juvenile magazine illustrations and her first book Flower Fairies of the Spring was published in 1923
Example
The Gregorian Tower Italian Torre Gregoriana
Tower of the Winds Italian : Torre dei Venti ) is round tower located above the Gallery of Maps which connects the Villa Belvedere with the Apostolic Palace in Vatican City The tower was built between 1578 and 1580 to design by the Bolognese architect Ottaviano Kascherino ( who was credited with building the Apostolic Palace ) mainly promote the study of astronomy for the Gregorian Calendar Reform which was commissioned by Pope Gregory XIII and promulgated 1582 It was then also known as the Tower of Winds The tower is now called the " Specola Astronomica Vaticana the Vatican Observatory
Example
Table 4: Some examples from the WikiText dataset.
Histogram of lengths of examples
40000
35000
Number of examples
25000
20000
15000
10000
5000
200
250
300
350
400
Number of takens
Figure 5: Distribution of sequence lengths in the WikiText dataset after filtering.

---

A.2 Selecting Model Configurations
Our model configurations follow the model width-over-depth (E/L) ratio from Llama Grattafioni et al. 2024) family of models. We plot E-L contours corresponding to non-embeddinng parameters each desired parameter count. Next, we plot the lines corresponding to different, fixed E/ L values. We then choose the model configurations nearest to the intersection between each contour and these lines. The contours, E/ lines and the selected model configurations are shown in Fig. 6
Koneersa 62x
Novara
Nonvera 0r+
Nenvrr
Params
L
E
0.5M
144
Now
IM
2
204
EL 04
235
4M
333
8M
408
12M
4
500
20M
577
32M
6
666
100
150
200
250
300
350
200
300
600
Figure 6: Selecting model configurations. Left: To select the model configurations for different model sizes, we choose model width (E) and number of layers (L) such that the E/L ratio is between 100 and 130 (following Llama 3), i.e., models lying at the intersection of contours and lines. The number of heads (H) is in all chosen configurations. Right: Chosen E and values across different model sizes.
0.0
Exst March
0.1
0.2
0.2
Mentic
2000
4000
5000
5000
10000
12000
2000 4000
5003
5003
LC000
22000
Cpach
Cpoch
Figure 7: Ablation on the prompt and generation lengths used for evaluation for "200 ex; 1M"
model. Left: Prompt length 50 tokens and results for generation lengths of 10 and are shown. Right: Two prompt and generation length combinations are shown: the default (50-30) and (30-70). Note that the trends still continue be the same across all these choices.
A.3 Choice of Prompt and Generation Lengths
In $3.4 we stated our choice of evaluating different memorization metrics on 30-token completions given 50-token prompt. We now show that these specific choices not fundamentally change the results observe. In Fig. 7 we plot EM in two scenarios: (a) keeping the prompt length as
25

---

0.8
0.0
East March
0.4
0.4
0.2
Stiters
Mines
Masch sleed)
Manck 00-20 wval
0
200
100
630
500
3000
1200
1400
1620
200 400
630
500
1000
1200
1400
1600
Ezoch
Esoch
Figure 8: Ablation on the prompt and generation lengths used for evaluation for "2000 ex; 8M" model. Left: Prompt length is 50 tokens and results for generation lengths of 10 and 30 are shown. Right: Two prompt and generation length combinations are shown: the default (50-30) and (30-70). Note that the trends still continue be the same across all these choices.
50 tokens and plotting results for generation lengths of 10 and 30; (b) varying both the prompt and generation lengths. We try configurations: (50-30) and (30-70). Fig. (sswws the same plots for "2000 ex; 8M" model. The trends still continue to be the same across all these choices.
10
08
Memortzaion Merics
0.0
Meric
0.2
VETCOR
Menoriation
0.4
5LC C0
Sected
4000
5000
5000
10000
12000
Epoch
Figure 9: Different metrics for memorization. Many metrics plotted to track memorization for "200 ex; 1M" (Left) and "2000 ex; 8M" model (Right). Note that all the metrics follow the same trend.
A.4 Other Metrics for Memorization
we plot different metrics we track for memorization for "200 ex; 1M" and "2000 ex; 8M" models trained on WikiText. All metries we tracked follow the same trend and hence we choose to only show EM for brevity.
In Fig. g
A.5 Model Sizes
We experiment with larger models to show evidence for the applicability of our findings larger model scales. We replicated our setup on model sizes 100M, 200M, 500M, and 1B, on datasets "Every 7th token same (10% sequences)" and "Monotonic Increasing with Jumps." We consistently observe that larger models take longer (aka 'overthinking") before starting to memorize sequences in these datasets (same as Fig. Please note that these training runs demonstrating overthinking can be very computationally expensive for larger models, they take much longer to start memorizing those datasets, limiting the model sizes we can include the study. However, these results up 1B scale conclude that overthinking does hold at large scales.
26

---

We also trained 200M and 500M models on the following datasets (one at time): "Monotonic Increasing". "Monotonic Increasing with Jumps". "In-Context Mapping". and "Random Digits", We observed different memorization profiles (same as Fig. 2ap. with "Random Digits" sequences being memorized last.
Hence, differentiation and overthinking are both applicable at larger model scales.
A.6 Attention Heads
In our initial experiments, we observed that the number of attention heads does not impact the results, and hence we chose to use a single attention head. We did more experiments with models having 12 attention heads and found that results still hold. Specifically, having 12 attention heads on 500 examples each from "Monotonic Increasing" and "Monotonic Increasing with Jumps" datasets. We observed different memorization profiles (same as Fig, Zap. We also trained models with 4M and 8M parameters on 500 examples of "Random Digits" data and noticed delayed memorization in the 8M model (like Fig- [4.. Hence the main results of study ('differentiation' and 'overthinking') are not impacted by a change in the number of attention heads.
train 4M-parameter model
B
Dataset ablations
B.1 Controlling the Degree of Shuffed-ness
In T show that the memorization of the dataset drops by 30% when the WikiText data is shuffled at the word level. We also check the effect of varying the degree of shuffled-ness between the original and word-level shuffled states. We shuffle at the sentence and multi-sentence levels. Morcover, we try shuffle variant where we preserve the 2- and 3-grams in the top 50th percentile (in terms of occurrence counts) as one unit and shuffle them with the rest of the sequence. This lets us control the extent of patterns in the dataset. We present the 'exact match' results on 0.5M (trained on 100 examples) and 1M (trained on 200 examples) models in Table Consistent with our claims, we observe that the memorization capacity of a model is dependent on the extent of exploitable structure present in the data. With reduced structure (increased degree of shuffled-ness) the model memorizes less data.
Shuffle variant
100 ex; 0.5M
200 1M
Baseline (original WikiText)
0.96
0.96
Shuffle 2-sentence units Shuffle sentences Preserve top 50% 3-grams
0.94
0.95
0.93
0.92
0.78
0.78
Preserve top 50% 2-grams
0.72
0.73
Word-level shuffled
0.68
0.63
Table 5: Effect of different shuffling strategies on memorization (exact match scores are shown).
B.2 Further Studies with Synthetic Datasets
To understand memorization dynamics more clearly, we do some more in-depth studies with our synthetic datasets by varying the different dataset hyperparameters. We present the results from these experiments with 4M models trained on 500 examples from the indicated datasets.
1. Vary the jump probability in MIJ dataset. We check for the following jump probabilities (in %): 10, 15, 20, 40, 50. As the probability of jumps increases (thereby increasing randomness), it takes longer to memorize the dataset, which is consistent with our claims.
2. Use or letters instead just after each jump for MIJI dataset. When more letters are used after the jump, memorization is faster. As the sequences are of finite length, with increased number of monotonically increasing letters, the randomness decreases, making the sequences easier to memorize.
27

---

Mac Famer Natee
3-
3.03
4:42
4.40
0.93
2/3
2-
RCL
200 430 2006
2000
4000
3000
230
023
3300
2000
4030
3003
Figure 10: Minimal model size for memorization scales linearly with dataset size. Maximum Exact Match and final training loss across different model and dataset sizes are plotted. When models have enough capacity memorize the dataset, the exact match is close to 1 (red region). When the models lack sufficient capacity, the exact match is close to (blue region). Minimal models for each dataset size lie at the boundary of these two regions.
3. Check for 4/5/6/8 tokens ago as in the "Repeat 5 Tokens Ago" dataset. Exact Match curves for all these runs (4/5/6/8 tokens ago) jump from 0 to 1 in the range of [230, 280] epochs without any particular order. This denotes that there no significant difference in the difficulty of these datasets in terms of memorizability.
400 ex; 2M
4000 ex; 12M
sequences maamoriuedd range appchs
Numoer of sequences first memorited in range spochs
20:2.%
4003
2043
230
2006
X5.5%%
1043
Gacleta
spach spachu
N Number of aequiizze finst forgotion in range of spachs
Number becLe lin: forgetien wpochn
3003
200
2500
2000
U803
503
L75%
1.22%
2.00%
3,70
100
100 (000
1853
2000 4837
5000
Figure 11: First Memorized and First Forgotten statisties for models trained on WikiText. First row denotes the model configurations. The percentage of total sequences that make up each bar is also written on top of it. We note that models memorize most sequences for the first time early on: larger models do it faster. Also, first forgotten and first memorized statistics are similar.
28

---

C
Note on Scaling
While the main subject of our study is the training dynamics of memorization, we make an interesting observation regarding how it scales. We trained models of different sizes on datasets of different sizes until the training loss saturated, and found that the size of the smallest model needed to memorize the training data (EM Z 0.95) scales linearly with the dataset size (Fig. 10
D Further Dynamics of Memorization
D.1 First-Memorized and First-Forgotten Statisties
In $4.1 we saw that memorization is dynamic and sequences switch between memorized and forgotten states. We also track the epoch when sequence is first memorized and first forgotten. These two distributions are plotted in Fig. [ for "400 ex; 2M" and "4000 ex: 12M" configurations trained on WikiText. We make two observations: (a) models do prefer to (first) memorize most sequences early on, with larger models (first) memorizing in fewer epochs; (b) firs-mmmorized and forst-forotttn distributions are similar. We also plot these metrics for mixtures of Wiki'Tet and Random Digits datasets for "2000 ex; 8M" model in Fig. 12 and make same observations. Additionally, we observe the two distinct phases (as in Fig. corresponding to the memorization of WikiText and random data.
0.25
0.5
400
midied
Wiliier Handors Set
kandon Sec
00
52.00%
.3.3%%
socch uastees
spOCh LACAME
of suquanceu First fongetien a ranga spschs
sequsnces forgotten in "argge cf spectr
200
100
Kendon
Nunden Sue.
4.3m
40-
24.8%
Epach Dackda
Eocch Buckdda
Figure 12: First Memorized and First Forgotten statistics for 8M-parameter models trained on a mixture of WikiText and Random Digits data. Topmost row denotes data in the mixture. The percentage of sequences (out of 2000) memorized across epochs is plotted. WikiText sequences are (first) memorized prior to Random Digits sequences.
the fraction of random
D.2 Using Softer Metrics to Measure the Dynamic Nature of Memorization
We showed that sequences continue to be forgotten and re-memorized throughout training (i.e., memorization is dynamic) by using exact match as the metric. Here, we check if the same observation
29

---

Epochs
Topics
Topic 0: for to and they that males be it are episode
Topic 1: the was by and to for of on
< 500
Topic 2: the of and in to was on as that with
Topic 3: of is her and she for the as to
Topic 4: is and to his it at that with he on
Topic 0: 16 1961 21 22
24
27
28 29 31 36
Topic 1: 16 1961 21 22
24
27
28
29 31 36
Topic 2: 16 1961 21
27
28
29 31 36
500 2000
Topic 3: the of and size castles distinctive has great
geography for
Topic 4: 16 1961 21 22 24 27 28
29 31 36
Table 6: Topic modeling results (top 5 topics) sequences memorized at different points during training for the "2000 ex; 8M" model.
S. No.
Sequence
1915 16, 1917 18, 1918 19, 1920 21,
1921
22, 1923
24, 1926 27, 1927 28, 1928 29, 1930
31, 1935 36,
1.
1936 37, 1948 49, 1949 50, 1950 51, 1955 56, 1957
58, 1959 60, 1961 62, 1963 64,
The size of these castles varied depending on the geography of the site, the decisions of the builder and the available resources. Analysis of the size of mottes has shown some distinctive regional variation East Anglia, for example, saw much larger mottes being built than the Midlands or London. while motte @-0 and @-@ bailey and ringwork castles took great effort to
Table 7: Sequences not memorized by "4000 ex; 12M" model.
holds under a "softer" metric. We track the fraction of times a sequence is sampled (out of 5 generations) with a temperature of 0.5. We call this sampling accuracy (Sace). We also track the status of each sequence at given timestep: 0 (forgotten), (memorized), as found via exact match. We report our findings on the configurations: "100 ex; 0.5M", "200 ex; 1M", and "400 ex; 2M".
We average status and 8ace across sequences and note that their trend across epochs is similar to Fig. [ (rising from 0 to 1) and that they almost overlap. This suggests that the assignment of sequence status using exact match agrees with this softer sampling accuracy metric.
The average Sace across the sequences with status 0 (forgotten) was about 10 15% while the average Sace across the sequences with status (memorized) was about 99%. This strongly suggests that sequences that tend to be forgotten, detected exact match, also tend to have low chance of being generated by the model.
Also, we use exact match since softer measures are not relevant in the context of PII or copyright infringement, which are real-world memorization issues.
D.3 Sequences Memorized across Different Epochs
To understand what type of sequences are memorized at different points during training. we perform topic modeling on different epoch buckets. The topics so obtained for the "2000 ex; 8M" model are shown in Table Note that the topics for epochs > 500 primarily have numbers. Moreover, we observe that the sequences that are never memorized also sometimes happen to be abundant in numbers. In Table 7 show such sequences for the "4000 ex; 12M" model.
30

---

D.4 Metries for Non-Memorized Sequences
We also track memorization metries for sequences that are memorized (EM = 0) any given point during training. We show three such metrics in Fig. [3] for "400 ex; 2M" model, and make two observations: (a) Initially, when nothing memorized, the model gradually memorizes parts of each sequence; (b) Later on, as more sequences get memorized, the ones that are being forgotten still have most of their parts memorized.
Figure 13: Metries for non-memorized sequences. Individual and Cumulative 4-gram BLEU, and 13-gram Jaceard Similarity are shown for "400 ex; 2M" model. Note that all metrics gradually rise until saturation.
31


Reviews:
[{"review_id": "F7hIW5WDGg", "reviewer": "Reviewer_eVay", "summary": "This paper investigates the dynamic and pattern-driven mechanisms of memorization in Transformer-based language models. The authors demonstrate that memorization is not a simple accumulation of training sequences nor determined by recency of exposure. It is driven by pattern recognition.", "questions": "Are there any reasons for choosing GPT-2 (2019) for the experiments? Typically, smaller models are chosen for interpretability reasons, but I didn't see a thorough interpretability study in the paper.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The authors validate that memorization in Transformer models is pattern-driven rather than simply a matter of sequence accumulation or recent exposure.\n- The memorization behavior of the models is demonstrated to be driven by patterns rather than gradual accumulation or recency.\n- The authors find the \"overthinking\" phenomenon, in which larger models exhibit delayed memorization on data.\n\nWeaknesses:\n- The largest model used in the experiments is only 85M parameters, this is hard to conclude that \"overthinking\" would also occur in larger models.\n- The paper lacks recent literature. I suggest that the authors pay more attention to recent papers on the understanding of memorization in language models, such as: https://arxiv.org/abs/2404.15146, https://arxiv.org/abs/2411.07175, https://arxiv.org/pdf/2404.13501, and https://arxiv.org/abs/2504.12523 etc.", "was_revised": true, "final_justification": "I updated my score because the authors provide the detailed related work and address my concern about the model size", "rebuttal": "We thank the reviewer for highlighting the strong results of our work, such as memorization being driven by patterns and larger models exhibiting delayed memorization (overthinking). They had a concern about the relatively small model sizes used in the paper and about the paper missing some references. We ran additional experiments, which further support our results and provide an updated section on related work. We will include them in the paper to make the resulting manuscript even better. We respond to their comments below.\n\n**[W1; Q1] Model sizes.** In our work, we find that larger models show delayed memorization on data with weak patterns. We call it overthinking. The reviewer expressed a concern regarding the validity of this result at larger model scales. We replicated the setup in the paper and performed new experiments on model sizes 100M, 200M, 500M, and 1B on datasets “Every 7th token same (10% sequences)” and “Monotonic Increasing with Jumps”. We consistently observe that larger models take longer before starting to memorize sequences in these datasets (same as Fig. 5). We note that these training runs demonstrating overthinking can be very computationally expensive for larger models, as they take much longer to start memorizing those datasets, limiting the model sizes we can include in the study. However, these results up to 1B scale conclude that overthinking does hold at large scales. We will include them in the final version.\n\n**[W2] Missing references.** The reviewer highlighted certain related work that is missing from the paper. We thank the reviewer for bringing these references to our attention. Below, we provide the updated related work section that covers all suggested references. We shall include it in the final version, making the related work coverage more holistic.\n\n## Related Work\nMemorization has been a central topic in machine learning since the earliest neural networks. We organize prior work on this topic into six themes: (1) classical and associative memory models, (2) memorization in modern language models, (3) privacy and data extraction, (4) memorization versus generalization and pattern-seeking behavior, (5) interpretability and training dynamics, and (6) related learning phenomena such as forgetting and curricula.\n\n### Classical and Associative Memory Models\nThe study of neural associative memories began with Hopfield networks [1], which showed that recurrent networks could store discrete patterns as stable attractors. These systems, along with bidirectional associative memories [2] and sparse distributed memory [3], were designed to explicitly encode and retrieve stored patterns from the network’s weights, leading to a conception of neural networks as content-addressable memories. \n\nWhile modern transformers use attention and contextual representations rather than discrete attractors, these classical models highlight that the ability to store and recall information has always been a fundamental property of neural architectures. Recent surveys emphasize that modern LLMs behave differently from these classical models, exhibiting implicit memory through distributed representations, attention mechanisms, and in-context learning rather than explicit associative storage [4]. This shift motivates a fresh look at memorization in the transformer era.\n\n### Memorization in Transformer Language Models\nIn modern models, memorization is less deliberate but equally real. Large-scale studies demonstrate that transformers memorize exact sequences from their training data. Starting with the early extraction experiments in GPT-2 [5], research has repeatedly shown that certain examples (especially those that are unique or duplicated) can be reproduced verbatim during inference [6, 7, 8, 9]. Empirical evidence indicates that memorization grows with model scale, dataset duplication, and training length. Strategies such as deduplicating training data [10] reduce this tendency.\n\nRecent work proposes alternative ways of thinking about memorization, including counterfactual memorization [11] (asking what a model would output if a particular example were removed from training) and adversarial compression views [12], which link memorization to how models compress information under overparameterization. Although our focus is on language models, memorization has been observed across modalities: for instance, extraction from diffusion models [13], code models [14], and large vision models [15] demonstrates that this is a general property of overparameterized models.\n\n### Privacy and Data Extraction\nMemorization directly affects privacy and intellectual property. Data extraction studies [5, 16, 6, 7, 9] have shown that sensitive information such as names, addresses, or copyrighted material can be recovered from trained models, even without direct access to their training data. Large-scale attacks on production LLMs [7] have scaled these demonstrations, showing that thousands of training examples can be extracted at low cost, and work on model stealing [17] has shown that the problem persists even when the adversary only queries a deployed model. These results motivate defenses: deduplication [10] and differential privacy-based training can reduce leakage, though at the expense of performance. The risk of leakage is not limited to NLP models but also applies to other modalities [13, 14, 15].\n\n### Memorization vs. Generalization and Pattern-Seeking Behavior\nWhile memorization is often cast as the opposite of generalization, work from deep learning has shown that the relationship is more nuanced. Networks can memorize random labels [18], but in structured datasets, models first capture simpler patterns and only later memorize irregularities [19]. This pattern-seeking inductive bias is particularly important in LMs: when patterns exist, models generalize; when they do not, models ultimately memorize specific examples. Evidence for such biases includes frequency-driven acquisition of syntactic rules [20], mapping of a memorization$-$generalization continuum [21], and studies on the long tail, where rare examples are memorized rather than generalized [22]. \n\nRecent taxonomies [23] emphasize that memorization itself has multiple modes, from simple recitation of exact strings to reconstruction based on partial patterns, suggesting that memorization and generalization can co-exist within the same model. Complementary perspectives view memorization as a form of compression [12], and phenomena like grokking [24] illustrate that generalization can appear abruptly after long periods of apparent overfitting, driven by latent structure-seeking. Our results echo these insights: memorization does not progress monotonically but is shaped by perceived pattern difficulty.\n\n### Interpretability and Training Dynamics\nAnother line of work studies when and how memorization occurs. Training-dynamics analyses have found phase transitions in LMs: loss drops accompanied by sudden acquisition of syntactic or structural knowledge [25]. Representation studies [26] have shown that language models acquire features (syntax, semantics) in a progressive, structured order, which may explain the order in which they memorize or generalize. Beyond training, post-hoc interventions attempt to control knowledge in already-trained models; recent work examines how to update models with new knowledge and balance memorization with reasoning [27]. These threads converge on the idea that memorization is not just an endpoint but a process: understanding it requires looking at how representations and behaviors evolve during training.\n\n### Related Learning Phenomena: Forgetting and Curricula\nMemorization is closely connected to forgetting and curricula. Catastrophic forgetting, where models overwrite previously learned information, was first documented in early connectionist systems [28, 29]. It remains an issue even in large models; fine-tuning can cause previously memorized examples to be forgotten [8]. New approaches, such as continual memorization [30], highlight the need to balance stability and plasticity.\n\nData ordering also shapes memorization: curriculum learning [31] biases models to learn easier, pattern-rich examples first, postponing rote memorization. These observations resonate with our findings that transformer LMs, even without explicit curricula, exhibit a natural ordering in what they memorize, preferring structured patterns first.\n\n**Following are all the references as DOIs. For arXiv papers, the DOI is the Crossref form 10.48550/arXiv.XXXX.XXXXX. For books / journal articles, the original DOI is used where available.**\n\n\n[1] 10.1073/pnas.79.8.2554\n\n[2] 10.1109/21.87054\n\n[3] Kanerva, P. (1988). Sparse Distributed Memory. MIT Press.\n\n[4] arXiv.2404.13501\n\n[5] arXiv.2012.07805\n\n[6] arXiv.2202.07646\n\n[7] arXiv.2311.17035\n\n[8] arXiv.2207.00099\n\n[9] arXiv.2210.17546\n\n[10] arXiv.2107.06499\n\n[11] arXiv.2112.12938\n\n[12] arXiv.2404.15146\n\n[13] arXiv.2301.13188\n\n[14] arXiv.2308.09932\n\n[15] arXiv.2310.05337\n\n[16] arXiv.2206.10469\n\n[17] arXiv.2403.06634\n\n[18] arXiv.1611.03530\n\n[19] arXiv.1706.05394\n\n[20] arXiv.2109.07020\n\n[21] arXiv.2311.05379\n\n[22] arXiv.2008.03703\n\n[23] arXiv.2406.17746\n\n[24] arXiv.2201.02177\n\n[25] arXiv.2309.07311\n\n[26] arXiv.1811.00225\n\n[27] arXiv.2504.12523\n\n[28] 10.1016/S0079-7421(08)60536-8\n\n[29] 10.1037/0033-295X.97.2.285\n\n[30] arXiv.2411.07175\n\n[31] 10.1145/1553374.1553380"}, {"review_id": "oxJyJaBJXP", "reviewer": "Reviewer_p7BT", "summary": "This paper empirically studies “discoverable” memorization (given a prompt, can the model exactly reproduce the subsequent tokens as in the training set) in language models via training dynamics. The main claim is that the model ‘seeks patterns’ during training for memorization, and does so in stages, each stage corresponding to patterns of certain complexity. Further, memorization of a sequence is not dependent on its recency in the training process. Overall, memorization is not simply copying verbatim sequences, but based on a notion of pattern recognition. They use toy examples as well as real-world language data to validate their hypotheses on models of different sizes.", "questions": "1. Model hyperparameters: I am not sure I completely follow the logic in selecting model sizes: why is E/L ratio a good metric for choosing language models for memorization? In fact, why the emphasis on embedding parameters - is there reason to believe that these parameters directly affect memorization or generalization in language models? The reasoning in the paper seems to be that some open source models use E/L in some (arguably large) range, which I did not completely follow in the above context. Also, why keep the number of attention heads = 1? Overall, I encourage the authors to rewrite this section for more clarity on these choices.\n\n\n2. \"Result 1: Memorization is neither a constant accumulation of sequences nor recency-biased.\" - I understand that constant accumulation / recency bias might intuitively make sense, but not sure if this has been claimed earlier? Mainly because I am not sure how a neural net like Transformer would implement ‘constant accumulation’ of training sequences. Could the authors clarify if this indeed happens (or if this could never happen) in some setup?\n\n\n3. The toy datasets used in the paper are interesting and do model many aspects of memorization: however I believe a little more in-depth study would help understand memorization dynamics more clearly. Concretely, take the example of monotonic increase with jumps: instead of just using 15% probability, maybe vary this probability from 10-50% and then see how dynamics are affected. Monotonic increase with jump and letter: maybe use 2/3/… letters instead of just 1 after each jump. Repeat 5 tokens ago: check for 4/6/8... tokens ago instead, etc.\n\nI am also curious about shuffled-WikiText: is there a way to control the degree of ‘shuffled-ness’? Maybe “more locally / more globally” shuffled than on a sentence / sequence level (e.g. multi-sentence shuffling)? It would be interesting to see what the results are on varying this characteristic.\n\nTo be clear, I do not expect the authors to implement all of this, and these are just some suggestions on how the study could be improved and made more systematic to derive better conclusions. \n\n4. Line 193: “We also observe two distinct type of curves: sudden jump vs gradual rise, which also characterizes the data the model is trained on” - could the authors clarify how it characterizes training data beyond being one of the 2 types of curves?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n- Studies a timely and important problem of memorization in language models. \n- Experiments are well-motivated and clearly formulated, using various types of toy datasets. \n- Paper is well written and easy to follow - the result takeaways at the beginning of each section are useful.  \n\nWeaknesses:\n\n- Figure 2 is difficult to read - could the authors improve the visibility of the text in the left portion of this figure?\n- Some model hyperparameter choices are not clearly motivated in the paper, specifically the emphasis on embedding parameters in Sec 3 and using only 1 attention head per layer. \n- While the toy datasets used are interesting and relevant to memorization, I believe these could be studied a little more systematically. Please see Questions section below for discussion on this and other issues that are not exactly weaknesses.", "was_revised": true, "final_justification": "My review was mainly about some clarifying questions (on model hyperparameters etc.), and some suggestions for ablations to improve their experimental analyses. They have addressed both these aspects in their rebuttal, and hence I've raised my score to 5.", "rebuttal": "We thank the reviewer for their insightful feedback and suggestions to improve the quality of our work. We are glad that they found our experiments well-motivated and liked the presentation of the paper. The reviewer raised a concern regarding some hyperparameter choices and suggested further experimentation with our datasets. Below, we provide results from new experiments that further strengthen our claims. We will include these results in the final version of the paper.\n\n**[W1] Figure 2 visibility.** We shall update the figure to be more readable and also include the same text in the Appendix.\n\n**[W2; Q1.1] Clarification regarding embedding parameters and selecting model configurations.**\n> Some model hyperparameter choices are not clearly motivated in the paper, specifically the emphasis on embedding parameters in Sec 3…\n\n> I am not sure I completely follow the logic in selecting model sizes: why is E/L ratio a good metric for choosing language models for memorization?\n\nWe believe there is a misunderstanding of what $E$ denotes. It is the size of the hidden dimension, i.e., model width. We will clarify this in the paper. We select model configurations for different parameter counts by maintaining a width-over-depth ratio ($E/L$) in the same range as present-day models like Llama 3.\n\n**[W2; Q1.2] Number of attention heads.**\n> Also, why keep the number of attention heads = 1?\n\nIn our initial experiments, we observed that the number of attention heads does not impact the results, and hence we chose to use a single attention head. We did more experiments with models having 12 attention heads and found that our results still hold. Specifically, we train a 4M-parameter model having 12 attention heads on 500 examples each from “Monotonic Increasing” and “Monotonic Increasing with Jumps” datasets. We observed different memorization profiles (same as Fig. 2). We also trained models with 4M and 8M parameters on 500 examples of “Random Digits” data and noticed delayed memorization in the 8M model (like Fig. 5). Hence the main results of our study (differentiation and overthinking) are not impacted by a change in the number of attention heads. We will include these results and clarify the choice of the number of heads in the paper. \n\n**[Q2] Clarification on constant accumulation.** The reviewer asks if we observe constant accumulation of training sequences in some setup and seeks further clarification on this aspect. Firstly, we would like to reiterate that by constant accumulation, we mean a non-dynamic memorization where a sequence once memorized is never forgotten. This hypothesis is implicit in how the memorization of 'Personally Identifiable Information’ (PII) is often framed in discussions around training on such sequences, since the model potentially processes such information only once in the large pretraining corpus. We show that even in the setting most friendly to constant accumulation (training on the same data for multiple epochs), it is not observed in our setup. The other prevailing hypothesis could be that memorization is governed by the recency of sequences seen, which we also show to not be the case.\n\n**[W3; Q3.1] Further studies using toy datasets.** We thank the reviewer for suggesting experiments with different variations of our toy datasets. We present the results from these experiments with 4M models trained on 500 examples from the indicated datasets.\n\n> Monotonic Increasing with Jumps: vary the jump probability from 10-50% and then see how dynamics are affected\n\nWe check for the following jump probabilities (in %): 10, 15, 20, 40, 50. As the probability of jumps increases (thereby increasing randomness), it takes longer to memorize the dataset, which is consistent with our claims.\n\n> Monotonic Increasing with Jumps and Letters: use 2/3/… letters instead of just 1 after each jump\n\nWhen more letters are used after the jump, memorization is faster. As the sequences are of finite length, with an increased number of monotonically increasing letters, the randomness decreases, making the sequences easier to memorize.\n\n> Repeat 5 tokens ago: check for 4/6/8... tokens ago instead\n\nExact Match curves for all these runs (4/5/6/8 tokens ago) jump from 0 to 1 in the range of [230, 280] epochs without any particular order. This denotes that there is no significant difference in the complexity of these datasets in terms of memorizability.\n\nThese results further strengthen our observations, and we will include these additional studies in the appendix.\n\n**[W3; Q3.2] Controlling the degree of ‘shuffled-ness’.**\nIn the paper (Table 3), we show that the memorization of the dataset drops by 30% when the WikiText data is shuffled at the word level. The reviewer suggested varying the degree of shuffled-ness between the original and word-level shuffled states. As suggested, we shuffle at the sentence and multi-sentence levels. Moreover, we try a shuffle variant where we preserve the 2- and 3-grams in the top 50th percentile (in terms of occurrence counts) as one unit and shuffle them with the rest of the sequence. This lets us control the extent of patterns in the dataset. We present the ‘exact match’ results on 0.5M (trained on 100 examples) and 1M (trained on 200 examples) models. Consistent with our claims, we observe that the memorization capacity of a model is dependent on the extent of exploitable structure present in the data. With reduced structure (increased degree of shuffled-ness), the model memorizes less data. We will extend Table 3 with these results.\n\n| Shuffle variant                                      | 100 ex; 0.5M | 200 ex; 1M |\n|-------------------------------------------------|-------------------|-----------------|\n| Baseline (original WikiText)                                        | 0.96              | 0.96            |\n| Shuffle 2-sentence units                      | 0.94              | 0.95            |\n| Shuffle sentences                        | 0.93              | 0.92            |\n| Preserve top 50% 3-grams          | 0.78              | 0.78            |\n| Preserve top 50% 2-grams          | 0.72              | 0.73            |\n| Word-level shuffled                                        | 0.68       | 0.63            |\n\n\n\n> **[Q4]** Line 193: “We also observe two distinct type of curves: sudden jump vs gradual rise, which also characterizes the data the model is trained on” - could the authors clarify how it characterizes training data beyond being one of the 2 types of curves?\n\nThis is a typographical error on our part. We will correct it as follows: “We also observe two distinct type of curves: sudden jump vs gradual rise, which is another characteristic feature of the data the model is trained on”."}, {"review_id": "pjgS8AsHfx", "reviewer": "Reviewer_mUn5", "summary": "This paper investigates the dynamics of memorization in transformer-based language models, focusing on how memorization evolves during multi-epoch training. The authors challenge the view that memorization is either a monotonic accumulation of data or driven solely by recency. Through controlled experiments on WikiText and synthetic datasets, they show that memorization occurs in bursts and is closely tied to the pattern complexity of the data. Models tend to memorize data with simpler or more salient patterns earlier and may “overthink” datasets lacking clear structure—delaying memorization. The paper introduces the notion of pattern-dependent memorization and highlights an intriguing phenomenon: larger models tend to spend more time trying to extract structure before resorting to rote memorization.", "questions": "I list my concerns and questions in the \"weaknesses\" section.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The paper offers a dynamic perspective on memorization by tracking how models evolve during training, rather than relying solely on final snapshots.\n\n- The idea that memorization is pattern-seeking (not just size- or exposure-driven) is well-supported through a variety of datasets with controlled structural complexity.\n\n- The observed \"overthinking\" behavior in larger models adds a novel insight into model capacity and inductive bias.\n\nWeaknesses: \n- The experimental setup focuses on relatively small models (mostly up to 12M non-embedding parameters), which may limit the direct applicability of findings to larger-scale LLMs.\n\n- The definition of \"pattern complexity\" remains somewhat qualitative; a more quantitative or formal measure would make the analysis more generalizable.", "was_revised": true, "final_justification": "Thanks for the detailed responses. All my concerns are addressed.", "rebuttal": "We thank the reviewer for their feedback. We are glad that they liked our experimental design and think that our claims are well-supported. Furthermore, we appreciate that they recognize our observations as novel insights into models’ inductive biases. They expressed a concern regarding the use of relatively small models and the definition of pattern complexity. Below, we respond to their queries by reporting results from new experiments and providing clarifications.\n\n**[W1] Model sizes.**\n> The experimental setup focuses on relatively small models (mostly up to 12M non-embedding parameters), which may limit the direct applicability of findings to larger-scale LLMs.\n\nWe experiment with larger models to show evidence for the applicability of our findings to larger model scales. \n\nWe replicated the setup in the paper and performed new experiments on model sizes 100M, 200M, 500M, and 1B on datasets “Every 7th token same (10% sequences)” and “Monotonic Increasing with Jumps”. We consistently observe that larger models take longer (aka overthinking) before starting to memorize sequences in these datasets (same as Fig. 5). Please note that these training runs demonstrating overthinking can be very computationally expensive for larger models, as they take much longer to start memorizing those datasets, limiting the model sizes we can include in the study. However, these results up to 1B scale conclude that overthinking does hold at large scales.\n\nWe further trained 200M and 500M models on the following datasets (one at a time): “Monotonic Increasing”, “Monotonic Increasing with Jumps”, “In-Context Mapping”, and “Random Digits”. We observed different memorization profiles (same as Fig. 2), with “Random Digits” sequences being memorized last.\n\nHence, differentiation and overthinking are both applicable at larger model scales. We will include these results in the final version of the paper.\n\n**[W2] Definition of ‘pattern complexity’.**\n> The definition of \"pattern complexity\" remains somewhat qualitative; a more quantitative or formal measure would make the analysis more generalizable.\n\nWe thank the reviewer for making this important point. We would like to clarify that there are two different perspectives on pattern complexity:\n1. Complexity of the underlying data-generation process / algorithm\n2. Learnability / Memorizability of the pattern by the model using gradient-based methods\n\nUnder the first perspective, for datasets with randomness (e.g., jumps, random repeats, etc.), the amount of randomness gives a gradation of complexity. For deterministic datasets (Monotonic Increasing, In-Context Mapping, etc.), the number of lines of the shortest data-generating program serves as a measure of complexity (Kolmogorov complexity). However, it is difficult to compare patterns across these two dataset classes.\n\nIn our paper, we focus on the learnability perspective to pattern complexity, which is tougher to define independently of the model. Our work highlights that the model has different affinities for memorizing different patterns, thereby suggesting that different patterns have different learnability. We believe that our work will motivate further analysis into this aspect of pattern complexity. We will include this discussion in the final version."}, {"review_id": "lWcMYwaAbD", "reviewer": "Reviewer_eqVj", "summary": "This paper studies the question “What affects which facts are memorized by a language model during pretraining and when/at what rate they are memorized?” The study is performed on small GPT-2 models trained on subsets of Wikitext, on small synthetic datasets containing, e.g., random digits, and combinations of both. The authors find that (1a) strings continue to be forgotten and re-memorized throughout training (1b) It is not always the most recent strings that are “freshly” memorized at any moment (2) subsets of the data that follow clearer patterns are memorized earlier, with clear phase shifts when training on mixture datasets (3) larger models delay memorization of strings with no patterns, which the authors speculate is because these models try to seek patterns to learn before memorizing a given sequence.", "questions": "- As mentioned in section 2 of Weaknesses, do one or both of the two metrics I suggested (or any others that measure a “softer” sense of forgetting) show that sequences are indeed forgotten and re-learned all the time?\n- Could you present more evidence or arguments as to why the results of section 4.4 should be interpreted as the model seeking patterns (as opposed to the larger model learning facts more slowly for some other reason)?\n- Could you clarify the use of the terms “pattern” and “generalization”? Ideally, a more formal definition would be welcome, as that would help delineate what results should be more or less surprising.\n- Out of all the results that are presented in the paper, which one would you say is the central contribution of your paper, and the one that summarizes the paper the best?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- From a scientific standpoint, I find the questions raised at the beginning of the paper very intriguing—specifically, what dictates which facts are memorized by a language model and how fast?\n- Some of the results here are surprising (at least to me). For example, the fact that a larger model might take longer to memorize some sequences is interesting, and to my knowledge, novel.\n- The setup is described clearly, and the description of the experiments is succinct but informative. Throughout, I had no trouble understanding what the current experiment was trying to do. \n\n**Weaknesses**\n\n- I don’t think that the setup adopted here is a faithful description of model pretraining. For example, Figure 1 trains on 400 examples for over 7000 epochs! In reality, we have millions or billions of examples for 1-4 epochs. Several of these results only seem to hold in the regime of 300+ epochs.\n- The claim that sequences are constantly forgotten and relearned warrants further study. If I am correct, the authors use greedy decoding (n=1) and use Exact Match to check if a sequence has been forgotten. It would be more suitable to instead measure if a sequence remained among the top few completions preferred by the model, measurable using either perplexity or the fraction of times the exact sequence was sampled when sampling several times with a temperature. I suspect that with these metrics, sequences will appear to have been forgotten much fewer times throughout training.\n- The word “pattern” is used rather loosely throughout the paper, which renders some of the takeaways ambiguous. I’ll try to give an example: the phrase “In the year…” might appear several times in this same order in the corpus, but I would not consider this trigram a “pattern”. In that case, I don’t find the results of Table 3 surprising - these sequences of words that always appear in the same order get jumbled when you shuffle the sequences, and hence, you get sequences that are harder to memorize. These issues might go away with a more rigorous definition of “pattern” and “generalization.”\n- Result 4 in the paper anthropomorphizes the language model, which leads to a leap in conclusions. For one, I think it would take much more evidence to make the case that (larger models memorize random facts slower) implies (larger models seek patterns to learn and hence take longer to memorize random facts). Besides, I don’t think any of this involves any “thinking” on the part of the model, because this describes the training process, not inference. Therefore, I find the term “overthinking” misleading.\n- More generally, I think the paper does not have a very clear direction in its current form. I will concede that it opens with an interesting question and has some very interesting results, but the results do not come together to tell a story or describe any particular phenomenon. In particular, I think the section on generalization to static v/s dynamic patterns does not fit well into a paper that is interested in understanding memorization.\n- While I don’t want to play the role of citation police too much, I do think that this paper needs a much more expansive study of related work—the related work section has no citations from before 2023, except for a 2017 paper. The problem of memorization has been studied for a very long time—as far back as 1989 (see [1] and [2] as examples).\n\n[1] McCloskey, M., & Cohen, N. J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem. In G. H. Bower (Ed.), Psychology of Learning and Motivation (Vol. 24, pp. 109–165).\n\n[2] Ratcliff, R. (1990). Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97(2), 285–308.", "was_revised": true, "final_justification": "I will paste the response to the authors here, as I think it summarizes the justification well.\n- I am glad to see an expansive list of related work. This addresses that concern.\n- The clarification of the narrative is welcome, and I would encourage the authors to also apply this to streamline their paper further.\n- I acknowledge that the authors do not wish to anthropomorphize the model. Yet, some readers and especially future work might mistakenly do so upon reading this paper, so I suggest that a different word than \"seeking\" be used when describing the delayed onset of learning, if possible. However, this is just a minor technical point.\n- The expansion of results to models up to 1B also greatly strengthens the paper, as do the results with soft metrics. I will still contend that using 100-1000 epochs is rather excessive and may skew takeaways.\n\nOverall, I find that the rebuttal is reasonably convincing. I will therefore raise my score to \"3: borderline reject\" keeping both the improvements and the lingering concerns in mind.", "rebuttal": "We thank the reviewer for their insightful feedback. We are grateful that they found our research questions intriguing, our results surprising and novel, and our setup clearly described. They demanded more support for our claim about the dynamic nature of memorization (sequences constantly being forgotten and re-memorized), which we provide via new experiments. They also sought clarification on other claims in the paper and highlighted missing citations. We respond to their comments below.\n\n**[W1] Differences from real-world model pretraining.** The reviewer highlighted that our multi-epoch training setup is different from the training of much larger real-world models that are typically trained for much fewer epochs.\n\nAs for model size, we found our results to be applicable up to 1B parameter scale. (details in response to Reviewer mUn5; omitted here due to length limits) We will include them in the final version.\n\nMoreover, in this work, we set a particular scope to understand memorization in a more synthetic setup. It would be quite interesting to extend it for real-world pretraining. For such large datasets, we need to measure semantic (as opposed to verbatim memorization), as information can recur in a paraphrased manner. We see our work as the first step to understanding these dynamics.\n\nFurthermore, real-world data may contain duplicates. [1] found instances of a 61-word sentence in C4 dataset, which was repeated over 60,000 times. [2] raised concerns about duplicate data in several social media datasets. Training on a dataset with such duplicates is similar to a multi-epoch training regime. While deduplication is done on large datasets, it is extremely computationally expensive, especially for detecting near-duplicates. Our setup applies to these duplicate sequences.\n\n[1] Lee et al., \"Deduplicating Training Data Makes Language Models Better.\", ACL 2022.\n\n[2] Mu et al., \"Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research.\", EMNLP 2024.\n\n**[W2; Q1] More support for the dynamic nature of memorization.** We showed that sequences continue to be forgotten and re-memorized throughout training (i.e., memorization is dynamic). The reviewer questioned the use of exact match and suggested using softer measures of forgetting. Following their suggestion, we track the fraction of times a sequence is sampled (out of 5 generations) with a temperature of 0.5. We call this sampling accuracy ($s_{acc}$). We also track the status of each sequence at a given timestep: 0 (forgotten), 1 (memorized), as found via exact match. Below we report our findings on the following three models (similar to Table 1): 0.5M (trained on 100 examples), 1M (200 examples), and 2M (400 examples).\n1. We average status and $s_{acc}$ across sequences and plot them across epochs. The two plots show a trend similar to Fig. 1 (rising from 0 to 1) and almost overlap. This suggests that the assignment of sequence status using exact match agrees with this softer sampling accuracy metric.\n2. The average $s_{acc}$ across the sequences with status 0 (forgotten) was about 10-15%, while the average $s_{acc}$ across the sequences with status 1 (memorized) was about 99%. This strongly suggests that sequences that tend to be forgotten, as detected by exact match, also tend to have a low chance of being generated by the model.\n\nWe also mention Fig. 14 (Appendix), where we plot metrics for forgotten sequences. The BLEU scores have a median of about 0.7. This suggests that multiple n-grams mismatch (as opposed to just a few tokens) when a sequence is deemed forgotten by exact match.\n\nAlso, we use exact match since softer measures are not relevant in the context of PII or copyright infringement, which are real-world memorization issues.\n\nWe thank the reviewer for their comment, as our claim is even stronger with these additional insights. We will include these results in the final version.\n\n**[W3; Q3] Definitions of ‘pattern’ and ‘generalization’.** We clarify these definitions below.\n\n*Pattern.* Assume an ideal (oracle) machinery $M$ that can learn any data generation process most efficiently. Then, for a dataset with patterns, $M$ learns a model better than unigram statistics. By ‘pattern’, we refer to any structure in the dataset that can be exploited by $M$.\n\nThe reviewer also expresses doubt on whether a repeating trigram like “In the year…” is a pattern. Since the trigram appears multiple times across the dataset, this shared context can be used to memorize sequences (e.g., making the prediction of “year” more likely given “in the”). Therefore, $M$ will utilize it, and hence it is indeed a pattern.\n\nThis also addresses reviewer’s concern about interpreting Table 3 results. When WikiText is shuffled, any shared n-gram context across sequences is lost. A model that could memorize WikiText is unable to memorize this shuffled dataset (without patterns, the dataset is harder to memorize and requires a model with more capacity).\n\n*Generalization.* In the context of our paper, a model is said to have generalized when it is shown to generate correct completions for unseen sequences that share patterns with the training dataset. We detail this in section 4.3 and in response to W5 below.\n\nWe will make these definition-related clarifications in the paper and include this discussion on the interpretation of results in the Appendix.\n\n**[W4; Q2] Clarification on “overthinking”.** The reviewer showed concern regarding the framing of our “overthinking” result, where a larger model takes longer to memorize certain datasets with weak or absent patterns.\n\nTo provide more context for this result, we refer to section 4.3, where we show generation statistics for models trained on “Monotonic Increasing with Jumps” dataset. This dataset contains monotonically increasing sequences of single digits with a 15% chance of a random jump to any digit. We observe that these models (4M and 8M) have 97.3% and 98.1% probability of generating a monotonically increasing digit, larger than the 85% probability of that happening in the dataset, suggesting that the models tend to strongly follow an identified pattern, with the larger model having more affinity. While this difference in probabilities for 4M and 8M models is small, so is the difference in epochs (~80) it takes them to memorize this “with jumps” dataset (Fig. 5).\n\nGiven these observations, a likely and simple explanation (Occam's razor) is that the delay in section 4.4 is caused by the pattern-seeking behavior, which is more prevalent in larger models. We do agree that this is a hypothesis, and we will soften the framing accordingly.\n\nWe use the term “overthinking” not to anthropomorphize the LM, but rather to provide a relatable term that conveys the phenomenon.\n\n**[W5] Clarification on the narrative.** The reviewer raised doubts about the story described in the paper. Below, we reiterate the results and how they fit the overall narrative.\n\nThrough this work, we highlight that memorization in LMs is pattern-seeking and call attention to its different aspects. To understand what governs memorization, we first check if it is a constant accumulation of sequences without forgetting, or if it is biased by recency. We find neither of these to be the case. (4.1) Next, via training models on real-world and synthetic datasets, we observe that models memorize different datasets at different rates, leading to dataset-specific memorization profiles. These datasets differ in the extent of patterns present in them, which is correlated with the memorization rate. (4.2) To understand whether the model *really* extracts patterns while memorizing the data, we test it on an unseen validation set that also shares the same patterns. We find that models trained on datasets with static lookups generalize, but the models trained on datasets with dynamic lookups do not. This suggests that pattern extraction happens for memorization in the former, while not in the latter. We also observe that models tend to apply patterns more often than they are present in the training dataset, suggesting a strong affinity, with larger models showing more of it. (4.3) Finally, we observe that larger models take longer to memorize datasets that have weak or no patterns. We suspect this is attributed to the pattern-seeking behavior. (4.4)\n\nIn this way, we view that the results presented in the paper highlight different dimensions of our central thesis (pattern-seeking behavior in memorization).\n\nThe reviewer also asked about a particular section:\n> In particular, I think the section on generalization to static v/s dynamic patterns does not fit well into a paper that is interested in understanding memorization.\n\nAs mentioned above, we use generalization to test *when* models exploit patterns to memorize. If they generalize, they must have extracted patterns; otherwise, individual sequences were independently memorized without regard to shared structure. Therefore, testing generalization allows us to differentiate between such rote memorization and the extraction of deeper patterns (data generation process), which is in line with our central thesis.\n\n**[W6] Missing citations.** We thank the reviewer for suggesting additional related work. We provide the updated related work section in response to Reviewer eVay. (omitted here due to length limits). We will include it in the final version.\n\n> **[Q4]** Out of all the results that are presented in the paper, which one would you say is the central contribution of your paper, and the one that summarizes the paper the best?\n\nAs mentioned for W5, we view our paper as highlighting different aspects of pattern-seeking behavior in memorization studied under a multi-epoch training regime. All results collectively convey that thesis. Since this core insight is founded in all results equally, no single result, on its own, fully summarizes the overall contribution."}]

Author rebuttal:

Meta-review: This paper investigates how transformer-based language models memorize information. The authors argue that memorization isn't just about storing data but is a process driven by pattern recognition, similar to how models generalize. Through experiments, they show that models learn in bursts, tackling simpler patterns first before moving to more complex ones. A key finding is a phenomenon they call "overthinking," where larger models delay memorizing random or pattern-less data, seemingly because they are searching for underlying structures to learn instead.

Initially, the reviewers had several key concerns. These included the use of small models, a lack of both recent and foundational related work, and the use of vague terms such as "pattern." They also questioned the experimental setup, such as training for an excessive number of epochs and using a strict metric for forgetting. In response, the authors expanded their experiments to include models up to 1B parameters, added a comprehensive literature review, and provided clearer definitions. These changes addressed the majority of concerns, leading three of the four reviewers to raise their scores to "Accept" or "Borderline Accept." One reviewer, while acknowledging the improvements and some resolved issues, still have remaining concerns about the training setup and that terminologies used. I think the authors should be able to find what the best terminology is to improve their presentation to the readers.
Decision: Accept (poster)