Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data
Authors: Xinyi Wang
Antonis Antoniades
Yanai Elazar
Alfonso Amayuelas
Alon Albalak
Kexun Zhang
William Yang Wang
Venue: ICLR 2025 Poster (2025)
Keywords: language model
pretraining data
n-gram
memorization
Primary area: foundation or frontier models, including LLMs

Abstract: The impressive capabilities of large language models (LLMs) have sparked debate over whether these models genuinely generalize to unseen tasks or predominantly rely on memorizing vast amounts of pretraining data. To explore this issue, we introduce an extended concept of memorization, distributional memorization, which measures the correlation between the LLM output probabilities and the pretraining data frequency. To effectively capture task-specific pretraining data frequency, we propose a novel task-gram language model, which is built by counting the co-occurrence of semantically related $n$-gram pairs from task inputs and outputs in the pretraining corpus. Using the Pythia models trained on the Pile dataset, we evaluate four distinct tasks: machine translation, factual question answering, world knowledge understanding, and math reasoning. Our findings reveal varying levels of memorization, with the strongest effect observed in factual question answering. Furthermore, while model performance improves across all tasks as LLM size increases, only factual question answering shows an increase in memorization, whereas machine translation and reasoning tasks exhibit greater generalization, producing more novel outputs. This study demonstrates that memorization plays a larger role in simpler, knowledge-intensive tasks, while generalization is the key for harder, reasoning-based tasks, providing a scalable method for analyzing large pretraining corpora in greater depth.

Published as a conference paper at ICLR 2025
GENERALIZATION V.S. MEMORIZATION: TRACING LANGUAGE MODELS' CAPABILITIES BACK TO PRETRAINING DATA
Xinyi Wang"", Antonis Antoniades'; Yanai Elazar Alfonso Amayuclas', Alon Albalak Kexun Zhang", William Yang Wang
University of California, Santa Barbara, 2 AAlen Institute for AI, Universitt of Washington, SynthLabs, *Carnegic Mellon University (xi fxinyi.wang, antonis (econnssssucee.....ooo @ucsb. edu, williamecs ucsb.edu
ABSTRACT
The impressive capabilities of large language models (LLMs) have sparked debate over whether these models genuinely generalize to unseen tasks or predominantly rely on memorizing vast amounts of pretraining data. To explore this issue, we introduce an extended concept of memorization, distributional memorization, which measures the correlation between the LLM output probabilities and the pretraining data frequency. To effectively capture task task-specific pretraining data fre- quency, propose novel task-gram language model, which is built by counting the co-occurrence semantically related n-gram pairs from task inputs and outputs in the pretraining corpus. Using the Pythia models trained the Pile dataset, we evaluate four distinet tasks: machine translation, factual question answering, world knowledge understanding, and math reasoning. Our findings reveal varying levels of memorization, with the strongest effect observed in factual question answering. Furthermore, while model performance improves across all tasks as LLM size increases, only factual question answering shows an increase in memorization, whereas machine translation and reasoning tasks exhibit greater generalization, producing more novel outputs. This study demonstrates that memorization plays a larger role simpler, knowledge-intensive tasks, while generalization the key for harder, reasoning-basee tasks, providing scalable method for analyzing large pretraining corpora in greater depth
INTRODUCTION
Large language models (LLMs), such GPT-4, have achieved remarkable performance across wide range of tasks, yet the debate persists regarding whether these models are truly generalizing to unseen test cases or merely memorizing their extensive training data (Magar and Schwartz) 2022 Srivastava et al., 2024 [ender et al.) [022 Merrill et al. 2024 [Shiib et al.] 2024). Previous research has primarily investigated memorization in LLMs through verbatim recall of long segments from the training corpus (Zhang et al., 2023 Jiang et al., 2024 Carlini et al.) (022). However, the exact reproduction of long text is relatively uncommon when examining high-level capabilities such as translation and reasoning, particularly when the task output is short, as with world knowledge questions. To advance this line of inquiry. a more flexible definition of memorization necessary.
Several works have explored the relationship between memorization and generalization (Feldman 2020 Feldman and Zhang 2020 Zhang et al. 2023), often employing counterfactual memorization. which measures the difference in model performance when specific training example is excluded. However, these studies typically use small models and datasets, or subsets of larger datasets, and focus primarily on quantifying how much model behavior depends on memorization. They do not fully examine how different model capabilities emerge from the interplay between memorization and
denotes equal contribution.
Our code is available at: http:sss/gitttbb..mmmaaaaatttttt htw.://msoooooooeeeeeeeeeeooo search

---

Published as a conference paper at ICLR 2025
Task-gram table
Task data
Input n-gram
Output n-gram
Alignment
Question (X]
Large Magellanio
A dwsirf gelsdy
The Large Magelianic Cloud
Filter
Cloud
Answer [y]
reaction of the
Congrees de los Estados Uniciss
x
markat
dwarr galaxy otbiting Milky Way
English (X)
WIMBO
Search
ni grem counts:
counts
gram
a[gag Magallanic
reaction market
Cloud, A dwert galaxo)
Spearna
results of the voto the American
cortelxtion
WMT
House of Representatiivs
Analysis
Spanich (X)
La reacción del mercado ante los resuttados votación
Source documents:
Aniv. Aniiv_400,jje 400,jsun
Congreso de los Estados Unidos,
Booksa 240.jaon
The Pile
n-gram probability
Figure I: Overview of our proposed analysis pipeline. For the selected evaluation tasks, we first construct a task-gram table by matching semantically similar n-grams from task inputs (æ) and targets (ÿ). These n-grams are then searched within the pretraining corpus, yielding their counts and source documents. We then build task-gram language model from the ohtained counts and then analyze their relationship with L.LM predictions.
generalization. Moreover, the limited scale of these analyses is partly constrained by the definition of counterfactual memorization, which requires expensive model retraining.
In this paper, we introduce new framework for understanding memorization at scale, enabling us to analyze various LLM capabilities. We define distributional memorization as the correlation between the distribution of LLM outputs and the distribution of pretraining data. Similarly, define distributional generalization as the divergence between the LLM's output distribution and the pretraining data distribution. For simplicity, we will refer to these concepts memorization and generalization throughout the paper.
Our approach to defining memorization and generalization requires estimating the distribution of LLMs' pretraining corpora, which is a challenging task given the sheer size of these datasets, often containing trillions of tokens. To address this, we propose novel method for modeling language distributions by counting semantically related r--ramm pairs extracted from a task's 's input-output pairs. For example, in machine translation, these n- o-grams would correspond phrase pairs from the source and target languages, as shown in Figure [ This allows us to construct set of n-gram pairs that characterize the task. Drawing inspiration from the phrase table used in machine translation (Passban et al., 2016), which consists of translation pairs, we refer to this set as the task's 's task-gram table. When n n-grams from the input and output co-occur within a document, they are often separated by significant distances. By counting these co-occurrences, we are able to model long-range, task- relevant dependencies Elazar et al. (2019) 2022). The resulting n-gram language model, built from the task-gu -ramm table, is referred to task-gram language model. In contrast, classical n- gram LMs capture only local lexical dependencies within a single n-gram. Although Liu et al. (2024) introduced co-gram LM, which extends n to infinity through backoff from infinitely long 7--grmm, the effective length after backoff remains limited and cannot capture the long-range dependencies modeled by our task-gram LM. These n-gram LMs can be viewed as empirical approximations of the pretraining data distribution since they follow the observed frequency of n-grams (or pairs) in the data. Using these n--ramm LMs, we measure the degree of memorization by correlating LLM-predicted probabilities with the probabilities generated by the n-gram LMs.
Our experiments focus on the Pythia model family (Biderman et 2023 pretrained on the Pile dataset (Gao et al.) 2022)) and evaluate performance across three tasks: translation (WMT (Callison | (2009)) factual question answering (TriviaQA (Joshi et al. 2017h) world knowledge questions (MMLU (Hendrycks et al., 2020)), and math reasoning (GSM&K (Cobbe et al.] 2021 The high-level overview of our analysis pipeline is depicted in Figure D We first construct a task- gram table using supervised task data, then search for the co-occurrence of n-gram pairs in the pretraining corpus using WIMBD (Elazar et al. 2024). Finally, we construct a task-gram LM from the co-occurrence counts and compare it with LLM predictions. Our results demonstrate that the task-gram LM effectively captures task-relevant data distributions, providing better explanations for L.MM behaviors compared to the oo-gram LM (Liu et al. 2024).
Burch al.
2

---

Published as a conference paper at ICLR 2025
Among the four tasks, our analysis reveals that TriviaQA exhibits the strongest memorization effect, with task performance highly correlated to the n-gram distributions in the pretraining data. In contrast, MMLU shows weaker memorization, and WMT exhibits only an insignificant memorization effect. Additionally, our results show that as the model size increases, the source of performance gains varies across tasks: for TriviaQA, improved memorization plays key role, while for MMLJ, WMT, and GSM8K, increased generalization is more crucial. These findings align with the nature of the tasks: TriviaQA relies on factual recall, MMLU and GSM8K require more complex reasoning, and WMT reflects transferable translation skills.
To complement our distributional memorization analysis, we also conduct gradient- based estimation of the training influence of pretraining documents on test examples. We find that, consistent with our distributional memorization results, pretraining documents have the largest impact on TriviaQA, followed by MMLU, and the least impact on WMT. Documents containing n-gram pairs from the task-gram table exert a greater influence than those containing only individual n-grams.
To our knowledge, this work presents one of the first comprehensive analyses of LLM capabilities by tracing their origins to pretraining corpora at scale. Our task-gram language model offers a scalable, generalizable approach for understanding LLM behavior across a variety of tasks
2
METHOD
Diverse abilitics have been observed from LLMs trained on large pretraining corpora. Many of them are distinct in nature, like knowledge retrieval and math reasoning. It is reasonable to hypothesize that these capabilities come from different subsets of texts from pretraining. To better identify these task-relevant texts, we propose to construct a task-gram table from a set of supervised task data Dr (z }; corresponding to task T,, characterize the task-relevant documents. More specifically, we mine semantically similar n- n-gram pairs (s" from corresponding task input x and output y respectively. i.e., s" S x., and s S y. We use the cosine similarity between the embeddings of the n-grams to measure the semantic closeness of the n-grams. The task-gram table is then constructed by all possible such n-gram pairs in Dr. The mined n-gram pairs capture task-specific supervision signals as the output n-gram can be viewed as "answering" the input n-gram. We formally define the mask-gram table as follows:
Definition 1. Denote all possible combinations of input-output n- n-gram pairs from task data Dr = ((s,, vi) by (T) = Uj =U(G,( (x:) x G., (3:),, where G.(.) denotes the set of all possible n-grams in a piece of text. Then the task-gram table Hn (T) is defined as:
(T( {(55.
Yr, s" # (5) E AT)T)),
where "T E (0, 1) is a threshold chosen as a hyperparameter, E is a pretrained text embedding model, and :) denotes the cosine similarity between two vectors.
Based on the dusk-gram table, can then construct a task-gram language model. which can describe the distribution of the characterized task-related data in a large pretraining corpus. In the following paper, we use C' as the counting function. We define the number of co-ocurrence of n-gram pair s²) in the same document in the pretraining copus D by C((s", s,), D). We also define the number of occurrences of a n-gram s in the pretraining corpus D by C((s", D). Then task-gram language model can be defined as follows:
Definition 2. A task-gram language model is defined over its v(s", 8d) (T) and corpus of interest DD define the following probability distribution
table (T). For
Pn. C((s",")) D)), ,D).
In practice, when we observe that an n-gram s" exists in an unseen text input, the chance of the corresponding n n-gram s appearing in the output can be estimated by (s")s))
Suppose we have an LLM pretrained on the corpus D. Considering the zero-shot setting where we prompt the L.M with an instruction text u and an input text J. Suppose s C x and 8 C y. we
"Code will he made available upon acceptance of the paper. 'In practice, we minimal instruction template indicate the input and output.
3

---

Published as a conference paper at ICLR 2025
want define LLM version of the above n-gram conditional distribution using the LLM predicted probability of s in the context of the concatenated testing example u @ x + y:
P
PLL @ x x 9(1m-1)) DESY
II
(2)
Here m denotes the location index of the n-gram s found in y. and f is each token in the tokenized n-gram sy. We can then formally define the distributional memorization by Spearman correlation p between the task-gran language model probabilities and LLM predicted probabilities of testing data: Definition 3. For a testing set Dj = {(T,, y.),, we denote all n-gram pairs found it by Galy)l (T') V(r,y) D/T). Then we define the extent of an LLM distributional memorize the pretraining corpus when performing task T as follows:
{(s", (V)("s (*) E (G)(
MLL,(LLM, D(T) - pllog P.,D (Y|x), log Pr)M(Y
(3)
where p denotes Spearman correlation, log Pa, (Y = (log (n. [ 8S) E p). and log PIIM(Y (log P [Vs"", $).
Similar to normal Spearman correlations, the significance of distributional generalization can be measured by p-value. The distributional generalization then defined as the opposite of distributional memorization: increased memorization implies decreased generalization, as the LLM predictions and the n-gram LM are more distributionally correlated, and vice versa.
Distributional memorization describes to what extent the LLM-predicted probability of the ground truth testing data be viewed as
knowledig intonsive tosis
Heasoning intensive tasks
task-grem
Dimticttioal
Comtunnto Ponenoee outioon
intin-gram
monotonic function of task-gram LM proba-
bility. This can be viewed a measure of the predictability of task-gram LM to the LLM probability. In the following sections, we show comprehensive empirical evidence of LLMs per- forming knowledge-intensive tasks depending on distributional memorization while perform- ing reasoning- intensive tasks depending on dis- tributional generalization, as shown in Figure
LLM
ULM sace
Diserbuuoraa
LLM size
LLM
3 EXPERIMENTAL, SETUP
Figure 2: Expected distributional memorization and gen- cralization trend for different types of tasks.
In this section, we introduce the datasets, mod- els, and tools we use for analyzing the memorization and generalization behaviors of LLMs.
Models and Pretraining Corpus We use a family of fully open-sourced, Transformerddecoder based LMs: Pythia (Biderman al. 2023), with a wide range of model sizes ranging from 13M to 12B parameters. All Pythia models are trained on Pile (Gao et al. 2020), a diverse pretraining corpus consisting of approximately 207B tokens. We also include some results with IB and 7B OLMo models (Groeneveld et al.) 2024), pretrained on Dolma (Soldaini al. 2024b) with 3T tokens.
We use four types of tasks: machine translation, factual question answering, world knowledge questions, and reasoning.
Downstream Tasks
For translation, we use the WMT-09 dataset (Callison-Buut Burrhh et al. 2009) with a 2.5K testing set. WMT is classic annual machine translation shared task with different languages. We chose WMT09 as our testing data instead of newer versions of WMT because WMT09 contains more European languages, which is more prominent in the Pile.
For factual question answering, we use the TriviaQA dataset (Joshi et al. 2017) with a 10K testing set, which is a knowledge-intenste question-ansee answering dataset with the questions originating from trivia enthusiasts. Since the answers are usually single words or short phrases, we regard the whole answer text the output n-gram s no matter the value of n.
For world knowledge questions, we use the MMLU benchmark (Hendrycks et al. 2020 covering 57 tasks including elementary mathematies, US history, computer science, law, and more. The aim of
4

---

Published as a conference paper at ICLR 2025
XMT
THAROA
KNLU
055
001437
427
580
441
10000
200.0
30000
+000
3303
2003
4000 2000 $300
20
2.0
30
Figure 3: Task performance v.s. n-gram pair count in the Pile with different Pythia model sizes, for four different tasks, from left to right: WMT, TriviaQA, MMLU, and GSM8K. For WMT, the z-axis shows the counts of n-gram pairs found in the testing of six different languages:
Hungarian, Czech, German, Italian, Spanish, and French, from left to right.
MMLU is to test models' world knowledge and problem-solving ability, so different from TriviaQA, there is large portion of questions that require logical/math reasoning skills.
For complex reasoning, we use the GSM8K dataset (Cobbe et al. (222], which contains linguistically diverse grade school math world problems with step-by-step solutions. To solve GSM8K, the LLM needs to first generate the chain-of- of-touhh-t (CoT) reasoning step, and then draw the final conclusion, which requires advance math and logical reasoning capabilities.
Searching over Pretraining Data Scale Given the scale of the LLM pretraining corpus, search- ing n-grams over the whole corpus 2 is non-trivial. We use What's 's In My Big Data? (WIMBD) (Elazar al., 2024) and the co-gram (Liu al. 2024) platform, which are designed search and retrivve documents in huge corpora. Both of them have indexed the Pile, allowing to search it using API calls. We use WIMBD for accurately counting the co-occurrence of n-gram pairs as co-occurrence frequency is usually low, and the approximate counting function in oo-gram sometimes fails to capture these www-rreyy n-gram pairs. We use o0-gram for counting the occurrence of single n-grams they appear more frequently in the corpus and the counting approximation has relatively small effect on the search results. We also use the oo-gram API to produce the 30-gram probability of single n-grams, detailed in Section 5
n-GRAM DATA FREQUENCY V.S. TASK PERFORMANCE
In this section, we investigate the overall relation between n-gram pair counts and the LLM task performance, which can viewed as a rough estimation of the importance of the task-related data. We confirm that the Pile is not contaminated by any of the datasets we used, by ensuring there are no large n-grams (n and 14) overlaps between the Pile and the testing data. This decontamination method is adopted from the GPT3 technical report (Brown 2020).
We estimate the probability of test example (I, v) appearing in the pretraining corpus the probability of any of the n-gram pairs from the task -ramm table found in (z, y) appearing:
PD,n x Σ (E(xE)))
C((s", D) (s
(4)
In Figure we plot the task performances v.s. the count of n- n-gram pairs per example as defined in Equation (4), for WMT (n = 2), TriviaQA (n 5). and MMLU (n = 3). For WMT, since the test set size for each language the same, thus we show the sum of count per language here. For TriviaQA, MMLU, and GSM8K, the z-axis represents bins that group test examples based on the number of n-gram pairs. For instance, the case of TriviaQA, data points at æ 0 correspond to test examples where the number of n-gram pairs falls between and 2000. In general, TriviaQA has more n-gram pair counts than WMT, MMLU, and GSM8K.
The WMT performance is evaluated by the BLEU score between greedily generated translation and the reference translations. The TriviaQA performance is evaluated by the accuracy of the exact match of the generated answer. The MMLU performance is evaluated by the accuracy of the option
5

---

Published as a conference paper at ICLR 2025
with the highest LM predicted probability. Since there four choices for each question, the random performance of MMLU is 25%. For GSM8K, since the performance of Pythia models is low (<' 5% accuracy), the accuracy plot is too sparse to show any trend. We compute the BERTScore (Zhagg ett al. 2020) (precision) between the model-generated chain-of-thoughts (CoT) and the ground truth CoT instead.
When the model size small (< 410m), WMT and TriviaQA have near-zero performance regardless of the n-gram pair count, while interestingly, MMLU reaches the lowest performance (<10%) significantly lower than random guessing when the 7-gram pair count is around 150. A closer inspection of the test examples in this interval reveals that they contain more reasoning or math problems, which appear to be harder for Pythia models. The noisier performance curve of MMLU and GSM8K is likely due to the weaker capabilities of Pythia models on this benchmark.
In general, all task performance increases when the number of resk-related n-gram pairs increases when the model size is large enough (> 410m). And the trend of performance improvement is more significant when the model size is larger. For GSM8K, the Pythia 2.8B model shows the most significant increasing trend, while larger models show a less significant trend. This seems to indicate that memorization plays essential role in LM's capabilities for all four tasks, and larger models memorize more. However, these performance curves can also be explained by the improved generalization ability of LMs when there is more relevant pretraining data. In the next section, use pre-defined distributional memorization and generalization to investigate the possible causes behind these performance trends.
n-GRAM DISTRIBUTION V.S. LLM DISTRIBUTION
In this section, we compute the distributional memorization Mem, D(T) as defined in Definition 3 for T WMT, TriviaQA, and MMLU respectively. In addition to computing the distributional memorization with the task-gram language model as defined in Definition 2 we consider computing another version of distributional memorization with an n-gram language model defined by single n- n-grams. Specifically, we consider the oo-gram language model (Liu et al. 2024) that uses better aligned with human written text compared with classical n--rrm LMs.
large as possible for predicting the probability of each token, which is shown to be
An oo- 3o-gram LM can be viewed n- a-gram count equals zero. This way, the probability of each token is dependent on its longest prefix that exists in the pretraining corpus. Considering concatenating the input and output text u @ x y as the context, such distribution can be written as
n-gram LM initialized with co, and then backoff when the
Pe.
II Poe, Pst.ptt/(tis-)) II &
(m: 11=-))=-))..
Here i is the location index of the token t; in the concatenated text u + x + and n, is the size of the longest prefix of 4 that can be found in the pretraining corpus, i.e. n; max {n' E [1:]]CC((i-----)-)--) > 0}. In practice we ignore the tokens with zero probability and set the oo-gram probability of this token to one, i.c., (s") = only when all its tokens have zero probability. Then the alternative version of distributional memorization using oo-gram LM is defined as:
Mem (LLM, DI7) = p(log P (Y),)( log PIAM(Y
(5)
with all the notations similarly defined as in Definition 3 We then compute the two versions of distributional memorization for the different datasets, WMT, TriviaQA, and MMLU, and show the results in Figure 4
Translation ability does not come from memorization. In Figure we do not show any distribu- tional memorization values of WMT because none of them are statistically significant. This indicates that while the translation performance is strongly positively correlated to the n-gram counts as shown in Figure 3 the performance gain does not come from initiating the pretraining data distribution.
To further investigate how different the LLM-generate translation text is from the pretraining data, show the number of novel n-gram pairs that L.LMs generated on WMT that have never been

---

Published as a conference paper at ICLR 2025
ThveggA
202.0
120RT our
19000
MHLU (knowidgg))
par (1w)) Wiireggnn (e
1.300
um pair (nnd) engiamm bwebs
80000
0.20
3280
0.20
2300
1.10
0:35
18m zin
Nin
1909
115m
1.8 7:hn
0.08
OSMEK
GGHBK
0300
-1.6
032
0522
-110
(--3)
0113
intiiggan 1--31
5ytha rreersiie sceld
Egram (ewd) pinepramm (om.)
0.50
0.4
MMLU (reesoring
innnigram (n-5)
Pystis
Figure 4: Visualization of distributional memorization with different-sized Pythia models on four tasks: WMT. TriviaQA, MMLU, and GSMBK. We also show results with OLMo models on GSM8K. For WMT, we show the number of new n-gram pairs generated by LLMs as the distributional memorization is not significant. For MMLU, we divide the tasks into two categories: snowledge-intensing and ceasoning-intensive For GSM8K, we show the Kendall tau ranking distance instead of Spearman correlation quantify the distributional generalization effect as the distributional memorization not significant. Solid lines show distributional memorization computed with our task-gram LM and dashed lines are computed with the oc-gram L.M. Statistical significant (p 0.05) values are marked with solid round markers while statistically insignificant values (p > 0.05) are marked with gray star markers.
in any pretraining documents in the top left panel of Figure 4 It shows that larger LLMs generate more novel n--ramms which indicates a larger discrepancy in text distribution from the pretraining data and better distributional generalization. This implies the performance increase in Figure 3 comes from a better generalization ability learned from more relevant data instead of memorization. Our hypothesis is that this generalization is the transfer of translation skills between different languages.
Such a result contradicts the observations in Merrill et al. (2024), which show that larger LLMs are less novel in n-gram generation. This is likely because they evaluate LLM generation using the prompts from an in-distribution dataset (validation set of the Pile) to the pretraining corpus, which might encourage LLMs to memorize. On the other hand, the translation dataset we use is which encourages LLMs to use its generalization capabilities.
Knowledge-intensive question-answering ability relies more on memorization. the top middle panel of Figure 4 we show that TriviaQA has significant distributional memorization effect, in terms of both task-gram LM and oo-gram LM. Mem,, (LLM, D(T) (> 0.35) is significantly more profound than Mem (LLM, DIT) (< 0.25) and Mem (LLM, DIT) (< 0.25). This indicates that LLMs memorize small, long-range parallel data pieces more than large, local data pieces.
Also, when the LM size increases, our Mem (LLM, D) (> 0.35) also increases. Since the larger model also has better performance as shown in Figure 3 this memorization behavior is highly correlated with LMs' factual QA ability. This might be because factual QA requires retrieving knowledge from training data, thus memorization plays a critical role in this task.
For MMLU shown in the two rightmost panels of Figure 4 divide the 57 MMLU tasks into two groups: wwwlledde-intensive and reasoning-intenn intensive. We consider knowledge-intone intensive tasks as tasks that can be answered by retrieving static knowledge, while reasoning-intents that need computation or logical reasoning over the knowledge [ In general, knowledge-intensive tasks show a more significant memorization effect than reasoning-intensive tasks. This discrepancy is the most profound when n = 3, while the memorization level of knowledge-intensive tasks and reasoning-intenss tasks similar when 5. This echoes our previous hypothesis that LLMs demonstrate a stronger memorization behavior when performing knowledge-intent intensive tasks.
tasks ones
*The detailed classification shown in the appendix.
7

---

Published as a conference paper at ICLR 2025
Recalling rare knowledge requires generalization. Similar to TriviaQA, the top right panel of Figure shows shows that our =em (LLM, D(T) (> 0.25) remains most pronounced for the knowledge- intensive MMLU tasks. The primary distinction between the MMLU tasks and TriviaQA is that Mem,, (LL.M, D))" decreases as the LLM size increases. This may be attributed to the fact that MMLU involves more specialized and less common knowledge compared to TriviaQA, making its occurrence in the pretraining corpus relatively infrequent. Consequently, for larger models to perform better on MMLU tasks, they may need adjust the probability of recalling this knowledge, resulting in decrease in distributional memorization.
Reasoning-intensive abilities rely more on generalization. The reasoning-intensive MMLU tasks in the bottom right panel of Figure 4 show a very different picture compared to the knowledge- intensive MMLU tasks and TAi In this case, our Mem (LLM, D)T) is the most significant, which indicates that the memorization of large text segments is more significant. This might be because some concepts can be meaningfully expressed in large text segments while the small text segments are meaningless in reasoning-intensie context. The decreasing trend of memorization when the model size increases also indicates that memorization is not the driving force performance improvement.
For GSM8K shown in the two bottom panels of Figure we did not observe a significant memoriza- tion effect with either Pythia models OLMo models. To quantify the distributional generalization, we substitute the Spearman correlation with the normalized Kendall tau ranking distance, which represents the fraction of data pairs that disagree on their rankings. For both Pythia models and OLMo models, the distributional generalization increases when the model size increases, while the LLMs' probabilities agree more with task-gram probabilities than the inf-gram probabilities. The GSM8K results confirm that generalization is the driving force of performance improvement.
Task-gram LM can better explain LLM predicted probabilities than co-gram LM. With both TriviaQA and MMLU, we observe that Mem (LLM, D(T) is always less or equal to our Mem, (LLM, D(T'). And larger LLMs always show less oo-gram memorization effect. This shows that our task-gram LM is better way to model the data distribution so that it is more correlated to what memorized by LLMs.
In general, the LLM shows decreased memorization and increased generalization when the model size increases. This implies LLMs leverage better generalization to solve hard tasks, whether they are hard in terms of the rarity of the knowledge or in terms of the requirement of reasoning. Our task-gram LM also better models the memorized data distribution than the oo-gram LM.
INFLUENCE OF n-GRAM DATA THROUGH PRETRAINING
Since our results so far only show correlations between pretraining data frequency and the LLM predictions, we wish to explore the causal relationship of training data on LLM predictions. this section, we estimate the influence of pretraining data related testing predictions by accumulating the gradient dot products through model checkpoints, as introduced in Pruthi et al. (2020).
More specifically, we approximate the influence of pretraining document at training time by the dot product between its pretraining loss gradient and the testing loss gradient. For an n- n-gram pair (s" (") E (T') found in some example (a. y), the test loss is defined as:
E(0, (x. y),s*) = log PLLM u, Iz 9[1mm-1)).
Here m denotes the location index of the n-gram s found in y, and 0 denotes all trainable parameters of the LLM. The training loss of a pretraining document d E D containing the n-gram pair (s" s") is defined
((0, d, s²) = log PLIM(s PLlM(s"(duAAI1)
Here h denotes the location index of the n-gram 8 found in d. Suppose we have k evenly spaced pretraining checkpoints of the LLM 01, 02...., Gs. then the influence of d on the testing example (z, v) through the pretraining is defined as:
In(d, (z, v)) (ar (7))((())))))))
Σ
Vo,((0,,d,5)) Ve, Vo, (z.
8

---

Published as a conference paper at ICLR 2025
Invi@AA
1000
5000
2000
2000
2002
509
000 121111
Figure 5: Training influence pretraining documents Pythia model size with WMT, TriviaQA, and MMLU. Green lines correspond to documents containing n-gram pairs, while blue lines correspond to documents containing only the output n-gram in n-gram pairs.
TriviaQA
GSM8K
Memorization
Generalization
Memorization
Generalization
Pythia (6.9B)
17%
9%
2.6%
2.8%
Pythia-Instruct (6.9B)
23.5%
23.2%
6.3%
7.3%
Pythia (12B)
28.7%
23.2%
2.7%
2.8%
OLMo (7B)
36.4%
29.8%
2.5%
3.1
OLMo-instruct (7B)
29%
10%
6.3%
7.9%
Table 1: Zero-shot accuracy TriviaQA and GSM8K test set with memorization encouraged task prompt (maximize counts) and generalization encouraged task prompt (minimize counts).
Here, we use (xx, y) to denote all n-gram pairs found (x, y) and exist the task lask-gram table (T)). Such influence function can be viewed the test example (z, y) that is induced by the training process whenever document is utilized the pretraining. To estimate how much influence the documents containing n-gram pairs have each testing task, we compute the average influence over the testing set D/- and randomly retrieved documents from pretraining corpus D for each testing example. The average influence obtained can then be written as:
estimation of the total reduction in loss on
11 In(D,T) D
Σ Σ (E.))ED;
In(di, (x,
Note that the computation of the full gradient is relatively expensive, so we choose relatively small R = 50. This analysis does not aim to cover the full pretraining corpus but to give complementary causal evidence the previous findings. We consider two retrieval schemes: retrieving documents where the test sample's 7--gram pair appears together. 2. retrieving documents that contain the output n-gram from the test sample's ' n-gram pair. In Figure 5] we plot the average influence of these two schemes by green and blue curves, respectively.
Across all three datasets, we observe that pretraining documents containing n n-gram pairs consistently contribute more the testing examples than documents containing only the output n-gram in r--ram pairs, over different model sizes. This suggests that our task-gram table identifies important task- relevant data better than single n-grams. The difference between n-gram pair and output n- graam is the smallest on WMT, as well as the value of the influence function, which echoes the insignificant memorization effect of LLMMs this task. In general, when LLM size increases, WMT and MMLU decrease data influence, while TriviaQA slightly increases, and has the highest influence value at the largest model size. This indicates that memorization is likely caused by more influence of the relevant data at training time.
7 PRACTICAL IMPLICATIONS: PROMPT OPTIMIZATION
An important observation of our study is that Knowledge-intensie tasks benefit from LLMs' distribu- tional memorization, while crasoning-intensii tasks benefit from LLMs' distributional generalization. Then is possible to design rewrite the prompt according to this principle improve LLM''s 's task performance, based on the hypothesis that the LLM generation distribution is strongly affected by the prompt distribution. More specifically, to encourage memorization, rewrite the task instruction

---

Published as a conference paper at ICLR 2025
to be more similar to pretraining data in terms of n-gram counts. To encourage generalization, can rewrite the prompt to be less similar to training data.
We implement simple prompt optimizer based on GPT4o and the WIMBD n-gram count feedback. More specifically, we instruct GPT4o (Achiam et al, 2023) to rewrite a given task prompt at each iteration, and give the average n-gram count in the pretraining corpus of the rewritten prompt to GPT4o in the next iteration as the reward. We instruct GPT4o to maximize this reward we want to encourage memorization, and instruct it to minimize this reward if we want to encourage generalization. Here, we show a maximization and a minimization result for TriviaQA and GSM8K respectively. We report zero-shot testing accuracy with the Pythia models and OLMo models in [ The meta prompt we used to perform such optimization and the optimized task prompts are included in Appendix F
Note that the lengths of the optimized prompts are not significantly different, while TriviaQA significantly benefits from the prompts that are more similar to the pretraining data, and GSM8K benefits from the prompts that are less similar. More sophisticated prompt optimization algorithms with more detailed distributional memorization feedback can be designed based on a similar idea. We leave the investigation of other possibilities for future work.
8 RELATED WORK
Understanding LLMs' capabilities from training data. Most work on understanding LLMs
analyzes their capabilities via synthetic experiments or small-scale studies (Arora and Goyal 2023
Prystawski et al. 2023 Wang et al.] 2024, Xie et al. 2022, Wang et al. [203]] Chan et al. 2022
Razeghi et al. 2023 Chen et al. 2024). ) despite the importance of scaling. Kirchenbauer et al.
(0244)
estimate dependencies between model capabilities and subsets of training data using kernel- -based
statistical evidence but rely on a small fraction of pretraining data due to computational constraints.
To address this, |lazar al. (2024) introduce WIMBD, system for efficient n-gram retrieval over massive datasets, while Merrill et al. (2024) develop an unbounded n--gram search method and find larger LLMs generate fewer novel n-grams. (Shaib al. (2024) demonstrate how syntactic templates in training data influence LLM outputs, and Liu et al. (2024) propose oo-gram language models to estimate text corpora distributions, focusing on local dependencies rather than complex capabilities.
In this work, we analyze the origins of LLMs' zero-shot capabilities by leveraging WIMBD and oo-gram frameworks for full-scale pretraining data exploration.
Memorization vs. generalization. LLM memorization, defined as exact recall of training data, has been extensively studied, including memorization of rare or private data (Zhang et al.) (022). test set contamination (Jiang et al.) 2024), and higher prevalence of verbatim recall in larger models (Carlini jet al. 2022). [Hartmann et al. (2023) provides a comprehensive summarization of different types of L.M memorization. The re relationship between memorization and generalization is explored in
Feldman (2020), showing that memorization can improve generalization. Extensions to this work quantify memorization via performance differences when specific training examples are included or excluded (Feldman and Zhang 2020: Zhang et al., 2023). y. However, this approach is infeasible for large-scale LLMs due to retraining requirements. We propose scalable definition of distributional memorization using n- n-gram counts to enable efficient large-scale analysis.
CONCLUSION
In this paper, we introduce the task-gram language model, a scalable approach for modeling the task-relevant distribution of pretraining data. We trace the capabilities of LLMs back to this data by defining distributional memorization, measured through the Spearman correlation between task-gram LM probabilities and LLM probabilities. Through extensive experiments using Pythia models across four distinct tasks, we find that LLMs tend to memorize more when engaged in simpler, Knowledge-intensive tasks, while they generalize more in harder, reasoning-intensive tasks. Our analysis provides a comprehensive examination of the origins of LLM capabilities and offers a scalable framework for investigating the fine-grat ine-grained tell-relevant characteristies of pretraining corpora.
'A More detailed discussion of related work can be found in Appendix
10

---

Published as a conference paper at ICLR 2025
REFERENCES
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
S. Arora and A. Goyal. A theory for emergence of complex skills in language models, 2023.
E. M. Bender, T. Gebru, A. McMillan-Mayjj Majoor, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 202/ ACM Conference on Fairness, Accountability, and Transparency, FAccT "21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery, ISBN 978 450033307 doi: 0114553442188.44459992 3445922
/tt:://dooi.oom URL https: httptttt//o.o..... otg////ooo..o..//////////// 1145/ 3tt.:///...1111/111111
S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
T. B. Brown. Language models are few-shot learners. arXiv preprint ar arXiv:2005.14/00 2020.
C. Callison- P. Koehn, C. Monz, and J. Schroeder, editors. Proceedings of the Fourth Workshop Statistical Machine Translation, Athens, Greece, Mar. 2009. Association for Computational Linguistics. URL httthht: /RLLatttps//acclanhc..ooo
org/ 909-0099-0000 -00000000
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memoriza- tion across neural language models. In The Eleventh International Conference on Learning Representations, 2022.
S. Chan, A. Santoro, A. Lampinen, J. Wang. A. Singh, P. Richemond, J. McClelland, and F. Hill. Data distributional properties drive emergent in-context learning in transformers. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 18878-18891. Curran Asso- ciates, Inc., 2022, URt.ttp:://proeecinss...uu neurip c paper files/paper) 2022 http://ieeee7cccceeeee Fttpe//e666ccca00eeeee006666666. Pawtssacce-aapeeeeeeee Conference pdf
Y. Chen, C. Zhao, Z. Yu, K. McKeown, and H. He. Parallel structures in pre-training data yield in-context learning. arXiv preprint arXiv:2402. 12530, 2024.
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers solve math word problems. arXiv preprint arXiv:21 14164. 2021.
Y. Elazar, A. Mahabal, Ramachandran, T. Bedrax-Weiss, and D. Roth. How large are lions? inducing distributions over quantitative attributes. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 9973-3983, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18633/1//119-1338
Y. Elazar, N. Kassner, S. Ravfogel, A. Feder, A. Ravichander, M. Mosbach, Y. Belinkov, Schütze, and Y. Goldberg. Measuring causal effects of data statistics on language model' sfactual predictions. arXiv preprint arXiv:2207. 14251, 2022.
Y. Elazar, A. Bhagia, I. Magnusson, A. Ravichander, D. Schwenk, A. Suhr, P. Walsh, D. Groeneveld, L. Soldaini, S. Singh, Hajishirzi, N. A. Smith, and J. Dodge. What's my big data?, 2024.
V. Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Anmual ACM SIGACT Symposium on Theory of Computing, pages 954-959,20 2020.
V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33. pages 2881 28911 Curran Asso- wwweeeeeeeeeeeeeeeeeeeeeeeeeeeee fttes/paper/
ciates, Inc., 2020. ht httpe//.....iittitt/////oo//oaoa

---

Published as a conference paper at ICLR 2025
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang. H. He, A. Thite, N. Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint ar XX11:2111.0002/ 2020.
D. Groeneveld, I. Beltagy, E. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. Jha, H. Ivison, I. Mag- nusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. Pe- ters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L.. Zettlemoyer, J. Dodge, K. Lo, L.. Soldaini, N. Smith, and H. Hajishirzi. OLMo: Accelerating the science of language models. In L..W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789-15809, Bangkok, Thailand, Aug. 2024. Association for Computational Linguisties. doi: 10. www.2222..ccllooogo.
URL
https: ac ann org/ 2024 awl.acclooooo 841
V. Hartmann, A. Suri, V. Bindschaedler, D. Evans, S. Tople, and R. West. Sok: Memorization in general-purpose large language models. arXiv preprint arXiv:2310.1836 2023.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. arXiv preprini 2XXi:2000000300 2020.
M. Jiang, K. Z. Liu, M. Zhong, R. Schaeffer, S. Ouyang, J. Han, and S. Koyejo. Investigating data contamination for pre-training language models. arXiv preprint S4Kiv:2401.00055 2024.
M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprchension. In R. Barzilay and M.-Y. Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1144 URLhttts::/ http///caaaattlcoocooeco org harg/P17-1144
J. Kirchenbauer, G. Honke, G. Somepalli, J. Geiping, D. Ippolito, K. Lee, T. Goldstein, and D. Andre. Lmd3: Language model data density dependence. arXiv preprint arXiv:2405.0633 06331, 2024.
Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Pupers, pages 79-86, Phuket, Thailand, Sept. 13-15 2005, URL https //aclantho logy. www.ooggyyooooo..oooo...ooooo. 2005 mt mtstmmmttttt pap.ttt-pppp....pepoo
J. Liu, S. Min, L. Zettlemoyer, Y. Choi, and H. Hajishirzi. Infini-gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling, 2024. URL
net / forum? Ld=u2vAyMeMMm
I. Magar and R. Schwartz. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 157-165, 2022.
W. Merrill, N. A. Smith, and Y. Elazar. Evaluating n-gram novelty of language models using rasty-dawg. arXiv preprint 24X.:22406.1366 2024.
P. Passban, Q. Liu, and A. Way. Enriching phrase tables for statistical machine translation using mixed embeddings. In Y. Matsumoto and R. Prasad, editors, Proceedings of COLING 2016. the 26th International Conference on Computational Linguistics: Technical Papers, pages 2582- 2591, Osaka, Japan, Dec. 2016. The COLING 2016 Organizing Committee. URL https Tlaclanthologe wwtp//l--22..00011
G. Pruthi, F. Liu, S. Kale, and M. Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920-19930, 2020.
B. Prystawski, M. Y. Li, and N. Goodman. Why think step by step? reasoning emerges from the locality of experience, In Seeetty-evennth Conference on Neural Information Processing Systems, 2023. URL hnttt:::://// httttp:://peezeviee.. wwwee/fooummiieeee rxx.XXXEXX.XXXXXxXXXXX
Y. Razeghi, H. Ivison, S. Singh, and Y. Elazar. Backtracking mathematical reasoning of language models to the pretraining data. In NeurIPS Workshop on Atributing Model Behavior at Scale, 2023. URL htt.ttttt///peeette
ew. net. http://fformmmm...kkkoo//re
12

---

Published as a conference paper at ICLR 2025
H. Schwenk and M. Douze. Learning joint multilingual sentence representations with neural machine translation. In P. Blunsom, A. Bordes, K. Cho, S. Cohen, C. Dyer, E. Grefen- stette, K. M. Hermann, L. Rimell, J. Weston, and S. Yih, editors, Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 157-167, Vancouver, Canada, Aug. 2017, Association for Computational Linguistics. doi: 10. 10.18663/11//17-26111 httpss
wwcclanhhoooy..rggg..o 2619
C. Shaib, Y. Elazar, J. Li, and B. C. Wallace. Detection and measurement of syntactic templates in generated text, arXiv preprint 2407.00211, 2024.
L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, V. Hofmann, A. H. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, A. Ravichander, K. Richardson, Z. Shen, E. Strubell, N. Subramani, O. Tafjord, P. Walsh, I.. Zettlemoyer, N. A. Smith, H. Hajishirzi, Beltagy, D. Groeneveld, J. Dodge, and K. Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024a.
L. Soldaini, Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint 1xXix:2402.00159 2024b.
S. Srivastava, A. PV, Menon, A. Sukumar, A. Philipose. S. Prince, S. Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. / 19450, 2024.
L. Wang, N. Yang. X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
X. Wang. W. Zhu, M. Saxon, M. Steyvers, and W. Y. Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL /tttps://openrrevieew fttt//oorumm11dd3vvv33
X. Wang, A. Amayuelas, K. Zhang, L. Pan, W. Chen, and W. Y. Wang. Understanding the reasoning ability of language models from the perspective of reasoning paths aggregation. arXiv preprint 2tl::2402.032665. 2024.
M. Xie, A. Raghunathan, Liang, and T. Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL net
https://openreviin htt:/eeeetttaiaee /ttp//ooondddddCcc
C. Zhang, D. Ippolito, K. Lee, M. Jagielski, Tramèr, and N. Carlini. Counterfactual memorization in neural language models. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
T. Zhang*, V. Kishore*, F. Wu* K. Q. Weinherger, and Y. Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https://openreve net forum? id Sww.umucesseeeeue......
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A LIMITATIONS
While this paper provides valuable insights into how large-scale pretraining corpus contributes to the emergent abilities of LLMs through n-gram search, there few limitations that we want to list out. First, the model we use, Pythia (Biderman et al.) 2023), and the pretraining corpus, Pile (Gao al. 2020 are slightly outdated and have been outperformed by many new open-source LLMs. However, most open open-source LLMs lack corresponding pretraining data and have limited model sizes and pre-training checkpoints, hindering scaling effect studies. The current WIMBD system also has limitations in searching larger corpora like Dolma (Soldaini al. (024a) (3T tokens), calling for improved searching and retrieval methods. The quality of task-relevant n-gram pairs highly sensitive to the filtering method, and while the current embedding similarity- basted----bddd-ee approach is effective, better filtering methods could significantly enhance the analysis, which is left for future research.
B ETHICS STATEMENT
The insights and methodologies developed in this paper have several significant implications for the broader field of artificial intelligence, particularly in the development and deployment of large language models (L.LMs). Understanding the balance between memorization and generalization within L.LMs crucial for both advancing the theoretical foundation of machine learning and addressing practical concerns related to their use.
Enhanced Model Interpretability: By extending the definition memorization and examining how LLMs utilize their pretraining data, our research contributes deeper understanding of the internal mechanics of these models. This improved interpretability can help researchers and practitioners diagnose and mitigate issues related to data bias, model robustness, and unexpected behaviors in Al systems.
Privacy and Security Considerations: Our findings have direct implications for privacy and security in AI. Demonstrating how LLMs memorize and potentially recall training data underscores the need for rigorous data handling and anonymization techniques. It raises awareness about the risks of inadvertent leakage of sensitive information, thereby informing policy and best practices for data usage in training large models.
Economic and Societal Impact: As LLMs become more integral to various industries, understanding their capabilities and limitations can have significant economic and societal implications. Our research can help businesses and policymakers make informed decisions about deploying these models, ensuring they are used ethically and effectively. This, in turn, can lead to more reliable and trustworthy Al systems, fostering greater public trust and acceptance.
C RELATED WORK DETAILS
Understanding LLMs' capabilities from training data Prystawski et al. (2023) and Wang et al.
(2024) discuss how the reasoning ability of language models is consequence of their pretraining data. Prystawski et al. 2023) discusses how chain-of-thought reasoning is effective in autoregressive language models because of local structure within pretraining data, and Wad[ al. (2024) derives novel conclusions from known facts by aggregating reasoning paths seen in pretraining data. On the other hand, Xie et al. (202)) and Wang et al. (2023) discuss how in-context learning is a by-product of learning the pretraining data distribution. They both suggest that language models learn to implicitly infer a latent variable from the given prompt, as the pretraining data is generated from some unknown latent variable. Additionally, Chan et al. (2022) propose that the distributional properties of training data drive emergent in-context learning behaviors in large language models, whereas Razeghi et al. (2023) shows the influence the pretraining data the mathematical abilities of LLMs. CCen et al. (2024) also highlights the significance parallel structures in pretraining data for the emergence of in-context learning.
14

---

Published as a conference paper at ICLR 2025
However, the small-scale nature of such analysis is antithetical to the commonly believed main driving factor behind the performance of LLMs: scaling. Recently, Kirchenbauer et al. (2024) proposes to provide statistical evidence of the dependence of target model capabilities subsets of its training data, by estimating the data distribution with an enbedding-induceed kernel. However, their estimation is based on very small portion of the pretraining data (around 0.3%) computing the embeddings of a huge dataset is very non-trivial. To get a better estimation of the whole distribution of the pretraining data, Elazar et al. (024) construct a retrieval system, WIMBD, that can efficiently search n-gram phrases over hundreds and thousands of GBs of pretraining data. Merrill et al. (2024) propose an efficient data structure that enables unbounded- -enghhhh-eehhhhhh n-gram searches massive pretraining datasets, and find that larger LLMs generate less novel large n-grams compared to human written texts, and (Shaib et al. 2024) shows how syntactic templates from the training data causes models to re-use such templates after training. Liu et al. (2024) proposes to build a prefix- efficient retrieval system for pretraining corpora, and then construct large-scale oc-gram language models to estimate the distribution of text corpora. However, such distribution only models local contextual dependency, which might not be useful for understanding complex LLM capabilities.
New methods and analysis to investigate these capabilities scale and to understand the role of scaling are needed to obtain useful insights into real-world LLMs. In this work, we aim to provide an in-depth analysis of the origin of the general zero-shot capabilities of LLMs, by performing full searches across the whole pretraining corpus with the WIMBD and oo-gram framework.
Memorization generalization The phenomenon of machine learning models being able to perfectly memorize the training data has been studied in many previous works. Most of them define LLM memorization as exactly recalling the training examples by designed prompting, including the memorization of rare long-tail data, like private information (Zhang et al., 2023). and the
contamination of testing sets (Jiang et al. 2024). Carlini et al. (2022) found that the exact copy and pasting behaviors are more prevalent in larger LMs. Hartmann et al. (2023 provides a comprehensive
summarization of I.I.M memorization.
Several papers have studied the interplay between memorization and generalization of training data. Feldman (2020) prove that memorizing the training data in fact required for optimal generalization on testing data. Works along this line (Feldman and Zhang. 2020; Zhang et al., (023) extend the original definition of memorization by quantifying the extent of memorizing a training example with the performance difference when including and excluding this specific example in training data. However, this definition is impractical for large-scale analysis of LLMs as it requires retraining an LLM from scratch to analyze one data point. In this paper, propose new definition of distributional memorization by using n-gram counts, which is more suitable for large-scale analysis with LLMs.
D EXPERIMENT DETAILS
Dataset choice The choice of dataset comes from the consideration of balancing different types of tasks, and we believe the combination of translation + factual QA + reasoning can well represent the spectrum of possible tasks. To enhance the selection of reasoning tasks, also add GSM8K new task as described in point I of the general response. We found both the performance v.s. n-gram count results and the distributional generalization results on GSM8K align with our previous findings.
Embedding model choice The choice of embedding model comes from accommodating the need for different tasks. We use the LASER embedding model for the Translation task because it is specialized for translation. We use E5 for other tasks because it is a embedding model that performs cosine-siminn caiine--imlllrit-bbaaee based retrieval.
sentence
Task-gram table construction To build filter all possible input-output n-gram pairs in a supervised dataset by the cosine similarity between their embeddings. Second, we search through the pretraining corpus and keep n- n-gram pairs that have nonzero counts.
table, we use two-step process. First,
For WMT, we use the Europarl (Koehn 2005) parallel corpus construct a large task- ook-gram table. For TriviaQA, we use its training set, while for MMLU, we use the testing set as there is no training set available. We employ two different models to filter the n-grams pairs. For translation tasks,
15

---

Published as a conference paper at ICLR 2025
the LASER embeddings (Schwenk and Douze 2017), which provide Linguage-agnoste sentence representations, in order to assess cross- lingull similarity. We mine n- gram pairs from the Europarl corpus, which consists of around 20 million parallel sentences extracted from the proceedings of the European Parliament (Koehn) 2005). For all other tasks, we use E5 (Wang et al. 2022), which are multilingual sentence representations trained using contrastive learning For TriviaQA, we mine the n-gram pairs from Tr'viaQA's training set. For MMLU, since the training is very small (100 500 examples for each task), we mine the n--rrmm pairs from the test sets directly. For GSM8K, due to the short time limitation during rebuttal, we also mine the n-gram pairs from the 1K test sets directly.
diverse range of tasks.
In WMT, the cosine similarity thresholds we use are: 0.85, 0.8, 0.75, and 0.7 for 2 to 5-gram pairs, respectively; For TriviaQA and MMLU, the values are 0.75 and 0.65 for 3 and 5-gram pairs, respectively. We use lower thresholds for larger n-grams because larger n-grams inherently impose stricter alignment, and are therefore less likely. We show a cosine similarity sensitivity ablation in Figure
=
1077
Ovdntuttonal memertatto
0.35
0.35
0.35
syan sal (n-1)
(0)3)
0.33
rgont par
0.25
0.5-
0.20
0.2-
0.15
0.12
16t JUm
420m 70m 100m Tythe 0og
1.4 2:55
5.50 126
16m
32m
425m T0m 260m
1.0 2:00 Pythas model sie dog scale!
6:90 12b
1dm 35m
70m Pycha 1og scald
160m
400m
3,46 2.50
5:00
Figure 6: Visualization of distributional memorization with different different-sized Pythia models TriviaQA with different cosine similarity threshold 7 E 0..7, 0.8). The trend of distributional memorization does change with different thresholds.
We perform our experiments on 8 GPU 40G A100 working stations. Below is the license information for the datasets we used:
Pile: MIT license.
master
Tulu: ODC-BY license. URL: htt.://s/uuggiiieeee.//////s////
tulu-v2-st http::/2ssffmmxxuurrmm
mixture
WMT-099 published with the WMT workshop. URL: https::/ww..tatttttttt
html
TriviaQA: Apache License 2.0.
URL: bttps://bbb.cs.wwhhngooooo
triviaqa/
MMLU: MIT license. wwl.:tttts///taabb..c/////eeeerr
Knowledge-intensive MMLU tasks:
prehistory', cecihesss_ehhisse 'philosophy',
fooral_disputes 'medical
'medical_geneticss" genetics',
human_aging', 'us_foreign_pollccy
1ogical_fallacies" intternationai_laaw", 'computer_securitty
sociooooy'', "professionaal_asychooll. 'marketing', "human_sexuality',
'anatomy', hhiighhchhhoouaaahhisttttro ottpppppiittoooooppppooo..eeaaaa 'public_relatiinns,,
hhiieh_schooLmmmcrreeroomeeee VelLnical_kkooweddgee" 'nutrition', werid_reliqions", " hhich_school_psycholeegs
securittyssuddies
hhigh_school_geograabh 'management', "global_facts""
shigh_school__wrld__hss
jurisspudencee', 'virology', 'astronomy', "miscelllnnes
Reasoning-intensive MMLU tasks:
16

---

Published as a conference paper at ICLR 2025
econometrics', 'professional Laapiooonnlllw,,,,,__ iesstraactaagebbaat 'eollege_medicine/
collleee_chemieeeeeeeeeeeeee chemistry', moral__cenarios', 'college "college_mathnatticss www.mmatiastss....aaaa
college
'hihh_ scchoooehhemiiieeeheehhcccchceea
apooeessionallaccouttttai colleee
rcollege_biolooy'' "high_schol__commpitreeeeeeeeeee hhigh_scholl_mahemtticc.
rcollegg_physsccs' "profeassional_mmdccieeee " "elemennaayymmthhmatic./
"machine_learni "elecriicaLengineeriii "high_sschoo_
concepttua__physicss "higl_school_statistiic "high_school_biologgy
OLMo-16
OLMo-76
OLMo-7B.instur
Accuracy
0.4
0.3
0.2
0.1
0.0
300
800
900
1200
grams
Figure 7: GSM8K accuracy v.s. n-gram pair count in Dolma with different OLMo model sizes.
E
PROMPT OPTIMIZATION DETAILS
Corpus
Dataset
Memorization
Generalization
TriviaQA
Deliver exact single word or concise phrase response to the factual question. (avg. 3-gram count: 557948.5)
Formulate distinctive and concise term phrase to clearly answer the factual ques- tion. (avg. 3-gram count: 1714.9)
Pile
GSM8K
Carefully analyze each math word problem presented, break it down step-by-step, and clearly state the final answer. (avg. 3-gram count: 45028.9)
Dissect each math word problem into straightforward, logical steps; solve each part systematically for precise solutions. (avg. 3-gram count: 43.1)
TriviaQA
Provide single word or concise phrase response to the following factual question. (Avg. 3-gram count: 5566475.5)
Provide clear and specific word or brief phrase in response to the factual question below. (Avg. 3-gram count: 43436.2)
Dolma
GSM8K
Solve the following math word problem by methodically breaking down into simple, clear steps to find the correct solution effi- ciently. (Avg. 3-gram count: 3736735.0)
Solve the upcoming math word problem by sequentially explaining each calculation and logical step, ensuring clarity and coher- ence in your solution. (Avg. 3-gram count: 4563.6)
Table 2: Prompts optimized for memorization and generalization for TriviaQA and GSM8K.
Text in the square brackets comments that are not involved in the actual prompt.
17

---

Published as a conference paper at ICLR 2025
Meta prompt for prompt optimization
Task Description..::
are lasked with optimizing a given prompt to guide an open-source language model (LM)
in completing specific task effectively. You will receive:
The current prompt for the task.
Its corresponding memorization score (Average trequency of task-related n-grams found the LM's pretraining corpus).
A few example input-outpa pairs illustrating the intended task. history of previous prompi optimization iterations.
OOptimization Goals
Clearly describe the intended task with a general instruction that effectively guides the LM perform the task.
[If trying encourage reasoning ...]
Minimizing the memorization score of the updated prompt. The memorization score reflects the distributional correlation between the prompt and the LM's pretraining corpus. A lower score encourages the LM to generate more novel outputs.
[If trying to encourage memorization ...]
Maximize the memorization score of the updated prompt. The memorization score reflects the distributional correlation between the prompt and the LM's pretraining corpus. higher encourages better alignment with the LM's learned knowledge.
Example onputt pairs..:
[Example onputt pairs for current task]
Output
Produce an updated prompt that balances clarity of task instruction with an lower memorization
F n-GRAM PAIR EXAMPLES
In this section, we present some representative examples collected from the analysis for the different tasks evaluating, including Translation and Question-Answering (MMLU, TriviaQA), In order to show examples from the different experiments, we show examples with different model sizes and number of n-grams.
G MODEL GENERATIONS VS. 7-GRAM FREQUENCY IN PRETRAINING
To further analyze the behavior model based on the pairs found in pretraining, we compared the aligmment of the generations of the model when that n-gram was generated, vs. n-gram frequency in the pretraining corpus. We found that as the r-gram frequency increased, the model n-gram pairs the model generated were more aligned.
18

---

Published as a conference paper at ICLR 2025
Source
(English. Spanish)
Result
Pythia 12b 4grra
The reaction of the market to the results of the vote the American House of Representatives, which refused to support the plan for the stabilization of the financial sector there, has manifested itself here as well.
(of the vote in, de la votación en)
La reacción del mercado ante los resultados de la votación en el Congreso de los Estados Unidos, que rechazó el plan para la estabilización del sector financiero allf, se ha manifestado aqui tambiēn.
in the, votación
(results of the vote, resultados de la votación)
results of, (has manifested iiself here, se ha manifestado aquí) (market the results, mercado (plan for the stabilization, plan para la establización)
resullados de)
resullados)
financial, estabilización sector financiero) (reaction of the market, La reacción del mercado)
(stabilization
Pythia 410m 4gram
reaction of the market
(of in, votación en)
resultado de votación Cámara Diputados, que aceptö la propuesta de la Comisión de Desarrollo Regional para Seguridad Social, ha manifestado aquí también."
the results the American House of Representatives, which refused to support the plan for the stabilization of the financial sector there, has manifested itself here as well.
the, la votación en el)
(results of the vate, resultados de votación) (has manifested itself here, manifestado aquí)
Figure 8: Examples of mined pairs for the translation Task (English to Spanish) using Pythia Models with 4-Gram analysis. Models evaluated include those with 12 billion and 410 million parameters.
Source
(English, Spanish)
Result
Pythia 12b 2grra
The new law is also meant to constrain the "golden parachutes" for the heads of the companies participating in the program establishes a committee which will oversee fund.
(new law, nueva ley)
La nueva ley también pretende limitar los "bonos de la parrilla" para los jefes de las empresas que participan en el programa y establece un comité que supervisará el fondo.
(companies participating, empresas participan) (program establishes, programa establece)
(committee (establishes committee, establece comité)
comilé supervisará)
(oversee fund, supervisará fondo) (heads companles, jefes empresas) (participating program, participan programa)
Pythia 410m 2gram
The new law is also meant to constrain the "golden parachutes" for the heads of the companies participating in the program and establishes a committee which will oversee fund.
(new law, nueva ley)
El programa de la nueva ley es también para limitar los "champios de la empresa" para los que participan en el programa y establece un comité que se encargará gestionar fondo.
(companies participating, empresas participan) (program establishes, programa establece) (establishes committee, establece comité)
(participating program, participan programa)
Figure 9: Examples mined pairs translation Task (English to Spanish) using Pythia Models with 2-Gram analysis. Models evaluated include those with 12 billion and 410 million parameters.
19

---

Published as a conference paper at ICLR 2025
Question
Choices
(Key, Value)
Answer
Category
Pythia 12b 3gram
These are chains of organizations
(a) Distribution
(management processes activitles, Distriuttion
(a)
marketing
that are concemed with the management of the processes and
(b) Vendor
Distribution
(c)
(organizations one,
activilies involved creating and moving products from producers and manufacturers to end-user customers. The organizations involved with any one jourey. collectively termed
Communication
Communication)
(management processes activities,
(d) Selling
Communication)
(activities involved creating, Communication)
(processes activities involved, Communication)
A 2-month-old female brought the office for her first routine health maintenance examination and for her immunization update. In order determine whether any contraindications exist for diphtheria, tetanus, pertussis (DtaP) immunization, the parents should be questioned regarding
(a) allergy to eggs
(diphtheria pertussis,
(d) previous seizures
professional medicine
previous seizures)
(b) Apgar scores
(tetanus pertussis diap, allergy eggs)
at birth
(routine health maintenance, gestational age at birth)
(c) gestational
age at birth
(pertusais dtep immunization, allergy to eggs)
(d) previous
seizures
Pythia 6.9 5gram
'In which of the following Asian countries would one find special
(a) Japan
(Tind special economic zones s82s,
(c) China
High School Geography
(b) South Korea
China)
(tollowing countries Chine)
economic zones (SEZs)?'
(c) China
(d) Vietnam
(asian countries would one find, China)
special economic,
China)
Figure 10: Examples of mined pairs for the MMLU Task using Pythia Models with bigram and 5-gram analysis. Models include 12 billion and 6.9 billion parameters.
Question
(Key, Value)
Answer
Pythia 12b 3gram
A tropical plant that grows several feet high, with large fleshy rool that contains an acrid milk juice, poisonous in its natural form, produces from the juice, extracted under pressure and purified, a staple foodstuff called what?
(fleshy root contains, tapioca) (tropical plant grows', "tapioca) (large fleshy root, 'tapioca) (grows several feel, "tapicca) (natural form produces, "tapicca)
Tapioca
Feel Like Making Love and The First Time Ever Saw Your Face were hit singles for which female artist?
(love first time, roberta flack)
Roberta Flack
(lime roberta flack)
(ever saw face, roberta flack)
(first time roberta flack) (feel like making, roberta flack)
Figure 11: Examples of mined pairs for the TriviaQA Task using Pythia Models (12b) with trigram analysis.
20

---

Published as a conference paper at ICLR 2025
ParAlgarant ScoreD SszreDattinn Diributio ngaamd
1.80
1.5
3.07
1.41
Alpamie Sooee 01-11
High mert Scare (0-1)
Figure 12: Model translation generations alignment for pair instances identifies in pretraining. As n-gram rises, the larger models are able to reproduce more aligned pairs from pretraining.
21


Reviews:
[{"review_id": "p0r2GT8fxb", "reviewer": "Reviewer_YUoe", "summary": "This paper proposes a \"task-gram language model,\" built on n-gram co-occurrences, to trace task-specific data in pretraining corpora. Through experiments on translation, question answering, and reasoning tasks using Pythia models, the paper finds that memorization is predominant in simpler, fact-based tasks like factual question answering, while generalization is more critical for complex tasks requiring reasoning.", "questions": "1. In Figure 2, how are n-gram occurrences counted when a sentence contains multiple task-related n-grams? Specifically, are overlapping n-grams counted individually or merged, and how does this approach impact the interpretation of memorization levels in longer and more complex sentences?\n2. For generative tasks with very short input descriptions, is this approach still applicable, and how does it handle sparse n-gram distributions?\n3. How does the methodology account for task-related patterns that are critical for accuracy rather than memorization, such as translation?\n4. The study shows that in knowledge-intensive MMLU tasks, memorization decreases as model size increases. Could this decrease be due to specific limitations in capturing rarer knowledge from larger models, or might it indicate a shift toward more generalized, inferential capabilities?\n5. Appendix E states that alternative methods (e.g., NER models) for identifying n-gram pairs yielded inconsistent results. Could you elaborate more about your implementation and the results?\n6. The experiment uses different cosine similarity thresholds across tasks (WMT v.s. TriviaQA and MMLU). How did you select these hyperparameters, and how sensitive are the results to these thresholds?\n7. Some tasks might share similar patterns or data structures (e.g., QA and factual retrieval), could memorization from one task inadvertently support generalization in another? Does the study address potential cross-task memorization effects, especially for overlapping or semantically similar tasks in the training data? The task-specific n-gram approach may suffice to take these cross-task effects into account. \n8. Beyond the analysis and findings, can you propose a potential method for actively adjusting the balance between memorization and generalization within models to improve performance across tasks?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed task-gram model offers an alternative approach to evaluating LLM behavior by connecting output distributions directly with pretraining data.\n- Experimental results reveal several interesting findings like how model size influences memorization versus generalization, showing that larger models tend to generalize more in complex tasks, an insight useful for understanding scaling effects.", "weaknesses": "- The task-gram model can be insufficient in capturing the complexity of today's generative tasks with short input descriptions. These tasks often demand high levels of abstraction and creativity from the LLMs to infer extensive information from minimal data, a process that cannot be well represented through n-gram frequency metrics. I recommend evaluating on more complex tasks like code generation (e.g., HumanEval) and math reasoning (e.g., GSM8K).\n- Using Spearman correlation between task-gram and LLM probabilities may conflate memorization with necessary task handling where n-gram overlap is essential. For example, high correlation in translation could reflect accurate language pairing rather than mere memorization. This might misinterpret critical task-related patterns as memorization.\n- Overall, the findings indicate that different tasks exhibit varying levels of dependence on memorization, which is not novel or surprising, as already discussed in previous works like [1].\n- Several methodological and experimental designs need clarification. See questions for details.\n\n[1] Hartmann, Valentin, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. \"Sok: Memorization in general-purpose large language models.\" arXiv preprint arXiv:2310.18362 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XqLeMNOAmo", "reviewer": "Reviewer_w9Gw", "summary": "The paper adds a thorough analysis of memorization vs generalization capability of LLMs taking into account the pretraining data. Using the Pythia model family and the Pile dataset, the paper proposes a task-based n-gram language model, which is built on downstream tasks, by extracting semantically related n-grams. This n-gram model is then used to compute correlation among seen/novel n-grams with the LLM output perplexity, to come up with a notion of distributional generalization. More correlated these metrics are (task n-gram vs LLM), the bigger the amount of memorization and vice-versa. The key contribution of the paper is this methodological approach, using which they show clear memorization and generalization aspect based on different downstream tasks.", "questions": "- In the definition of task-gram language model, what happens when the corresponding n-gram of the denominator is rare or absent? I assume you filtered them out from the computation, but wondering what is the % of n-grams you had to drop per dataset.\n- How does your setup account for the synonyms of the n-grams? This is probably going to be more of an issue for smaller datasets such as MMLU, and can likely be a confound to the results?\n- In your opinion, does the varying amount of samples in test datasets pose as a confound? Since MMLU is the smallest dataset, perhaps one experiment could be done by taking bootstraps from WMT and TriviaQA in the same sample sizes?\n- Not sure I follow the reasoning behind using *both* $\\infty$-gram and WIMBD. Wouldn't this add a bias? Curious how plots change by using only one or the other.\n- For TriviaQA results, any insight on why does $\\infty$-gram reduces with scale? \n- Here is a hypothetical scenario: suppose if you add the task data progressively into the LLM training (by finetuning), then the task-n-gram probability can be controlled from 0 to max smoothly. Since this is finetuning, the model will be more incentivized to memorize the data aggressively. What kind of distributional generalization curves you would expect to see?\n\n\n## Suggestions\n\n- From a cursory glance i missed that Figure 3 scales of MMLU knowledge and reasoning are different, which is important to highlight in my opinion. I understand making a single axes removes the ability to show the trend (stuff will be a straight line), but if there is a way to visually highlight the range of scales it would be great.\n- Section 2 can have a small plot showing the expected trend of memorization vs generalization, to ground the readers how to interpret the later plots.\n- Typo: L499 - ~pertaining~ -> pretraining", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is exceptionally well written - I found it very enjoyable to read it end to end. \n- The methodological contribution is strong and makes sense to compute correlation between task specific n-grams and LLM perplexity. This method can likely be extended to multiple different tasks easily, with the caveat of the availability of the pretraining data distribution.\n- While memorization vs generalization debate has a long list of papers in the literature, this paper makes an important contribution by computing the statistics _at scale_ by considering full pretraining data distributions.\n- The contributions builds upon relevant literature ($\\infty$ gram and WIMBD).\n- Good choice of downstream tasks tested, ranging from translation, to factual to math.", "weaknesses": "- Perhaps some more downstream datasets could have been considered apart from MMLU, as it is known the dataset contains several annotation issues [1]. \n- The argument of MMLU requires more reasoning than generalization due to math is also not completely answerable. It would be interesting to dive into that question using GSM8k, given recent results point more to memorization issues [2].\n- Not much of a weakness per se, but a limitation (which the authors duly acknowledge) that the results are conditional on the Pile dataset.\n\n\n[1] https://arxiv.org/pdf/2406.04127v2\n[2] https://arxiv.org/abs/2410.05229", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vkBCkdiLm2", "reviewer": "Reviewer_QYo1", "summary": "The authors propose a framework for measuring the correlation between the outputs of an LLM and n-grams present in its training data.\n\nTo that purpose, they first propose a task-gram language model, which approximates the distribution of task-related data in the model's training data. They then define the distributional memorization metric (and its complementary metric, the distributional generalization) as the correlation between the probabilities of the task-gram language model and the probabilities predicted by the LLM on test data.\n\nIn the second part, the authors measure the distributional memorization/generalization of a series of LLMs of increasing size on three different tasks requiring increasing generalization capabilities. They show that the model's performance increases with the number of task-related n-grams found in the pretraining data, and that knowledge-intensive tasks rely on memorization, while reasoning-intensive tasks rely on generalization. The authors cross-check their findings by using a gradient-based method to measure the influence of training n-grams in the model outputs, which yields results that are consistent with the previous findings.", "questions": "- Appendix: The example n-grams in Figure 5 appear to contain stop words, while those in Figures 6-8 do not. Could the authors clarify whether stop word removal was performed when mining n-gram pairs? Since this seems to be the most computationally expensive part of this work, limiting the number of obtained pairs may help with scaling. More generally, my intuition is that systematic enumeration of all n-gram pairs is probably superfluous. Have the authors considered performing NER-based filtering post-mining?\n\n- In Figure 2, the smaller models are depicted to obtain a ~10% accuracy on MMLU, which is below random chance, when the number of N-grams in the training set is 150. Do the authors have any qualitative insight on why this may be happening?\n\n- The Pile was checked for potential overlaps with the training datasets using exact n-gram matches with n > 14. As this value is quite large, especially if stopwords were removed, could the authors explain why this particular value was chosen?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper focuses on the important question of the generalization-memorization tradeoff and proposes a concrete and clearly defined metric to measure it on a given LLM, which is valuable for future research.\n\nThe author's work is well-written and structured, clear, and easy to read. The experimental methodology is sound, with some possible flaws and limitations described in the appendix.\n\nI appreciate the fact that the authors confirmed their findings using the gradient-based method and that they checked for possible cross-contamination of the training dataset.", "weaknesses": "My main concern regarding this work is that the distributional memorization/generalization metrics depend on a large number of hyperparameters (choice of dataset and embedding model, cosine similarity thresholds), but although the authors justify some of the choices they made regarding those, they do not measure how sensitive the metrics are to the value of those parameters. While I understand that the proposed method is computationally expensive, I believe that this needs to be addressed to assess the relevancy and usefulness of the proposed metric and strengthen the author's claims. In addition, some choices are unclear. For example, different values are picked for the 5-gram threshold choices for WMT compared to the other datasets, but this is not justified.\n\nA smaller point of criticism is that the authors define distributional generalization as the divergence between the distribution of the LLM’s output and that of the pretraining data. While necessary, this criterion is insufficient: for example, smaller LLMs tend to fall into degenerate/repetitive distributions which do not conform to natural language syntax, and would therefore obtain a high generalization score. Performing fluency measurements or qualitative discussion of the results would help with this aspect.\n\nAn editing mistake seems to appear in the paper in Def. 3 : \"we define the extent of an LLM distributional memorize the pretraining corpus\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A7LBebjT3D", "reviewer": "Reviewer_EBoK", "summary": "The paper explores the debate between generalization and memorization in LLMs. It assesses how model outputs correlate with pretraining data frequencies, and proposes a task-gram language model to capture task-specific data distributions.", "questions": "see weakness.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper provides a thorough investigation of the interplay between memorization and generalization in LLMs. Evaluates models on a range of tasks (translation, factual QA, reasoning), providing insights into how different tasks influence memorization and generalization.", "weaknesses": "1. This paper, published at the ICML 2024 Workshop on Foundation Models in the Wild, does not present significant new insights or findings. The contributions appear incremental and may not substantially advance the current understanding in the field.\n\n2.  The conclusion that \"factual question answering shows an increase in memorization with increasing LLM size\" is not novel. Prior research has extensively demonstrated factual knowledge are stored in the neurons of FFNs. The increasement of factual answering task is due to having more neurons in the lareger models.\n\n[1] Knowledge Neurons in Pretrained Transformers\n\n[2] Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space\n\n[3] Do Localization Methods Actually Localize Memorized Data in LLMs?\n\n[4] Locating and editing factual associations in GPT\n\n[5] Calibrating factual knowledge in pretrained language models\n\n3. Lack of practical implications. It is not clear about the practical implications of its findings for improving LLM training and deployment strategies.\n\n4. Despite covering diverse tasks, the study heavily relies on specific datasets like The Pile and utilizes Pythia models. This reliance may limit the generalizability of the findings to other datasets and model architectures.\n\n5. The paper uses the MMLU benchmark to assess reasoning capabilities. However, MMLU is designed to evaluate a model's understanding ability across 57 subjects, including elementary mathematics, history, computer science, and law, and can not effectively reflect reasoning ability. For demonstrating reasoning skills, datasets like MGSM would be more appropriate.\n\n6. The use of WMT-09 raises concerns due to its age and the potential for data contamination, where overlap between training and test data might exist. It is advisable to use more recent datasets, such as WMT23 or WMT24, to avoid these issues and ensure that the evaluation reflects the model's true capabilities.\n\n7. Relying solely on Spearman correlation as a primary metric may not fully capture the nuances of memorization and generalization in language models. The paper should include additional experiments or provide mathematical justification to demonstrate why Spearman correlation is an appropriate and sufficient metric for measuring these aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)