How Much Can We Forget about Data Contamination?
Authors: Sebastian Bordt
Suraj Srinivas
Valentyn Boreiko
Ulrike von Luxburg
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Contamination
Forgetting
Scaling
Optimization
Primary area: deep_learning->large_language_models
TL;DR: We show that moderate amounts of data contamination are forgotten by the end of LLM training runs.

Abstract: The leakage of benchmark data into the training data has emerged as a significant challenge for evaluating the capabilities of large language models (LLMs). In this work, we challenge the common assumption that small-scale contamination renders benchmark evaluations invalid. First, we experimentally quantify the magnitude of benchmark overfitting based on scaling along three dimensions: The number of model parameters (up to 1.6B), the number of times an example is seen (up to 144), and the number of training tokens (up to 40B). If model and data follow the Chinchilla scaling laws, minor contamination indeed leads to overfitting. At the same time, even 144 times of contamination can be forgotten if the training data is scaled beyond five times Chinchilla, a regime characteristic of many modern LLMs. Continual pre-training of OLMo-7B corroborates these results. Next, we study the impact of the weight decay parameter on example forgetting, showing that empirical forgetting occurs faster than the cumulative weight decay. This allows us to gauge the degree of example forgetting in large-scale training runs, indicating that many LLMs, including Llama 3 405B,  have forgotten the data seen at the beginning of training.

How Much Can We Forget about Data Contamination?
Sebastian Bordt
Suraj Srinivas 2
Valentyn Boreiko'
Urrike von Luxburg'
Abstract
their training data and the benchmark questions used for evaluation (Brown et al., 2020; Dubey et al., 2024). What is more, research on memorization (Carlini et al., 2019; 2021) shows that text sequences from the training data are sometimes encoded within the model, including machine learning datasets (Grynbaum & Mac, 2023; Liang et al., 2023; Nasr et al., 2023; Bordt et al., 2024).
The leakage of benchmark data into the training data has emerged significant challenge for evaluating the capabilities of large language mod- els (LLMs). In this work, we challenge the com- mon assumption that small-scale contamination renders benchmark evaluations invalid. First, we experimentally quantify the magnitude of bench- mark overfitting based on scaling along three di- mensions: The number of model parameters (up to 1.6B), the number of times an example is seen (up to 144), and the number of training tokens (up to 40B). If model and data follow the Chinchilla scaling laws, minor contamination indeed leads to overfitting, At the same time, even 144 times of contamination can be forgotten if the training data is scaled beyond five times Chinchilla, a regime characteristic of many modern LLMs. Continual pre-training of OLMo-7B corroborates these re- sults. Next, we study the impact of the weight decay parameter on example forgetting, showing that empirical forgetting occurs faster than the cu- mulative weight decay. This allows us to gauge the degree of example forgetting in large-scale training runs, indicating that many L.Mss, includ- ing Llama 3 405B, have forgotten the data seen the beginning of training.
While the fact that data contamination can lead invalid performance evaluations is now well-established (Magar & Schwartz, 2022; Li & Flanigan, 2024; Yang et al., 2023; Jiang et al., 2024), little is known about the precise con- ditions under which this is the case. The main reason for this is that the training data is usually unknown, and con- tamination identified via clever research designs that work around this restriction (Golchin & Surdeanu, 2023; Oren et al., 2024; Deng al., 2024). Because modern foundation models are sometimes trained for over million gradient steps (Dubey et al., 2024), it is unclear whether a single update on contaminated data at some point during training necessarily impacts downstream evaluations. And indeed, there is quite some evidence that language models need to see samples repeatedly to have any impact on the final model. For example, many papers on memorization have found that it occurs only when sample is frequently re- peated in the training data (Carlini et al., 2022; Biderman et al., 2023; Huang et al., 2024b). The same is true for research on knowledge acquisition, where a fact needs to be paraphrased many times before it finally remembered by the model (Allen-Zhu & Li, 2023; Cao et al., 2024; Chang et al., 2024).
1. Introduction
A core principle of machine learning is that a model should not be trained on the test set used for evaluation (Donoho, 2017). For foundation models trained on Internet-scale data, there are increasing concerns that this principle is violated due to the leakage of benchmark evaluation data into the training data (Xu et al., 2024; Oren et al., 2024). Indeed, many L.M developers have found overlap between
In this work, we study the impact of data contamination in controlled setting.' This means we train language models from scratch on datasets where we explicitly insert contami- nated examples (Jiang et al., 2024). We begin by quantifying how the overall magnitude of benchmark overfitting (or the cross-entropy loss of an observed sample) changes as we scale along three critical dimensions: model parameters, (2) the number of training tokens, and (3) the number of repetitions of an example in the training data (Section 4. 4). Holding the other two dimensions fixed, we find that the effect of scaling is monotone in each dimension.
(1) the number of
UUniversity of Tübingen. Tübingen AI Center, Germany Bosch Research North America & Bosch Center for Artificial Intelligence (BCAD), Sunnyvale, USA. Correspondence to: Sebastian Bordt <sebastian. uni uni-tucbingee tucbingen.de>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Code is available at https: / http//gtttuuuooogoo tm mol-tuebinggn// /orttobbbggg//ooootttootgnggenn htt.:/oortttgngooootoa
nat innion/.

---

How Much Can We Forget about Data Contamination?
First, similar many other works, we find that the tendency of model to overfit increases in the number of parameters (Goodfellow et al., 2016; Zhang et al., 2017; Carlini et al., 2022). Second, and this is also expected, we find a clear scaling in the number of repetitions, where more frequently repeated observations exhibit stronger overfitting (Carlini et al., 2022; Huang et al., 2024b). More surprisingly, we find that the effect of contamination can vanish as we increase the number of training tokens, up to the point where 12 repetitions of an entire dataset in the training data have no impact on the downstream evaluation on that same dataset.
2. Related Work
Data Contamination. The GPT-3 paper (Brown et al., 2020) uses an n-gram-based approach to differentiate be- tween "clean" and "dirty" benchmark questions. This ap- proach has since been used in many LLM reports (Chowd- hery et al., 2023; Touvron et al., 2023), including Llama 3 (Dubey et al., 2024), where it is estimated that there might be performance gain of up to 8 and 14 percentage points on PiQA and HellaSwag, respectively. The GPT-4 techni- cal report (Achiam et al., 2023) remarkably concluded that "contamination overall has very little effect on the reported results". This has since given rise to literature that aims to detect (Oren et al., 2024), mitigate (Li et al., 2024), and estimate the effect of (Yang et al., 2023; Bordt et al., 2024) data contamination under various assumptions, but crucially without access to the training data. This literature often chal- lenges the conclusion that contamination overall has little effect in GPT-4 (Xu et al., 2024). In closely related work, Jiang al. (2024) show that inserting benchmark questions into the pre-training data can lead to overfitting. Kocyigit et al. (2025) investigate the impact of data contamination on machine translation.
Our investigation reveals that the forgetting dynamics of gradient descent (Tirumala et al., 2022; Jagielski et al., 2023) the reason why increasing the number of tokens alleviates the impact of contamination. Concretely, we show that training five times Chinchilla (Hoffmann et al., 2022) of clean data can cause a model to forget even 144 times repeated training examples (Section 4.2). Forgetting also occurs for larger models, a point that we demonstrate using OLMo-7B (Groeneveld et al., 2024) (Section 4.3). What the reason for the forgetting? We show that exposure to novel data is important. Interestingly, models tend to exhibit the strongest overfitting on examples seen repeatedly throughout training, even compared to those seen during the end (Section 4.2).
Forgetting. In machine learning, the term forgetting is frequently associated with "catastrophic" " forgetting, where learning new tasks hurt the performance at previously solved tasks (Lopez-Paz & Ranzato, 2017). In the context of LLMs, catastrophic forgetting can occur during fine-tuning (Luo et al., 2023) or continual learning (Huang et al., 2024a). In contrast, this paper studies forgetting as potential "natural" phenomenon of learning (Toneva et al., 2019). Tirumala et al. (2022) study forgetting in language modeling and find, similar to Toneva et al. (2019), that forgetting can be exponentially slow. In contrast, Jagielski et al. (2023) find that models empirically do forget examples over time. Pagliardini et al. (2025) propose to add second momentum term to the Adam W optimizer, and show that this slows down the forgetting of past gradients.
Because running pre-training experiments is expensive, we also ask to what degree forgetting can be explained by the training dynamics of gradient descent. We find that the weight decay parameter and learning rate schedule of the AdamW optimizer (Loshchilov & Hutter, 2019) play a key part in forgetting past training examples (Section 5). Con- cretely, we demonstrate that the cumulative weight decay (Section 5. ) bounds empirical forgetting (Section 5.2). This approach allows us to gauge the degree of forgetting in large- scale training runs by analyzing the optimization hyperpa- rameters (Section 5.3). It also allows us to approximate how the final model parameters balance the gradient updates from different stages of training. Our analysis indicates that many LLMs, including OLMo-7B and Ilama 3 405B (Dubey et al., 2024), have forgotten the data seen at the beginning of training.
Data Attribution. Data attribution methods (Koh & Liang, 2017; Ilyas et al., 2022: Park al., 2023) aim to identify data points responsible for specific model behaviors. We ask how much model's benchmark performance is influenced by seeing the example during training, which broadly falls within this field (Grosse et al., 2023; Choe et al., 2024). Im- portantly, we directly measure the influence of contaminated examples through explicit retraining, avoiding the approx- imation errors that can occur when using data attribution methods (Koh & Liang, 2017; Ghorbani & Zou, 2019) for large-scale models (Basu et al., 2021; Bac et al., 2022)
Taken together, our main contribution to show that the impact of individual examples in the training data depends on the precise characteristics of the setting. There are set- tings where the effect can be significant; Chinchilla training is an important example (Section 4.1). However, there are equally realistic settings where individual examples don't 't matter- including quite likely the data-intensive training runs of many recent L.LMs (Gemma Team, 2024).
Supplement Sections A. and A.2 provide an overview of the most important takeaways and limitations of our work.
2

---

How Much Can We Forget about Data Contamination?
3. Background and Methods
et al., 2021), PiQA (Bisk al., 2020), BoolQ (Clark et al., 2019), MMLU (Hendrycks al., 2021), and HellaSwag (Zellers et al., 2019). This means that every evaluation contains questions from all seven benchmarks. To con- struct the mixed contamination data, we first concatenate the different benchmarks. We then partition the set of all benchmark questions into subsets ranging from 10,000 to 2,000 questions so that each subset contains all benchmarks in equal weight: HellaSwag: 19.58%, SociallQA: 8.27%, PiQA: 19.7%, MMLU: 21.82% BooiQ: 6.48% ARC-Easy: 5.92%, and WinoGrande: 18. 16%. A holdout set of 10,000 benchmark questions is never added to the training data. The other subsets are added to the training data, repeated either 4, 12, 36, or 144 times.2 2 We consider exact contam- ination, that is we contaminate the training data with the same texts that the model is later evaluated on. We insert benchmark questions individually and at random positions into the training data. Models are evaluated zero-shot via the likelihood assigned to different sentence completions (Gao, 2021). For more discussion and details about how contami- nation is performed, see Supplement A.3 and Supplement A.4.
This Section gives additional details on the research ques- tions and lays out our experimental setup.
Research question
How does the presence of text in the training data influence the final model's performance on that same text?
While it is well known that an individual data point can be influential if the training data and model are small (Koh & Liang, 2017), we are concerned with the large-data regime where the influence of any individual example may vanish (Basu et al., 2021; Bae et al., 2022). To further clarify this setup, Section 3.1 gives a brief overview of recent devel- opments in scaling the training data of L.LMs. Section 3.2 details the used benchmarks and explains how contami- nate the training data. Section 3.3 discusses the problem of near-duplicate benchmark questions.
Models and Training Data. We train language models of up to 1.6B parameters using the architecture and hyperpa- rameters from the GPT-3 paper (Brown et al., 2020, Table 2. 1). For this, we adopt the llm.c codebase. The training data is the 100BT split of the FineWeb-Edu dataset (Lozhkov et al., 2024). We also train OLMo-7B (Groeneveld et al., 2024) using the corresponding code and data (Soldaini et al., 2024).
3.3. We Filter Near-Duplicate Benchmark Questions
Our method requires no side effects from contaminating the training data with one question on the evaluation of another question. However, upon eloser inspection, it turns out that all the commonly used benchmarks from the literature con- tain questions that are either near-duplicates or where the context of one question contains the answer to another ques- tion (for example, because the same text document was used to create multiple questions). To address this problem, we perform extensive filtering, removing duplicate questions where the length-normalized Levenshtein distance falls be- low a certain threshold (Levenshtein, 1966; Navarro, 2001). The issue of duplicate benchmark questions is detailed in Supplement A.5, where we also describe an experiment to verify that near-duplicate questions do not invalidate our method.
3.1. We Consider the Regime of n-times Chinchilla
According to the Chinchilla scaling law, for every doubling of model size, the number of training tokens should also be doubled. The Chinchilla model itself has 70 billion (B) parameters and was trained on 1.4 trillion (T) tokens; sug- gesting that the number of training tokens should be roughly 20x the number of model parameters (Hoffmann et al., 2022, Table 3). While the Chinchilla paper was highly influential, modern language models are trained on significantly more tokens (Sardana& & Frankle, 2024). For example, the OLMo- 7B model was trained on 2.46T tokens, 17.5x the amount suggested by Chinchilla (Groeneveld et al., 2024). Simi- larly, the Llama 3 70B model was reportedly trained on 15T tokens, at over 10x Chinchilla (Dubey al., 2024; Meta AL, 2024). The same holds for almost all recent 7B-parameter LLMS (Gemma Team, 2024). In this paper, we count the number of tokens a model is trained on as multiple of its Chinchilla tokens.
4. Experimental Results
We now present our main experimental results. We begin in Section 4.1 by discussing the scaling in model parameters, training tokens, and repetitions in the training data. The following Section 4.2 discusses various experiments on for- getting. The first two sections rely on training small GPT-3
IIn preliminary experiments, we found that these numbers pleasantly cover the range from statistically significant contam- ination complete overfitting. We also considered repeating observations single time, as (Jiang et al., 2024). However. we found this often leads to accuracy differences of about one or two percentage points, just within the margin of our confidence intervals, which is undesirable.
3.2. We Evaluate on a Mix of Different Benchmarks
We evaluate the impact of data contamination using a mix of seven different benchmarks: ARC-Easy (Clark et al., 2018), Social-I-QA (Sap et al., 2019), WinoGrande (Sakaguchi
3

---

How Much Can We Forget about Data Contamination?
60%
60%
60%
Accuracy Difference
50%
Accuracy Difference
50%
Accuracy Difference
50%
40%-
40%-
40%
30%
30%-
30%
20%
20%
20%
10%
10%
10%
09
0%
0%
1
350M
774M
1.68
15
124M
350M
774M
1.68
Model Parameters
Chinchilla Tokens
Model Parameters
(a) Scaling the Model
(b) Scaling the Data
(c) Chinchilla Scaling
Holdout
times repeated
12 times repeated
32 times repeated
144 times repeated Contamination
Figure 1. Benchmark overfitting due to contamination. (a) We train different models on 7B tokens. (b) We train 124M parameter models on increasingly many tokens. (c) We train models according to the Chinchilla scaling laws. The figure depicts the accuracy difference in percentage points between the holdout (normalized to zero) and the contaminated examples. The results across mix of seven different benchmarks, as outlined Section 3.2. Different colors indicate different levels of contamination. Mean and bootstrapped
90% confidence intervals.
depicts the accuracy difference between the holdout and contaminated examples percentage points. In Figure 1a, we see that the accuracy difference due to contamination increasing in the number of model parameters. For 124M parameter model trained on 7B tokens, the overfitting due to 4 times contamination is percentage points. For 1.6B parameter model train on the same dataset, it is 20. Next, Figure lb shows that the accuracy difference is decreasing in the number of training tokens. For 124M parameter model trained at 2x Chinchilla, the accuracy difference due to 12 times contamination 18 percentage points. For a 124M parameter model trained at 15x Chinchilla, the respective accuracy difference is within the confidence interval of the holdout. From Figure 1, we also see that the accuracy difference is increasing in the number of times an example is repeated. For a 350M parameter model trained on 7B tokens, the accuracy difference is 11, 25, 44, and 51 percentage points for 4, 12, 32, and 144 times repeated contamination, respectively.
Table 1. Accuracies of the Chinchilla-optimal models. The table depicts the absolute accuracies the Chinchilla-optimal models for holdout and contaminated benchmark questions.
Model
Holdout
4x
12x
32x
144x
124M
42.22
48.14
56.92
80.70
96.45
350M
44.72
55,69
69.90
89.20
95.50
774M
49.16
64.76
81.30
92.95
96.05
1.6B
52.06
67.61
82.32
91.85
95.40
models. Section 4.3 complements this with an analysis of OLMo-7B (Groeneveld et al., 2024).
4.1. Contamination Scales with Model, Data, and Repetitions
We conduct three different experiments understand how the effect of data contamination scales with the number of model parameters, training tokens, and the number of times contaminated example is seen. First, we train increasingly large models on the same dataset of 7B tokens. Second, we train 124M parameter models on increasingly many tokens. Third, we train increasingly large models according to the Chinchilla scaling laws (Hoffmann al., 2022), meaning that the number of training tokens scales linearly with the model parameters. In all experiments, contaminate the training data uniformly random with benchmark ques- tions.
Because the accuracy difference increases in the number of model parameters and decreases in the number of to- kens, the interesting question is how it behaves if model parameters and tokens are scaled jointly. A natural start- ing point is to double the number of training tokens for every doubling of model parameters, as specified by the Chinchilla scaling laws (Hoffmann et al., 2022). Figure Ic depicts the accuracy difference due to contamination train increasingly large Chinchilla-optimal models. While there is no clear monotone pattern, we see that moderate amounts of contamination can lead to significant overfitting. For the 774M parameter model, times repeated contam- ination leads to an accuracy difference of 15 percentage
Figure depicts the results of all three experiments. Because we are interested in the performance difference between the holdout data and the contaminated examples, Figure
4

---

How Much Can We Forget about Data Contamination?
-2.0
15.0%
6.0%
CE Loss Difference
Accuracy Difference
Accuracy Difference
-1.5
4.0%
10.0%
2.0%
-1.0
0.0%
4.0%%
0
3
11
13
15
-5.0%
4x
12x
32x
144x
4x
12x
32x
144x
Chinchilla Tokens
Number Repetitions
Number Repetitions
(a) Forgetting on Novel Data
(b) After Chinchilla
(c) After Chinchilla
-2.0
6.0%-
CE Loss Difference
Accuracy Difference
Accuracy Difference
30.00%
-1.5
4.0%
3.0%
20.0%
-1.0
10.0%
-0.5
1.0%
0.0%
0.0%
First 40%
Middle 20% Last 40%
Rist
20%
40%
Chinchilla Tokens
Avg. Contamination Position
Avg. Contamination Position
(d) Forgetting on 100M tokens
(e) 124M 15x Chinchilla
(f) 774M 1x Chinchilla
Holdout
times repeated
12 times repeated
32 times repeated
144 times repeated Contamination
Figure 2. The forgetting dynamic of neural network training. (a) The development of the cross-entropy loss difference between contaminated holdout benehmark questions over the course training. Contamination occurs hetween first and second Chinchilla and 2 the x-axis). (b) Accuracy differences after training for 3 Chinchilla. (c) Accuracy differences after training for Chinchilla. (d) Same as (a). (f)+(f) The accuracy difference depends on the average position of an example in the training data. Mcan and bootstrapped 90% confidence intervals.
points, suggesting that under Chinchilla training, single time of contamination can lead to overfitting of as much as 3 percentage points.
uniformly over the course of training like in the previous Section 4.1, we perform the contamination between the first and second Chinchilla." Figure 2a depicts the development of the difference in cross-entropy loss between contaminated and clean benchmark questions over the course of training, We strong peak after Chinchilla, which is expected and shows the effect of contamination. What is interesting to us is the rate at which the cross-entropy loss difference decays as we continue training. After training for ad- ditional Chinchilla (2.5B tokens for the 124M parameter model), has already decayed significantly. However, the difference still visible Figure 2a. Figure 2b depicts the corresponding accuracy differences at this point, and see that all contamination levels still lead to overfitting. As we continue training, the cross-entropy loss difference be- tween contaminated and holdout questions further narrows.
4.2. Contamination Can be Completely Forgotten
In the previous Section 4.1, we saw that the accuracy dif- ference due to contamination decreases in the number of tokens up to the point where even 12 repetitions of bench- mark question in the training data can become insignificant. In this Section, we identify the forgetting dynamic of neural network training the reason for this effect. We discuss how quickly forgetting occurs, whether examples are com- pletely forgotten, and what kind of repetition makes a model remember.
To study the effect of forgetting, we train a 124M param- eter model 15x Chinchilla. Instead of contaminating
"Note that the model is already fairly trained after the first Chinchilla, meaning that the contamination not very early dur- ing training. This is important because there evidence observations are more quickly forgotten the model has not yet learned representations (Jagielski et al., 2023; Cao et al., 2024; Huang al., 2024b). This is not the setting studying here.
"To contaminate training data a 774M model single time with 10,000 benchmark questions, we need to insert ~0.55 million tokens into 15.5 billion training tokens, about 0.003% of the training data.
5

---

How Much Can We Forget about Data Contamination?
Accuracy Difference
25.0%
20.0%
Before Contaminaton on
OlMo-18, rescaled
Mi
Accuracy Difference
12.5%
15.0%
Accuracy Difference
12.5%
15.03:
Accuracy Difference
12.0%
10.0%
7B
15.00 15.0%
10.0%
10.07
8.0%
1.5%
6.0%
30.0%
5.0%
5.0%
4.0%
5.0%
2.5%
2.5%
0.0%
0.0%
0.0%
0.0%
10000
20000
10500
20000
10000
20000
10600
20000
Gradient Step
Gradient Step
Gradient Step
Gradient Step
(a) HellaS wag
(b) WinoGrande
(c) ARC-Easy
(d) PiQA
Figure 3. Contamination and Forgetting in OLMo-IB and OLMo-7B. We contaminate intermediate OLMo checkpoints four times with different benchmarks. This causes an average accuracy increase of percentage points for the 7B model. then continue pre-training for 13% of the remaining training time (or 25000 gradient steps), leading a reduction of 87% of the accuracy increase due to contamination. The solid line depicts the result for OLMo-7B, and the dashed line depicts the results for OLMo-IB. To account for the difference in the number of model parameters, scaled the curve of the IB parameter model by factor of 5.9 (the actual parameter ratio between the two models), after which the forgetting curves of both models align remarkably well (the plot depicts approximately 3500 gradient steps for the IB model, and 25000 gradient steps for the 7B model). Different colors correspond to different benchmarks, and the grey line depicts the clean accuracy before contamination. Supplement Figures 9 and 10 depict the results for both models separately.
From Figure 2c, which depicts the accuracy differences af- ter forgetting for a total of 5 Chinchilla, we see that the effect of contamination is eventually completely forgotten in the sense that there is no longer any accuracy difference between contamination and holdout benchmark questions.
towards the beginning nor the end but as uniformly dis- tributed throughout training as possible exhibit the strongest overfitting, suggesting that this spaced form of repetition helps the model remember (the middle peak is the most pronounced both in Figure 2e and Figure 2f).
The result that contamination can be completely forgotten is in contrast to some previous work on forgetting which have found that forgetting approaches stable baseline Tirumala et al. (2022, Figure 10), or that certain examples are never forgotten (Toneva et al., 2019). To understand this differ- ence, observe that many previous works on forgetting have not trained on a continuous stream of data. Instead, they have trained on the same training set for multiple epochs. Consequently, we modify our forgetting experiment to re- peatedly train on the same 100M tokens after the second epoch. The result of this experiment is depicted in Figure 2d and should be compared to Figure 2a. Interestingly, this sim- ple modification causes the effect of forgetting to stabilize at level strictly larger than zero. We conclude that exposure to novel data important for forgetting, an observation similar to Jagielski et al. (2023).
4.3. Contamination and Forgetting in OLMo
In the previous sections, we trained small GPT-3 models from scratch. In this Section, we complement this analysis by continual pre-training from intermediate OLMo- and OLMo-7B checkpoints (Groeneveld et al., 2024). Similar to the analysis in Section 4.2, we insert the benchmark data at a specific point into the training data and then measure the subsequent forgetting. Unlike in the previous Section, we now insert the entire benchmark data - we already have "clean" baseline from the original OLMo training runs. We insert each benchmark question four times, and contam- inate with four different benchmarks: HellaSwag (Zellers et al., 2019), WinoGrande Sakaguchi et al., 2021, ARC-Easy (Clark et al., 2018), and PiQA (Bisk et al., 2020).
Figure: 3 depicts the result of the experiment. The leftmost point in every plot corresponds to the uncontaminated model, and the following four points depict the effect of contamina- tion. The plots then depict how the effect of contamination is forgotten
To further understand the impact of forgetting, we now ask whether examples seen late during training influence model behavior more strongly than examples seen early during training. To study this question, we average all the different uniform contamination levels from the models in the pre- vious Section 4.1 (to gain statistical power) and consider the amount of overfitting depending on whether question is seen, on average, in the beginning, middle, or end of training. The result of this experiment is depicted in Fig- ure 2e and Figure 2f. As expected under forgetting, we see that benchmark questions seen early during training ex- hibit the smallest amount of overfitting. Interestingly and somewhat unexpectedly, questions that are neither clustered
continue training. Similar to the results in the previous sections, the immediate effect of contamination is significant, leading to an average accuracy increase of 17 percentage points across the different benchmarks for the 7B model. At the same time, the effect of contamination decays considerably as we continue training. After contin- ual pre-training for 13% of the remaining training time, the evaluation on WinoGrande and ARC-Easy for the 7B model is no longer significantly different from the uncontaminated evaluation. On HellaSwag and PiQA, the effect of contam-
6

---

How Much Can We Forget about Data Contamination?
CE Loss Difference
Weight Decay
-1.5
0.8
-1.0
0.4
-0.5
0.0
0.0
9550
9600
9650
9530
10500
12000 14000
95.50
40000
60000
Gradient Step
Gradient Step
Gradient Step
Gradient Step
(a) Weight Decay 50
(b) Weight Decay 5
(c) Weight Decay
(d) Weight Decay 0.1
Figure 4, Empirical forgetting occurs faster than the cumulative weight decay. We continue training the contaminated 124M parameter model from Section 4.2. The figure depicts empirical forgetting (dashed line) and the cumulative weight decay (solid line) for four different choices of the weight decay parameter. Forgetting depends strongly on the weight decay parameter, as is evident from the very different x-axis scales, ranging from 120 gradient steps (a) 62500 gradient steps in (d).
ination has decayed to approximately 2 percentage points of overfitting. This demonstrates that the effect of forgetling during pre-training significant, even for 7B parameter model.
5.1. The Cumulative Weight Decay
Consider the parameter update of Adam W at gradient step 2 consists two decoupled updates: A weight decay update given by
Interestingly, we find that forgetting in OLMo exhibits scaling behavior. To see this, note that Figure 3 also depicts the result of the forgetting experiment with OLMo-IB I.. How- ever, we rescaled the forgetting curve the IB parameter model by factor of 5.9, meaning that any given point on the x-axis, the 7B parameter model has seen 5.9 times as many tokens the 1B parameter model. We choose the factor 5.9 because it corresponds the actual parameter ratio between the two models (OLMo-1B 1B has 1, 1,176,7644 766,416 parameters, whereas OLMo-7B has 6,888,095,744 parame- ters). With this rescaling, the forgetting curves of the two models are surprisingly well aligned, suggesting that for- getting with model that has 5.9 times as many parameters requires 5.9 times many tokens. Of course, this observa- tion is well aligned with the results on scaling in Section 4.1.
0, 01-1-0777 77000-1-
and a gradient update given by
0-=0; - To m 0g +c).
(2)
Here, Og are the model parameters, 7/ the learning rate, is the weight decay parameter, and ri, and 6 are first- and second-order moment estimates of the gradient (Loshchilov & Hutter, 2019; PyTorch Contributors, 2024). Denoting the model weights at initialization by lg. and the adaptive gradient by 9r- = /it/////// + +e), we can iterate (I) and (2) to obtain
T
0, = w do - Σ
(3)
1=1
where the cumulative weight decay is computed follows
5. Weight Decay and Forgetting
m
II (1 7A) 1=1+1
(4)
In the previous Section 4, we have seen that forgetting is an important empirical property of LLM training. Which fac- tors contribute to the phenomenon? In this section, we show that the weight decay parameter and learning rate schedule of the AdamW optimizer play a part in forgetting past train- ing examples. This offers a novel perspective on the weight decay parameter, usually seen in terms of generalization and training stability (Van Laarhoven, 2017; Zhang et al., 2019; Lewkowycz & Gur-Ari, 2020; Andriushchenko al., 2024).
Equation (3) shows that the model weights after gradient steps are a weighted average of the initial model weights and all the adaptive gradient updates up time step In other words, the weights w/² determine the influence of the gradient update at time & on the model weights at time lg. As recently observed by Wang & Aitchison (2024), the weights Oy can also be understood as an exponential moving average of past gradient updates. With weight decay, w decays to zero, as the following proposition describes.
'Here refer the number of tokens in the experiment. The total number of tokens seen by the two models during pre- training up to this point is approximately the same.
Proposition 1. (Cumulative Weight Decay) Fix e O. Let T T= 12 - fi be the number of gradient steps that have passed since step t1. Let 7vvg = 7 Et,+1 Ys be the average learn- ing rate between gradient steps 1 and fg. If sufficiently
7

---

How Much Can We Forget about Data Contamination?
Cumulatiiv Weight Decay
Curulattive Weight Decay
Cumulative Weight
0.6-
0.6-
0.6
0.4-
0.4-
0.4
0.2
0.2-
02
0.
10000 20000 30000 40000 soboo coooo 70000 Gradient Steo
0.0
0
200000
400000
600000
0.0
200000
400000 600000
800000 100000 Gradient Step
Gradient Step
(a) 124M 15x Chinchilla
(b) OLMo 7B
(c) Llama 3 405B
0%
10% 20% 30%
40% 50% 60%
70%
80% 90%
Figure 5. Cumulative weight decay and approximate model weight composition for different large-scale training runs. Top Row: The cumulative weight decay wi? as defined in equation (4) for different training runs. The figures depict the decay of the gradient updates for every decile of the training run, indicated in different colors. Bottom Row: The approximate composition of the final model weights in terms of the gradient updates from different deciles of the training run. The deciles indicated in colors depicted in the legend below the plot. The cumulative weight decay can be casily computed using this Jupyter Notebook.
large, that is T 2
log(1/s) Anoss
then
the empirical rate of forgetting. This is evident from the very different x-axis scales in the different plots Moreover, the dashed line always lies below the solid line, mcaning that empirical forgetting occurs faster than the cumulative weight decay. This is despite the fact that weight decay is not necessary for forgetting (see Supplement Figure 11).
S e. 1
l
We present a proof in Supplement A.8.
What is the relationship between cumulative weight decay and forgetting? The adaptive gradient 9t, contains the rele- vant information about the training examples seen at time t1. If 9., is no longer part of the model weights because ti has decayed to zero, then the model should no longer depend on the examples seen at time h1 In other words, training examples seen at time I should be forgotten. However, the data seen at time f1 still influences fr via the trajectory of gradient descent, even if the direct contribution of 9t, has decayed to zero. Because of this, we now investi- gate the relationship between forgetting and weight decay experimentally.
5.3. Cumulative Weight Decay in Large-Scale Training
Given Proposition and the experimental results in Section 5.2, hypothesize that example forgetting occurs least as fast the cumulative weight decay. Based on this hypoth- esis, we now gauge the degree of forgetting in large-scale training runs. The key insight here is that the cumulative weight decay depends only on the learning rate schedule and the weight decay parameter. This means that we can study the cumulative weight decay even large-scale training that far exceed our experimental budget. Ideally, would of course want experimental evidence on forgetting in large-scale training, akin to the analysis in Section 4.1. Since this is not possible within our experimental budget, we resort to a heuristic analysis of example forgetting via the cumulative weight decay.
5.2. Empirical Forgetting Occurs Faster than the Cumulative Weight Decay
We now study the causal effect of weight decay on forgetting by intervening on the weight decay parameter, keeping ev- erything else constant. We consider the contaminated model from Section 4.2 after two Chinchilla and continue training with four different choices of the weight decay parameter. Figure 4 depicts the result of this experiment. Every plot in Figure 4 depicts two curves. The cumulative weight de- cay is depicted as a solid line. The empirically observed forgetting for 144 times repeated contamination is depicted as a dashed line (compare Figure 2a). From Figure 4, we see that increasing the weight decay parameter increases
Figure 5 depicts the cumulative weight decay w for three different models: The 124M parameter model from Section 4.2, OLMo 7B (Groeneveld et al., 2024), and Llama 3 405B (Dubey et al., 2024) (where we assume the model trained with a weight decay of 0.).. The figure depicts the evolution of the term w/ for ten fixed choices of t1, one at each decile of the training run. As discussed above, we can interpret the curves in Figure heuristic forreetinoorgettnng curves
8

---

How Much Can We Forget about Data Contamination?
that indicate the degree to which the data seen at different stages is forgotten we continue training. For the 124M model depicted in Figure 5a, even the initialization is not completely forgotten at the end of training (the blue curve is still significantly larger than zero). For OLMo-7B, however, the weights of the gradients of the first 40% of training decay to zero until the end of training, meaning that the data seen during this time is likely forgotten (Figure 5b). The same holds for Llama 3 405B, where the first 10% of the training data is likely forgotten (Figure 5c).
This means that one has to be careful when extrapolating our results to other contexts, especially (Carlini et al., 2019; Jagielski et al., 2023). This is because empirical forgetting might behave differently for random strings or otherwise uniquely identifiable information (Car- lini et al., 2021).
privacy setup
Impact Statement
This paper studies the empirical properties of training deep neural networks. Because our study concerns the scientific question of model evaluation, we don't believe it raises significant ethical concerns. That being said, contamina- tion and memorization have broader implications, including copyright and privacy.
By integrating w the cumulative weight decay even al- lows us to approximate the composition of the final model weights in terms of the gradient updates from different deciles of training. This depicted in the bottom row of Figure 5.
Acknowledgments
6. Discussion
We would like to thank Jonas Geiping and Dirk Groen- eveld for helpful discussion about data contamination. This work is supported by the Tübingen AI Center, the Ger- man Research Foundation through the Cluster of Excellence "Machine Learning- New Perspectives for Science" (EXC 2064/1 number 390727645), and the CZS Institute for Arti- ficial Intelligence and Law.
In this work, we have connected the literature on data con- tamination (Jiang et al., 2024) with research on forgetting (Tirumala et al., 2022; Jagielski et al., 2023). Through care- ful controlled experiments, we have shown that the impact of contamination depends on the joint scaling of model, data and contamination overlooked in the existing literature (Yang et al., 2023; Oren et al., 2024; Jiang et al., 2024). Perhaps surprisingly, we have also seen that the impact of contamination can be com- pletely forgotten, an effect that is especially relevant for training runs that operate in the large-data regime (Groen- eveld et al., 2024; Gemma Team, 2024). As a consequence, large-data LLM pre-training also appears to be empirically stable with respect to individual benchmark questions (Bous- quet & Elisseeff, 2002). Interestingly, we have also seen number of ways in which L.LM pre-training is different from other learning setups. For example, we have seen that forget- ting behaves very differently when training on a continuous stream of novel data as opposed multi-epoch training on limited dataset (Tirumala et al., 2022; Muennighoff et al., 2023).
aspect that has largely been
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. OpenAl, 2023,
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309. 14316, 2023.
Andriushchenko, M., D' Angelo, F., Varre, A., and Flammar- ion, N. Why do need weight decay in modern deep learning? In NewrIPS, 2024.
Bae, J., Ng. N., Lo, A., Ghassemi, M., and Grosse, R. B. If influence functions are the answer, then what is the question? In NeurIPS, 2022.
The "contamination problem" as studied in this paper has interesting connections to many areas of machine learn- ing, including privacy (Graves et al., 2021; Jagielski et al., 2023), data attribution (Kirchenbauer et al., 2024), and gen- eralization (Bousquet & Elisseeff, 2002; Hardt al., 2016; Mania et al., 2019). The connection to data attribution arises from the fact that the presence or absence of a benchmark question in the pre-training data sometimes appears to be irrelevant for the model behavior on that same datapoint, suggesting that it does not make sense to attribute model behavior to individual datapoints in this regime.
Basu, S., Pope, P., and Feizi, S. Influence functions in deep learning are fragile. In ICLR, 2021.
Biderman, S., Prashanth, U., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raff, E. Emergent and predictable memorization in large language models. In NewrIPS, 2023.
An important limitation of our work that our empirical results on forgetting are restricted to benchmark questions.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020.

---

How Much Can We Forget about Data Contamination?
Bordt, S., Nori, H., Rodrigues, V., Nushi, B., and Caruana, R. Elephants forget: Memorization and learning of tabular data in large language models. In COLM, 2024.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
Bousquet, O. and Elisseeff, A. Stability and generalization. The Journal of Machine Learning Research, 2002.
Deng, C.,, Zhao, Y., Tang. X., Gerstein, M.. and Cohan, A. Investigating data contamination in modern benchmarks for large language models. In ACL, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Voss. A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, F.,, Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In NeurIPS, 2020.
Donoho, D. 50 Years of Data Science. Journal of Computa- tional and Graphical Statistics, 2017.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Cao, B., Tang, Q.. Lin, H., Jiang, S., Dong, B., Han, X., Chen, J., Wang, T., and Sun, L. Retentive or forgetful? diving into the knowledge memorizing mechanism of language models. In LREC-COLING, 2024.
Gao, L. Multiple choice normalization in Im evaluation. Blog Post, 2021. URL https: : wlw.o...eeeee.rr... ai/multip whappeee-chiieeeeeee wetteeoooommaaaaiiiiiee
Carlini, N., Liu, C., Erlingsson, Ú. Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX secu- rity symposium (USENIX security 19). 2019.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac h, A., Li,
H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou,
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2021.
A. A framework for few-shot language model evaluation, 2024. URL ht / /gi thub. homm/Eleuthheree harness.
Gemma Team. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.00/18, 2024.
Carlini, N., Ippolito. D., Jagielski, M., Lee, K., Tramer, F,, and Zhang, C. Quantifying memorization across neu- ral language models. arXiv preprint urXiv:2202.07646. 2022.
Ghorbani, A. and Zou, J. Data shapley: Equitable valuation of data for machine learning. In ICML, 2019.
Chang, H., Park, J., Ye, S., Yang. S., Sco, Y., Chang, D.-S., and Seo, M. How Do Large Language Models Acquire Factual Knowledge During Pretraining? In NeurlPS, 2024.
Golchin, S. and Surdeanu, M. Time travel in 1lms: Trac- ing data contamination in large language models. arXiv preprint arXiv:2308..88 08493, 2023.
Choe, S. K., Ahn, H., Bae, J., Zhao, K., Kang, M., Chung, Y., Pratapa, A., Neiswanger, W. Strubell, E., Mitamura, T., al. What is your data worth gpt? Ilm-scale data valuation with influence functions. arXiv preprint 1rXiv:2405.13954 2024.
Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016.
Graves, L., Nagisetty, V., and Ganesh, V. Amnesiac machine learning. In AAAI, 2021.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W. Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 2023.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kin- ney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. Olmo: Accelerating the science of language models. In ACL, 2024.
Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., et al. Study- ing large language model generalization with influence functions. arXiv preprint arXiv:2308. 032966 2023.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In NAACI, 2019.
10

---

How Much Can We Forget about Data Contamination?
Grynbaum, M. M. and Mac, R. The Times Sues OpenAl and Microsoft Over A.I. Use of Copyrighted Work, 2023.
Lewkowyez, and Gur-Ari, G. On the training dynamies of deep networks with L2 regularization. In NeurlPS, 2020.
Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. In ICMI., 2016.
Li, C. and Flanigan, J. Task contamination: Language models may not be few-shot anymore. In AAAI, 2024.
Li, Y., Guerin, F., and Lin, C. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In AAAI, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In ICLR, 2021.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C. A., Manning, D., Re, C., Acosta-Navaae D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F,, Rong, F,, Ren, H., Yao, H., WANG, J., Santhanam, K., Orr, L., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N., Guha, N., Chatterji, N. S., Khattab, O., Henderson, P., Huang, Q. Chi, R. A., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023.
Hoffmannn, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L.., Hendricks, L. A., J., Clark, A., et al. Training compute-optimal large language models. In NeurlPS, 2022.
Huang. J., Cui, L., Wang, A., Yang, C., Liao, X., Song, L., Yao, J., and Su, J. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In ACI. Annual Meeting, 2024a.
Huang. J., Yang. D., and Potts, C. Demystifying verbatim memorization in large language models. arXiv preprint arXiv:2407. 17817, 2024b.
Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. In NeurlPS, 2017.
Ilyas, A., Park, S. M., Engstrom, L., Leclere, G., and Madry, A. Datamodels: Predicting predictions from training data. In JCML, 2022.
Loshchilov, and Hutter, F. Decoupled weight decay regu- larization. In ICLR, 2019.
Jagielski, M., Thakkar, O., Tramer, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A. G., Papernot, N., and Zhang, C. Measuring forgetting of memorized training examples. In ICLR, 2023.
Lozhkov, A., Ben Allal, L., von Werra, L., and Wolf, T. Fineweb-edu, 2024. URL co co/dataseta. HuggingFaceFW/ Serecceee///ineeeeeee fineweb- edu.
Jiang, M., Liu, K. Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, S. Investigating data contami- nation for pre-training language models. arXiv preprint arXiv:2401.0m 06059, 2024.
Luo, Y., Yang, Z., Meng, F,, Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08 300.0077777 2023.
Kirchenbauer, J., Honke, G., Somepalli, G., Geiping, J., Ippolito, D., Lee, K., Goldstein, T., and Andre, D. Lmd3: Language model data density dependence. arXiv preprint arXiv:2405.0633/ 2024.
Magar, and Schwartz, R.
Data contamination:
From memorization to exploitation. arXiv preprint arXiv:2203.08222, 2022.
Mania, H., Miller, J., Schmidt, L., Hardt, M., and Recht, B. Model similarity mitigates test overuse. NeurIPS, 2019.
Kocyigit, M. Y., Briakou, E., Deutsch, D., Luo, J., Cherry, C., and Freitag, M. Overestimation in IIm evalua- tion: A controlled large-scale study on data contami- nation's impact machine translation. arXiv preprint 17111 2025.
Meta AL. Introducing Meta Llama 3: The most capable openly available LLM to date. Blog Post, 2024. URL https: //ai www...t...ommmm cewea..commblli com/blog/ meta 1lamalaam33 3/.
Koh, P. W. and Liang, P. Understanding black-box predic- tions via influence functions. In ICML, 2017.
Muennighoff, N., Rush, A., Barak, B., Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data-constrained language models. In NeurIPS, 2023.
Levenshtein, V. Binary codes capable correcting dele- tions, insertions, and reversals. Proceedings of the Sovier physics doklady, 1966.

---

How Much Can We Forget about Data Contamination?
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F,, and Lee, K. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:231 1.70055, 2023.
Tirumala, K., Markosyan, A., Zettlemoyer, L.,, and Agha- Janyan, A. Memorization without overfitting: Analyz- ing the training dynamics of large language models. In NeurIPS, 2022.
Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A.. Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019.
Navarro, G. A guided tour to approximate string matching. ACM computing surveys (CSUR), 2001.
Oren, Y.. Meister, N.. Chatterji, N., Ladhak, F., and Hashimoto, T. B. Proving test set contamination in black box language models. In ICLR, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models, 2023.
Pagliardini, M., Ablin, P., and Grangier, D. The AdEMAMix Optimizer: Better, Faster, Older. In ICLR, 2025.
Van Laarhoven, T. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.0053: 2017.
Park, S. M., Georgiev, K., Ilyas, A., Leclerc, G., and Madry, A. Trak: Attributing model behavior at scale. In ICML, 2023.
Wang, X. and Aitchison, L.. How to set adamw's weight decay as you scale model and dataset size. arXiv preprint arXiv:2405. 13698, 2024.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
Xu, C.,, Guan, S.. Greene, D., Kechadi, M., et al. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.044 2024.
PyTorch Contributors. AdamW. Pytorch Documentation, 2024. URL https: //tp::///////tt //tpt/rchtccccccccooooooooocccc orgo prgg/doc///... /2.cs/22.4. generated/ opt in AdamM.htmm
Yang, S., Chiang, W.-L.,, Zheng, L.., Gonzalez, J. E., and Stoica, I. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04852 2023.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 2021.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sentence? In ACL, 2019.
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social IQa: Commonsense reasoning about social interactions. In EMNLP-IJCNLP. ACL, 2019.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking general- ization. In ICLR, 2017.
Sardana, N. and Frankle, J. Beyond chinchillaooptmmal: Accounting for inference in language model scaling laws. In ICML, 2024.
Zhang, G., Wang, C., Xu, B., and Grosse, Three mecha- nisms of weight decay regularization. In ICLR, 2019.
Singh, A. K., Kocyigit, M. Y., Poulton, A., Esiobu, D., Lomeli, M., Szilvasy, G., and Hupkes, D. Evaluation data contamination in llms: how do we measure it and (when) does matter? arXiv preprint arXiv:241 1.03923, 2024.
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkinson,
D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, Lucy, L., Lyu, u, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, 7.,, Strubell, F., Subramani, N., Tafjord, O., Walsh, P., Zettlemoyer, L., Smith, N. A., Hajishirzi, H., Beltagy. I., Groeneveld, D., Dodge, J., and Lo. K. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In ACI. 2024.
12

---

How Much Can We Forget about Data Contamination?
A. Appendix
A.1. Takeaways
Here we list some important takeaways from our work.
1. The impact data contamination is not fixed, but depends on the scale model, data, and contamination.
2. If the scale of the data is large compared to the scale of the model and contamination, then the causal effect of contamination can be zero.
3. The interplay of two opposing mechanisms determines the impact of seeing a benchmark question during pre-training. Directly after seeing the question, there is spike in the likelihood that corresponds to overfitting. Subsequently, there is forgetting effect, meaning that the impact of the text on the model decays we continue training.
4. LLM pre-training is fairly stable with respect to individual benchmark questions.
5. LLM pre-training, where the model continuously exposed to novel data over many gradient steps, differs from multi-epoch training and small-data fine-tuning setups because of forgetting.
6. The weight decay parameter in the Adam optimizer has a causal effect on forgetting.
7. Cumulative weight decay might be useful heuristic for forgetting in large-scale training runs.
A.2. Limitations
Here list some important limitations of our work.
1. The experiments in this paper were conducted only with benchmark questions, and only for the setup of exact contamination.
2. Forgetting might behave differently for random strings other uniquely identifiable information.
3. Further experiments would be required to validate our results at larger scale (> 7B parameters).
A.3. Additional Discussion of Data Contamination Assumptions and Setting
Here, we discuss our data contamination approach in a bit more detail.
In this paper, we consider only exact contamination. This means we contaminate the training data exactly with the text the model later evaluated on. the litcrature, it has been shown that non-exact contamination (re-worded questions, translation into different language) can affect benchmark performance, too. For example, Yang et al. (2023) have shown that 13B parameter Llama 2 Model (Touvron al., 2023) can achieve an accuracy increase of over percentage points after training re-phrased benchmark questions. We decided against considering non-exact contamination in this paper because the models we train from scratch are much smaller than those for which non-exact contamination results have been shown. This means these models are less capable of making sense of related information, potentially leading us to underestimate the effect of non-exact contamination for realistic training runs.
In addition, we consider contamination with individual benchmark questions, inserted into the training data at random positions. We consider this setup because we are interested in contamination from the perspective of leakage, where individual benchmark questions may enter the training data via different documents (for example, as quotes in Wikipedia articles, case described in Brown al. (2020)). This contrasts with the setup where dataset is present in the training data as long contiguous string, which we conjecture might have a similar impact but be easier detectable (Oren et al., 2024). The fact that we contaminate with benchmark questions also sets apart from related works that study data contamination and memorization for random strings and uniquely identified objects (Carlini et al., 2019; 2021). It is worth highlighting that the results between these two setups might differ, especially considering the time it takes to forget an example.
We only consider pre-training.
13

---

How Much Can We Forget about Data Contamination?
Table Overview of benchmarks used the paper. This tahle documents the experiments with GPT-3 models. The first rows provide the dataset split and corresponding number of benchmark questions. The third row provides the number of questions that were removed fourth row provides the number questions that were
from the dataset after filtering each dataset for near-duplicate questions. removed after additionally filtering for near-duplicate questions across all the different datasets combined. The fifth row provides the dataset's weight in dataset splits used in experiments.
WinoGrande
HellaSwag
Social-I-QA
ARC-Eau Easy
MMLU
PIQA
BoolQ
Split
Validation
Train
Train
Validation
Test
XL, Train
All
Size
10,042
16,113
33,410
3,269
14,042
40,398
5,197
Filtered
1,416
7,386
29,756
409
4,423
21,944
2,568
Cross-Filtered
3
15
0
Weight 19.58%
19.77%
8.27%
6.48%
21.82%
18.16%
5.92%
HellaSwag:
woman stands holding violin against herself. The woman plays the violin. The woman stops playing the violin.
ARC-Easy: Question: student playing with a small
toy boat [...] The boat moves toward the shore because the waves transfer Answer: eneryy.
HellaSwag: A man is standing outside holding a violin. He begins to play the violin. he stops and sets the violin to his side.
MMLU: Question: wave transfers
A nswer: energy
Figure 6. Language modeling benchmarks frequently contain near-duplicate questions. We perform extensive filtering for duplicates using fuzzy string matching. The figure depicts near-duplicate from HellaSwag and a cross-benchmark duplicate from ARC-Easy/MMLU.
A.4. Additional Details on Evaluation and How Contamination was Performed
Benchmark Questions and Evaluation. We use code from OLMo (Groeneveld et al., 2024) to format the different benchmark questions. This code is again based in part on the EleutherAl Evaluation Harness (Gao et al., 2024). The benchmark questions are multiple-choice, and the different options are presented the model zero-shot as possible sentence continuations. The prediction is the sentence continuation with the largest likelihood. For the small GPT-3 models, normalize by the number of tokens (Gao, 2021). For OLMo, we rely on the evaluation framework that is part of the code repository.
Inserting benchmark questions into the training data. batch LLM training data consists B sequences of S tokens, resulting in batch size of x S. For example, OLMo-1B trained with B = 2048 and S 20448 the batch for a single gradient step contains ~4M tokens (Groeneveld al., 2024). Individual sequences in a batch usually contain multiple texts separated by special end-of-text token. We insert benchmark questions random positions into the pre-training data, separated the beginning and end with the end-of-text token.
A.5. Filtering Near-Duplicate Benchmark Questions
Upon close inspection of the different benchmarks, it turns out that there are exact and near-duplicate questions both within and across benchmarks. Consider, for example, the following examples from HellaSwag (Zellers et al., 2019):
HellaSwag 1: person is seen riding board along the water with kite on top. more clips are shown of the person riding back and fourth the board.
and
HellaSwag 2: person is seen riding a board along the water with a kite on top. More clips are shown of the
14

---

How Much Can We Forget about Data Contamination?
person riding back and fourth on the board. the person continues to ride the board along the water.
Note that these are the ground-truth options of two different benchmark questions. Similar patterns can be observed for many benchmarks, for example, because single text document was used to create different benchmark questions. Here is another example from PiQA (Bisk et al., 2020):
PiQA 1: Goal: how do you close a cabinet? Solution: push the door shul.
and
PiQA 2: Goal: how do you close cabinet? Solution: shut the door.
Near-duplicate benchmark questions present a challenge for our methodology. This is because one question might end up in of benchmark questions that we highly contaminate with, whereas the other question might end up in the purportedly "clean" set of benchmark questions that we use for holdout evaluation. To tackle this problem, we perform fuzzy string matching between the ground-truth options (that is, the potential contamination data) of all benchmark questions, randomly removing one question for every detected duplicate. We use the Python package rapidfuzz.
Summary Statistics. Table 2 depicts summary statistics about the different benchmarks, including the number questions that were filtered during the duplicate- datasets, especially Social-i-QA, we had to apply very aggressive filtering to avoid any side-effects during contamination. Hence, the number of removed questions per dataset does not necessarily reflect the actual number of duplicates, but the level of filtering that had to be applied to remove all duplicate questions.
stage. We see that the number of filtered questions is significant. On some
Experimental verification that filtering worked. We verify that our filtering procedure worked by training two models: One that is heavily contaminated (obtaining an accuracy of over 97%), and another model that did not see any contamination. We then evaluate both models on a set of 10,000 benchmark questions that are holdout even for the contaminated model. The contaminated model obtains an accuracy of 42.2%, (95%-CI: 41.2% 43.2%) on the holdout, while the clean model obtains an accuracy of 41.9% (95%-CI: 41.0% -42.9%) Because the observed accuracy difference small in absolute terms and lies within the confidence interval, we conclude that there are significant side-effects in our evaluation procedure.
A.6. Measuring the Overlap Between the Benchmark Questions and the Pre-Training Data
Here, we analyze the overlap between the 44000 de-duplicated benchmark questions and the pre-training data. Note that we did not de-contaminate the pre-training data (we use Web-Edu "as-is"). In addition, note that our experimental design is valid even if the pre-training data is already contaminated (if there is already contamination, our experiments measure the causal effect of inserting the benchmark questions k-additional times). Our implementation of fuzzy string matching used to de-duplic deppucl the benchmark questions is not efficient enough to match all benchmark questions against the pre-training data. Hence, we follow Singh et al. (2024) and report the length of the longest common substring between a benchmark question and the pre-training data. For simplicity, we report the overlap between all benchmark questions and the entire 100B token split of Web-Eu.. In our experiments, the training data a random subset of these 100B tokens.
Figure 7 depicts the overlap between the benchmark questions and the pre-training data. For every benchmark question, we compute the length of the longest common substring divided by the length of the benchmark question (that is, value of I means that the benchmark question occurs verbatim in the pre-training data). Figure 7 depicts the result of this computation in aggregated form. From Figure 7, we see that there is little overlap between the pre-training data and Hellaswag, PiQA, Social-i-QA, and Winogrande. For ARC-Easy, MMLU, and especially BoolQ, we find that there exist benchmark questions that have very high overlap with the pre-training data. Depending on the format of the questions and the nature of the overlap, some of these cases essentially correspond verbatim contamination. In other cases, there might be a high overlap between the benchmark question and the pre-training data, but it is not necessarily clear to what degree the text in the pre-training data actually reveals the true answer to the benchmark question. On BoolQ, for example, many of the overlaps with the pre- pre-training data occur for the descriptive contexts of the questions, which are the same for all answer options. Or consider the following example from ARC-Easy, where the pre-training data contains all four answer options.
Are-Easy: Question: In Colonial America, people used ice to help keep foods fresh. They cut the ice from lakes and ponds during the winter and stored the ice ice houses. They sometimes used hay as an insulator to prevent
15

---

How Much Can We Forget about Data Contamination?
2002
1790
700
590
705
175
192
Prequency
1590
502
Frequency
1252-
1002
300
75
103
0. 62
80 62 03 24
64 b5 B3 as 10
8.
Normalized lonoth
Normalized length
Normalized length of
Normalized length
(a) Hellaswag
PiQA
Social-i-QA
(d) BoolQ
300
1000
Frequeccy
Frequency
Frequency
390-
800
8oc
200
00 02 03 05 d6
0. ca 0. 6.4 85 d& d. on na Normalized length LCS
6a ca ca 05 d. as be aJ on da 1o
Normaized length of LCS
Normalized length LCS
(c) MMLU
(f) WinoGrande
(g) ARC-Easy
Figure 7. Overlap between the benchmark questions and the 100B token split of Web-EEu (Single Best Match), For every benchmark question, search in the pre-training data for the longest common substring. The figure depicts the distribution of the length of the longest common substring, normalized by the length of the benchmark question, for the different benchmarks.
the ice from melting. you wanted to build an icehouse today, which of the following would be the best material to use as an insulator? Answer: foam blocks
and
Pre-training data: Colonial America, people used ice to help keep foods fresh. They cut the ice from lakes and ponds during the winter and stored the ice in ice houses. They sometimes used hay as an insulator to prevent the ice from melting. you wanted build icehouse today, which of the following would be the best material
use insulator?
(A) dried leaves
(B) foam blocks
(C) plastic wrap
(D) rock salt
In order to build a wooden table, [...]
For the correct interpretation of Figure 7, note that is depicts the length of the single best match between the benchmark questions and the entire pre-training corpus of 100B tokens. For comparison, Figure depicts the distribution of the length of the 10 best matches in the pre-training data. From Figure (a), we see that searching for the 10 best matches does not affect the overall frequency distribution on Social-i-QA. for which there was no significant overlap the first place. In contrast, for BoolQ, MMLU, and ARC-Easy, for which there was significant overlap, we observe that the frequency distribution of the 10 best matches shifts significantly the left, indicating that regions of high overlap occur less than 10 times in the overall pre-training data.
In summary, we find that there are benchmark questions that have high overlap with the pre-training data, and that these questions are mostly from BoolQ, MMLU and ARC-Easy. At the same time, most of our 44000 benchmark questions do not have any significant overlap with the pre-trame pee-trannigg data as measured by the longest common substring metric. For those questions that have a high overlap, such an overlap usually occurs less than 10 times the 100BT split of FibeWeb-FF
16

---

How Much Can We Forget about Data Contamination?
3500
2550
6002
3000
2020
PeQQuency
Frequency
Forquency
Frequency
1002
2300
1920
2000
1000
1002
100
02
0.0
de di on Normalized lenoth
as 07 Ga
Normalized longth
Normalized length
Normaizze length
(a) Social-i-QA
(h) BoolQ
(c) MMLU
(d) ARC-Easy
Figure 8 Overlap between the benchmark questions and the 100B token split of Wib-Web-Edu (10 Best Matches). For every benchmark question, we search in the pre-training data for the 10 longest common substrings. The figure depicts the distribution of the length of the longest common substrings. normalized by the length of the benchmark question, for the different benchmarks.
More research is needed to investigate different types of overlap between benchmark questions and the pre- pre-training data, and whether they lead to benchmark overfitting (sce also Jiang et al. (2024), who investigate n n-gram-base based overlap).
A.7. Reproducibility
The code for this paper is available at gettubb.cm/mnee www..////wwtcteeeeeeeeeeeeeeeeee tuebingeen/foreettnng contamination. It relies the OLMo codebase, available at github.come catiuuu.oomaalenai/O....
and the llm.c codebase, available at htthub..com/krrattyyll... Our code is fully reproducible, including the random positions in which the benchmark questions were inserted into the training data. We trained on the 10OBT split of the FineWeb-Edu dataset, available at Model training relied on Pytorch (Paszke et al., 2019) and was performed on 8xA100 nodes for all experiments except the continual pre-training of OLMo-7B, which ran for 6 weeks on 4xH100.
A.8. Proof of Proposition
Fix > Let T t2 - t be the number gradient steps that have passed since step f1. Let Yavg = 4 Eli+1 % be the average learning rate between gradient steps I and 22. If sufficiently large, that
T 2 log(1/c)
(5)
Xaag
then
wh S c.
(6)
Proof.
Without loss of generality, let f = and T = 1z. We have from equation (4):
w I (1 nd).
We want to guarantee that
II (m))
f=1
First, apply the logarithm
log(1 7i)) loge.
(8)
Now, for 0 < < it holds that log(1 - z) < z. This means that the inequality (7) is satisfied if
2 log c.
(9)
im
17

---

How Much Can We Forget about Data Contamination?
Denoting Tave 7 Ei=1 Nis this is the same
log(1/6
T
Yavg
85.0%
Blfooe Conternination
68.0%
75.0%
07.595
Accuracy
80.0%
Accuracy
Accuracy
Accuracy
92.5%
15.0%
82.00
70.0%
60.0%
85.05-
80.0%
77.5%
$8.0%
60.0%
60.0%
309000
571000
373000
363000
571000
369000
373000
369000
371000
373000
Gradient Step
Gradient Step
Gradient Step
Gradient Step
(a) HellaS wag
(b) WinoGrande
(c) ARC-Easy
(d) PiQA
Figure 9. Contamination and forgetting in OLMo-1B. We contaminate the OLMo- IB checkpoint at gradient step 369,000 four times with different benchmarks. This causes an average accuracy increase 15 percentage points. remaining training time, leading to a reduction of 96% of the accuracy increase due to contamination. In this figure. different colours correspond different benchmarks, and the grey line depicts the clean accuracy without contamination. Mean bootstrapped 90% confidence intervals.
continue pre-training for the
92.5%
95.0%%
Contamination
90.0%
90.0%
Accuracy
Accuracy
75.00
Accuracy
Accuracy
87.5%
85.0%
05.0%
30.0%
70.0%
75.0%
92.5%
80.0%;
70.0%
200000
310000
220000
300000
210000
300000
300000
SL0000
320000
200000
200000
320000
Gradient Step
Gradient Step
Gradient Step
Gradient Step
(a) HellaSwap wag
(b) WinoGrande
(c) ARC-Fasy
(d) PiQA
Figure 10. Contamination and forgetting in OLMo-7B. We contaminate the OLMo-7B checkpoint at gradient step 300,000 four times with different benchmarks. This causes an average accuracy inerease of percentage points. We then continue pre-training for 13% the remaining training time, leading to a reduction of 87% of the accuracy increase due to contamination. this figure, diffcrent colours correspond different benchmarks, and grey line depicts the clean accuracy without contamination. Mean bootstrapped 90% confidence intervals.
18

---

How Much Can We Forget about Data Contamination?
-2.0-
15.0%
6.0%
CE Loss Differnnce
Accuracy Difference
Accuracy Difference
4.0%
-1.5
10.0%
2.0%
-1.0
-0.5
0.0%
www. 00sw
4.0%
0
11
13
15
-5.0%
4x
12x
32x
144x
4x
12x
32x
144X
Chinchilla Tokens
Number Repetitions
Number Repetitions
(a) Forgetting on Novel Data
(b) After Chinchilla
(c) After Chinchilla
Holdlout
times repeated
12 times repeated
32 times repeated
144 times repeated Contamination
Figure 1f. Forgetting without weight decay. We perform the forgetting experiment from Section 4.2 without weight decay (instead of the default weight decay of 0.1). This figure depicts the same quantities Figure 2 in the main paper. We see that there is significant forgetting without weight decay. At the same time, forgetting without weight decay occurs somewhat slower, especially for 144 times repeated contamination. In particular, the accuracy differences depicted (b) slightly larger than the corresponding accuracy differences in Figure 2, and the difference (c) is statistically significant (compare Figure 2).
19


Reviews:
[{"review_id": "IAYyhion2h", "reviewer": "Reviewer_KE3e", "summary": "This paper provides a very important perspective in data contamination of LLM and show that not all data leakage will lead to false evaluation in benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for your positive assessment.\n\nIt seems that you ask under what conditions we can be confident that a benchmark evaluation is not contaminated. This is an interesting question that lies somewhat beyond the scope of our paper. In our paper, we demonstrate that moderate amounts of exact contamination do not necessarily lead to benchmark overfitting. This implies that minor mistakes in data pre-processing and filtering might not lead to benchmark overfitting. To provide stronger guarantees that a benchmark evaluation is \"totally safe\", we would require a much deeper understanding of the learning dynamics of LLMs.\n\nWe would be happy to answer any additional questions.", "claims_and_evidence": "Strengths:\n1. This paper question the severity mentioned in the previous paper. The assumption or the settings of data contamination might not be practical or not common in LLM.\n2. Besides the problem, this paper gives a comprehensive evaluation on the property of forgetting.\n3. The paper is well-written and the problem is interesting.\n\nQuestion:\n1. Although the paper point out this problem, people still have the question when would the benchmark be totally safe. And is there any method to improve or gaurantee the fairness of the benchmark?", "methods_and_evaluation_criteria": "Yes. Their evaluation is more practical than the previous paper.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are comprehensive.", "relation_to_broader_scientific_literature": "It questions the reaonability in previous settings of data containation paper.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "doclgC3rxB", "reviewer": "Reviewer_Mm41", "summary": "This paper investigates the impact of data contamination in large language models (LLMs), challenging the assumption that minor contamination invalidates benchmark evaluations. Through controlled experiments, the authors study how contamination effects scale with model size (up to 1.6B parameters), training tokens (up to 40B), and example repetitions (up to 144x).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review of our paper. We are happy to hear that you appreciate our experimental design. Below, we give answers to your questions/comments.\n\n*“The proof assumes constant learning rates”*\n\nTo clarify, the proof does not assume constant learning rates (in the proof, the learning rate is denoted $\\gamma_i$ and depends on the gradient step). We conjecture that the reviewer observed that $\\lambda$ in the proof is constant - this is the weight decay (which is usually constant, but the algebra in the proof does not require this). \n\n*“I recommend that the authors open-source the code and experimental data logging of this work in order to verify reproducibility and increase the transparency of the work.”*\n\nThe code is available at https://github.com/icml9771/code (the link is currently in the supplement; we will move it to the first page). We additionally commit to open-source our Weights & Biases logs and model checkpoints. \n\n*“Could paraphrased or semantically similar contamination (vs. exact matches) alter the conclusions, especially for larger models?”*\n\nThis is an interesting question. The reviewer is correct in observing that we decided to consider only exact contamination because non-exact contamination might behave qualitatively differently for larger models (see also Supplement A.1. Additional Discussion of Data Contamination Assumptions and Setting). For example, larger models might observe a kind of ``emergence’’ phenomenon where they can suddenly make efficient use of rephrased samples.\n\nThat being said, we agree with the provided reference Xu et al. (2024), which states that exact contamination is more severe than other forms of contamination (e.g. the last paragraph on page 4 in Xu et al. (2024)). This means that the results in our paper should provide a heuristic upper bound for what would happen with paraphrased or semantically similar contamination. \n\nOf course, ultimately experimental evidence would be required for other forms of contamination, too. \n\nWe would be happy to answer any additional questions.", "claims_and_evidence": "The claims are supported by systematic experiments and theoretical analysis. Evidence includes:\n\n1. Controlled scaling experiments showing monotonic trends in contamination effects.\n\n2. Weight decay analysis linking optimization hyperparameters to forgetting.\n\n3. Validation via OLMo-7B continual training However, extrapolation to larger models (e.g., Llama 3 405B) relies on theoretical weight decay bounds rather than direct empirical validation.", "methods_and_evaluation_criteria": "Benchmarks (ARC-Easy, HellaSwag, etc.) are filtered for duplicates to isolate contamination effects. Contamination is inserted randomly, mimicking real-world leakage. Evaluation via accuracy gaps between contaminated and holdout data is appropriate. A limitation is the focus on smaller models (≤7B), which may not fully capture dynamics in larger LLMs.", "theoretical_claims": "Proposition 1 (cumulative weight decay bounds forgetting) is proven in the appendix. The proof assumes constant learning rates, which may not hold in practice, but the core intuition—weight decay reduces past gradient influence—is valid.", "experimental_designs_or_analyses": "Experiments are methodical but limited to smaller models due to computational constraints. The extrapolation to larger models is plausible but unverified empirically. The OLMo-7B experiments add credibility, but testing on >10B parameter models would strengthen conclusions.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "I recommend that the authors open-source the code and experimental data logging of this work in order to verify reproducibility and increase the transparency of the work.", "questions_for_authors": "Could paraphrased or semantically similar contamination (vs. exact matches) alter the conclusions, especially for larger models?\n\nFor example the \"Semantic Level\" and \"Information  Level\" contamination in the Xu et al. 2024 (Benchmark data contamination of large language models: A survey)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F2oHqymVzJ", "reviewer": "Reviewer_T1JX", "summary": "The paper investigates the impact of data contamination in LLMs, specifically addressing whether small-scale contamination significantly affects benchmark evaluations. The authors analyze contamination effects along three dimensions: model size, number of training tokens, and repetition of contaminated examples. They find that while contamination can lead to overfitting under certain conditions, large-scale training beyond Chinchilla-optimal regimes can mitigate or even eliminate its impact. Empirical experiments demonstrate that continual pre-training can effectively erase contamination effects, with weight decay playing a key role in forgetting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review of our paper and insightful questions. Below, we give detailed answers to your questions/comments.\n\n*“The main limitation is the relatively small scale of experiments, though this is understandable given computational constraints.”*\n\nRunning experiments with a 7B parameter model is the largest we can afford on an academic compute budget. We hope that, inspired by our work, future work will study forgetting and contamination as part of a large-scale training run. \n\n*“Your experiments provide insights into LLM forgetting, but it is unclear what actionable takeaways arise from these findings. How do you envision these results informing model training or evaluation practices?”*\n\nA key insight of our work is that the causal effect of a small number of data points in the LLM pre-training data on final model behavior can be zero. If model training is in this large-data regime, this has several important practical implications:\n\n- Our work questions the common practice of grouping benchmark questions into “clean” and “contaminated” questions based on the maximum found overlap between a question and the training data (for example, n-gram overlap) to reason about benchmark overfitting (Brown et al. 2020, Touvron et al., 2023, Dubey et al., 2024). Specifically, our work shows that the mere existence of an evaluation data point in the LLM pre-training data is not necessarily an indicator of benchmark overfitting. We show that it is necessary to consider other factors, such as the frequency of the contamination.\n- Our work highlights the importance of when a contaminated sample is seen. If the samples are seen early during pre-training, overfitting is mitigated due to the forgetting phenomenon. This is directly **actionable**: If we suspect benchmark overfitting during an early pre-training stage on relatively unfiltered internet data, the practitioner can consider continual pre-training (“mid-training”) on clean data to mitigate the overfitting.\n- Our work suggests that filtering benchmark questions from the mid-training and post-training data is much more critical than filtering the pre-training data (additional factors like the learning rate schedule come into play). This has immediate and direct implications for the design of training datasets.\n\nWe would be happy to further elaborate on these points.\n\n\n*“Section 5 suggests weight decay contributes to forgetting, but the analysis is mainly based on gradient decay without considering how earlier gradients influence the optimization trajectory. [...] what is the key insight of this section? Are you proposing weight decay as a primary mechanism, or just one possible factor?”*\n\nLet us outline the motivation for Section 5 in the paper and our interpretation of the experimental results. \n\nGiven that forgetting is a key empirical property of LLM pre-training, it is natural to ask: Is there a simple factor in the pre-training pipeline that explains the forgetting? Asking this question, it seemed straighforward to consider the weight decay parameter: After all, weight decay is a mechanistic process that gradually removes the influence of earlier gradient updates on the model weights (this is what we formally describe in Proposition 1). \n\nIn our experiments, it turned out that the weight decay parameter does indeed influence forgetting in the sense that increasing the weight decay parameter leads to faster forgetting (Figure 4). The experiments also revealed that the empirical rate of forgetting occurs faster than the cumulative weight decay. This makes a lot of sense to us, given how weight decay works mechanistically. At the same time, the experiments also demonstrated that weight decay is not necessary for forgetting (Supplement Figure 8) – though forgetting without weight decay occurs at a slower rate. We don’t see this as a problem for our analysis—it simply means that forgetting is a multifaceted phenomenon not exclusively driven by weight decay.\n\nSection 5.3 suggests that the mechanism of forgetting via weight decay is likely relevant in large-scale LLM pre-training.\n\nTo summarize, the key insight in Section 5 is that the weight decay parameter has a causal effect on forgetting that is likely relevant in large-scale LLM pre-training. \n\n*“What role does data attribution play in your findings, and how does it relate to the core contributions of the paper?”*\n\nOur paper's controlled experiments resemble the “leave-one-out” (LOO) procedure in data attribution. Concretely, our experiments directly “leave-out” entire groups of benchmark questions, thus simulating a kind of “group” LOO procedure.\n\nMoreover, our result that the (average) causal effect of inserting individual benchmark questions into the pre-training data can be zero suggests that we might not be able to attribute “importance” to individual data points in the LLM pre-training regime.\n\nWe would be happy to answer any additional questions.", "claims_and_evidence": "The experimental results support the paper’s main conclusions regarding the impact of data contamination and forgetting dynamics in large-scale training. However, the section on weight decay raises some questions, which will be discussed below.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are reasonable for the problem. The main limitation is the relatively small scale of experiments, though this is understandable given computational constraints.", "theoretical_claims": "I checked Proposition 1, and it appears to be correct. No issues were found.", "experimental_designs_or_analyses": "The experimental conclusions are sound and align well with expectations. No issues were found.", "relation_to_broader_scientific_literature": "The paper provides findings on LLM forgetting through controlled experiments, contributing to the understanding of data contamination and memory dynamics. However, it is unclear how these conclusions translate into actionable insights, which I would like to ask the authors about.", "essential_references_not_discussed": "The paper does not strongly rely on prior literature, so additional references are not essential.", "other_strengths_and_weaknesses": "**Strengths:**\n1. The paper includes a large number of experiments with well-designed and solid methodologies, leading to conclusions that align with expectations.\n2. This paper is well-written.\n\n**Weaknesses:**\n1. The practical actionable insights of these empirical findings are unclear.\n2. The experiments are limited in scale (model size), though this is understandable given computational constraints.\n3. The analysis of weight decay is relatively shallow. While the perspective is interesting, the analysis mainly focuses on gradient decay, ignoring how earlier gradients influence the optimization trajectory. Simply showing that gradients decay to zero may not be sufficient. Additionally, the paper states that weight decay is not necessary for forgetting, making the core insights of Section 5 unclear. Does this section propose a possible explanation? If so, it is presented in a shallow way and is not clearly necessary, raising questions about its contribution.\n\nAnother point of confusion is the mention of data attribution. While the paper references it, neither the experimental results nor the theoretical analysis provide any direct connection to data attribution methods. This is not necessarily a weakness, but it raises questions about why data attribution is discussed if it does not play a role in the findings.", "other_comments_or_suggestions": "First line of Section 4: \"We being\" -> \"We begin\"", "questions_for_authors": "1. Your experiments provide insights into LLM forgetting, but it is unclear what **actionable** takeaways arise from these findings. How do you envision these results informing model training or evaluation practices?\n2. Section 5 suggests weight decay contributes to forgetting, but the analysis is mainly based on gradient decay without considering how earlier gradients influence the optimization trajectory. Given that the paper also states weight decay is not necessary for forgetting, what is the key insight of this section? Are you proposing weight decay as a primary mechanism, or just one possible factor?\n3. The paper references data attribution, but the experiments and theoretical analysis do not directly connect to it. What role does data attribution play in your findings, and how does it relate to the core contributions of the paper?\n\nOverall, I really appreciate the experimental design of this paper. If Questions 1 and 2 are addressed convincingly, I would be happy to increase my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PGmeRivHqR", "reviewer": "Reviewer_F1rX", "summary": "- the paper studies the effect of data contamination during the pre-training of language models, through a series of *controlled* contamination experiments\n- the paper studies the effect by considering: (1) scaling the amount of contamination (i.e. repetitions), (2) scaling the model size, (3) scaling the (unrelated) pre-training data size, (4) scaling data and model jointly (Chinchilla), and (5) if and how weight decay plays a role\n- the paper reports multiple findings, for example:\n  - small-scale contamination may or may not be \"forgotten\" (i.e. the contamination stops contributing to benchmark gains), depending on how much (unrelated) pre-training data there are (as relative to Chinchilla optimal)\n  - more repetitions of contamination uniformly increases benchmark performance, but the increase depends on both model scale and data scale\n  - exposing to *novel* pre-training data is more effective at forgetting contamination than to old pre-training data \n  - repetition is an important factor, perhaps arguably more so than seeing them later in training\n\n## update after rebuttal \n\nI appreciate the authors' rebuttal and will keep my score and my assessment that the paper should be accepted.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review of our paper. We are happy to hear that our paper is “a pleasure to read”! Below, we respond to your questions/comments.\n\n*“deduplicating the contamination benchmark data (e.g. HellaSwag) from the pre-training tokens (e.g. FineWeb) is also recommended for a truly unbiased evaluation.”*\n\nWhile it will not be possible to re-run the experiments with de-duplicated pre-training data, we commit to adding a Supplement Section that reports the overlap between the pre-training data and the benchmark questions that we contaminate with.\n\nPreliminary experiments with a random subset of benchmark questions and a random subset of the pre-training data indicate that the overlap between the benchmark questions and FineWeb-edu is small (as measured by the fuzzy string metric used to de-duplicate the benchmark questions). \n\n\n\n*“Key references are cited and discussed in the paper, although prior work's contributions can be highlighted a bit more.”*\n\nThank you for the additional reference [3]; we will incorporate it. We think that the reviewer's positioning of our contribution in the literature is quite fitting! We will revise the relevant parts of the paper to highlight the contributions of prior work better.\n\n\n\n*“Fig 1, 2: when you scale more pre-training data (\"Chinchilla Tokens\"), are the extra tokens fresh tokens,”*\n\nYes, they are fresh tokens. We will replace or qualify the usage of the term “epoch,” which can indeed be confusing in our setting. \n\n\nThank you again for providing such a high-quality review. We would be happy to answer any additional questions.", "claims_and_evidence": "Yes, the claims in the paper are supported by extensive experiments to my understanding", "methods_and_evaluation_criteria": "Yes, the proposed research question, methods, and evaluation data make sense overall. \n\n- One minor weakness is that for some of the experiments, the authors used the 124M model, which is perhaps too small for meaningful performance on some of the benchmark datasets (e.g. MMLU). \n- Another minor weakness is that FineWeb-Edu contains only text and largely no math or code; this means the models may be biased towards being a text model as opposed to a general purpose model like GPT-4. Though I understand that this could be an artifact of using LLM.c which primarily uses FineWeb-Edu.", "theoretical_claims": "The paper is mostly empirical. The analysis component for weight decay (sec 5.1) seems to make sense.", "experimental_designs_or_analyses": "Overall, the experiments of the paper are well-designed to comprehensively answer the proposed research question. The findings are well supported by the experiment results.\n\nComments:\n- (minor) deduplicating the contamination benchmark data (e.g. HellaSwag) from the pre-training tokens (e.g. FineWeb) is also recommended for a truly unbiased evaluation. Since the authors report accuracy gaps between holdout and contaminated, this is perhaps OK.\n- (minor, mentioned earlier) depending on the model size (small vs big), pre-training data (FineWeb vs other pre-training mix), or even architecture (GPT-3 vs newer ones), the pre-trained model may not perform meaningfully on some of the benchmark data (e.g. the 124M model trained on 1x Chinchilla can be basically random guessing at MMLU). This could affect the takeaways, but on a macroscopic level the results are consistent and make sense.", "relation_to_broader_scientific_literature": "- this paper can be viewed as a more systematic, comprehensive, and rigorous execution of Jiang et al. (2024) [2] to understand the effect of data contamination in the pre-training stage\n- related work like [1] discussed the effect of gradual forgetting of past training data over the course of training, which is related to this paper's finding that benchmark data can be forgotten (e.g. in abstract \"Lllama 3 405B, have forgotten the data seen at the beginning of training.\").\n- this paper can be viewed as an intersection of several parts of the relevant literature: contamination analysis (many references in paper), memorization analysis (e.g. [1]), optimization analysis, data selection (e.g. use stale vs fresh pre-training data [3]) \n\n\n[1] https://arxiv.org/abs/2207.00099 \n[2] https://arxiv.org/abs/2401.06059\n[3] https://arxiv.org/abs/2305.16264", "essential_references_not_discussed": "N/A to my understanding. Key references are cited and discussed in the paper, although prior work's contributions can be highlighted a bit more.", "other_strengths_and_weaknesses": "Strength\n- the paper is well-written and a pleasure to read! \n\nWeaknesses: see prior sections", "other_comments_or_suggestions": "- Section 4: \"We being in\" --> \"We begin in\"", "questions_for_authors": "- Fig 1, 2: when you scale more pre-training data (\"Chinchilla Tokens\"), are the extra tokens *fresh* tokens, or repetitions of the same set of 1x Chinchilla tokens (e.g. same 20B tokens for an 1B model)? My understanding is fresh tokens, but since Fig 2 (b, c) says \"Epochs\", I'm not too sure", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the impact of data contamination in large language models (LLMs), challenging the assumption that minor contamination invalidates benchmark evaluations. Through controlled experiments, the authors study how contamination effects scale with model size (up to 1.6B parameters), training tokens (up to 40B), and example repetitions (up to 144x).

The paper is well-written and includes a large number of experiments with well-designed and solid methodologies. The weight decay analysis linking optimization hyperparameters to forgetting is interesting. Validation is conducted via OLMo-7B continual training. The experiments are limited in scale (model size), though this is understandable given computational constraints. 

The paper reports multiple findings:
(i) small-scale contamination may or may not be "forgotten" (i.e. the contamination stops contributing to benchmark gains), depending on how much (unrelated) pre-training data there are (as relative to Chinchilla optimal)
(ii) more repetitions of contamination uniformly increases benchmark performance, but the increase depends on both model scale and data scale
(iii) exposing to novel pre-training data is more effective at forgetting contamination than to old pre-training data
(iv) repetition is an important factor, perhaps arguably more so than seeing them later in training

Recent, very related work:
[1] Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation
https://arxiv.org/pdf/2501.18771?
Decision: Accept (poster)