How much can we Forget about Data Contamination?

## Abstract

Abstract The leakage of benchmark data into the training data has emerged as a significant challenge for evaluating the capabilities of large language models (LLMs). In this work, we use experimental evidence and theoretical estimates to challenge the common assumption that small-scale contamination renders benchmark evaluations invalid. First, we experimentally quantify the magnitude of benchmark overfitting based on scaling along three dimensions: The number of model parameters (up to 1.6B), the number of times an example is seen (up to 144), and the number of training tokens (up to 40B). We find that if model and data follow the Chinchilla scaling laws, minor contamination indeed leads to overfitting. At the same time, even 144 times of contamination can be forgotten if the training data is scaled beyond five times Chinchilla, a regime characteristic of many modern LLMs. We then derive a simple theory of example forgetting via cumulative weight decay. It allows us to bound the number of gradient steps required to forget past data for any training run where we know the hyperparameters of AdamW. This indicates that many LLMs, including Llama 3, have forgotten the data seen at the beginning of training. Experimentally, we demonstrate that forgetting occurs faster than what is predicted by our bounds. Taken together, our results suggest that moderate amounts of contamination can be forgotten at the end of realistically scaled training runs.

## 1 Introduction

A core principle of machine learning is that a model should not be trained on the test set used for evaluation []. For foundation models trained on Internet-scale data, there are increasing concerns that this principle is violated due to the leakage of benchmark evaluation data into the training data [, ].
Indeed, many LLM developers have found overlap between their training data and the benchmark questions used for evaluation [, ]. What is more, research on memorization [, ] shows that text sequences from the training data are sometimes encoded within the model, including machine learning datasets [, , , ].

While the fact that data contamination can lead to invalid performance evaluations is now well-established [, , , ], little is known about the precise conditions under which this is the case. The main reason for this is that the training data is usually unknown, and contamination identified via clever research designs that work around this restriction [, , ]. Because modern foundation models are sometimes trained for over a million gradient steps [], it is unclear whether a single update on contaminated data at some point during training necessarily impacts downstream evaluations. And indeed, there is quite some evidence that language models need to see samples repeatedly to have any impact on the final model. For example, many papers on memorization have found that it occurs only when a sample is frequently repeated in the training data [, , ]. The same is true for research on knowledge acquisition, where a fact needs to be paraphrased many times before it is finally remembered by the model [, , ].

In this work, we study the impact of data contamination in a controlled setting. This means we train language models from scratch on datasets where we explicitly insert contaminated examples []. We begin by quantifying how the overall magnitude of benchmark overfitting (or the cross-entropy loss of an observed sample) changes as we scale along three critical dimensions: (1) the number of model parameters, (2) the number of training tokens, and (3) the number of repetitions of an example in the training data (Section [4.1](#S4.SS1)). Holding the other two dimensions fixed, we find that the effect of scaling is monotone in each dimension. First, similar to many other works, we find that the tendency of a model to overfit increases in the number of parameters [, , ]. Second, and this is also expected, we find a clear scaling in the number of repetitions, where more frequently repeated observations exhibit stronger overfitting [, ]. More surprisingly, we find that the effect of contamination can vanish as we increase the number of training tokens, up to the point where 12 repetitions of an entire dataset in the training data have no impact on the downstream evaluation on that same dataset.

Our investigation reveals that the natural forgetting dynamics of gradient descent [, ] is the reason why increasing the number of tokens alleviates the impact of contamination. Concretely, we show that training on five times Chinchilla [] of clean data can cause a model to forget even 144 times repeated training examples (Section [4.2](#S4.SS2)).
Forgetting the bulk of the impact of a training example can occur rapidly, a point that we demonstrate using OLMo-1B [] (Section [4.3](#S4.SS3)). What is the reason for this rapid forgetting? We show that exposure to novel data is important. Interestingly, models tend to exhibit the strongest overfitting on examples seen repeatedly throughout training, even compared to those seen during the end (Section [4.2](#S4.SS2)).

Because running pre-training experiments is expensive, we also ask to what degree forgetting can be explained by the training dynamics of gradient descent. We show that the weight decay parameter and learning rate schedule of the AdamW optimizer [] play a key part in forgetting past training examples (Section [5](#S5)). Concretely, we derive a simple theory of forgetting via cumulative weight decay (Section [5.1](#S5.SS1)) and show that it provides an upper bound on empirical forgetting, which usually occurs faster (Section [5.2](#S5.SS2)). The key point is that this approach allows us to gauge the degree of forgetting present in any training run for which the optimization hyperparameters are known. It even allows us to approximate how the final model parameters balance the gradient updates from different stages of training. Overall, our analysis indicates that many LLMs, including OLMo-7B and Llama 3 405B [], have forgotten the data seen at the beginning of their training run.

Taken together, our main contribution is to show that the impact of individual examples in the training data depends on the precise characteristics of the setting. There are settings where the effect can be significant; Chinchilla training is an important example (Section [4.1](#S4.SS1)). However, there are equally realistic settings where individual examples dont’t matter - including quite likely the data-intensive training runs of many recent LLMs [].

## 2 Related Work

Data Contamination. The GPT-3 paper [] uses an n-gram-based approach to differentiate between “clean” and “dirty” benchmark questions. This approach has since been used in many LLM reports [, ], including Llama 3 [], where it is estimated that there might be a performance gain of up to 8 and 14 percentage points on PiQA and HellaSwag, respectively. The GPT-4 technical report [] remarkably concluded that “contamination overall has very little effect on the reported results”. This has since given rise to a literature that aims to detect [], mitigate [], and estimate the effect of [, ] data contamination under various assumptions, but crucially without access to the training data. This literature often challenges the conclusion that contamination overall has little effect in GPT-4 [].

Forgetting. In machine learning, the term forgetting is frequently associated with “catastrophic” forgetting, where learning new tasks hurt the performance at previously solved tasks []. In the context of LLMs, catastrophic forgetting can occur during fine-tuning [] or continual learning []. In contrast, this paper studies forgetting as a potential “natural” phenomenon of learning []. Tirumala et al. [] study forgetting in language modeling and find, similar to Toneva et al. [], that forgetting can be exponentially slow. In contrast, Jagielski et al. [] find that models empirically do forget examples over time. In concurrent work, Pagliardini et al. [] propose to add a second momentum term to the AdamW optimizer, and show that this slows down the forgetting of past gradients.

Data Attribution. Data attribution methods [, , ] aim to identify data points responsible for specific model behaviors. We ask how much a model’s benchmark performance is influenced by seeing the example during training, which broadly falls within this field [, ]. Importantly, we directly measure the influence of contaminated examples through retraining, avoiding the approximation errors that can occur when using data attribution methods [, ] for large-scale models [, ]

## 3 Background and Methods

This Section gives additional details on the research questions and lays out our experimental setup.

While it is well known that an individual data point can be influential if the training data and model are small [], we are concerned with the large-data regime where the influence of any individual example may vanish [, ]. To further clarify this setup, Section [3.1](#S3.SS1) gives a brief overview of recent developments in scaling the training data of LLMs. Section [3.2](#S3.SS2) details the used benchmarks and explains how we contaminate the training data. Section [3.3](#S3.SS3) discusses the problem of near-duplicate benchmark questions.

Models and Training Data. We train language models of up to 1.6B parameters using the architecture and hyperparameters from the GPT-3 paper []. For this, we adopt the [llm.c](https://github.com/karpathy/llm.c) codebase. The training data is the 100BT split of the FineWeb-Edu dataset []. We also train OLMo-1B [] using the corresponding [code](https://github.com/allenai/OLMo) and data [].

### 3.1 We consider the regime of n-times Chinchilla

According to the Chinchilla scaling law, for every doubling of model size, the number of training tokens should also be doubled. The Chinchilla model itself has 70 billion (B) parameters and was trained on 1.4 trillion (T) tokens; suggesting that the number of training tokens should be roughly 20x the number of model parameters []. While the Chinchilla paper was highly influential, modern language models are trained on significantly more tokens []. For example, the OLMo-7B model was trained on 2.46T tokens, 17.5x the amount suggested by Chinchilla []. Similarly, the Llama 3 70B model was reportedly trained on 15T tokens, at over 10x Chinchilla [, ]. The same holds for almost all recent LLMs at the 7B parameter scale []. In this paper, we count the number of tokens a model is trained on as a multiple of its Chinchilla tokens.

### 3.2 We evaluate on a mix of seven different benchmarks

We evaluate the impact of data contamination using a mix of seven different benchmarks: ARC-Easy [], Social-I-QA [], WinoGrande [], PiQA [], BoolQ [], MMLU [], and HellaSwag []. This means that every evaluation contains questions from all seven benchmarks. To construct the mixed contamination data, we first concatenate the different benchmarks. We then partition the set of all benchmark questions into subsets ranging from 10,000 to 2,000 questions so that each subset contains all benchmarks in equal weight: HellaSwag: 19.58%, SocialIQA: 8.27%, PiQA: 19.7%, MMLU: 21.82%, BoolQ: 6.48%, ARC-Easy: 5.92%, and WinoGrande: 18.16%. A holdout set of 10,000 benchmark questions is never added to the training data. The other subsets are added to the training data, repeated either 4, 12, 36, or 144 times.(^1^11In preliminary experiments, we found that these numbers pleasantly cover the range from statistically significant contamination to complete overfitting. We also considered repeating observations a single time, as in []. However, we found this often leads to accuracy differences of about one or two percentage points, just within the margin of our confidence intervals, which is undesirable.) We consider exact contamination, that is we contaminate the training data with the same texts that the model is later evaluated on. We insert benchmark questions individually and at random positions into the training data. Models are evaluated zero-shot via the likelihood assigned to different sentence completions []. For more discussion and details about how contamination is performed, see Supplement [A.1](#A1.SS1) and Supplement [A.2](#A1.SS2).

### 3.3 We filter near-duplicate Benchmark Questions

Our method requires that there are no side effects from contaminating the training data with one question on the evaluation of another question. However, upon closer inspection, it turns out that all the commonly used benchmarks from the literature contain questions that are either near-duplicates or where the context of one question contains the answer to another question (for example, because the same text document was used to create multiple questions). This is illustrated in Figure [1](#S3.F1), which depicts two near-duplicate questions on HellaSwag and questions from ARC-Easy and MMLU that are cross-benchmark duplicates. To address this problem, we perform extensive filtering, removing duplicate questions where the length-normalized Levenshtein distance falls below a certain threshold [, ]. This is documented in Supplement [A.3](#A1.SS3), where we also describe an experiment to verify that our method is not invalidated by near-duplicate questions.

Figure: Figure 1: Language modeling benchmarks frequently contain near-duplicate questions. We perform extensive filtering for duplicates using fuzzy string matching. The figure depicts a near-duplicate from HellaSwag and a cross-benchmark duplicate from ARC-Easy/MMLU.
Refer to caption: /html/2410.03249/assets/x1.png

## 4 Experimental Results

We now present our main experimental results. We being in Section [4.1](#S4.SS1) by discussing the scaling in model parameters, training tokens, and repetitions in the training data. The following Section [4.2](#S4.SS2) discusses various experiments on forgetting. The first two sections rely on training small GPT-3 models. Section [4.3](#S4.SS3) complements this with an analysis of OLMo-1B [].

### 4.1 Contamination scales with Model, Data, and Repetitions

Figure: (a) Scaling the Model
Refer to caption: /html/2410.03249/assets/x2.png

We conduct three different experiments to understand how the effect of data contamination scales with the number of model parameters, training tokens, and the number of times a contaminated example is seen. First, we train increasingly large models on the same dataset of 7B tokens. Second, we train 124M parameter models on increasingly many tokens. Third, we train increasingly large models according to the Chinchilla scaling laws [], meaning that the number of training tokens scales linearly with the model parameters. In all experiments, we contaminate the training data uniformly at random with benchmark questions.

Figure [2](#S4.F2) depicts the results of all three experiments. Because we are interested in the performance difference between the holdout data and the contaminated examples, Figure [2](#S4.F2) depicts the accuracy gap between the holdout and contaminated examples in percentage points. In Figure [2(a)](#S4.F2.sf1), we see that the accuracy gap due to contamination is increasing in the number of model parameters. For a 124M parameter model trained on 7B tokens, the overfitting due to 4 times contamination is 5 percentage points. For a 1.6B parameter model train on the same dataset, it is 20. Next, Figure [2(b)](#S4.F2.sf2) shows that the accuracy gap is decreasing in the number of training tokens. For a 124M parameter model trained at 2x Chinchilla, the accuracy gap due to 12 times contamination is 18 percentage points. For a 124M parameter model trained at 15x Chinchilla, the same accuracy gap is within the confidence interval of the holdout. From Figure [2](#S4.F2), we also see that the accuracy gap is increasing in the number of times an example is repeated. For a 350M parameter model trained on 7B tokens, the accuracy gap is 11, 25, 44, and 51 percentage points for 4, 12, 32, and 144 times repeated contamination, respectively.

Figure: Table 1: Accuracy of the Chinchilla models.

Because the accuracy gap increases in the number of model parameters and decreases in the number of tokens, the interesting question is how it behaves if model parameters and tokens are scaled jointly. A natural starting point is to double the number of training tokens for every doubling of model parameters, as specified by the Chinchilla scaling laws []. Figure [2(c)](#S4.F2.sf3) depicts the accuracy gap due to contamination as we train increasingly large Chinchilla-optimal models. While there is no clear monotone pattern, we see that moderate amounts of contamination can lead to significant overfitting. For the 774M parameter model, 4 times repeated contamination leads to an accuracy gap of 15 percentage points, suggesting that under Chinchilla training, a single time of contamination can lead to overfitting of as much as 3 percentage points.(^2^22To contaminate the training data of a 774M model a single time with 10,000 benchmark questions, we need to insert $\sim$0.5 million tokens into 15.5 billion training tokens, about 0.003% of the training data.)

### 4.2 Contamination can be completely Forgotten

Figure: (a) Forgetting on Novel Data
Refer to caption: /html/2410.03249/assets/x6.png

In the previous Section [4.1](#S4.SS1), we saw that the accuracy gap due to contamination decreases in the number of tokens up to the point where even 12 repetitions of a benchmark question in the training data can become insignificant. In this Section, we identify the natural forgetting dynamic of neural network training as the reason for this effect. We discuss how quickly forgetting occurs, whether examples are completely forgotten, and what kind of repetition makes a model remember.

To study the effect of forgetting, we train a 124M parameter model for 15 epochs. Instead of contaminating uniformly over the course of training like in the previous Section [4.1](#S4.SS1), we perform the contamination between the first and second Chinchilla.(^3^33Note that the model is already fairly trained after the first Chinchilla, meaning that the contamination is not very early during training. This is important because there is evidence that observations are more quickly forgotten if the model has not yet learned representations [, , ]. This is not the setting we are studying here.) Figure [3(a)](#S4.F3.sf1) depicts the development of the difference in cross-entropy loss between contaminated and clean benchmark questions over the course of training. We see a strong peak after 2 Chinchilla, which is expected and shows the effect of contamination. What is interesting to us is the rate at which the cross-entropy loss difference decays as we continue training. After training for 1 additional Chinchilla (2.5B tokens for the 124M parameter model), it has already decayed significantly. However, the difference is still visible in Figure [3(a)](#S4.F3.sf1). Figure [3(b)](#S4.F3.sf2) depicts the corresponding accuracy gaps at this point, and we see that all contamination levels still lead to overfitting. As we continue training, the cross-entropy loss difference between contaminated and holdout questions further narrows. From Figure [3(c)](#S4.F3.sf3), which depicts the accuracy gaps after forgetting for a total of 5 Chinchilla, we see that the effect of contamination is eventually completely forgotten in the sense that there is no longer any accuracy difference between contamination and holdout benchmark questions.

The result that contamination can be completely forgotten is in contrast to some previous work on forgetting which have found that forgetting approaches a stable baseline Tirumala et al. [, Figure 10], or that certain examples are never forgotten []. To understand this difference, observe that many previous works on forgetting have not trained on a continuous stream of data. Instead, they have trained on the same training set for multiple epochs. Consequently, we modify our forgetting experiment to repeatedly train on the same 100M tokens after the second epoch. The result of this experiment is depicted in Figure [3(d)](#S4.F3.sf4) and should be compared to Figure [3(a)](#S4.F3.sf1). Interestingly, this simple modification causes the effect of forgetting to stabilize at a level strictly larger than zero. We conclude that exposure to novel data is important for forgetting, an observation similar to Jagielski et al. [].

To further understand the impact of forgetting, we now ask whether examples seen late during training influence model behavior more strongly than examples seen early during training. To study this question, we average all the different uniform contamination levels from the models in the previous Section [4.1](#S4.SS1) (to gain statistical power) and consider the amount of overfitting depending on whether a question is seen, on average, in the beginning, middle, or end of training. The result of this experiment is depicted in Figure [3(e)](#S4.F3.sf5) and Figure [3(f)](#S4.F3.sf6). As expected under forgetting, we see that benchmark questions seen early during training exhibit the smallest amount of overfitting. Interestingly and somewhat unexpectedly, questions that are neither clustered towards the beginning nor the end but as uniformly distributed throughout training as possible exhibit the strongest overfitting, suggesting that this spaced form of repetition helps the model remember (the middle peak is the most pronounced both in Figure [3(e)](#S4.F3.sf5) and Figure [3(f)](#S4.F3.sf6)).

### 4.3 Contamination and rapid forgetting in OLMo-1B

Figure: (a) HellaSwag
Refer to caption: /html/2410.03249/assets/x13.png

In the previous sections, we trained small GPT-3 models from scratch. In this Section, we complement this analysis by pre-training from an intermediate OLMo-1B checkpoint []. Similar to the analysis in Section [4.2](#S4.SS2), we insert the benchmark data at a specific point into the training data and then measure the subsequent forgetting. Unlike in the previous Section, we now insert the entire benchmark data – we already have a “clean” baseline from the original OLMo-1B training run. We insert each benchmark question four times and contaminate with four different benchmarks: HellaSwag [], WinoGrande , ARC-Easy [], and PiQA [].

Figure [4](#S4.F4) depicts the result of the experiment. The effect of contamination is visible from the five leftmost points of every plot. The leftmost point corresponds to the uncontaminated model, and the next four points each depict the effect of one time contamination. Again, we see that the immediate effect of contamination is significant, leading to an average accuracy increase of 15 percentage points across the different benchmarks. At the same time, we also see that the effect of contamination decays considerably as we continue training. To contextualize this result, note that Figure [4](#S4.F4) depicts less than 2000 gradient steps. The pre-training stage of OLMo-1B model consists of 739,328 gradient steps. This means that Figure [4](#S4.F4) depicts less than 1% of the total forgetting until pre-training is done.

## 5 What is the role of weight decay in forgetting?

Figure: (a) 124M 15x Chinchilla
Refer to caption: /html/2410.03249/assets/x17.png

In the previous Section [4](#S4), we have seen that forgetting is an important empirical property of LLM training. In this Section, we show that the weight decay parameter and learning rate schedule of the AdamW optimizer play a key part in forgetting past training examples. This offers a novel perspective on the interplay between these two parameters, usually seen in terms of generalization and training stability [, , , ].

### 5.1 Weight Decay as a Mechanism for Forgetting Training Examples

Consider the parameter update of AdamW at gradient step $t\geq 1$. It consists of two decoupled updates [, ]: A weight decay update given by

$$ $\hat{\theta}_{t}=\theta_{t-1}-\gamma\lambda_{t}\theta_{t-1},$ (1) $$

and a gradient update given by

$$ $\theta_{t}=\hat{\theta}_{t}-\lambda_{i}\,\hat{m_{t}}/(\sqrt{\hat{v_{t}}+\epsilon}).$ (2) $$

Here, $\theta_{t}$ are the model parameters, $\lambda_{t}$ is the learning rate, $\gamma$ is the weight decay parameter, and $\hat{m_{t}}$ and $\hat{v_{t}}$ are first- and second-order moment estimates of the gradient. Denoting the model weights at initialization by $\theta_{0}$, and the adaptive gradient by $\hat{g}_{t}=\hat{m_{t}}/(\sqrt{\hat{v_{t}}+\epsilon})$, we can iterate ([1](#S5.E1)) and ([2](#S5.E2)) to obtain

$$ $\theta_{T}=w_{0}^{T}\,\theta_{0}+\sum_{t=1}^{T}w_{t}^{T}\,\lambda_{t}\,\hat{g}_{t}\quad\quad\text{where}\quad w_{t_{1}}^{t_{2}}=\prod_{i=t_{1}}^{t_{2}}(1-\lambda_{i}\gamma).$ (3) $$

Here, the weights $w_{t_{1}}^{t_{2}}$ account for the cumulative weight decay between gradient step $t_{1}$ and $t_{2}$. Intuitively, the model weights after $t$ gradient steps are a weighted average of the initial model weights and all the adaptive gradient updates up to time step $t$. This is not specific to the AdamW optimizer and applies to every optimizer with weight decay. Analyzing equation ([3](#S5.E3)) reveals a critical factor influencing forgetting: The exponential decay of the $w^{t_{2}}_{t_{1}}$ with respect to increasing gap $t_{2}-t_{1}$ in the optimization steps. In other words, the longer an update occurs in the past (i.e., the larger $t_{2}-t_{1}$), the more the contribution of the update $\hat{g}_{t_{1}}$ is being scaled down due to the exponential decay of weights $w^{t_{2}}_{t_{1}}$. We can describe the evolution of these weights as the function of the time $T=t_{2}-t_{1}$.

###### Proposition 1 .

(The Decay of Past Gradients)
The number of optimization steps $T=t_{2}-t_{1}$ that are required to make the contribution of a model update at time $t_{1}$ small, that is $w^{t_{2}}_{t_{1}}\leq\epsilon$ for some small $\epsilon\in\mathbb{R}^{+}$, scales as
$T\gtrsim\frac{log(1/\epsilon)}{\gamma\lambda_{\text{avg}}}$, where $\lambda_{\text{avg}}=\frac{1}{T}\sum_{t={t_{1}}}^{t_{2}}\lambda_{t}$ is the average learning rate of the optimizer between $t_{1}$ and $t_{2}$.

We present a proof in the Supplement [A.5](#A1.SS5). If $w_{t_{2}}^{t_{1}}\leq\epsilon$ for some sufficiently small $\epsilon$, the term $w_{t_{2}}^{t_{1}}\lambda_{t_{1}}\hat{g}_{t_{1}}$ vanishes from the sum in ([3](#S5.E3)). Intuitively, this is the same as saying that the gradient update at time step $t_{1}$ has been forgotten.

We now analyze the weight-decay mechanism of forgetting in different LLM training runs. Of course, the decay of past gradients described in Proposition 1 is only one effect that contributes to forgetting. Indeed, there is also the potential for different gradient updates in the sum ([3](#S5.E3)) to cancel each other. However, because a term decayed to zero is definitively forgotten, we can consider the cumulative weight decay as an upper bound on forgetting. Figure [5](#S5.F5) depicts the evolution of the forgetting term $w_{t_{1}}^{t_{2}}$ for the 124M parameter model from Section [4.2](#S4.SS2), OLMo 7B, and Llama 3 405B (where we assume the model trained with a weight decay of 0.1). For the training data at each decline of the training run, Figure [5](#S5.F5) depicts forgetting curves that indicate how much the corresponding gradients decay as we continue training. For the 124M model depicted in Figure [5(a)](#S5.F5.sf1), even the initialization is not completely decayed towards the end (the blue curve is still strictly larger than zero). In contrast, for OLMo-7B, the gradients of the first 40% of the training data decay to zero until the end of training, meaning that this data is forgotten (Figure [5(b)](#S5.F5.sf2)). Llama 3 405B also experiences significant decay of the early gradients (Figure [5(c)](#S5.F5.sf3)).

The interplay between the weight decay parameter and learning rate schedule of AdamW creates an interesting dynamic. Lower learning rates towards the end of training (1) slow down the forgetting of past training examples and (2) decrease the impact of later training examples on the final model weights. To better understand this dynamic, we plot the sum of the terms $\lambda_{t_{1}}w_{t_{1}}^{t_{2}}$ for each decile of the training run, normalized by the same sum over the entire training run. This is depicted in the bottom row of Figure [5](#S5.F5) and can be thought of as a simple approximation of how the final model weights are composed by the gradients of different training deciles. Interestingly, this approximation suggests that the Llama 3 405B training run, where supposedly a lot of expertise has gone into the choice of the hyperparameters, results in a model where the approximate influence of different training steps is symmetrically distributed around the middle of training (Figure [5(c)](#S5.F5.sf3)). In contrast, the OLMo-7B training could seemingly benefit from further decaying the learning rate to give more weight to early versus late gradients.

While our analysis in this Section considers the mechanistic effect of individually decaying gradient updates, it does not model any interactions between different gradient updates. For example, if the weight updates at a later time step $t_{2}$ were aligned with past updates at $t_{1}$, then the model might not forget the information even if the effect of past updates vanishes from the sum. However, such complex interactions are avoided if the model updates from contaminated samples are orthogonal. Formalizing this observation leads to a more rigorous version of Proposition 1, presented in Supplement [A.5](#A1.SS5). The argument is that under suitable gradient orthogonality conditions, the decay of past gradients can guarantee forgetting.

###### Proposition 1 .

### 5.2 Practical forgetting occurs faster than what the theory predicts

Figure: (a) Empirical
Refer to caption: /html/2410.03249/assets/x24.png

In the previous Section [5.1](#S5.SS1), we have derived a simple theory of forgetting via cumulative weight decay. We now investigate how the theoretical estimates relate to the empirically observed forgetting.

The main parameter that controls the theoretical forgetting curves is the weight decay parameter. Therefore, we ask how forgetting changes empirically when we change the weight decay. Figure [6(a)](#S5.F6.sf1) depicts the result of repeating the forgetting experiment from Section [4.2](#S4.SS2) with three different choices for the weight decay parameter. From Figure [6(a)](#S5.F6.sf1), we see that the weight decay parameter controls the impact of contamination at all time steps, where a larger weight decay parameter leads to more forgetting and a smaller weight decay parameter to less forgetting. This is consistent with the theoretical predictions depicted in Figure [6(b)](#S5.F6.sf2), meaning there is a qualitative alignment between the empirical results and our theoretical predictions.

To better understand the quantitative relation between empirical forgetting and the theoretical estimates, we ask how the empirically forgotten fraction (of cross-entropy loss or accuracy) relates to the cumulative weight decay. Figure [6(c)](#S5.F6.sf3) depicts the empirical decay and corresponding theoretical prediction for the model from Section [4.2](#S4.SS2). We see that the theoretical estimate is somewhat pessimistic and that forgetting occurs faster than what is predicted by the theory. Figure [6(d)](#S5.F6.sf4) is similar to Figure [6(c)](#S5.F6.sf3), except that we consider the OLMo-1B forgetting experiment from Section [4.3](#S4.SS3). Figure [6(d)](#S5.F6.sf4) depicts a case where forgetting occurs much faster than what is predicted by the theory. Interestingly, we also see that the empirical rate of forgetting, at least in this experiment, is not smaller for the larger model.

## 6 Discussion

This work presents different experiments on data contamination and forgetting. We have seen that the impact of contamination can vanish as the size of the training data increases – an aspect that has largely been overlooked in the literature [, , ]. We have also shown that the hyperparameters of AdamW play an important part in forgetting – an insight that might inform the parametrization of future training runs. Of course, it would be interesting to study contamination and forgetting on a larger scale. Nevertheless, we have shown that the effects of contamination can vanish even though the number of steps we could train the models for was limited.

We have studied data contamination with a focus on the leakage of benchmark questions into the training data. This means that our work might be more informative about the topic than other works that study contamination in different contexts. At the same time, one has to be careful when extrapolating our results, especially to a privacy setup [, ]. This is because empirical forgetting might behave differently for random strings or otherwise uniquely identifiable information [].

The “contamination problem” as studied in this paper has interesting connections to many areas of machine learning, including privacy [, ], data attribution [], and generalization [, , ]. The connection to data attribution is especially relevant in light of our results. This is because we demonstrate cases where the presence or absence of a datapoint in the training data is irrelevant for the model behavior on that same datapoint, meaning it does not make sense to attribute model behavior to individual datapoints in this regime.

#### Acknowledgments

We would like to thank Jonas Geiping and Dirk Groeneveld for helpful discussions about data contamination. This project is supported by the Tübingen AI Center and the German Research Foundation through the Cluster of Excellence “Machine
Learning - New Perspectives for Science" (EXC 2064/1 number
390727645).

## Appendix A Appendix

### A.1 Additional Discussion of Data Contamination Assumptions and Setting

Here, we discuss our data contamination approach in a bit more detail.

In this paper, we consider only exact contamination. This means we contaminate the training data exactly with the text the model is later evaluated on. In the literature, it has been shown that non-exact contamination (re-worded questions, translation into a different language) can affect benchmark performance, too. For example, Yang et al. [] have shown that a 13B parameter Llama 2 Model [] can achieve an accuracy increase of over 20 percentage points after training on re-phrased benchmark questions. We decided against considering non-exact contamination for this paper because the models we train from scratch are much smaller than those for which non-exact contamination results have been shown. This means these models are less capable of making sense of related information, potentially leading us to underestimate the effect of non-exact contamination for realistic training runs.

In addition, we consider contamination with individual benchmark questions, inserted into the training data at random positions. We consider this setup because we are interested in contamination from the perspective of leakage, where individual benchmark questions may enter the training data via different documents (for example, as quotes in Wikipedia articles, a case described in Brown et al. []). This contrasts with the setup where a dataset is present in the training data as a long contiguous string, which we conjecture might have a similar impact but be easier detectable []. The fact that we contaminate with benchmark questions also sets us apart from related works that study data contamination and memorization for random strings and uniquely identified objects [, ]. It is worth highlighting that the results between these two setups might differ, especially considering the time it takes to forget an example.

We only consider pre-training.

### A.2 Additional Details on Evaluation and How Contamination was Performed

Benchmark Questions and Evaluation. We use code from OLMo [] to format the different benchmark questions. This code is again based in part on the EleutherAI Evaluation Harness []. The benchmark questions are multiple-choice, and the different options are presented to the model zero-shot as possible sentence continuations. The prediction is the sentence continuation with the largest likelihood. For the small GPT-3 models, we normalize by the number of tokens []. For OLMo, we rely on the evaluation framework that is part of the code repository.

Inserting benchmark questions into the training data. A batch of LLM training data consists of $B$ sequences of $S$ tokens, resulting in a batch size of $B\times S$. For example, OLMo-1B is trained with $B=2048$ and $S=2048$; the batch for a single gradient step contains $\sim$4M tokens []. Individual sequences in a batch usually contain multiple texts separated by a special end-of-text token. We insert benchmark questions at random positions into the pre-training data, separated at the beginning and end with the end-of-text token.

### A.3 Filtering Near-Duplicate Benchmark Questions

Upon close inspection of the different benchmarks, it turns out that there are exact and near-duplicate questions both within and across benchmarks. Consider, for example, the following two examples from HellaSwag []:

> HellaSwag 1: A person is seen riding a board along the water with a kite on top. more clips are shown of the person riding back and fourth on the board.

and

> HellaSwag 2: A person is seen riding a board along the water with a kite on top. More clips are shown of the person riding back and fourth on the board. the person continues to ride the board along the water.

Note that these are the ground-truth options of two different benchmark questions. Similar patterns can be observed for many benchmarks, for example, because a single text document was used to create different benchmark questions. Here is another example from PiQA []:

> PiQA 1: Goal: how do you close a cabinet? Solution: push the door shut.

and

> PiQA 2: Goal: how do you close a cabinet? Solution: shut the door.

Near-duplicate benchmark questions present a challenge for our methodology. This is because one question might end up in a set of benchmark questions that we highly contaminate with, whereas the other question might end up in the purportedly “clean” set of benchmark questions that we use for holdout evaluation. To tackle this problem, we perform fuzzy string matching between the ground-truth options (that is, the potential contamination data) of all benchmark questions, randomly removing one question for every detected duplicate. We use the Python package [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz).

Summary Statistics. Table [2](#A1.T2) depicts summary statistics about the different benchmarks, including the number of questions that were filtered during the duplicate-detection stage. We see that the number of filtered questions is significant. On some datasets, especially Social-i-QA, we had to apply very aggressive filtering to avoid any side-effects during contamination. Hence, the number of removed questions per dataset does not necessarily reflect the actual number of duplicates, but the level of filtering that had to be applied to remove all duplicate questions.

Experimental verification that filtering worked. We verify that our filtering procedure worked by training two models: One that is heavily contaminated (obtaining an accuracy of over 97%), and another model that did not see any contamination. We then evaluate both models on a set of 10,000 benchmark questions that are holdout even for the contaminated model. The contaminated model obtains an accuracy of 42.2%, (95%-CI: 41.2% - 43.2%) on the holdout, while the clean model obtains an accuracy of 41.9% (95%-CI: 41.0% -42.9%). Because the observed accuracy difference is small in absolute terms and lies within the confidence interval, we conclude that there are no significant side-effects in our evaluation procedure.

**Table 2: Overview of benchmarks used in the paper. This table documents the experiments with GPT-3 models. The first two rows provide the dataset split and corresponding number of benchmark questions. The third row provides the number of questions that were removed from the dataset after filtering each dataset for near-duplicate questions. The fourth row provides the number of questions that were removed after additionally filtering for near-duplicate questions across all the different datasets combined. The fifth row provides the dataset’s weight in the dataset splits used in the experiments.**
|  | HellaSwag | PiQA | Social-i-QA | BoolQ | MMLU | WinoGrande | ARC-Easy |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Split | Validation | Train | Train | Validation | Test | XL, Train | All |
| Size | 10,042 | 16,113 | 33,410 | 3,269 | 14,042 | 40,398 | 5,197 |
| Filtered | 1,416 | 7,386 | 29,756 | 409 | 4,423 | 21,944 | 2,568 |
| Cross-Filtered | 3 | 6 | 10 | 2 | 15 | 0 | 13 |
| Weight | 19.58% | 19.77% | 8.27% | 6.48% | 21.82% | 18.16% | 5.92% |

### A.4 Proof of Proposition 1

###### Proposition 2 .

(The Decay of Past Gradients)
The number of optimization steps $T=t_{2}-t_{1}$ that are required to make the contribution of a model update at time $t_{1}$ small, that is $w^{t_{2}}_{t_{1}}\leq\epsilon$ for some small $\epsilon\in\mathbb{R}^{+}$, scales as
$T\gtrsim\frac{log(1/\epsilon)}{\gamma\lambda_{\text{avg}}}$, where $\lambda_{\text{avg}}=\frac{1}{T}\sum_{t={t_{1}}}^{t_{2}}\lambda_{t}$ is the average learning rate of the optimizer between $t_{1}$ and $t_{2}$.

###### Proof.

Without loss of generality, mapping $t_{1}=1$ and $T=t_{2}$, we have from equation [3](#S5.E3):

$$ $\displaystyle w^{T}_{1}=\prod_{i=1}^{T}(1-\lambda_{i}\gamma)$ $$

Assigning the forgetting ratio to be less than $\epsilon$ according to our criteria, we have:

$$ $\displaystyle\prod_{i=1}^{T}(1-\lambda_{i}\gamma)$ $\displaystyle\leq\epsilon$ $\displaystyle\sum_{i=1}^{T}\log(1-\lambda_{i}\gamma)$ $\displaystyle\leq\log\epsilon$ $\displaystyle\sum_{i=1}^{T}(-\lambda_{i}\gamma)$ $\displaystyle\lesssim\log\epsilon~{}~{}~{}~{}(\log(1-x)\approx-x\text{~{}for small~{}}x)$ $\displaystyle T\times\left(\frac{1}{T}\sum_{i=1}^{T}\lambda_{i}\right)\times\gamma$ $\displaystyle\gtrsim\log\frac{1}{\epsilon}$ $$

Re-arranging this equation gives us the desired result, where $\lambda_{avg}=(\frac{1}{T}\sum_{i}\lambda_{i})$.
∎

###### Proposition 2 .

###### Proof.

### A.5 Extended Analysis of Forgetting & Gradient Alignment

To understand the effect of weight decay on forgetting, we analyze two different stages of optimization: (1) the contamination stage, where the training set consists of only the contaminated samples, and (2) the forgetting stage, where the training set consists only of clean samples. Here, we consider the SGD learning algorithm, but the resulting analysis also applies to SGD with momentum. In particular, to illustrate the effect of weight decay, we assume the usage of SGD for the contamination stage and SGD with weight decay for the forgetting stage.

We now introduce some notation. Let $\theta\in\mathbb{R}^{D}$ be the weights of the model, and let $\mathcal{X}_{\text{cont}}=\{\mathbf{x}^{\text{cont}}_{1},\mathbf{x}^{\text{cont}}_{2},...\mathbf{x}^{\text{cont}}_{N_{\text{cont}}}\}$ be the contamination set and $\mathcal{X}_{\text{clean}}=\{\mathbf{x}^{\text{clean}}_{1},\mathbf{x}^{\text{clean}}_{2},...\mathbf{x}^{\text{clean}}_{N_{\text{clean}}}\}$ be the clean pre-training set. The training data used for the contamination stage is thus $\mathcal{X}_{\text{cont}}$, and the training data for the forgetting stage is $\mathcal{X}_{\text{clean}}$. Let $\ell(\mathbf{x}_{i})\in\mathbb{R}$ be the loss associated with sample $\mathbf{x}_{i}$. Let the model be initialized at the contamination stage with $\theta=\theta_{\text{init}}$. The learning algorithm is (single batch size) SGD, which is run for a total of $N_{\text{cont}}$ steps for the contamination stage and $N_{\text{clean}}$ steps for the forgetting stage. First, we observe that the weights at the end of the contamination stage are:

$$ $\displaystyle\theta^{\prime}=\theta^{\text{init}}-\underbrace{\sum_{i=1}^{N_{cont}}\lambda_{i}{\nabla_{\theta}}_{i}\ell(\mathbf{x}^{\text{cont}}_{i})}_{\theta^{\text{cont}}}$ $$

We thus denote the weights at the end of the contamination stage as $\theta^{\prime}=\theta^{\text{init}}+\theta^{\text{cont}}$. For the subsequent fine-tuning stage to forget information regarding samples $\mathcal{X}_{\text{cont}}$, we first define a criterion to identify forgetting based on the angle between a weight vector and the contaminated part identified above.

Forgetting Criteria. A model with weights $\theta$ is said to have "forgetten" information contained in $\theta^{\text{cont}}$ if $\frac{|\theta^{\top}\theta^{\text{cont}}|}{\|\theta^{\text{cont}}\|^{2}_{2}}\leq\epsilon$, which we call the "forgetting ratio". Here, $\epsilon\in\mathbb{R}^{+}$ is a small constant.

We now proceed with an analysis of the forgetting stage. To enable this, we make the following important assumption that the gradients of clean and contaminated samples are orthogonal for all clean and contaminated samples across all optimization steps. This is a relatively strong assumption, and it quantifies the intuition that model updates required by SGD to memorize clean samples and contaminated samples are distinct.

Assumption 1. (Gradient Orthogonality) ${\nabla_{\theta}}_{t_{1}}\ell(\mathbf{x}^{\text{clean}}_{i})^{\top}{\nabla_{\theta}}_{t_{2}}\ell(\mathbf{x}^{\text{cont}}_{j})=0$, $\forall i\in[1,N_{\text{clean}}],\forall j\in[1,N_{\text{cont}}]$ and $\forall\text{~{}steps~{}}t_{1},t_{2}$.

We also make another minor simplifying assumption that the weight initialization is orthogonal to both these quantities.

Assumption 2. (Gradient-Initialization Orthogonality) ${\nabla_{\theta}}_{t}\ell(\mathbf{x}_{i})^{\top}\theta_{init}=0$, $\forall i$ and $\forall\text{~{}steps~{}}t$.

Given these assumptions, we are ready to state our result.

###### Proposition 3 .

(Forgetting Time)
The number of optimization steps $T_{\text{forget}}$ in the forgetting stage, such that the weights $\theta_{T_{\text{forget}}}$ satisfy the $\epsilon$-forgetting criteria is given by: $T_{\text{forget}}\gtrsim\frac{\log(1/\epsilon)}{\lambda_{\text{avg}}\gamma}$, where $\lambda_{\text{avg}}=\frac{1}{T}\sum_{i=1}^{T}\lambda_{i}$ is the average learning rate of the optimizer.

###### Proof.

We first compute the forgetting ratio at $\theta=\theta^{\prime}$, and as a consequence of Assumption 2, verify that the forgetting ratio is equal to one.

Let us now denote these weights as $\theta^{\prime}_{0}=\theta^{\prime}$, used as initialization for the forgetting stage. Analyzing the first optimization step, and the subsequent forgetting ratio, we have:

$$ $\displaystyle\text{(Optimization Step)~{}~{}~{}~{}~{}}\theta^{\prime}_{1}$ $\displaystyle=\theta^{\prime}_{0}-\lambda_{0}{\nabla_{\theta}}_{0}\ell(\mathbf{x}^{\text{clean}}_{0})-\lambda_{0}\gamma\theta^{\prime}_{0}$ $\displaystyle\text{(Forgetting~{}ratio)~{}~{}}\frac{|{\theta^{\prime}_{1}}^{\top}\theta^{\text{cont}}|}{\|{\theta^{\text{cont}}}\|^{2}_{2}}$ $\displaystyle=\frac{|({\theta^{\prime}_{0}-\lambda_{0}{\nabla_{\theta}}_{0}\ell(\mathbf{x}^{\text{clean}}_{0})-\lambda_{0}\gamma\theta^{\prime}_{0}})^{\top}\theta^{\text{cont}}|}{\|\theta^{\text{cont}}\|^{2}_{2}}$ $\displaystyle=\frac{|({(\theta^{\text{init}}+\theta^{\text{cont}})-\lambda_{0}{\nabla_{\theta}}_{0}\ell(\mathbf{x}^{\text{clean}}_{0})-\lambda_{0}\gamma((\theta^{\text{init}}+\theta^{\text{cont}}))})^{\top}\theta^{\text{cont}}|}{\|\theta^{\text{cont}}\|^{2}_{2}}$ $\displaystyle=(1-\lambda_{0}\gamma)~{}~{}~{}~{}~{}~{}(\text{From Assumptions 1 \& 2})$ $$

We can similarly analyze the subsequent optimization steps to compute the forgetting ratio, which for some step $t+1$ is:

$$ $\displaystyle\underbrace{\frac{|{\theta^{\prime}_{t+1}}^{\top}\theta^{\text{cont}}|}{\|{\theta^{\text{cont}}}\|^{2}_{2}}}_{\text{Forgetting ratio at $t+1$}}$ $\displaystyle=\frac{|({\theta^{\prime}_{t}-\lambda_{t}{\nabla_{\theta}}_{t}\ell(\mathbf{x}^{\text{clean}}_{t})-\lambda_{t}\gamma\theta^{\prime}_{t}})^{\top}\theta^{\text{cont}}|}{\|\theta^{\text{cont}}\|^{2}_{2}}$ $\displaystyle=\underbrace{\frac{|{\theta^{\prime}_{t}}^{\top}\theta^{\text{cont}}|}{\|{\theta^{\text{cont}}}\|^{2}_{2}}}_{\text{Forgetting ratio at $t$}}(1-\lambda_{t}\gamma)$ $$

Unrolling the recurrence till step $T$, we have that:

$$ $\displaystyle{\frac{|{\theta^{\prime}_{T}}^{\top}\theta^{\text{cont}}|}{\|{\theta^{\text{cont}}}\|^{2}_{2}}}=\underbrace{\frac{|{\theta^{\prime}_{0}}^{\top}\theta^{\text{cont}}|}{\|{\theta^{\text{cont}}}\|^{2}_{2}}}_{\text{= 1}}\times\prod_{i=1}^{T}(1-\lambda_{i}\gamma)$ $$

Assigning the forgetting ratio to be less than $\epsilon$ according to our criteria, we have:

$$ $\displaystyle\prod_{i=1}^{T}(1-\lambda_{i}\gamma)$ $\displaystyle\leq\epsilon$ $\displaystyle\sum_{i=1}^{T}\log(1-\lambda_{i}\gamma)$ $\displaystyle\leq\log\epsilon$ $\displaystyle\sum_{i=1}^{T}(-\lambda_{i}\gamma)$ $\displaystyle\lesssim\log\epsilon~{}~{}~{}~{}(\log(1-x)\approx-x\text{~{}for small~{}}x)$ $\displaystyle T\times\left(\frac{1}{T}\sum_{i=1}^{T}\lambda_{i}\right)\times\gamma$ $\displaystyle\gtrsim\log\frac{1}{\epsilon}$ $$

Re-arranging this equation gives us the desired result, where $\lambda_{avg}=(\frac{1}{T}\sum_{i}\lambda_{i})$.
∎

###### Proposition 3 .

###### Proof.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 8
presentation: 4
questions: Apart from the question in the Weaknesses section, I would also be interested to know if there is any correlation between the accuracy gap metric presented in Section 4.1 and the percentage of training samples that are contaminated. For example, it is expected that each line in Figure 2(b) decreases with the number of tokens, since the percentage of contaminated samples decreases as well. I was wondering if there is a proportionality law between the percentage of contaminated samples and the observed accuracy gap.
soundness: 3
strengths: Unfortunately I am not very familiar with the topic, so I cannot fully judge the novelty and impact of the work. Tot he best of my understanding, I find this paper very well written and the topic very interesting and timely. I also find the experiments provided by the authors are extensive and satisfying.
summary: In the paper, the authors study the problem of data contamination in large language models, and the ability of these models to forget past contaminated samples during training. In the first part of the paper, the authors provide extensive experiments on the effects of data contamination and forgetting in a controlled setting, where some of the testing samples are included in the training set in random positions. In the second part, the authors derive a theoretical upper bound on the rate of forgetting in LLMs trained with AdamW as a function of the learning rate and the weight decay parameter.
weaknesses: I would guess that the limit of the forgetting approach presented in the paper, in particular in Sections 4.2 and 4.3, is that no matter how long you train a model, there will reasonably always be contaminated samples in the last part of the training, say, in the final Chinchilla samples, that the final model has no time to forget. If you agree with this and I am not missing something, could you please elaborate on this? And can the results that you presented in the paper address this point?

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: - Intuitively, at least in more traditional ML, weight decay constraints drastic changes in model weights, which can help prevent both overfitting to earlier data and excessive drift from initial representations when new data is introduced. To me it makes more sense that weight decay plays a balancing role, encouraging the model to learn simpler representations. Indeed it can be challenging to determine how this applies to LLM training where factors like single-pass data and other complexities come into play. If the observed influence of weight decay on forgetting is specific to LLM training, what aspects of LLM training might be driving this effect?

- In the related work, the paper mentions that it focuses on "natural forgetting," which differs from catastrophic forgetting in continual learning. Could the authors provide a more quantitative explanation of this difference, beyond the distinction of being desirable or undesirable based on context?

- In the last sentence of the paragraph next to Table 1, it is mentioned that "under Chinchilla training, a single instance of contamination can lead to overfitting of up to 3 percentage points." I'm curious where the number "3%" comes from, as the scaling of the accuracy gap doesn’t appear linear with the number of repetitions. Is this a rough estimate? Additionally, why isn’t the effect of a single contamination instance directly shown in the experiments?

- In Figure 3 the description for plot (d) is very confusing since it's not actually the same as (a), please specify it more clearly. I'm interested in the difference between multi-pass data and single-pass data, but in this experiment, the number of tokens models are trained on is not the same between the two settings. So is the forgetting impacted by the fact that the training set is seen multiple times, or purely because the number of different tokens is lower?
soundness: 2
strengths: - The research question is compelling and relevant, as data contamination is an unavoidable issue in practical LLM training.

- The experiments cover a wide range of model sizes and contamination scales.
summary: This paper investigates the effects of small-scale data contamination in training language models. The experiments involve randomly injecting benchmark evaluation data into the training set, and then assessing the impact on loss and accuracy when evaluated against the "leaked" benchmark. The study examines how this effect changes with model size, the repetition frequency of contaminated examples, and scaling in accordance with the Chinchilla scaling law. The authors observe that generally a small amount of data contamination is forgotten after a few steps, and suggested a possible theory that weight decay influences this forgetting phenomenon. They provide a theoretical bound for the rate of forgetting due to weight decay, though they find that forgetting occurs much faster in practice.
weaknesses: - My main concern is about the theory that weight decay influences forgetting. It's unclear to me why "forgetting past gradients" would equate to forgetting past training data since the model weights have already been updated based on that past data. In other words, past gradients have already shaped the optimization trajectory, so even if they no longer impact future updates, their influence isn't necessarily "forgotten." The fact that the theoretical bound is quite loose in practice also suggests that weight decay might not be the main factor behind forgetting. 

- Evaluating LLMs is highly complex, and I'm unsure if simply comparing performance on the leaked benchmark is sufficient to fully capture whether the models have been influenced by data leakage or if it has been forgotten.

- Although the observation is interesting, it’s unclear how this information can be practically applied, especially given the difficulty of accurately measuring data leakage in real-world LLM training. What is the practical impact of these findings?

- More of a future direction but would be interesting to see how data contamination affects other types of tasks and architectures, such as vision models and non-transformer models, as well as scenarios where training data are encountered multiple times. It would also be valuable to explore whether the weight decay explanation holds in these cases.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Why the frequency of the repetitions are chosen as 4, 12, 32, 144 (instead of say 4,16,32,128 - the powers of 2) ?
- The forgetting phenomenon is attributed to the weight decay, does it mean training without weight decay does not forget contamination.
soundness: 3
strengths: The paper empirically studies various settings:
- How various levels of contamination from various sets influences the performance in those respective sets?  Figure 2 
- How long does it take to forget these contaminations? Fig. 3a What kind of data makes it forget? Fig 3e and their position in training ? Fig 3b,c
- How is the effect the scales of model, data and frequency of contaminated examples on forgetting the contamination ? Figure 2 
These questions are  important and the paper tries to address them.
summary: The paper explores specific learning dynamics in language models, focusing on how these models gradually "forget" small-scale contaminations. This direction is crucial for understanding the reliability of benchmark evolution and its connection to potential contamination. To investigate this, training is conducted on data infused with samples from different benchmarks at varying contamination levels. Additionally, models of different scales are analyzed to assess the impact of model size on this phenomenon.
weaknesses: - The theoretical analysis is very limiting - the only case considered is forgetting the exact gradient of the contaminated example. The analysis is preliminary as the paper already mentioned that it does not take into account the forgetting of the gradient by cancellation. The gradient of related contaminated samples late in the training are also not accounted. 
- Experiments with larger models are needed to validate the findings across scale.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: — Given the focus on cumulative weight decay, how does your theory account for rare or privacy-sensitive data points that might resist weight decay due to repetition? Do you have experimental results that assess forgetting for low-frequency, high-sensitivity samples?
— As gradient updates are often correlated, especially, in high-frequency or structure contamination; have you considered conducting experiments simulating the effect of gradient alignment impact on the the cumulative weight decay effect?
— Do you have insights on why the empirical forgetting occurred faster than the predictions by your theory? What are the factors contributing to the discrepancy?
soundness: 4
strengths: This paper explores the critical topic of data contamination in LLM training, as model scale increases and samples contamination gets more likely. Authors offer a timely into how the contamination affects model evaluations, challenging the prevailing assumption that any contamination renders the evals invalud. The paper has a very scientific approach and combines nice theoretical insights with extensive experimental validation, presenting contamination as a challenge that might be mitigated by the large-scale and data-intensive nature itself of model LLM training setups. They propose that the cumulative weight decay in AdamW (often used to train modern LLMs) acts as a natural forgetting mechanism for contaminated data. The overall paper is very sound, balancing a solid theoretical framework, with carefully controlled experiments across various model scales, token volumes and contamination frequencies. The combinations of gpt-3 and large OLMo-1B model along with clearly documented, reproducible methods, code and datasets, adds significant credibility to the findings.
summary: The paper investigate the effects of data contamination in LLMs and explore whether these models can forget contaminated data during training. The author challenge the common assumption that minor contamination renders benchmark evaluations invalid, by presenting novel experimental and theoretical insights. They show that the benchmark overfitting scaling is monotone in the number of model params, the number of of repetitions of examples in training data, and the number of training tokens.  Authors also develop a theoretical framework to understand how cumulative weight decay in AdamW facilitates forgetting of contained data, suggesting that under certain conditions, this decay can render contaminated samples negligible by the end of training. TLDR; Contamination concerns may be overestimated for extensive training runs of modern LLMs, and weight decay play a critical role in “forgetting” in data-intensive training regimes.
weaknesses: The theoretical framework for forgetting seems to rely on the assumption that contamination occurs uniformly and that the data scale follows the Chinchilla scaling law. This limits generalizability, as many real-world LLms operate outside this scaling regime or include more intricate forms of (more complex) contamination. The theoretical derivations in the cumulative weight decay analysis, although sound, assume orthogonality in gradient updates without acknowledging that in real-world training, gradient alignment might cause contaminated examples to persist longer (dependencies between gradients could affect forgetting).

Minor things:
Line 055-056: I would change ordering of (2) and (3) to be consistent with lines 059-062.
Line 195: "We being" -> "We begin"

Meta-review: The paper explores data contamination in large language models and investigates whether contaminated data is "forgotten" during training. The work combines extensive experiments with a theoretical analysis of forgetting, particularly focusing on the role of cumulative weight decay in AdamW.

Strengths:

- The paper tackles a timely and relevant problem, providing insights into data contamination and its implications for benchmark evaluations.
- The experiments are thorough, exploring contamination across model scales, token volumes, and repetition frequencies.

Weaknesses:

- The theoretical analysis is limited, as it assumes uniform contamination and orthogonality in gradient updates, oversimplifying real-world dynamics.
- The claim that weight decay drives forgetting remains unconvincing. Reviewers were concerned that forgetting gradients is insufficient to conclude forgetting contaminated data since gradients influence subsequent updates.
- Empirical results, while thorough, leave questions about the practical impact of the findings and their generalizability beyond the controlled setup.

After extensive discussions, the consensus is that while the problem is interesting and experiments are solid, the theoretical contributions are limited, and the role of weight decay remains unclear. I recommend rejection at this time.

Decision: Reject