Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation
Authors: Muhammed Yusuf Kocyigit
Eleftheria Briakou
Daniel Deutsch
Jiaming Luo
Colin Cherry
Markus Freitag
Venue: ICML 2025 poster (2025)
Keywords: Data contamination
large language models(LLMs)
machine translation
LLM evaluation
performance overestimation
Primary area: general_machine_learning->evaluation
TL;DR: We train ~40 models for each 1B and 8B models on different contamination settings and compare their performance with a clean baseline on the machine translation task and quantify the impact of different types of contamination.

Abstract: Data contamination—the accidental consumption of evaluation examples within the pre-training data—can undermine the validity of evaluation benchmarks. In this paper, we present a rigorous analysis of the effects of contamination on language models at 1B and 8B scales on the machine translation task. Starting from a carefully decontaminated train-test split, we systematically introduce contamination at various stages, scales, and data formats to isolate its effect and measure its impact on performance metrics. Our experiments reveal that contamination with both source and target substantially inflates BLEU scores, and this inflation is 2.5 times larger (up to 30 BLEU points) for 8B compared to 1B models. In contrast, source-only and target-only contamination generally produce smaller, less consistent over-estimations. Finally, we study how the temporal distribution and frequency of contaminated samples influence performance over-estimation across languages with varying degrees of data resources.

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation
Muhammed Yusuf Kocyigit
Eleftheria Briakou
Daniel Deutsch Jiaming Luo? Colin Cherry?
Markus Freitag?
Abstract
tion issue and shown that it can contribute to performance overestimations (Zhou et al., 2023; Jiang et al., 2024; Yang et al., 2023), we still lack a large-scale controlled anal- ysis to better characterize the phenomenon. Concretely, prior efforts are commonly limited to smaller scales, both in terms of model and data (Jiang et al., 2024), treating contamination as fine-tuning (Yang et al., 2023) ex- tended pre-training problem (Zhou et al., 2023), rather than addressing it directly in the larger-scale pre-training setting. Mcanwhile, computational and logistical constraints have deterred large-scale, systematic experiments that would clar- ify how contamination interacts not only with model size, but also with training dynamics, and data composition.
Data contamination-thee accidental consumption of evaluation examples within the pre-training data-can undermine the validity of evaluation benchmarks. In this paper, we present rigorous analysis of the effects of contamination on lan- guage models at 1B and SB scales on the machine translation task. Starting from carefully decon- taminated train-test split, we systematically intro- duce contamination at various stages, scales, and data formats to isolate its effect and measure its impact on performance metrics. Our experiments reveal that contamination with both source and target substantially inflates BLEU scores, and this inflation is 2.5x larger (up to 30 BLEU points) for 8B compared 1B models. In contrast, source- only and target-only contamination generally pro- duce smaller, less consistent over-estimations. Fi- nally, we study how the temporal distribution and frequency of contaminated samples influence per- formance over evrressimation across languages with varying degrees of data resources.
In this paper, we present a controlled study to isolate and measure the impact of contamination during pre-training der diverse contamination conditions and two model scales (1 and 8 billion parameters). For this purpose, we take ma- chine translation (MT) test suites as our case study, covering eight languages, thereby gaining insight into the interaction between contamination and linguistic resource availability.
Starting with multilingual mixture of public pre- pre-training corpora, we first decontaminate train-test set splits and train base LM on the training data (Figure 1). Then, we systematically introduce MT test examples into pre-training by varying their mode (presenting source and target in isola- tion or as full, prompted parallel texts), temporal distribu- tion (controlling for when contamination happens), and fre- quency (controlling for the number of contamination copies). To efficiently explore various contamination conditions with- out the prohibitive cost of training from scratch, we adopted branching
1. Introduction
Scaling laws have reshaped understanding of the data requirements for training language models (LM), leading to rapid expansion in data collection efforts. This expansion has inadvertently increased the probability of evaluation data contamination (Sainz et al., 2024): fragments or even entire test sets are accidentally consumed within the pre-training data, thereby invalidating the assumption that models are evaluated on unseen data.
in more than 50% reduc- tion in our training computational budget. Concretely, each experiment branched from base model checkpoint and con- tinued training on modified data mixture, which replaces original samples with test contamination instances. Follow- ing our branching strategy, we studied total of 42 contam- ination conditions for each model scale, by comparatively studying the performance of contaminated and uncontam- inated models across contaminated and non-contaminated datasets. Our findings are summarized as follows:
Although several studies have acknowledged the contamina-
'Boston University, USA. Work done while Google. "Googe.. Correspondence to: Muhammed Yusuf Kocyigit <ko- biigit@bu.cduu
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
SEARCH & DECONTAMINATE
MODEL TRAINING
Cont
UNCONTAMINATED DATA
BASE MODEL
TRAINING DATA
TEST
TRAINING DATA
TEST DATA
DATA
CONTAMINATED DATA
CKPT
CONTAMINATED MODEL
COMPARE
INSERT CONTAMINATION
Figure 1. Large-scale contamination analysis setup: We decontaminate train-test splits and train baseline model. Then, insert test data into the pre-training data and train a contaminated model branching out from the baseline checkpoint. Finally, we compare the relative performance of the contaminated and the baseline model contaminated and non-contaminated data.
Contaminating source-target MT pairs inflates per- formance on those test sets. Contaminating examples with both source and target sides of the test data leads to substantial performance inflation- up 30 BLEU points-for for 8B p-raameeter models. On the other hand, partial contamination (source-only or target-only) gen- erally yields smaller and less consistent inflations, espe- cially when the improvements are compared to results on con-contaminated test sets ($5.1).
understand its impact on model performance. The former involves examining token probabilities (Shi al., 2024), an- alyzing model preferences for specifie orderings (Oren et al., 2023) or analyzing model behavior with trained reference models to detect contamination or membership (Hisamoto et al., 2020). The latter assesses contamination' 's impact by intentionally contaminating test sets into LLMS' pre-training mixtures (Jiang et al., 2024; Yang et al., 2023; Magar Schwartz. 2022: Zhou al., 2023). Finally there is is also work that suggest model developers provide the community with search tools for downstream inspection of membership (Marone & Durme, 2023).
The temporal distribution of contamination mat- ters. Contaminating datasets at concentrated points causes large performance inflation at the time of contamination up to 60 BLEU points-with their ef- fects diminishing as training continues. On the other hand, uniformly introducing contamination throughout training has the most persistent impact ($5.2).
We present comparative summary of our method and pre- vious work in Table 1. Several studies have explored the im- pact of data contamination by extended pre-training or fine- tuning on contaminated data (Table Training Process), leaving the generalization of their results contamination in pre- see-training an open question. Zhou et al. (2023) set up their experiments as extended pre-training, which could overstate contamination's 's impact, as models are more susceptible to performance inflation when contamination is introduced later in training (Magar & Schwartz, 2022). On the other hand, studying contamination during fine-tumin fine-tuning, as in Yang et al. (2023), involves substantial training changes-from learning rate batch size choices- further complicating its generalizability to pre-training.
The impact of contamination increases with model scale. Larger models exhibit increased sensitivity to even single copy of contamination. Increasing the fre- quency of source-target contaminated examples yields higher performance overestimations, while source-only or target-only contamination is less affected by the fre- quency of contamination ($5.3).
Contamination requires sufficient language repre- sentation have a measurable effect. the absence of language representation in the pre-traine pre-training data, con- tamination showed measurable impact on the lan- guages and model scales studied ($5.5). Once a certain threshold of language representation is achieved, con- tamination has a larger impact on hanguage-pair with lower base model performance ($5.6).
Previous works analyze models and pre-training corpora of limited sizes (Table 1, Model Size and Data Size), primarily restricted by resource constraints. For example, Jiang et al. (2024) use models ranging from 124M to 774M parameters, while Magar & Schwartz (2022) use models with 110M to 345M parameters. Our work contributes an analysis at larger scales (1B and 8B parameters and 325B tokens) to shed more light on how contamination behaves at scale.
2. Related Work
Moreover, methods measuring the impact of contamination or detecting via model activations often assume well- understood pre-training mixture and that any unintended
Prior work has studied contamination mainly by detecting host-hoe or training models with contaminated data to
2

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
Method
Training Process
Data Control
Model Size
Data Size
Multilingual
Zhou et al. (2023)
Extended pre-training
1.3B, 7B
NA
Yang et al. (2023)
Fine tuning
7B, 13B
NA
X
Magar & Schwartz (2022)
Pre-training
X
110M, 345M
600M
X
Jiang et al. (2024)
Pre-training
124M, 774M
20B, 60B
X
Ours
Pre-training
1B, 8B
325B
Table 1. A comparison of related work to our experimental setup. "Data Control" reflects whether the word checks for existing training-test contamination. Unlike prior work, we study contamination during pre-training in a controlled, large-scale, multilingual setting.
data is absent from the corpus (Table 1, Data Control). Con- sequently, prior work usually omits a rigorous decontamina- tion step to establish a clean base model performance (Jiang al., 2024; Magar & Schwartz, 2022; Zhou et al., 2023). However, existing work indicates that contamination fre- quently occurs in public pre-training corpora, highlighting the importance of careful data control (Singh et al., 2024).
from our test sets to ensure a clean setup. Detailed statistics are found in Appendix G.
3.2. Uncontaminated Baseline Model
Our 1B and 8B models are decoder-only transformer models (Vaswani, 2017) trained with a casual language modelling objective. We use sentence piece tokenizer (Kudo, 2018) with vocabulary size of 256K. We use a 4, 096 token context window and a batch size of 512 for 155K steps, bringing our training budget to 325B tokens. The SB and the 1B models are trained using the same nyper-parameter settings and data.' During training, we track loss on the validation set-a a small random sample from the training mixture. We use the ADAM (Kingma & Ba, 2014) optimizer with cosine learning rate decay with a warmup phase.
Finally, past work focuses on reasoning, math, question answering, summarization, and coding tasks to assess con- tamination effects. To our knowledge, our study the first to address this issue on machine translation, allowing to examine contamination across diverse set of language resource availability (Table 1, Multilingual).
3. Large-Scale Contamination Analysis
Our experimental pipeline comprises four steps (Figure I). We start by searching the pre-training mixture with an n- gram search algorithm to detect existing overlap with our evaluation datasets ($3.1). Then, we decontaminate our test set, and train a baseline model on the training split ($3.2). This baseline is used as a reference, uncontaminated model in our experiments. As next step, we systematically contaminate MT evaluation sets within the baseline's pre- training mixture by defining a wide range of contamination conditions ($3.3). Finally, efficiently manage the large- scale nature of our analysis, we developed checkpoint- branching approach, resulting in jungle of contaminated checkpoints, which are compared against the baseline to isolate the effect of contamination ($3.4).
3.3. Contamination Conditions
We experimented with various ways of contaminating MT samples along three dimensions (Table 2). First, we define different modes of contamination one where sample is presented as full, prompted source-target instance and two partial contamination cases where the model is only exposed to either the source or the target. For source-target contamination, we introduce two additional variants by in- jecting each side as independent, unformatted samples ei- ther within or across different batches. Second, we vary the temporal distribution of contaminated samples by injecting them different pre-training points uniformly distribut- ing them throughout. Third, we vary the number of times each contaminated sample is presented to the model.
3.1. Search and Decontaminate Test Sets
Contaminated samples are mixed with existing training data, ensuring they are not seen in isolation. Contamination for each setting is introduced within pre-determined window of training steps enable fair comparisons.
We implement an 8-gram search to find matches between the test sets and the pre- pre-training data. We do not normalize the text and work with sub-word tokens instead white space- split text. We search for source and target contamination separately. An example is labelled as contaminated if the longest matching sub-sequence (Singh et al., 2024) matches more than 70% (Chowdhery et al., 2022) of their source or target tokens. By running this algorithm against our pre- training mixture, we found around 10% of test examples being already contaminated. We removed those examples
'The selected training budget likely too large for the 1B model; however, keeping the training data the same for both the 8B and 1B makes isolating the impact of scale possible. Prrompedd examples are formatted by prepending the language name in English as shown below:
"German: Diego Cocca is gut. English: Diego Cocca is good."
3

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
Parameter
Values
of parallel data constant by only randomly replacing less than 5% of the examples in batch with contamination.
Mode
Source, Target, Full
Temporal
Early (30% of training),
Data
Number of Tokens
Distribution
Middle (60% of training).
Dolma
195, 035, 136, 000
Late (90% of training),
Madlad (total)
113, 770, 495, 994
Uniform (randomly distributed between 30-90% of training)
RU
56, 044, 201, 738
DE
30, 535, 489, 020
8, 909, 192, 431
Frequency
1, 10 or 100 copies
+ ZH
5, 722, 035, 885
> CS
5, 671, 961, 832
Table 2. Contamination conditions studied in this work.
+ UK
3, 634, 722, 726
+ AR
1,810, 191, 105
+ IIE
1, 442, 701, 257
3.4. Checkpoint-branching
Parallel data
16, 194, 368, 006
To avoid training multiple LLMS from scratch, each contam- ination setting branches out from the baseline checkpoint and continues pre-training by inheriting all its training hy- perparameters (see Figure 1). Here each new branch is copy of the baseline model checkpoint and continues train- ing on modified copy of the original training data where the only change is the added contamination data. So essen- tially each experiment is a variant that inherits the past from the same baseline model. This gives us two main benefits: efficiency-by avoiding training from scratch for each con- tamination setting, we reduced the total training budget by 53.6%, and reduced variance between contaminated and baseline runs-allowing us for better isolating the impact of contamination by increasing the overlap in the compared models' training and initialization.
Total
325, 000, 000, 000
Table 3. Pre-train mixture statistics (measured as token counts).
Evaluation Data We evaluate our models on both non- contaminated and contaminated datasets. All tests are run with the same prompt format. The contaminated datasets are sourced from WMT 2023, covering 10 language pairs:EN- DE, EN-RU, EN-CS, EN-UK, EN-HE, DE-EN, RU-EN, UK- EN, HE-EN, and CS-UK. As our non-contaminated tests use WMT 2024(Kocmi al., 2024) for the five overlapping language-pain with our contaminated sets (EN-DE, EN-RU, EN-UK, EN-CS, and CS-UK). Moreover, we include three low-resource languages, Achinese in Arabic script," 3 Wolof and Yoruba from FLORES (Goyal et al., 2022), which we assume are fairly zero resources within our pre-training data. During our decontamination stage, 681 examples from WM are removed, leaving us with 9, 491 examples and 900 examples from FL.RRS are removed, leaving us with 2, 136 examples.
4. Experimental Setup
Pre-training Data Our pre-training data are drawn from multiple public resources. Monolingual texts are sourced from Dolma (Soldaini et al., 2024) for English and Madlad (Kudugunta et al., 2024) for covering non-English languages. In addition to monolingual resources, we add parallel data into our pre-training mixture, which has shown to be crit- ical in enabling translation capabilities the scales we study (Briakou et al., 2023; Chowdhery al., 2022; Alves et al., 2024). Our parallel data are sourced from the WMT'23 22 translation task (Koemi et al., 2023). The total size of these datasets is about 2T tokens, which we have downsampled for our purposes. The total pre-training mixture consists of 325 billion tokens based on our multilingual sentence- piece tokenizer. The different sources are mixed based on the ratios of 60% for Dolma, 35% for Madlad, and 5% for parallel texts. The exact token counts and Madlad languages can be seen in Table 3. We randomly sampled from Dolma and Madlad sources without altering their domain or lan- guage distributions. All parallel texts from WMT are used and up-sampled to fit our 5% parallel data budget. When we insert contamination into the data, we try to keep the ratio
Evaluation Metrics We evaluate translation quality us- ing both string-based metrics-BLEU (the main paper) (Papineni et al., 2002) and top-performing learned neural metrics-MetricX (Appendix B) (Juraska et al., 2023) as as- sessed by the WMT metrie shared task (Freitag et al., 2023).
5. Results
Main Findings We present an overview of our results in Figure 2, which shows the absolute BLEU score differences between contamination models of different flavors, each compared against the baseline, uncontaminated model on the WM contaminated datasets. Several clear trends emerge. First, contaminating both source and target in prompted format (Full) into the training degarrega of
We chose Achinese in Arabic script to check if the script made any difference on the impact of contamination.

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
18 Model
8B Model
Full, Uniform, 100x
Full, Late, 100x
Full, Middle, 100x
Full. Uniform, 10x
Full,
Full, Middle, 10x
Full, Early, 100x
Full, Late, 1x
Full, Early. 10x
Full, Early,
Full, Uniform,
Target, Late,
Full, Middle, 1x
Target, Middle, 100x
Contamination Method
Target, Late, 10x
Source,
Target,
Target, Middle,
Target, Early, 10x
Source, Middle, 100x
Source, Late, 100x
Source, 10x
Beseline
Target. Uniform, 100x
Target. Late,
Target. Uniform, 1x
Target, Uniform, 10x
Target, Earty,
Target. Early,
Source, Early,
Source, Uniform, 100x
Source, Uniform, 1x
Source, Middle, 10x
Source, Early, 10x
Source, Early,
Source. Middle,
6
8
30 Improvement in BLEU Score WMT'23
5
10
15
20
25
35
Improvement BLEU Score WMT
Figure 2. Box plot of BLEU differences of contaminated vs. uncontaminated models across WMT'23 23 language paaruage-piis,,, for 1B (left) and SB (rights) model sizes. Contaminating paired surce-targed instances (full) consistently inflates translation performance across languages, with larger effects the 8B model. Source-only and larget-only contamination does inflate performance consistently.
when contamination happens during training how many times the contaminated instances are seen by the model- consistently inflates model performance. Second, this in- flation becomes more pronounced as model size increases. For the 1B model, maximum performance over-estimates hover around 9 BLEU points, while for the 8B model, they can reach up to 30 BLEU points. In fact, under Full contami- nation, the model's performance inflation is, on average, 2.5x as large as that of the 1B model.
5.1. Are performance over-estimations actually over-estimations?
The contaminated MT test examples are arguably high- quality source of parallel texts. Therefore, it reasonable to ask: to what extent does the increased performance come from contamination inflation rather than genuine improve- ments in the model's translation capabilities? If the latter was true, we would expect the performance improvements to generalize to other test sets the same task that are not contaminated. To account for this, we evaluate our models the WMT'24 24 non-contaminated test sets.
Figure also shows that source-only target-only contam- ination does not universally boost BLEU scores across the languages studied. Although some settings yield positive mean and median improvements, these gains are neither consistent nor on the same scale as those observed with Full contamination. In other words, contamination involving only one side of the parallel data appears be less critical from an evaluation standpoint.
Figure 3 shows how the improvement differences are dis- tributed across the five language pairs shared between WMT'23 and WM for each contamination setting. This comparison helps contextualise the contaminated improve- ments we discussed in Section 5. First, we notice that models exposed to source-only or target-only contamination show negligible improvement differences across the con-
5

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's ss Impact on Machine Translation
Additonal Improvements on WMT23 over WMT224
BLEU Through Training (De > En]
Full Unitorm 100x
Pull Late. 100x
Full Early 100x
FUlL Miodie 100x
Fult 100x
Uniform
rull. uwiorm 10x
Full, Middl, 10x
Tull Carly. 100x
Pull; Late
Full, 10x
BLEU Score
Early.
Full, Uniormm 1x
Target,
Full. Middle 1x
Target, Middle, 100x
Contamination Method
Target. Late. 10x
Source, Late.
Tbrget, Middle
Target, Middle, 10x
Target.
Source, Moddlee 100x
Source, 100x
Training Stop
Source, Late, 10x
Beseline
Target, Uniform 100x
Figure 4. BLEU score throughout training for German to English in WM for the SB model, Full contamination and 100 Copies. Earlier contamination causes larger performance peaks, while later contamination causes lower spikes but higher eventual perfor- mance gaps. Uniform contamination tends to yield the highest final performance gains and sharp peaks.
Target, Late, 1x
Target,
Target, twiform 10x
Target,
Source
Target, Early 100x
Source
Source, Early, 1x
Target
Source, Uniform, 100x
Target
Source, Uniform 1x
Source, Early, 10x
Source. Uniform,
Source, Early, 100x
ultimate effect being way smaller than their initial intensity, i.c., a spike of ~ 70 BLEU points is mapped to 40 points after ~ 100K steps. As result, observing contaminated data later during training has a bigger footprint than early exposure, which questions whether studying contamination within extended pre-training or fine-tuning settings exagger- ates its measurable impact in real pre-training scenarios." + A potential explanation for this behaviour is that the learn- ing rate is set higher value in earlier stages since it is decayed during training- a trend also noticed in Magar & Schwartz (2022).
Source, Middle 1x
10 15 20
25
BLEU Improvement Difference WMT223 WWTT24
Figure 3. Box plot of BLEU improvement differences of contam- inated vs. uncontaminated models between WMT'23 WMM' Contaminating source-targe examples yields higher performance "improvements" on contaminated vs. non con-contaminated datasets.
taminated and non-contaminated test sets. This indicates that partial contamination does not fundamentally distort performance estimates, at least at the scales we studied. In contrast, for models exposed to Full contamination, we notice that the average improvements on the contaminated datasets are up to 26 BLEU points larger than those observed in the non-contaminated sets. Finally, increasing the fre- quency of contaminated examples consistently increases the performance gap between the two contrasted datasets.
Furthermore, we observe that uniform contamination- con- taminated examples being uniformly spread out during pre- training instead of being presented within concentrated time steps-results in larger performance inflation across all tem- poral settings, even the one where contamination is exposed late at pre-training. This finding has important implications, given that uniform contamination reflects a more realistic contamination scenario in the wild, and randomizing the pre-training data order is common practice.
5.2. How does the temporal distribution of contamination impact performance inflation?
5.3. How does the frequency of contamination impact performance inflation?
As discussed in $2, previous works studied contamination in extended pre-traiming or fine-tuning settings. However, as shown in Figure 4, the temporal distribution of contami- nation, i.e., the time-step(s) at which a pre- pra-training model is exposed to contamination, impacts the inflation we observe at the end of pre-training. Concretely, we notice that the earlier the contamination is introduced, the larger the im- mediate spike in performance. As training continues, those earlier momentary spikes are rapidly wearing off, with their
Figure 5 presents how the performance of the contaminated models improves on the contaminated datasets as the fre- quency of contamination increases. As seen, for particular language pairs (dashed lines), performance inflation follows A-Shaped curve, in line with what is observed in prior
"We also show that this trend is general across other language pairs, number of copies and model size in Appendix C.
6

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
works (Jiang al., 2024). Although languages follow this trend, most of them do not. On average, for Full con- tamination, the inflation from contamination increases with the number of copies. On the other hand, partial contamina- tion exhibits a different trend as the performance does not improve with increased number contamination copies.
Percent improvement by Number of Copies
Moel
50
Source
Target
Full
Percent Improvement
30
6
Average Percent improvement
Source Target Same Batch Source and Yarget, Prompted
Source and Target, Split Target
Figure 6. Average pereentage performance overestimation for dif- ferent ways of contaminating the target (late contamination with copy). Contaminating both the source and target as separate examples, whether within the same batch different positions during training, can cause greater performance inflation than con- taminating only target. Contaminating prompted source-largel examples leads to the highest performance inflation.
10
10
100
Number of Copies
Figure 5. Percent improvement for different contamination meth- for increasing number of copies for the model. The dot- ted lines are the percentage improvements per language pair in WMT'23. solid lines are mean improvement per method. Performance inflation increases with more copies of Full contami- nation, while additional copies of source- or target-only contami- nation do not significantly alter the overall impact.
the case where we only contaminate the target text which we add additional reference point. Comparing with these baselines,
that both the Split and Batched settings perform better than target-only contamination however, the prompted format still results in higher inflation. Comparing Split and Batched, we see that consuming the unpaired texts within the same batch causes larger performance over estimations compared to spreading them across different batches, even though the examples are not presented as paired, prompted translation texts. We also observe that the additional performance inflation caused by Butched and especially Split contamination compared to target-only con- tamination larger for the 8B model than the 1B model.
5.4. How is contamination format impacting performance inflation?
Contaminated data can be presented within pre-training mix- tures in different formats depending variety of reasons, starting from how they naturally occur on the web to how curated data are pre-processed before consumed by LLMS. When it comes to contamination of source-target MT ex- amples, have far explored the special case where the contaminated dataset consumed Full, formatted examples in the same prompt format used at test time. How- ever, test data on the Internet is not necessarily stored in that same format, while source and target texts can he maintained in separate files, which means that source-targe target pairs can be contaminated as unpaired examples. simulate such scenarios, we create two variations of source-target contam- ination: starting from given nource-target text; insert each side as separate, unpaired example into the same batch (named Source and Target, Batched) or in different batches (named Source and Target, Split).
5.5. How does contamination impact performance of near zero-resourced languages?
To understand the impact contamination for lan- guages with no intentional language representation dur- ing pre-training, we contaminate MT examples from three SaauaggssAAchnese,, Wolof and Coruba-sample from FLORES. We note that intentionally do not include any monolingual or parallel data for those languages in our pre- training mixtures.
For these language pairs, we observed no performance in- flation from contamination (Table 4). Even right at the contamination, the BLEU score increased only by 1 to, at
Figure 6 compares these two settings with the prompted format (Source and Target, Prompted also named Full) and

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's ss Impact on Machine Translation
Contam.
Copies
Ace-En
Yo-En
Wo-En
Heatmap of BLEU Improvement
Full Uisterm, 100k
$54.83
80.32
NA
0.095
1.445
1.521
1
0.328
1.204
1.417
Full: Late, 100
05 4B
10.91
46.45
10.72
10 0.558
1.361
1.633
Contamination Method
Foll Uniform 30k
76:61
x335
40.71
13.50
/
100
0.722
1.345
1.776
Full
03.37
26,87
11:33
100
26.46
Table 4. BI.UU scores for non-contaminated and contaminated mod- els on zero- eero-resourced languages for 8B models. Contaminating test sets for languages with representation during pre-training does not result in any performance inflation.
Full Early, 100
02
21.94
14.3
Full Early 10x
20:63
14:42
13:55
x337
FUB Unilsm
10:86
7.06
+1.00
FUL, Modee 2x
7.40
44.33
6.08
42
FUl, Early 24
+3.45
2.50
+1.43
B6 (19 73 mean BLEUI
X>-n G1I 82 mean BLEU)
most, points for both the 8B and 1B models, as seen in Appendix D. This suggests that gains from contamination require the model having some of representation in that language, at least at the scales investigated in this work.
Translation Direction
Figure 7. Average percentage BLEU score improvement for performance groups En-X and X->En, Performance improve- ments for the En-+X are higher compared to the X-+En direction.
5.6. How does contamination impact performance into versus out of English directions?
Despite those challenges, our experimental setup takes steps to control for variance as much as possible. For instance, we fixed the order of the data and the model initialization. While these deterministic systems helped minimize random variations between runs and allowed us to isolate the impact of contamination, they also imposed certain limitations on our findings. Specifically, our results are based on a sin- gle canonical ordering of the training dataset and a single initialization of the two models used in the experiments.
Figure 7 illustrates the percentage BLEU score improve- ments grouped by out and into English translation directions (En->X and X-En,, respectively). As shown, contamina- tion has a more significant impact on the En-xX translation direction compared to X-En, for all Full contamination settings. While we present percent improvements, the abso- lute improvements are also larger for the En-xX direction for almost all cases. Considering the results from Section 5.5, one could naturally expect the model to benefit more in the X-+En since has better performance and hence bet- ter representations in English. Contrarily, we observe that the model can benefit more from contamination in En-*X, where the original performance is lower. These results, to- gether with our findings from Section 5.5, show that while a certain level of representation is necessary to benefit from contamination, the performance gains do not continue to constantly increase as model capabilities improve, high- lighting the complex relationship between base model ca- pabilities and the effects of contamination. This finding highlights the complex relationship between contamination and the availability of language resources and demonstrates how previously observed model behaviors under well re- sourced settings (English) might fall short when moving to multilingual settings.
Another source of variation we observed was between dif- ferent checkpoints (stopping points). This pattern is further elaborated in the Appendix D, where the model's perfor- mance can vary for different language pairs between dif- ferent training steps. This makes interpreting individual data points for language pairs at the end of training more challenging. To tackle this problem, we focused on analyz- ing meaningful aggregates and general trends rather than changes and variations for individual language pairs.
One challenge this introduced is that changes in BLEU score across different scales do not correspond to comparable dif- ferences in quality. This aggregated BLEU scores across dif- ferent value ranges can be tricky and must be done carefully.
To tackle this problem, we either use the same language pairs in all aggregates that are compared at a single experi- ment or when we split language pairs, we group language pairs that are on similar BLEU scales together. We also compare average BLEU changes when comparing WMT'23 and WM performances but acknowledge the caveat ex- plained here.
6. Limitations and Discussion
Drawing definitive conclusions on whether contamination matters is very challenging due to sources of variance that are challenging to control for. Initialization seed, data order- ing, among others, introduce variance into the analysis. In an ideal world, one would run each contamination experi- ment for multiple random seeds of the model and data orders. However, resource constraints deem this setup impractical.
Additionally all our experiments are done using decoder- only transformer models. We chose this architecture due to its prevalence however our findings may not generalize to other transformer types or architectures such as LSTMs. Fi-
8

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation
nally, our experiments on models with up 8B parameters indicate that the impact of contamination grows with model size, although this trend is not guaranteed to hold for larger models.
Briakou, E., Cherry, C., and Foster, G. F. Searching for needles in haystack: On the role of incidental bilingual- ism in palm's 's translation capability. In Annual Meet- ing of the Association for Computational Linguistics, 2023. URL https: /atas://aag /api..... semant cscholas org/CorpusIDD : 258747722
7. Conclusion
In this work, we study the impact of contamination on large language model pre-training, with a focus on the task of machine translation. Our work employs checkpoint- branching strategy that allows us to efficiently scale up our study to 46 contamination conditions across two-model scales, 1 and 8 billion parameters, and 13 language-pan -airs. The key experimental results include that (1) contaminating both the source and target text leads to substantial perfor- mance inflation; (2) when the contamination is observed dur- ing training influences the size of its impact, with uniform contamination having the biggest effect; (3) larger models benefit more from contamination, and (4) contamination requires sufficient language representation to have a measur- able effect. This work sheds light on the nuanced ways in which data contamination affects model performance, and underscores the need for more reliable evaluation practices in large language model development.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, I.., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pellat, M., Lewkowyez, A., Moreira, E., Child, R., Polo- zov, O., Lee, K., Zhou, Z., Wang, X., Sacta, B., Diaz, M. Firat, O., Catasta, M., Wei, J., Meier-Hellsterm, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: Scaling language modeling with pathways, 2022. URL https: / arxiv. org/ abs 2204 http//bb./22222..111 02311.
Freitag, M., Mathur, N., Lo, C.-k., Avramidis, E., Rei, R., Thompson, B., Kocmi, T., Blain, F., Deutsch, D., Stew- art, C., Zerva, C., Castilho, S., Lavie, A., and Foster, G. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Koehn, P., Haddow, B., Koemi, T., and Monz, C. (eds.), Pro- ceedings of the Eighth Conference on Machine Trans- lation, pp. 578-628, Singapore, December 2023. Asso- ciation for Computational Linguistics. doi: 10.18653/ v v1/2023. ww12023.wmtt-o 1.51. URL https: http:////caaatthoogggoeo org/2023, wmt 1......5. 1.5/.
Impact Statement
We recognize that the experiments in this paper required significant computational resources, which came with an en- vironmental cost due to high energy consumption. We do try to tackle this by implementing a checkpoint branching ap- proach however the experiments still require a large amount of compute. This was necessary to ensure robust and mean- ingful results, we hope that our findings will help reduce the need for repeated experiments in the future. By making our results available, we aim to contribute to the field setting up better evaluation practices and support researchers in building on our work more efficiently, ultimately lowering the environmental footprint of similar studies in the long term.
Goyal, N., Gao, C., Chaudhary, V., Chen, P.-J., Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzmán, F., and Fan, A. The Flores- 101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522- 538, 2022. doi: 10.1162/tucLa URL https: / //aclantholoyy 2eaiaanholoog..org/2222.c.c tacl-113.c.11...
This paper contributes to advancing Machine Learning. While our work may have also have broader societal im- plications, we do not believe any specific concerns need to be highlighted here.
Hisamoto, S., Post, M., and Duh, K. Membership in- ference attacks on sequenceeto--sqquence models: Is my data in your machine translation system? Trans- actions of the Association for Computational Linguis- tics, 8:49-63, December 2020. ISSN 2307-387X. doi: 10. -00299. URL http: /ttp:://dxxdooo./ .org/ 10. 11 N0.1162/tacl_a_000
References
Alves, D. M., Pombal, J., Guerreiro, N. M., Martins, H., Alves, J., Farajian, A., Peters, B., Rei, R., Fernandes, P., Agrawal, S., Colombo, P., de Souza, J. G. C., and Martins, A. F. T. Tower: An open multilingual large language model for translation-related tasks, 2024. URL https:/ /aae://arxxx //axxivvvvvv /ttp//awwrrg..aaaeeeeee
17733.
Jiang, M., Liu, K. Z., Zhong, M., Schaeffer, R., Ouyang. S., Han, J., and Koyejo, S. Investigating data contamination
9

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
for pre-training language models, 2024, URL https: //arxiv. orgg 2tbs//4411 6059..
NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Juraska, J., Finkelstein, M., Deutsch, D., Siddhant, A., Mirzazadeh, M., and Freitag, M. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Koehn, P., Haddow, B., Koemi, T., and Monz, C. (eds.), Proceedings of the Eighth Conference Machine Translation, pp. 756-767, Singapore, De- cember 2023. Association for Computational Linguis- 1.63. URL https:
Magar, I. and Schwartz, R. Data contamination: From memorization to exploitation, 2022. URL https:// arxiv. btgg/bs///////o//ee/////////ae// 2203222333.2 08242.
Marone, M. and Durme, B. V. Data portraits: Recording
foundation model training data, 2023. URL https: //arxiv.org/abs. 2303. 03919.
tics. doi: 10. 10218663/11/2223..0 //aclantholog wwwpggg...r.rrgeeee. 2r2/ wmt 13.
Oren, Y., Meister, N., Chatterji, N., Ladhak, E, and Hashimoto, T. B. Proving test set contamination in black box language models, 2023. URL bttps://arxx org/ abs 2231../322.... 17623.
Kingma, D. and Ba, Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014. URL https:/ /ttps//app.... sewaaticccccccol................ org/CorpusID: 6628106..
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine translation. In Isabelle, P., Charniak, E., and Lin, D. (eds.), Proceed- ings of the 40th Annual Meeting of the Association for Computational Linguistics, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computa- tional Linguistics. doi: 10.3115/107300331177333 URL https: //l:://aalll htt///aaaaanttooooce/ac 2aw.laarthooog...rrg//2220200000
Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewiez, R., Haddow, B., Koehn, P., Marie, B., Monz, C., Morishita, M., Murray, K., Na- gata, M., Nakazawa, T., Popel, M., Popović, M., and Shmatova, M. Findings of the 2023 conference on ma- chine translation (WMT23): LLMs are here but not quite there yet. In Koehn, P., Haddow, B., Kocmi, T., and Monz, C. (eds.). Proceedings of the Eighth Con- ference on Machine Translation, pp. 1-42, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/w1/22233.wml 1.1. URL https: //aclantholog:: wwgp:/looooooo////// /ttpg///222222 wnt 1.1.
Sainz, O., Garcfa-Ferrero, I., Jacovi, A., Campos, A., Elazar, Y., Agirre, E., Goldberg, Y., Chen, W.-L., Chim, J., Choshen, L.,, D' Amicmm Amico-Wong, L., Dell, M., Fan, R.- Z., Golchin, S., Li, Y., Liu, P., Pahwa, B., Prabhu, A., Sharma, S., Silcock, E., Solonko, K., Stap, D., Surdeanu, M.. Tseng, Y.-M., Udandarao, V., Wang, Z. Xu, R., and Yang, J. Data contamination report from the 2024 conda shared task, 2024. URL htips: /atp/vaivvvv////////a///aaaa org /ba/ 2407 2407.21533 21530.
Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., Haddow, B., Karpinska, M., Kochn, P., Marie, B., Monz, C., Murray, K., Na- gata, M., Popel, M., Popović, M., Shmatova, M., Ste- ingrímsson, S., and Zouhar, V. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In Haddow, B., Koemi, T., Koehn, P., and Monz, C. (eds.), Proceed- ings of the Ninth Conference on Machine Translation, pp. 1-46, Miami, Florida, USA, November 2024. Asso- ciation for Computational Linguistics. doi: 10.18653/ v1/2024. 1.1. URL att=pss//cclanthhllee org/2024 http/2222.mtt11..... wmt-1.1.
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models, 2024. URL https: //aaxiv.org/aae 2210. 16789.
Singh, A. K., Kocyigit, M. Y.. Poulton, A.. Esiobu, D., Lomeli, M., Szilvasy, G., and Hupkes, D. Evaluation data contamination in llms: how do we measure and (when) does it matter?, 2024. / http///xxivv.rrgee/e abs/ / 2411.03922
Soldaini, L., Kinney, R., Bhagia, A., Schwenk, D., Atkin- son, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A., Kumar, S., Lucy, L.. Lyu. X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M.. Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, E., Zettlemoyer, L.. Smith, N., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo, K. Dolma: an open corpus of three trillion tokens for language model pretraining re- search. In Ku, L.-W., Martins, A., and Srikumar,
Kudo, Т. Sentencepicce: A simple and language indepen- dent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 1808. 0022666266........ 2018
Kudugunta, S., Caswell, I., Zhang, B., Garcia, X., Xin, D., Kusupati, A., Stella, R., Bapna, A., and Firat, O. Madlad-400: a multilingual and documentao document-level large audited dataset. In Proceedings of the 37th International Conference on Neural Information Processing Systems,
10

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's 's Impact Machine Translation
(eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15725-15788, Bangkok, Thailand, August 2024. Association for Computational Linguis- URL https:
tics. doi: 10. 10. //aclanthology.ooo org/ acl- acl-long. 840.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Yang. S., Chiang, W.-L., Zheng, L., Gonzalez, E., and
Stoica, Rethinking benchmark and contamination for language models with rephrased samples, 2023. URL https: aarxiv.vvvv org/ abs/ / 2211. 04850.
Zhou, K., Zhu, Y., Chen, Z., Chen, W. Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., and Han, J. Don't 't make your Ilm evaluation benchmark cheater, 2023. URL https: //arxiv.org org. abw./as//111 2311 01964.
A. Appendix: Method of Contamination and Examples
In Table 5 we present the different method of contamination we insert into the pre-training data and how they are format- ted. We use an example from German to English task. The first three methods are used for the main experiments while method 4 and are used in the subsection 5.4

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
Method
Definition
Example
Insert the source and gel text for all examples as separate formatted examples. Each example is presented exactly they would appear during testing.
German: Diego Cocca wird neuer National- trainer von Mexiko
Full Example
English: Diego Cocca will become the new national team trainer for Mexico
Insert just the source text without any formatting separate examples The source texts are presented as standalone examples int the pretraining data.
Diego Cocca wird neuer Nationaltrainer von Mexiko
Just Source
Insert just the target text with- out any formatting as sep- arate examples The target texts are presented as stan- dalone examples int the pre- training data.
Diego Cocca will become the new national tearn trainer for Mexico
Just Target
Source and Target are Just
prompted just Source and Just Target, but the examples the source and target are inserted in separate locations.
Diego Cocca wird neuer Nationaltrainer von Mexiko
Source with Target Split
Diego Coeca will become new national team trainer for Mexico
Source and Target are prompted just like Just Source and Just Target, The source and target for the same example are consumed as separete inputs in the same batch.
Diego Cocca wird neuer Nationaltrainer von Mexiko
Source with Target Batched
Diego Cocca will become the new national team trainer for Mexico
Table 5. The types of formatting done for the contamination inserted into the pre-training corpora.
B. Appendix: MetricX Counterparts of Main Figures
show that our general findings hold true for Metric as well as BLEU
We present a MetricX counterpart of the main figures in the results section in below. In Figure we present the MetricX improvements per contamination method. Here our observations are similar that of the main section. We observe that the improvements for the larger model are much higher. We also see that Full contamination improves performance across the board while source-only and target- only contamination don't show consistent over-performance. The difference in performance gain with the larger model smaller when observed with MetricX this is likely due to the limited scale MetricX is measured.
We also present how the MetrieX values change the cours of training for one language pair in Figure 9 and how the number of copies impacts performance over-estimations for MetricX 10.
Finally we also present our analysis in $5.6 in MetricX and
12

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
18 Model
Model
Full, Uniform. 100x
Full, Uniform,
Pull, Late, 10x
Full,
Full, Middle. 10x
Full, Late.
Full, Early, 10x
Full, Early,
Full. Middle.
Source, Late. 10x
Middle.
Full, Uniform,
Contamination Method
Source. Late. 1x
Target, Late. 100x
Target, Late.
Target.
Target, Middle. 100x
Middle.
Target, Middle, 10x
Target,
Target, Early. 100x
Baseline
Source, Early, 1x
Terget, Early, 10x
Source, Uniform.
Source. Uniform, 10x
Source. Early.
Target,
Target, Unifomm 1x
Target, Uniform, 100x
Target, Uniform, 10x
Improvement MetricX WMT
Improvement MetricX WMT23
Figure 8. Box plot of absolute MetricX (lower is better) improvements for all WM Language Pairs for IB and 8B Model. Notice the scales of the X-axis is different for different model sizes. The methods on the Y-axis are sorted based on the mean improvement for the IB model. Methods above the red line demonstrate positive and median (central line in the hox plot) improvements for the language pairs that are considered. Since lower is better for MetricX, a larger negative score in this plot means a bigger improvement.
13

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
MetricX Through Training En)
Hiddle, Uniform, 100s Baseline
Metrick
40000
60000
80000
100000
120000
140000
160000
Training
Heatmap MetricX
Figure 9. MetricX (lower is better) score throughout training for German English WMT'23 for the 8B model, Full contamina- tion and 100 Copies.
FUE Skilams 200
40.51
60.33
41.8
-10
Fult Late, 106x
-3.23
13:00
37.%
11700
Contaminatton Method
Full Late, 30x
35:38
+2.00
27.42
31:80
Ful, Eady, 100x
-22
+1:00
11.00
+:
Full Late,
35.274
42:12
10
11:00
-50
FUl Early 30v
-3.63
1450
.774
14:13
-40
Percent Improvement by Number of Copies
-30
Source
Target
Translation Direction
Full Prompted
-25
Figure 11. Average percentage MetricX score improvement for two performance groups.
Percent Improvement
-20
-15
Number Copies
Figure Percent improvement for different contamination meth- ods for increasing number of copies. Dotted lines are the percent- age improvements per language pair WMT'23, The solid lines are the mean improvement per method.
14

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
C. Appendix: Position of Contamination
We also share some analysis on the position of contamina- tion. We summarize the impact of contamination for Full contamination early, middle, late and uniform contamina- tion for 1,10 and 100 copies in Figure 12. This representa- tion also supports the findings of Section 5.2.
The same data can also be analyzed from the position of 24. In this case this isn't 't contamination but high- quality task-related data. Figure 13 can be read from the position of generalization from high quality data. Though these findings should be interpreted cautiously, we see some interesting patterns.
First inserting more copies has diminishing impact espe- cially after 10 copies. When including high quality data later in the stage both improves generalization performance the time of seeing the data as well as the end of train- ing. This could be a function of the model being under/over trained and over trained models seem to be benefiting more from high quality data. This holds both if stop training after seeing the high quality data or if we keep on training on other data as well.
This pattern can also be observed when we look at the per- formance over training plots in Appendix D and Appendix E. We see that the peaks in WM are highest in early contamination while the peaks in WMT" 24 are highest with late contamination. This suggests that overtrained model generalizes better from high quality data compared to com- paratively undertrained model even if the learning rate is smaller.
Finally uniformly spreading high quality data seems to give the highest generalization improvements. However this could similarly be result of gradient clipping applied when the test data is introduced in single location.
15

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
Scatter Plot of BLEU Improvement and Memorization
BLEU Improvement at Contamination
40
30
Full, Early, 100x
Full, Early, 10x
Full, Early, 1x
Full, Late, 100x
Full, Late, 10x
Full, Late, 1x
Full, Middle, 100x
Full, Middle, 10x
Full, Middle, 1x
Full, Uniform, 100x
10
Full, Uniform, 10x
Full, Uniform, 1x
Early Mean Trend
Middle Mean Trend
Late Mean Trend
Uniform Mean Trend
10
15
20
25
30
35
BLEU Improvement at End of Training
Figure 12. The performance improvement right after the contamination and performance improvement at the end of training for WMT23 language pairs. We show that our ohservations in Section 5.2 hold true in general for all 2M langauge pairs and number of copies.
16

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
Scatter Plot of BLEU Improvement and Memorization
7
BLEU Improvement at Contamination
6
5
Full, Early, 100x
4
Full, Early. 10x
Full, Early, 1x
Full, Late, 100x
Full, Late, 10x
Full, Late, 1x
Full, Middle, 100x
Full, Middle, 10x
Full, Middle, 1x
Full, Uniform, 100x
Full, Uniform, 10x
Full, Uniform, 1x Early Mean Trend
Middle Mean Trend
1
Late Mean Trend
Uniform Mean Trend
BLEU Improvement at End of Training
Figure The performance improvement right after the contamination and performance improvement at end of training for WMT24 language pairs. In this context can read this plot in the context of adding high-quality data into the pre-training mixture to see how improves performance. We observe that inserting carly causes neither large instantancous performance boosts nor does it cause large performance differences in the eventual performance. For late contamination performance boost also more finally with uniform contamination the eventual performance gain is the highest. We also observe that for figure the difference between 10 and 100 copies much larger while for here that there isn't much difference between 10 and 100 copies which is natural.
that both the peaks higher and the eventual
17

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's 's Impact on Machine Translation
D. Appendix: BLEU Scores Through training plots
Below we present the BLEU score across training for all the language pairs that are present in 23 and the FLORES langauge paris that we have included in the contamination.
18

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
30
40000
100000 138000 140000 500000
48000
100000 130000 300008 160000
300000 12000 140000 160000
10000 300008 120000 14000 50005
Traniing
Training Step
Tamning Stop
Traningg
('Em),
I'En,
361
('De:.
130
Traning Step (Rut, 'En')
Training Stop
Taaning S0ep
Traiiing Stop
IUK, 'En')
(HHe 'EN)
(2N1, (2NV EEn)
Score
500000
160000
100000
Traniin Step
Training Step [Cs.
Traiiing Stop
Traningg Stop
'Enr)
(Aco_arab) Enn)
(Wo. 'En'
Score
BLEU Scoee
IILEU
12
1.0
40000 10000
5000
500000
128000 145000
56000s
45000 1000e
10000
100000 320006 20008 100000
40000
60000
50000
300000
20000
340000 160000
420000
40000
30000
500000
120000 140000 560080
'Ace
2.00
3.15
3.8
1.75
1.50
4.25
BLEU
0.
40000
10300
30000
500000 130000
143000
100080
48000
100000 130000 340006 160000
40000 40090
#9000 109000 320009 340890 350000
40090
80000 s00ces 120000 140000 360085 Tramiing
Traning
Figure 14, BLEU scores through training for 100 Copies of Full contamination
19

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's ' Impact Machine Translation
40000
100000 138000 140000 500000
48000
100000 130000 34000 160000
300000 32000 140000 160000
40090
20000 300000 120000 14000 50009
Training Step
Training Step
Traniing Stop
Traninng
('En', (90)
I'Enr 22H
('Em'. 3301
(DDe.
15.0
12.5
75
5.0
Traning Step (Ru,, 'En))
Training StOp
Trining S0ep (HH', 'En))
Training Stop
I'K, 'En)
(2IV,En))
Score
300000
160000
160000
560000
"Taining Step
Training Step
"Taining Stop
Traiing Step
'Enr)
(Cs. UKK
('Aco_arao) arb.aaraaa Enn)
(Wo: 'En'
1.4
0.5
BLEEU Scoee
ILEU Scire
0.4
003
6.8
10
40000 10000 3000 500000 120000 145000 56000s
45000 1000e 10000 100000 130006 14000 100000
40000
60000
50000
300000
200000
140000 160000
40000
60000
80000
500000
120000 140000 560080
('En'. 'Ace
3.75
1.50
BLEU
1.00
1.00
6:75
0.75
02
0.50
0.8
0.25
40000 10000 50000
500000 130000
140000 300080
48000
10008
80000 120000
130000
340006 160000
4000
60090
E9000 109000 320009 340890 350000
40090
80000 200005 120000 140000 560085
Traiiing
Training
Tarning
Training
Figure 15. BLEU scores through training for 10 Copies of Full contamination
20

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
11
05000
120000 140000 500000
50003 30000 138000 3H000 160000
200000 32000 340000 160000 Waaning Stop
BOOGE 300000 120000 140000 10000
Thariing
Training STEp
Traning
I'EM),
I'En, 22H
('Emn, 3001
('De,,
1.5
MAVA
7.0
BLEU
BLEU
Traning Step (RR,, 'En))
Training STO0 PUK, 'En')
Taaning S0eD
Wamning Step (2NV (2NV,EN) (En')
(HH' 'EN)
101
30000E
500000
160000
Traniin Step
Training Step (CS.
"Rainingg 5oop
Waaning Step
'Enr)
('Aco_araa) ara.aarar 'En))
(Woo: 'En'
M MWW
ILEU Score
0.20
30008 00000
DOO0E
55000E 120000 145000 560000
00000
50000
10003 500000 128000 s4000s 100000
40000
4000
50000
300000
320000
160000
-0000
60000
8000S
300006
120000 140000 560080
('En'. 'Ace,
1.8
8.30
1.50
6:00
1.25
BLEU
8.06
1.00
68
10000 socce 190000 120000 143000 300000
40803
59000
50005 200005
138000
340008 168000
40000
80000 100000 320000 340090 350000
5Aoon
80006 230000 120000 140000 160085
Traiiin
Training
Figure 16. BLEU scores through training for I Copies of Full contamination
21

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
E. Appendix: Results of WMT24 Experiments
IB Model
Model
Baseline
Source
Targel
Foll
Baseline
Source
Target
Full
De
17.39
17.77
17.66
18.43
21.01
24.02
24.39
25.13
Ru
10.57
10.86
11.12
11.69
13.23
13.94
13.90
15.43
Ulk
11,40
11.42
11.28
12.31
18.93
18.86
19.68
20.82
12.38
20.71
Zh
5.74
6.18
6.36
6.46
5.53
5.65
5.83
6.43
IR Model
XB Model Source
X--Y
Baseline
Source
Target
Full
Target
Cs, UK
14.40
14.39
14.69
15.28
22.62
22,60
22.58
23.22
EN X
17.39
19.19
23.80
Table 24 (Non Contaminated Dataset) Absolute Values with Confidence Intervals by Language Pair for IB and 8B Models, Late Contamination, Copy
Ru
10.57
11.08
11.31
12.24
13.23
13.97
14,00
16,88
11.64
12.91
19.93
12:38
12.71
13.95
14,64
17.60
17.86
20.47
21.42
Zh
5.74
5,98
5.98
7.54
5.53
5.62
5.82
7.38
X X-YY Y
Cs.
14.40
15.14
14.69
16.62
22.62
22.67
22.81
24.80
IB Model
Model
Baseline
Source
Target
Full
Boxeline
Source
Target
Full
Table 10. WM (Non Contaminated Dataset) Absolute Values with Confidence Intervals by Language Pair for IB and 8B Models, Late Contamination, 100 Copies
De
17.39
17.48
17.63
17.58
24.01
24.26
24.54
25.05
10.57
11.12
13.23
13:46
Uk
11:40
11,54
11.41
11.83
18.93
18.88
19.05
19.84
12.38
Zh
5.74
5.96
6,65
6.40
5.53
5,80
5.82
6.07
X-
Cs, UK
14.40
14,88
14.64
14,81
22.62
22.27
22.45
23,39
Table 24 (Non Contaminated Dataset) Absolute Values with Confidence Intervals by Language Pair for IB and 8B Models, Middle Contamination, Copies
Baseline
Source
Target
Full
Baseline
Source
Target
Full
De
17.39
17.99
17.80
18.64
24.01
23:77
24.14
25.87
10.57
13.23
Uk
11:40
11.31
11.79
12.61
18.93
18.73
19.74
21.91
7h
5.74
6.07
6.55
7.47
5.53
5.64
6.19
7.27
X
Cs. Uk
14.40
14.56
14.89
15.91
22.62
22.56
22.79
24.43
Table 24 (Non Contaminated Dataset) Absolute Values with Confidence Intervals by Language Pair for 1B and 8B Models, Late Contamination, Copies
IB Model
8B Model
Baseline
Targot
Full
Target
EN X
17.39
18.02
24.05
Ru
10.57
10.95
11.17
11.58
13.23
13.49
13.64)
14.81
Uk
11.40
11.46
10.99
11.42
18.93
18.94
18.96
20.40
12:38
12.90
17.61
15.33
17.60
17:25
16.35
17.82
Baseline
Source
Target
Full
Baseline
Source
Target
Full
5.74
5.48
7.01
5.53
5.72
5.31
X Y
De
17.39
17.31
17.18
17.83
21.01
24.18
24.40
25.56
CR.
14.40
14.55
14.30
15.37
22.62
22.46
22.73
24.06
10.71
13.23
14.99
Ul
11.40
11.51
11.38
11.76
18.93
18.86
18.99
19.90
Table II. WM (Non Contaminated Dataset) Absolute Values with Confidence Intervals by Language Pair for IB and 8B Models, Middle Contamination, 100 Copies
17.60
18.42
2h
5.74
5.95
6.03
6.33
5.53
5.37
5:71
6.00
X
Cs, Uk
14,40
14.37
14.56
14.96
22.62
22.52
22.86
24.21
Table 24 (Non Contaminated Dataset) Absolute Values with Confidence Intervals by Language Pair for 1B and 8B Models, Middle Contamination, 10 Copies
22

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
Pdt Life, 100s Ful. HUSE,, 100x
32.5
LPS 1004
MiGdA 1004
3000
MiAdo, 1004
27.3
BLEU SOORE
BLEU
1000R0 Training
Training Step
Traning Snep
UK
20
300a
PLL. Earty 180s
PULL Rarl)
Saveting
Bassiine
BIEU Score
12
Training Step
Thaining
Stamning Step
Figure 17. 24 BLEU scores through training for 100 Copies of Full contamination
KUL MEDEA 124
27
Fal. MadA 2D4
BLEU Score
BLEU
22
29
Training Step
Training Step
Training Step 2h
Pulk Earty,
Fal. Earyy
13
Fall, Earty,
Unitans,
Bacoting
22
Racolive
11
BLEU
32
140000
390000
140000
140000
120000
140000
560000
Training
Training
Tranning Step
Figure 18. WM BLEU scores through training for 10 Copies of Full contamination
23

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's s Impact Machine Translation
Latte Witdee
LANE
24
Mide
Scot
BLEE0
BLEU
100080 Training Step uk
Training
Traiiin Step 2th
Karty uniters,
guibors
0.0
BLEU
140000
30008
120000
140000
100000
120000
140800
160000
Training
Training
Trining
Figure 19. WM BLEU scores through training for Copies of Full contamination
24

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
F. Appendix: Impact of Full contamination on WMT'23 Language Pairs
IB Model
SB Model Source
Beseline
Source
Target
Full
Boseline
Tirget
EN
X
De
21.71
21.59
21.52
23.37
30.95
31.95
32.11
34.34
Ru
13:42
13.89
13.88
14.78
15.42
14.53
14.77
16.97
IB Model
8B Model
14.06
14.68
14.57
15.67
22.65
23.75
24.28
25.65
Baseline
Source
Target
Full
Baseline
Source
Target
Full
Uk
12.94
12.60
12.83
13.59
21.96
21.16
21.42
24.31
EN X
Hc
9.72
10.00
10.02
10.69
18.27
18.76
18.51
20.02
5.83
4,86
4.88
4.76
3.64
3,63
3.35
3.27
4.47
5.04
5.51
5.23
4.97
5.11
4.90
4.79
Ru
6.10
5.96
6.04
5.77
4.93
5.04
5.50
4.31
X
EN
Cs
7.30
7.26
7.35
6.86
4.50
4.19
De
33.87
UK
8.15
8.29
8.12
7.70
5.27
5,02
4.76
3.89
Ru
22.55
22.96
22.99
23,87
28.41
27.60
27.92
30.50
10.62
9.96
9.97
9.79
5.54
5.54
4.91
UK
20.33
Ja
10.27
10.54
10.40
9.85
5.67
5.17
5.27
4.59
He
25.05
25.14
24.41
26.68
38.62
37.53
37.88
40.99
X
11.31
11.34
12.70
19.31
De
7.34
7.08
7.11
6.67
5.21
5.09
5.11
4.37
Ju
4.97
5.22
5.53
5.83
10.42
9.89
10.32
12.40
9.61
8.76
8.71
8.40
6.97
7.13
6.94
5.59
1K
11.11
11.70
12.14
11.18
8.11
7.89
7.71
7.57
Cs
11.24
11.68
11.32
12,03
20.39
19:44
19.51
22.67
8.95
9.24
9.28
8.88
5.26
5.67
5.45
5.26
2h
7.86
8.17
7.55
7.50
5.52
5,34
5.51
4.73
Table 15. Absolute BLEU by Language Pair for IB and 8B Models, Late Contamination, 1 Copies
In
10.36
9.90
9.57
9.96
8.45
8.60
8.74
7.79
X Y-+Y
Cs, Uk
11.01
10.68
10.64
10.46
6.06
6.32
4.75
IR Model Target
8B Model
Table 12. Absolute MetricX (lower is better) by Language Pair for IB and 8B Models, Late Contamination, Copy
Baseline
Full
Target
EN X
De
21.71
21.82
33.09
Model
Model Source
Ru
13:42
13.84
13.76
16.93
15.42
14.25
15.08
22.56
Baseline
Source
Target
Full
Baseline
Target
Full
14.46
17.57
22.65
24.82
UK
12.94
12.28
13.44
16.19
21.96
21.84
23.40)
31.58
X
He
9.72
9.94
10.36
12.33
19.42
25.43
De
5.83
5.05
4.90
4.38
3.64
3.68
3,37
2.38
4.47
5.14
6.13
6.46
4,97
5.66
4.57
8.39
6.10
5.23
CA
7.30
7.14
7.16
6.08
4.50
4.65
4.44
3,94
EN
UR
8.15
8.18
7.90
6.63
5.27
5,08
3,10
De
26.46
26.11
26.43
29.96
33.59
33.69
34,71
42.60
He
10.62
9.59
9.84
8.95
5.54
5.30
5.24
3,98
Ru
22.55
23.03
22.99
26.67
28.41
27.41
28.07
34.38
Ia
10.27
10.22
9.85
8.85
5.67
5,43
5.12
3,51
UK
20.23
19.93
19:44
22.37
27.55
27.53
26.65
34.72
He
25.05
25.11
25.06
31,50
38.62
37.95
39,00
49.90
X EN
11.31
11.05
11.64
14.38
19.81
19:41
19.62
7.34
7.01
7.15
6.07
3.75
Jn
4.97
5.61
5.90
7.27
10.42
10.29
10.76
17.11
Ru
9.61
8.75
8.87
7.30
6.97
7-46
7.04
4.56
10.86
8.54
X->Y
Hc
8.95
8.93
9.55
7.63
5.26
5.49
5.46
4.94
C.
11.24
11.15
11.82
14.89
20.39
19.29
20.12
29.12
7,86
7.45
7.58
6.94
5.38
4.28
Ja 10.36
10.14
10.00
9.06
8.45
8.59
8.72
6.52
Table 16. Absolute BLEU by Language Pair for BB and Models, Late Contamination, 10 Copies
Cs. Lik
11.01
10.72
10.93
9.52
6.06
6.78
6.38
3,50
Table 13. Absolute MetrieX (lower is better) by Language Pair IB and 8B Models, Late Contamination, 10 Copies
Model Source
8B Model
Baseline
Targer
Full
Baseline
Source
Target
Full
EN
X
18 Model
8B Model
De
21.71
21.24
21.20
28.70
30.95
31.09
33.13
47.55
Bascline
Source
Target
Full
Bascline
Target
Ru
13:42
14.06
13.74
14.73
14.80
15.57
19.93
19.15
22.65
15.42
22.86
14.54
15.53
26.02
27.61
EN
X
Uk
12.94
12.24
13.70
18.83
21.96
22.18
24.02
40.30
5.83
9.72
13.61
20.23
Ru
6.10
6,00
6.12
4.50
4.93
5.06
4,88
3,01
Jn
4.47
4.67
5.92
5.98
4.97
6.71
4.48
9.87
7.30
7.10
7.19
4.50
UK
8.15
8.19
8.18
5.90
5.27
4,85
4.40
2.76
De
26.46
26.36
27.11
32,06
33.59
33.47
35.66
47.56
Ho
10.62
9.86
10.08
8.76
5.54
5.54
5.16
3.41
Ja
10.27
10.29
10.24
8.66
5.67
5.27
4,88
3,02
UK
20.23
20.59
20.20
23.11
27.55
27.57
27.68
40.71
X EN
He
25.05
24.65
24.74
34.11
38.62
36.72
39.94
59.93
7.34
7.11
7.01
5.60
5.21
4.98
3.56
Zh
11.31
11.39
11.37
15.96
19.81
19.16
19.62
32.56
Ru
9.61
8.80
8.85
7.19
6.97
7.01
6,80
4.28
4.97
5.15
5.77
8.82
10.23
11.34
22.94
11.11
12.25
11.78
9.96
8.11
8.22
7.82
5.00
X Y
He
8.95
8.86
9.08
7.42
3.26
5.56
5.46
3.12
CR.
11.24
11.35
11.75
17.63
20.39
18.88
21.23
37.34
7,86
7.50
7.84
Ja 10.36
9.84
10.02
8.58
8.45
8.44
8.36
6.32
Table 17. Absolute BLEU by Language Pair for 1B and 8B Models, Late Contamination, 100 Copies
Cs. Uk
11.01
11.10
10.68
9.36
6.06
6.75
5.84
3.04
Table 14. Absolute MetrieX (lower is better) by Language Pair for IB and 8B Models, Late Contamination, 100 Copies
G. Appendix: Search and Decontamination
For the search and decontamination step we search the tok- enized pre- treiiranning corpus with the bpe tokenized examples. We first split the evaluation examples into 8-grams and then
25

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
search each 8-gram. Once find all matches for all 8- grams in the pre-training corpora we extend each match to the left and right as much as can. Then we take the longer token match similar to Singh et al. (2024). Once we have the longest match we calculate the number of overlapping tokens. We use the overlapping percentage of tokens both source and target field to filter examples as contaminated or not.
Decontaminating these examples could be done by either removing the relevant examples from the training data or removing these examples from the test set. After analyzing the overlap statistics we choose to do the latter and decon- taminate our test set by removing these examples from the test set.
First we present histograms of contamination scores for all language pairs in WMT'23 23 and the three language pairs from FLORES that we use in our testing. Here we observe that language pairs from FLORES have more contamination compared to language pairs from 23.' Second we observe that contamination in 2M seems to be mostly partial. While the relationship with partial contamination and performance is still studied, for practical reasons we followed an pre-established cutoff of 0.7 that was introduced by Chowdhery et al. (2022).
Search Results for WMT23 and Flores Test Sets
Below Threshald (1 1064)
When we check for the threshold in any field source or target and analyze the test sets we see in Figure 20 that the actual examples that are contaminated are around 10% of the test data where most of these examples are what we can call single field coniiiiinttiooooonmmnnntttoo in just source just target) There is small subset of examples where we see both the source and the target is contaminated. When we analyzed this subset we observed that they are generally simple sentence structures that can frequently occur on the internet. In Table 18 we are presenting some examples that are in this group.
AS Data (12527)
Abavel Iesesood(ATT (14633
Target AT
Soutcee 2T (425)) Both AT (60)
Figure 20. Riverplot of clean, contaminated and different forms of
contamination for the threshold 0.7.
We also check the 24 test data and the contamination level with our main corpus and find that 162 examples are contaminated in their source field above the threshold 0.7 and we remove these examples from the WMT'24 24 test data that we use for our experiments.
26

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's s Impact Machine Translation
Cs-uk
de-en
en-cs
375
250
390
300
Count
330
04
0.3
0.4
0.0
04
0.3
en-de
en-he
en-ja
150
300
Count
350
230
300
68
en-ru
en-uk
en-zh
300
250
300
Count
200
300
200
150
300
04
8.0
0.0
0.4
he-en
ja-en
ru-en
340
200
Count
100
0.
0.
uk-en
zh-en
ace-Arab-en
Count
150
200
100
300
0.
0.4
8.0
0.0
04
10
wo-en
yo-en
230
250
0.5
Count
150
100
0.4
0.
Figure 21. Histogram of contamination scores. Here the scores calculates max(s (Ssouven, Starget) where Scouree is the percentage of overlapping tokens with the longest contamination that contains any tokens from the source field and Storget similarly for target.
27

---

Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact Machine Translation
Indx
Source
Target
schwarze, lange Haare mit Pferdeschwanz
long, black hair with ponytail
2
Sieben Leichtverletzte nach Auffahrunfali auf A5 Schlossplatz Stuttgarter gedenken der Opfer der Erddbebenkatastropht
Seven slightly injured in rear-end collision A5 Schlossplatz Stuttgart commemorate the victims of the earthquake disaster
3
4
3D Drucker Prusa i3 DIY
3D Printer Prusa i3 DIY
5
Photos from my visit to Ghana in 2011.
Fotografie z mé návštěvy Ghany roce 2011. Byla to katastrofa pro Apple Maps roce 2012? OLIVIER DOULIERY AFP prostiednictvím
6
Was it the 2012 Apple Maps disaster?
OLIVIER DOULIERY/AFP via Getty Images
služby Getty Images
8
For more information our Privacy Policy.
Dalăí informace naleznete našich Zásadách
ochrany soukromí.
9
First medical aid is also provided on round-the- clock basis.
První lékaïská pomoc je poskytována nepřetršité.
10
Follow Metro Sport for the latest news on Face- book, Twitter and Instagram.
Sledujte Metro Sport a jeho nejnovější zprávy na Facebooku, Twitteru a Instagramu.
11
This is the best way earn the press's trust. Follow him on Instagram: @awr hawkins. Reach him directly at awrhawkins @ @breitbart.com.
Ten nejlepší způsob, jak si získat dúvěru tisku. Sledujte ho na Instagramu: @awr hawkins.. Obratte
12
13
přímo awrhawkins wrrriinttteeeebbb...oommm Soud mu uložil devítimésíční podmíněný trest pokutu.
něj adrese
14
It handed him a nine-month suspended sentence and a fine.
15
give this book 10 stars!
Této knize dávám 10 hvězdiček!
16
Good, but would like find something better Good umbrella, would buy again if had to
Dobré, ale chtelo by to najít nèco lepšího
17
Dobrý deštník, koupil bych si ho znovu, kdybych musel
18
but it happens enough to be annoying.
ale stává se to docela často na to, aby to bylo nepříjemné.
19
Nothing like the previous Stylo phones, MASSIVE DISAPPOINTMENT.
Nic jako předchozí telefony Stylo, VELKÉ ZK- LAMANÍ.
20
Some Super Bowl Commercials Can't Wait to See
Einige Super Bowl-Werbespots die ich unbedingt sehen will
Table 18. Examples where hoth and target contaminated above the 0.7 threshold.
28


Reviews:
[{"review_id": "BBNZqaneYa", "reviewer": "Reviewer_7tAa", "summary": "The paper analyzes the influence of data contamination on LLMs trained for machine translation.  Their testing controls factors such as the modes of contamination, the temporal distribution of contaminated samples, and the frequency the contaminated samples are presented.  From their experimentation they demonstrate that (1) parallel data contamination provides a much larger impact than monolingual data contamination, (2) uniformly distributed data contamination has the most persistent impact, (3) larger models are more influenced by data contamination, and (4) data contamination is most prevalent once the model already has a language understanding capability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and feedback. We respond to the reviewer’s comments below. \n\n1. Question 1 (training compute): All models are implemented as continued pre-training starting off baseline checkpoints. For instance, models with late contamination are only trained for 10% of the entire training (given we kickstart the model from the 90% of the baseline’s mark). In practice, this means that our checkpoint branching approach reduces the compute requirements about 60% (compared to the compute budget we would have spent, if all models were trained from scratch.\nThe experiments being expensive is definitely a challenge however we have undertaken this cost for the community and shared our findings openly that can answer critical questions in this area as well as inform future research. We think it is an important contribution of this paper that we have committed to rigorously experimenting the impact of contamination given the computational requirements and not a weakness. \n2. Question 2 (generalization to different architectures): We acknowledge that our findings are limited to decoder-only LLMs, which constitute the predominant LLM paradigm in recent years. \n3. Question 3 (frequency of contamination in pre-training):  While in principle, pre-training data deduplication is a common practice; in multilingual settings, upsampling under-represented languages is an equally common tactic. The latter is dictated by the lack of resources for many languages, and inevitably can lead to data being repeated more than once. As a result, it is reasonable to assume that once a test instance bleeds into pre-training data, it might end up seen by the model more than once.\n4. On the novelty of our findings: The more intuitive result, i.e., contamination leads to performance overestimation, has been hypothesized by many works; however, which data conditions contribute to this overestimation and to what extent, has not been studied by prior work. Our study contributes empirical evidence that contamination could, but not always lead to performance overestimation, a behavior that depends on various factors ranging from the way contaminated test sets are presented in the pre-training data, to how well the contaminated languages is represented in the pre-training distribution (Section 5 and 5.1). \n5. On the choice of Machine Translation (MT) test sets: MT test sets naturally let us study how data contamination interacts with pre-training resource requirements (which are different across the many languages we studied with this work). As evidenced from our findings, whether contamination has a measurable impact on downstream performance does connect to the language’s representation during pre-training.", "claims_and_evidence": "Yes, claims are supported by evidence, but they would be more convincing if a wider range of models were tested on.  (Not a big deal as I understand this would require an incredible amount of compute and would most likely lead to the same outcomes.)", "methods_and_evaluation_criteria": "Yes, the methods and evaluation make sense for the problem.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, experimental design and analysis is sound.", "relation_to_broader_scientific_literature": "The paper contributes a thorough analysis of the influence of data contamination for machine translation LLMs.  This is a prevalent issue that has not been explored to the same level in a prior publication.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1) Propose a checkpoint-branching scheme that reduces variance and increases computational cost of their experimentation.\n2) This is the first paper that provides experimentation with data control that explores the influence of data contamination for LLMs when they are applied to machine translation.\n\nWeaknesses:\n1) The experimentation required an extensive amount of computation and there is a lack of documentation outlining the amount of compute required in case future people want to replicate the methodology for alternative seq2seq tasks.\n2) A lot of the conclusions drawn from experimentation were not incredibly insightful.  For instance, (1) and (3) are already known by the community.\n3) It does not seem that testing the influence of the number of copies of the test set in the training data is entirely necessary.  It seems unlikely that the training data would contain 10-100 copies of the test set ever.  Unless I am misinterpreting something these results appear as though they were included to exaggerate the influence of data contamination on machine translation.  Is this something that would be seen in real world data?\n4) The conclusions were made on a single architecture, and it is unclear if the conclusions would extend to alternative architectures.\n5) The testing is only done for machine translation, and it does not explore if similar results would hold for alternative seq2seq problems.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1) What is the overall amount of compute required for all experiments in terms of GPU hours? How many models were trained from your checkpoint branch?\n2) Do you think similar trends would carry over to alternative architectures?\n3) Are there any circumstances where there might be 10-100 copies of the test set in the training set?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jEedE17dxP", "reviewer": "Reviewer_J6EW", "summary": "This paper presents a controlled large-scale study on how data contamination impacts machine translation evaluation in large language models. The researchers created a carefully decontaminated train-test split, then systematically reintroduced contamination under controlled conditions across different modes, temporal distributions, and frequencies on both 1B and 8B parameter models.\n\nThe key findings of the paper shows that the full contamination mode (where both the source and target text of the test set exist) can dramatically inflate performance metrics (up to 30 BLEU points for 8B models). Corroborating previous study, contamination effect is more evident when it is introduced later during training, but the paper also found that uniform contamination leads to the most significant effect. Last but not least, contamination requires a sufficient language representation to have a measurable effect. These findings highlight the critical importance of properly decontaminating evaluation benchmarks to avoid substantially overestimating model capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review and insightful comments. We will use the reviewer’s feedback to revise the writing of Section 3.4 and include missing citations. \n\n1. Details on contamination score calculation: The contamination score is calculated as the number of tokens in the longest overlap, over to the number of tokens in the eval example. If more than 70% of the tokens are contained in the longest matching overlap we label an example as contaminated and remove it from our test set.  \n2. Choice of WMT23/24 datasets: The WMT23/24 datasets used in our experiments are the standardized datasets, established by WMT (Conference on Machine Translation) annual conference. \n3. Contamination from WMT’24: Source segments from WMT’24 are generally standard pieces of texts that are frequently found in news articles or general webtext. Since WMT sources input segments from online sources, it is possible that frequently used sentences might be found in any pre-training data. For this reason, we make sure that we run decontamination tests for all datasets we use. \n4. On the ranking of Figure 2: The presented methods are sorted based on the average performance across language pairs. They are shown in descending order for both Figs 2 and 3. We will clarify this in the next version of the paper. \n5. Checkpoint branching details: In our branching method, each branch is effectively a copy of the baseline model, with all the optimizer and meta data inherited from the checkpoint. As training continues, the only difference is the pre-training mixture that the new model (continued pre-training) is exposed to. This mixture is a copy of the baseline one with random examples replaced by the contaminated instances. Our setup covers 42 contamination experiments, leading us to 42 “branches”.  We will include those details in the next version.", "claims_and_evidence": "Following the breakdown from the introduction of the paper:\n\n- Contaminating source-target MT pairs inflates performance on those test sets. -> adequately supported\n- The temporal distribution of contamination matters. -> adequately supported\n- The impact of contamination increases with model scale. -> supported with limitation (only data points for 1B & 8B is supplied), but acknowledged in the paper\n- Contamination requires sufficient language representation to have a measurable effect -> adequately supported", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I checked everything that's presented in Section 3 and 4 carefully. While I don't spot issues with what's presented, I find the level of details given for checkpoint-averaging a little lacking. See more details in \"Other Strengths And Weaknesses\".", "relation_to_broader_scientific_literature": "There are existing work that looks at membership detection of certain data point into a models pre-training data [1][2][3][4], each of which has established some sort of detection algorithm: min k% prob [1], membership inference attacks [2], and bloom filters [3], infini-gram [4]. There are also papers that examine the impact of data contamination on fair LLM evaluation [5][6]. However, to the best of my knowledge, none of them has studied systematically how introducing data contamination in the pre-training data in different ways can introduce different effects in the resulting model, so I think the novelty of the findings in this paper is significant.\n\n- [1] https://arxiv.org/pdf/2310.16789\n- [2] https://aclanthology.org/2020.tacl-1.4.pdf\n- [3] https://arxiv.org/pdf/2303.03919\n- [4] https://arxiv.org/pdf/2401.17377\n- [5] https://aclanthology.org/2023.findings-emnlp.722.pdf\n- [6] https://aclanthology.org/2024.findings-acl.716.pdf", "essential_references_not_discussed": "The paper has already cited [1]. I would recommend also discussing [2] because they are tackling a similar data contamination problem from a different (the system user's) angle, even though it predates the boom of LLMs. [3] and [4] should also be discussed in the context of alternative decontamination methods, which is something I would like to see added (more details in \"Other Comments Or Suggestions\").", "other_strengths_and_weaknesses": "This is a high-quality controlled study on the effect of data contamination on the inflation of machine translation evaluation result. The problem studied here has significant real-world implications and is of value to other tasks other than machine translation as well. The presentation quality of the paper is high, with the core methods and findings very clearly presented.\n\nThe main complaints I have about the paper is the lack of details in the checkpoint branching method (Section 3.4). This is the most significant experimental design choice of this paper, and yet is very briefly glossed over. This is not at all sufficient for subsequent studies to reproduce their results.", "other_comments_or_suggestions": "I would like to see two main things added in the next draft.\n\n### More details on checkpoint branching\n\nI would like to see Section 3.4 significantly re-written to better present this method.\n\n* Start with the motivation. Stress what you have given up compared to multiple full re-training runs (essentially, you have fixed initialization seed and data sampling order), and what you get out of this (re-use some compute from other runs).\n* When you introduce a \"branch\", how do you implement it? Do you just add extra samples to the training data? What about learning rate scheduler?\n* How do you introduce uniform contamination?\n* In total, how many \"branches\" did you have to introduce for one set of experiment? (e.g. for 1B, single-copy, full contamination)\n\n### More discussions on the choice of decontamination method\n\nIn its current shape, the paper only mentions their choice of decontamination method in Appendix G. I think this is an important experiment design that is worth talking in the main paper as well. Moreover, this should be treated as an intentional choice, since what this paper adopts is not the only contamination detection/decontamination method out there (see [1][2][3][4] above). The paper should acknowledge these other methods and discuss how the other methods relate to their method.\n\nI also spot a few minor typos/style comments:\n\n* L74 left col. -- \"up to 60 BLEU points\", not 30?\n* L83 right col. -- \"previous works\" -> previous work\n* L85-86 right col. -- use -> used\n* L205-216 left col. -- for readers that's not familiar with how WMT datasets are set up, probably give some justification for why you take 2023 as contaminated dataset and 2024 as non-contaminated dataset? (But then you also report in Appendix G that there are also some contamination in 2024 dataset, which I'm not sure how it happened?)\n* Figure 2 -- I'm confused about how the methods are ranked? Specifically, there seems to be some methods below \"baseline\" that's doing worse than baseline? (Same with Figure 3)", "questions_for_authors": "Please clarify the following:\n\n* The questions I listed in \"Other Comments Or Suggestions\" about checkpoint branching\n* The ranking of setups in Figure 2\n* What does \"threshold of 0.7\" mean in Appendix G (L1343 and L1359) -- basically, here is the relevant quote I found from the section 8 of the PALM paper that was cited:\n\n> \"So, we were able to split each dataset into a “contaminated” and “clean” subset based on whether at least 70% of the 8-grams in question, prompt, or target were seen at least once our training data.\"\n\nbut I was confused about how exactly it is computed, and how it related to what's been discussed there.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "81aJ3EthJ2", "reviewer": "Reviewer_x73P", "summary": "This paper investigates the impact of data contamination on machine translation. In particular, the paper tests factors including source contamination, target contamination and temporal distribution.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your review of our paper.\n\nFirst, we respectfully disagree that the contribution of our paper is insignificant. The key assumption of data contamination is motivated by the hypothesis that consuming test data leads to an overestimation of a model’s performance. While intuitive, this hypothesis has not been rigorously tested at this scale, and our work aims to fill that gap in the literature. Further, we suspect many researchers care about testing this hypothesis but are unable to run the experiments that we did due to the computational cost. Therefore, we feel our contributions are significant and our findings are very relevant to the research community.\n\n1. There are methods for detecting contamination by looking at the overlap between the training and test sets, including https://arxiv.org/abs/2411.03923. Our train-test decontamination stage is inspired by the findings of this paper. Our implementation is very efficient because it’s highly parallelizable and uses a Bloom filter combined with an exact search to search more than 100k 8-grams from the test data in trillions of training tokens in less than a day.\n\n2. There are papers focused on finding contamination with just access to model activations. However this task is shown to be harder than initially expected (https://arxiv.org/abs/2402.07841.) Our experiments and findings regarding contamination in section 5.4 can partially highlight why this task can be harder than initially hypothesized. We show that models don’t naively memorize text in the pretraining data suggesting that model behaviour might not be as easily separable between what is in the pre-training data and what is not. Recent work also shows that models can verbatim generate text that is explicitly not in their pre-training data https://arxiv.org/abs/2503.17514. This suggests that models learn more general representations of language and understanding contamination and the impact on evals requires understanding knowledge formation and access in large language models.", "claims_and_evidence": "The claims are supported by abundant experiments from multiple data source.", "methods_and_evaluation_criteria": "No significant flaws in method and evaluation.", "theoretical_claims": "Not applicable for this paper.", "experimental_designs_or_analyses": "No significant problems in experiments or analyses.", "relation_to_broader_scientific_literature": "This paper extends the study of data contamination to the field of machine learning.", "essential_references_not_discussed": "No missing reference found.", "other_strengths_and_weaknesses": "Strength: Abundant experiment to support the claim\n\nWeakness: Although it conducts a thorough investigation on the impact of data contamination in machine translation, there is no suggestion on how to resolve this issue. Significance can be further improved by e.g., methods to detect data contamination in the pretraining data, or method to detect data contaminated pretrained models.", "other_comments_or_suggestions": "See the previous comment.", "questions_for_authors": "1. Is there an efficient way to detect data contamination in the pretrained dataset, i.e., overlap between pretrained and test data?\n2. Is it possible to determine whether there is data contamination, if you only have access to the pretrained model and test dataset, but no access to the pretrained data? In other words, is there a clear separation between the performance of a model pretrained with non-contaminated data, and that of a model pretrained with contaminated data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHY4lpzr8a", "reviewer": "Reviewer_QNfP", "summary": "This paper presents a controlled study on the impact of data contamination on machine translation. They decontaminate their train-test splits and train two decoder-only models of different sizes (with 1 and 8 billion parameters). Then, they add test data into the pretraining data and train a contaminated model branching out from the baseline checkpoint. Finally, they compare the relative performance of the contaminated and the baseline model on contaminated and non-contaminated data. Their main findings are as follows:\n\n(i) contaminating with both source and target sides leads to substantial performance inflation on those test sets;\n\n(ii) partial contamination (source-only or target-only) leads to smaller/less consistent inflation;\n\n(iii) the temporal persistence of contamination matters;\n\n(iv) contamination has more impact on larger models;\n\n(v) contamination requires sufficient language representation to have a measurable effect.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive review.  We will extend the discussion section to include a discussion on the broader implications of our findings. \n\nOn the choice of BLEU as a metric: We acknowledge that a string-based metric (like BLEU) has limitations. For this reason, we already accompany all of our evaluations with an additional learned metric (i.e., MetricX) which has been widely adopted by the field. All drawn conclusions are consistent across both reported metrics. Our choice to surface BLEU in the main paper is based on its wider adoption within the ML community.", "claims_and_evidence": "Their main claims/findings are summarized above (i-v); they are all well-supported through experiments. (iv) seems to hold in practice but this is only tested for two model sizes (this is acknowledged in the paper).", "methods_and_evaluation_criteria": "The method seems appropriate. Using branching to save computation sounds reasonable. It would be nice to explore other types of contamination (eg., paraphases) but this is not really needed. However, I can’t understand the reason for drawing conclusions based on BLEU scores. Its issues are well known, check for instance [1, 2]. The authors include Metric-X results in the Appendix, so my question is:\n\n(1) Is there a reason to report BLEU in the main paper and Metric-X results only in the Appendix?\n\n[1] Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (Freitag et al., WMT 2022)\n\n[2] Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent (Freitag et al., WMT 2023)", "theoretical_claims": "No proofs are presentend but I think none are needed (the paper focuses on empirical findings).", "experimental_designs_or_analyses": "The experimental design is sound. In particular the authors searched and found that ~10% of test set examples were already contaminated (Section 3.1). This shows the importance of their decontamination step (the first step in Fig. 1), which is missing in previous work according to Table 1.", "relation_to_broader_scientific_literature": "Table 1 provides a good summary, but I think this is the first work doing this analysis for machine translation.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "In addition to what I pointed out above, the paper is exceptionally well-written and is easy to follow; the analysis setup is sound, and most of their choices are well-justified. The main weakness is the reliance on BLEU as the main evaluation metric to draw conclusions. I’d like to see some discussion justifying this choice.", "other_comments_or_suggestions": "Minor comments:\n\n- Typo in L208 (missing white space after “language pairs:”)\n- It would be valuable to expand a bit the discussion on the broader implications of these findings, eg., how to adapt existing evaluation protocols/benchmarks to mitigate this issue?", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of data contamination in LLM-based MT on two model sizes (1B and 8B scales) through controlled experiments. The results suggest that contamination with both source and target substantially inflates BLEU scores, while source-only and target-only contamination generally produce smaller, less consistent over-estimations. 

While the results are mostly predictable and the novelty is fairly limited, the reviewers agree that this is a solid empirical paper whose findings are valuable to the research community to better understand the phenomenon of data contamination. I therefore recommend acceptance.
Decision: Accept (poster)