The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Authors: Yifan Sun
Han Wang
Dongbai Li
Gang Wang
Huan Zhang
Venue: ICML 2025 poster (2025)
Keywords: Data Contamination
Large Language Models
Evaluation
Primary area: deep_learning->large_language_models
TL;DR: We design a systematic and controlled pipeline along with two novel metrics, fidelity and resistance, to provide a fine-grained and comprehensive assessment of existing Benchmark Data Contamination mitigation strategies for Large Language Models.

Abstract: Benchmark Data Contamination (BDC)—the inclusion of benchmark testing samples in the training set—has raised increasing concerns in Large Language Model (LLM) evaluation, leading to falsely inflated performance estimates and undermining evaluation reliability. To address this, researchers have proposed various mitigation strategies to update existing benchmarks, including modifying original questions or generating new ones based on them. However, a rigorous examination of the effectiveness of these mitigation strategies remains lacking. In this paper, we design a systematic and controlled pipeline along with two novel metrics—*fidelity* and *contamination resistance*—to provide a fine-grained and comprehensive assessment of existing BDC mitigation strategies. Previous assessment methods, such as accuracy drop and accuracy matching, focus solely on aggregate accuracy, often leading to incomplete or misleading conclusions. Our metrics address this limitation by emphasizing *question-level* evaluation result matching. Extensive experiments with 10 LLMs, 5 benchmarks, 20 BDC mitigation strategies, and 2 contamination scenarios reveal that no existing strategy effectively balances fidelity and contamination resistance. No semantic-preserving strategy yields a significant improvement in resistance over the vanilla case (*i.e.*, no benchmark update) across *all* benchmarks, while semantic-altering strategies sacrifice fidelity for resistance. These findings underscore the urgent need for designing more effective BDC mitigation strategies. Our code repository is available at https://github.com/ASTRAL-Group/BDC_mitigation_assessment.

The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Yifan Sun Han Wang"
Dongbai Li
Gang Wang'
Huan Zhang'
Abstract
1. Introduction
Benchmark Data Contamination (BDC)tth inclusion of benchmark testing samples in the training set-has raised increasing concerns in Large Language Model (LLM) cvaluation. leading falsely inflated performance estimates and undermining evaluation reliability. To address this, researchers have proposed various mitigation strategies to update existing bench- marks, including modifying original questions or generating new ones based on them. However, a rigorous examination of the effectiveness of these mitigation strategies remains lacking. In this paper, we design systematic and controlled pipeline along with two novel fidelity and contamination resistance-to provide a fine-grained and comprehensive assessment of existing BDC mitigation strategies. Previous assessment methods, such as accuracy drop and accuracy matching, focus solely on aggregate ac- curacy, often leading to incomplete or mislcading conclusions. Our metrics address this limitation by emphasizing question-level evaluation result matching. Extensive experiments with 10 L.LMs, 5 benchmarks, 20 BDC mitigation strategies, and contamination scenarios reveal that no existing strategy effectively balances fidelity and con- tamination resistance. No semantic-preserving strategy yields significant improvement in resistance over the vanilla case (i.e., bench- mark update) across all benchmarks, while semantic-altering strategies sacrifice fidelity for resistance. These findings underscore the urgent need for designing more effective BDC mitigation strategies. Our code repository is available www.oo///SSAa.... Group/ BDC mi tnc_miiiiatiinnn_ooo www.miiiiiiitoosssssssssee
Benchmarking Large Language Models (LLMs) has re- cently become critical area of focus (White al., 2024; Xia et al., 2024; Guha et al., 2024; Zeng al., 2024; Lin et al., 2024; Ni et al., 2024a). driven by the rapid increase in their number and capacity (Achiam et al., 2023; Dubey et al., 2024; Team, 2024b; Team et al., 2023; 2024). Reli- able and high-quality evaluation benchmarks are essential to provide comprchensive and accurate assessments of LLM capabilittes. However, modern LLMs are trained on vast amounts of web-scraped data, concerns have emerged re- garding benchmark samples inadvertently appearing in their training sets. Consequently, it is challenging to determine whether the model just simply memorizes answers to dif- ficult test questions to achieve a better performance (Oren et al., 2023; Zhu et al., 2024b). This phenomena, known as Benchmark Data Contamination (BDC), results in falsely inflated performance metrics, thereby undermining the reli- ability of evaluation conclusions (Zhou et al., 2023; Sainz. et al., 2023; Dominguez-Olmedo et al., 2024).
Synonym Replacement:
At what tempereture
At which benchmark test sample Aro-Chalenge
Eveluction Analysis Extension
How does the presence of impurtiees and atmospherii infuence the freezing point?
Figure I. Illustration of BDC mitigation strategies. BDC miti- gation strategies, such as synonym replacement and analysis ex- tension (Ying al., 2024), update benchmark questions to reduce the risk of direct memorization.
To mitigate BDC, creating new benchmark datasets from scratch is potential solution, but this process is often pro- hibitively expensive and labor-intensive! Moreover, some existing benchmark datasets, such as MMLU (Hendrycks et al., 2020) and GSM8K (Cobbe et al., 2021), are already
https:///itthie
Equal contribution 'University of Illinois Urbana-Chaan Champaign. Correspondence to: Yifan Sun <yifan50@illinoti idd Huan Zhang <huan @ huan- hwan-zhang.cee
For instance, curating the GPQA datasct (Rein et al., 2023), which contains 448 multiple-choo choice questions written by domain experts, required over $120,000 (Rein, 2024). Similarly, the re- cently introduced HLE benchmark (Phan et al., 2025) has allocated $500,000 to collect high-quality henchmark questions.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Scenario
Acc.
Scenario
Acc.
Q4
Contaminated
80%
How much drop is good?
Clean
40%
migiiinttgttoion really good?
Mitigated
30%
Mitigoted
40%
(a) Accuracy drop
(b) Accuracy matching
Not matching!
The limitations of existing approaches for assessing BDC mitigation strategies. (a) Accuracy drop measures the performance decline between contaminated accuracy and mitigated accuracy, but does not account for the clean accuracy, making it unclear how much drop indicates effective mitigation. (b) Accuracy matching requires that the mitigated accuracy restores clean accuracy. However, as shown in the example, even when the accuracies match, the question-level evaluation results differ significantly (e.g., correctly answering the 1st and 2nd questions versus the 4th and 5th). This discrepancy suggests that the updated benchmark may evaluate different aspects of model capacity compared to the original benchmark. As a result, the mitigation strategy may fail to preserve the original benchmark's s evaluation objective and could be ineffective.
Figure2. 2.
metric. Focusing solely on matching the scalar accuracy is not sufficient and can even be misleading. For example, in the case shown in Fig. 2(b), even if scalar accuracy aligns, the strategy fails to recover the clean question-wise eval- uation results. Consequently, the mitigation strategy may alter the original benchmark's 's evaluation objective, putting its effectiveness into question.
Table 1. Definition of different evaluation scenarios based on the contamination status of the LLM and the benchmark version used.
Scenario
LLM
Benchmark
Cleun
Uncontaminated
Original
Contaminated
Contaminated
Original
Mitigated
Contaminated
Updated
of high quality and accurately reflect real-world question distributions within their respective domains. Rather than retiring such well-established benchmarks, ongoing efforts aim to update them or generate new questions based on these benchmarks to mitigate BDC (Zhu et al., 2023b; 2024a;b; Ying al., 2024). For example, straightforward approach is to paraphrase original questions, reducing the risk of models naively leveraging memorized answers.
In this paper. we present a comprehensive and rigorous framework for assessing BDC mitigation strategies (Fig. 3). We identify two key desiderata for an effective strategy: (1) Fidelity: For high-fidelity strategy, if the clean LLM answers the original question correctly, it also answers the updated question correctly; if it fails on the original question, it also fails on the updated version.
(2) Contamination Resistance: For a contamination-resistann strategy, even if the LLM has been contaminated by the original dataset, its ability to answer each question in the updated benchmark remains unchanged.
Our Research Question
Each BDC mitigation strategy yields an updated benchmark. We focus on a thorough and rigorous examination towards the effectiveness of different BDC mitigation strategies.
By employing the normalized Hamming distance and jointly evaluating these metrics, our framework emphasizes question-wise matching, offering a fine-grained and multi- faceted assessment of mitigation strategies.
However, it is crucial to assess the effectiveness of differ- ent BDC mitigation strategies systematically. For example, whether surface paraphrasing can indeed alleviate the effects of BDC is under question. Nevertheless, current practices for assessing BDC mitigation strategies have clear limita- tions, as illustrated in Fig. 2: (a) Accuracy drop. Some previous studies regard a mitigation strategy as successful if the contaminated LLM''s ' accuracy on the updated bench- mark (i.e., mitigated accuracy) is lower than its accuracy on the original benchmark (i.e., contaminated accuracy) (Zhu et al., 2024a). However, without referencing the model's performance on the original benchmark before any contami- nation (i.e., clean accuracy), it is unclear how much of a drop is meaningful. (b) Accuracy matching. Other works assess mitigation strategies by comparing clean accuracy with mit- igated accuracy, expecting them to match (Zhu et al., 2023b; 2024b; Ying et al., 2024). Yet, accuracy is only an aggregate
Our main contributions are summarized as follows:
We identify the limitations of existing approaches for assessing BDC mitigation strategies and propose two novel metrics, fidelity and contamination resistance ($3).
We design a scientific and controlled pipeline to assess BDC mitigation strategies. Different from previous stud- ies, extensive checks are performed to confirm that each LL.M-henchmak pair is uncontaminated prior to manual contamination, ensuring the validity of clean evaluation re- sults. Two contamination recipes that simulate real- real-world data contamination scenarios are examined ($4).
Through experiments with 10 LLMs, 5 benchmarks and 20 BDC mitigation strategies, we find that none of the ex- isting strategies achieves strong fidelity and contamination resistance simultaneously. While some semantic- semantic-preserving strategies offer statistically significant improvements in re-
2

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Pick uncontaminated LLM--benhhmee pair.
Apply each mitigation strategy to benchmark.
Vidate
Must pass All BDC detection methods:
Mitigation
LLM
Token prabrbbbbtbbbbbkooeee.e
Genarenion-basee IS Guuusiinn
(e.g. paraphrasing)
Order bosed Sharded Ronk Compartion Test
Original Benchmork
Updatedd Benchmark
Contaminate LLM original benchmark.
Obtain evaluation vectors.
3
4
Q2
Q3
Q5
Evaluate
Fine-tuning
LLM
LLM
Evaluate
Uncontominated LLM
Contaminated LLM
Original Benchmork
(c)
Evaluate
Derive Fidelity Contamination Resistance.
5
Strotegles
Fidelity (Evah
Eval
matching(
Resistance
(c)
(b)
1/5
Update
Uplate
Resistance = matching(Evai((
Eval
= m
Vendlla (No miliiioo)
(b)
(c)
3/5
Fidelity
Figure 3. Overview of our pipeline for assessing BDC mitigation strategies: (1) We select LLM- benehmark pair and ensure it passes three BDC detection methods to confirm is uncontaminated, a crucial step for reliable "clcan" evaluation results ($4.2). (2) Each mitigation strategy is applied separately the original benchmark produce updated benchmark; 20 strategies are examined total ($4.3). (3) The uncontaminated LLM is fine-tuned on the original benchmark dataset. Two contamination recipes (mild and intensive) are tested to ensure robust conclusions and three validation checks performed confirm the effectiveness of the contamination process ($4.4). (4) Evaluation vectors are computed for: (a) uncontaminated L.MM with the original benchmark, (b) uncontaminated LLM with the updated benchmark, and (c) contaminated LLM with the updated benchmark ($4.5). (5) Fidelity and resistance derived based the degree of matching between these evaluation vectors ($3). An effective mitigation strategy should achieve high scores in both metries.
sistance over the vanilla case (i.e., no benchmark update) on certain benchmarks, none consistently yields such gains across all benchmarks. On the other hand, semantic-altering strategies compromise fidelity for resistance. These findings highlight the need for designing more effective mitigation strategies ($5).
Reneration-basee detection methods prompt the model to predict information that should not be inferable from the input (Deng et al., 2023; Golchin Surdeanu, 2023b;a; Chang al., 2023). For instance, TS TS-guessing checks the model correctly predict the content of masked incorrect choice. Accurate predictions suggest prior expo- sure to the instance. (3) Order-based detection methods focus on the tendency of models to memorize the order of samples and options, identifying models as contaminated it exhibits a strong preference for the original sequence over permutations (Oren et al., 2023; Ni et al., 2024b).
2. Related Work
BDC Detection. This line of research focuses on detecting BDC and flagging specific model-benchhart pairs where contamination may be present. With access to the train- ing corpus, contamination can be detected through n-gram ILM-as-a-judee (Yang et al., 2023). However, access to the training corpus is often unrealistic (Ravaut et al., 2024). Black-box meth- ods, which do not require such access, can generally categorized into three types: (1) Token probability- detection methods leverage predicted token probability distributions (Zhang al., 2024b; Dong al., 2024; Ye et al., 2024; Yax et al., 2024). For example, Min-K% Prob (Shi et al., 2023) flags contamination if the model as- signs unusually high logits to the lowest K% of tokens. (2)
BDC Mitigation. Existing research seeks to mitigate the impact of BDC through two primary strategies: curating new benchmarks and updating existing benchmarks (Xu et al., 2024). Recent works have proposed novel benchmarks to address contamination (Li al., 2024b; Zhu al., 2023a; Jain et al., 2024; Li et al., 2024a; Qian al., 2024; Wu et al., 2024; Zou et al., 2024; White et al., 2024). While effective, this approach is costly and time-intensive, requiring signifi- cant human effort for labeling and maintenance. An alter- native strategy focuses on updating existing high-quality benchmarks, maximizing the utilization of well-established
overlap (Brown et al., 2020)
3

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
benchmarks while being more cost-effective and automated. Some methods modify evaluation samples while preserv- ing their semantics (Zhu et al., 2024b;a; 2023b; Li et al., 2024c; Wang et al., 2021; Xia et al., 2024; Haimes et al., 2024; Zheng et al., 2024). Others generate new samples with altered semantics based on original questions, using advanced LLMs (Ying et al., 2024). However, the latter case, the quality of generated samples is often limited by the task-specific capabilities of the underlying LLMs used in the generation process.
corresponding updated question correctly (or incorrectly). Formally, the evaluation vectors on D and DS for the clean model M should match:
R(M, D) R R(M.
It is important to clarify why high fidelity is necessary. Low fulelity does not necessarily mean the updated benchmark is of poor quality. Rather, it signals that the updated bench- mark has undergone excessive modifications relative to the original, which can introduce two practical issues: (i) Dif- ficulty or objective drift: The updated questions may no longer be appropriate for LLM evaluation. They could be- come too difficult, too trivial, or shift focus to unintended skills or knowledge domains. This requires human annota- tors not only to provide revised answers but also to assess whether the benchmark remains suitable for evaluation. (ii) Answer invalidation: The modifications may alter the se- manties of the questions such that the original answers longer correct, necessitating manual verification to en- sure correctness. In either case, such a strategy can no longer be considered as a fully automated mitigation strategy due to the need for manual post-hoc inspection.
3. Method
We focus exclusively on BDC mitigation strategies that up- date existing benchmarks, since introducing entirely novel ones can be difficult automate and incurs high costs. With- out clear and thorough understanding of how well these mitigation strategies work, benchmark developers and eval- uation practitioners risk making unnecessary changes to existing benchmarks that fail to actually reduce the impact of BDC. In this section, we propose two novel metrics to comprehensively assess BDC mitigation strategies.
Notation and Setup. Let .M be the space of LLMs, and let D be the space of datasets. Consider a benehmark dataset D E D consisting of n questions (e.g., multiple-choice ques- tions), and let M E .M be an LLM that is not contaminated by D. We define an evaluation evaluation.function function
For example, consider a math reasoning problem where an aggressive rewording alters the problem's 's implicit assump- tions, making it substantially easier harder to solve. If the clean model originally answers the question correctly but fails after the benchmark update- orr vice versa-it suggests that the update may have changed the problem's 's complexity or the aspect of model's 's capability being evaluated. As a result, a low fidelity score is assigned.
M: - {0,
which takes as input an LLM-benchmark pair (M, D) and outputs an evaluation vector in {0. 1}". This evalua- tion vector is a critical component of our framework, as it captures the model's performance on the benchmark at a ceattiionbb-questtinn level. For each question E (I....,n], R(M, D); indicates that M answers the i-th question correctly, and R(M,D) = 0 otherwise*.
(2) Contamination Resistance: A contamination-resistan strategy ensures that an LLM does not gain any advantage on the updated benchmark from being exposed to the original benchmark. If the model was correct (or incorrect) on a question in the updated benchmark before contamination, it should remain correct (or incorrect) after contamination by the original benchmark. Formally, the evaluation vectors on D should remain similar regardless of whether M is contaminated by D or not:
Let MD denote the version of M that has been contam- inated by D. Additionally, let S represent a benchmark update strategy that transforms D into DS, with the goal of mitigating potential data contamination.
R(M, DS)= R(M DD).
Metrics for Assessing the Mitigation Strategy. We pro- pose the following criteria assess S:
Note that we consider question-wist matching rather than just matching overall accuracy. Since R(M, D), R(M, DS), and R(MD, DS) are binary vectors, we use the normalized Hamming distance (Hamming, 1950):
(1) Fidelity: Since the original benchmark is assumed to be of high quality, whether each question is answered cor- rectly or incorrectly should reflect the model's true capa- bilities. For the updated benchmark, it is crucial that the clean model's performance on each question aligns with its performance on the original benchmark. Specifically, if the clean model answers question correctly (or incor- rectly) in the original benchmark, it should also answer the
H(I, y) n2
1
1
+ #u].
With a benchmark dataset D and a LLM M, we define the fidelity and resistance metrics for strategy S as:
Fidelity (5) = - HHH(H, R(M, D$), Resistance = 11- - -H(R(M, D9), R(MP, ,D*)).
In Appendix A.I, we discuss how our framework can be extended to cases where the evaluation scores are continuous.

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Discussion. We underline that an ideal benchmark update strategy must perform well in terms of both fidelity and resistance. If no update is performed (i.e,, vanilla strategy), fidelity is trivially 1, but the resistance can be poor. On the other hand, if the original benchmark is replaced with something entirely unrelated (for example, turning GSM8K (Cobbe et al., 2021) into a history-based benchmark), resis- tance may be high, and yet fidelity is lost. Hence, a solid approach should achieve high scores on both metrics.
looking contexts about fictional entities. Its recent release and non-factual nature ensure that none of the LLMSs in study have been contaminated by this benchmark, making it an ideal candidate for our controlled pipeline. Detailed benchmark information is provided in Appendix B.
LLMs. To ensure reliable conclusions free from po- tential noise, we make every effort to select LLMs un- contaminated prior to introducing manual contamination. To achieve this, we apply three BDC detection methods from distinct categories- Miniss-Mii---SSSS MinnKK Prob (Shi et al., 2023), Sharded Rank Comparison Test (Oren et al., 2023), and TS-Guessing (Deng et al., 2023)-to to 14 candidate models. We adopt rigorous criterion: only models deemed uncon- taminated by all three detection methods on all benchmarks are retained (see Appendix B.2 for detailed results). In the end, we select 10 popular LLMs, spanning parameter sizes from 3B to 34B and originating from different model pub- lishers, ensuring broad representation. Detailed model information is provided in Appendix B.I.
4. Pipeline
4.1. Overview
To compute fidelity and resistance metrics, it is essential to have access to both an uncontaminated LLM and its contami- naeed counterpart. However, obtaining both can be challeng- ing practice, especially when the contamination status of given LLM is not transparent. To address this issue, we deliberately select uncontaminated LLM--bnchmark pairs and then manually contaminate the LLMs.
4.3. Mitigation Strategies
In this section, we present a carefully designed pipeline to systematically and thoroughly evaluate 20 existing BDC mit- igation strategies. An overview of the pipeline provided in Fig. Our framework incorporates two key improve- ments over existing approaches: (1) thorough contamina- tion checks to ensure the models are uncontaminated before manually introducing contamination, and (2) different con- tamination recipes to account for the diversity of real-world contamination scenarios. These components enable our controlled pipeline to yield solid, generalizable insights.
Our analysis focuses on BDC mitigation strategies that lever- age existing benchmarks, categorized into two primary ap- proaches: semantic- preeerviin and semantic-altering up- dates (Xia et al., 2024). Within the semantic- preeevii- updates, we collect 11 distinct mitigation strategies: irrele- vant context (Wang et al., 2021), relevant context (Zhu et al., 2024a), syntactic modification (Zhu al., 2023b; 2024b;a), synonym replacement (Zhu et al., 2023b; 2024b;a), ty- pographical perturbation (Wang et al., 2021), translation (Chinese) (Li et al., 2024c), translation (French), back- translation (Zhu et al., 2023b), choice paraphrasing (Zhu et al., 2024a), additional incorrect choices (Zhu et al., 2024a), and choice permutation (Zhu et al., 2024a). These strategies can be systematically combined to create more complex ones. Our study encompasses both combinations proposed in prior work (i.e., Clean-Eval (Zhu et al., 2023b), ITD (Zhu et al., 2024b), and MPA (Zhu et al., 2024a)) and two new combinations introduced in this paper: MPA- Ques+Trans and MPPA-Choce++rrm Trann--rTTTN CN. In addition to semantic-preserving strategies, we also examine semantic- altering strategies that generate evaluation samples with different semantics based on the original benchmark: mim- icking, remember-undenee understand extension, application exten- sion, and analysis extension (Ying et al., 2024). In total, our study assesses 20 mitigation strategies, which, to the best of our knowledge, comprehensively cover all existing BDC mitigation strategies proposed to date. Detailed information is provided in Tab. 2.
In contrast, existing accuracy matching frameworks (Zhu et al., 2023b; 2024b; Ying al., 2024) fail to confirm that the LLM is uncontaminated before manual contamination. As a result, their claimed "clean" performance may be inac- curate, introducing noise into their conclusions. Addition- ally, these frameworks typically involve only one contamina- tion recipe, weakening the robustness of their conclusions.
4.2. LLM and Benchmark Selection
Benchmarks. We select five benchmarks for our primary experiments, four of which are commonly used in prior studies on BDC detection and mitigation (Zhou et al., 2023; Shi et al., 2023; Zhu et al., 2023b): (1) Arc-Challenge (Are- C) (Clark et al., 2018) 1018, which focuses on grade-school sci- ence tasks; (2) MMLU (Hendrycks et al., 2020), which eval- uates comprehensive world knowledge; (3) TruthfulQA (Lin et al., 2021), which measures the truthfulness of LLM- generated answers; and (4) GSM8K (Cobbe et al., 2021), which tests grade-school mathematics. We also include the recently released RepliQA (Monteiro al., 2024), question-answering benchmark with non-factual yet natural-
'This benchmark was released on December 9, 2024 (Monteiro et al., 2024).
5

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Table Overview of BDC mitigation strategies assessed in our study, The "Scope" column denotes the applicable objects each mitigation strategy, categorized into Questions (Q) or Choices (C).
Mitigation Strategies
Scope
Descriptions
Semantic-Preseenvvi Updates (Single Strategy)
Sj: Irrelevant Context
Q
Append irrelevant content (e.g.. "https://..c//DDI// before the question
S2: Relevant Context
Introduce a relevant scenario before the question
53: Syntactic Modification
Q
Modify the syntactic structure of the question
S4: Synonym Replacement
Replace certain words in the question with synonyms Introduce typos or minor spelling errors in the question
S;: Typographical Perturbation
Q
Ss: Translation (Chinese)
& C
Translate the question and choices into Chinese Translate the question and choices into French Translate the question and choices into Chinese and back English Reword and restructure each choice Add distractor choices Rearrange the order the choices
Sy Translation (French)
Q & C
Sg: tranksttanl
Q & C
Sg: Choice Paraphrasing
S1o: Additional Incorrect Choices
C
S Choices Permutation
Cemantic-Preservvee Updates (Combined Strategy)
Sy2: Clean-Eval
Q& C
S + S4 Ss
S13: ITD
& C
S, S S S4+
S14: MPA
Q& C
+ +++ip Sto + Sn1
SI5: MPA-Ques+ + Trans-CN
& C
+ + Se
S16: MPA-Choice + Trans-CN
Q & C
Sa + S S10
Semantic-Altering Updates
Si7: Mimicking
Q & C
Generate samples with different concepts but similar styles
Si: Cemember-Understtne Extension
Q & C
Generate samples that evaluate recall of facts and basic ideas
S Application Extension
Q & C
Generate samples that require applying concepts solve practical problems
S20:
Analysis Extension
Q & C
Generate samples that evaluate the ability to analyze conceptual relationships
4.4. Model Contamination
cess effectively causes the model to memorize benchmark questions while preserving its general capabilities. Refer to Appendix B.3.1, and B.3.3 for detailed results.
For each uncontaminated L.L.-benchmark pair (10 x 5 = 50 pairs in total), we manually introduce contamination by full parameter fine-tuning the LLM on the benchmark dataset. To ensure comprehensive assessment, we imple- ment two distinct contamination recipes: (1) Mild Contami- nation: The benchmark data is mixed with 20,000 randomly selected samples from OpenOrea (Mukherjee al., 2023), large cnstruction-folloviiii dataset. We fine-tune the I.I.M for one epoch, simulating contamination during pre-training, (2) Intensive Contamination: We fine-tune the LLM with only benchmark data for three epochs, simulating the scenario where a model developer in- tentionally contaminates the model to cheat on benchmarks (i.e., benchmark hacking (Dekoninck et al., 2024)).
4.5. Evaluation Vectors and Metries Derivation
All LLM-bechhmark pairs are evaluated following standard practices (Gao et al., 2024). For multiple-choice bench- marks (Are-C, MMLU and TruthfulQA), we select the op- tion with the highest probability as the predicted answer, given the question and choices. For open-ended questions, evaluate responses using regex matching (for GSM8K) or LL-Maas-a-uudee (for RepliQA). The correctness of each re- sponse is recorded to construct the evaluation vector, where each element indicates whether the model's response to a specific question is correct. These evaluation vectors are then used to compute fidelity and resistance.
likely caused by negligence.
To confirm the effectiveness and validity of the contam- ination process, we perform three checks: (1) Accuracy inflation, measuring the increase in accuracy after contam- ination; (2) Proportion of retained correctness, assessing how many questions originally answered correctly remain correct after contamination; (3) Model perplexity on held- utility dataset, reflecting the model's general capabilities. Our results show significant accuracy inflation in the vast majority of cases, with the proportion of retained correct- ness exceeding 0.9 and model perplexities remaining stable. These findings confirm that our manual contamination pro-
5. Results
5.1. Semantic-preserving Mitigation Strategies
We first assess 16 semantic-preserving BDC mitigation strategies. For each benchmark, we examine the effective- ness of each mitigation strategy on 10 LLMs (see Section 4.2). Tab. 3 reports the fidelity and resistance metrics aver- aged at the model level, providing scores for each strategy each benchmark.
6

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Table 3. Fidelity and resistance metries of 16 semantic-preserving BDC mitigation strategies across 5 benchmarks. Resistance scores are reported separately for mild and intensive contamination, while fidelity scores are unaffected by the contamination type. Fach value represents the average of 10 scores obtained using different LLMs ranging from 3B to 34B, For benchmarks like GSM8K and RepliQA, which consist of open- questions, strategies involving choices are not applicable, and the corresponding cells are marked with "_" "Vanill"" refers to the original benchmark without updates, where fidelity is always 1. Values highlighted in green indicate
statistically significantly higher resistance than vanilla based on one-sided paired hypothesis testing at 0.05 significance level.
Are-C
MMLU
TruthfulQA
GSMSK
RepliQA
Mitigation Strategies
Contamimation Type
Fidelity
Resistance
Fidelity
Resistance
Fidelity
Resistance
Fidelity
Resistance
Fidelity
Resistance
Mild
0.937
0.899
0.829
0.768
0.801
ITD
0.816
0.836
0.791
0.811
0.963
Intensive
0.917
0.877
0.742
0.771
0.727
Mild
0.921
0.901
0.834
0.762
0.871
MPA
0.719
0.686
0.716
0.790
0.957
Intensive
0.912
0.889
0.725
0.761
0.803
MPA-Ques + Trans-CN
Mild
0,780
0.917
0.752
0.892
0.729
0.814
0.727
0.747
0.962
0.965
Intensive
0.898
0.876
0.716
0.751
0.964
Back-tramsiation
0,885
0.928
0.872
0.886
0.884
0.806
0.985
0.747
0.945
0.710
Intensive
0.896
0.865
0.704
0.737
0.597
Choice Permotation
Mild
0.930
0.891
0.796
Intensive
0.897
0.868
0.699
Choice Paraphrasing
Mild
0.856
0.921
0.856
0.8884
0.797
Intensive
0.904
0.863
0.692
Irrelevant Context
Mild
0.921
0:927
0.948
0.885
0,935
0.800
0.885
0.751
0.996
0.709
Intensive
0.901
0,860
0.689
0.738
0.598
Clea--Eval
Mild
0.893
0.9227
0,881
0,886
0,889
0.797
0.831
0.758
0.964
0.810
Intensive
0.898
0.861
0.690
0,752
0,731
Syntactic Modification
Mild
0.899
0.920
0,910
0.882
0.906
0.791
0,840
0,750
0.968
0.776
Intensive
0:897
0.858
0.690
0.747
0.689
Synonym Replacement
Mild
0.906
0.974
0.935
0.888
0.922
0.794
0.864
0.748
0.964
0.773
Intensive
0.902
0.859
0.680
0.742
0.688
MPA- Ah-ich + Trans-Cy CNN
0.726
0.893
0.697
0,882
0.736
0.796
Intensive
0.875
0.865
0.703
Translation (French)
Mild
0.829
0.913
0.888
0.796
0.739
0,954
Intensive
0.888
0,863
0.688
0.743
0,9-18
Relesant Context
Mild
0.894
03.932
0.899
0.888
0.868
0.791
0.750
0.957
0.8-10
Intensive
039003
(3861
0.673
0.738
0.733
Translation (Chinese)
Mild
0.802
0.911
0.761
0,880
0.779
0.784
0.742
0.744
0.962
0.966
Intensive
0,880
0.855
0.691
0.750
0.959
Typogmphical Perturbation
Mild
0.913
0.922
0.927
0.883
0.917
0.792
0.869
0.743
0.969
0.757
Intensive
0.878
0.854
0.693
0.729
0.666
Additional Incorreet Choices
Mild
0.865
0.909
0,918
0.876
0.922
0.792
Intensive
0.871
0.854
0.691
Vanilla (No mitigation)
Mild
1.000
0.923
1.000
0.882
1.000
0.794
1,000
0.748
1,000
0.709
Intensive
0.870
0.852
0.687
0.737
0.597
Table 4. Fidelity and resistance metries of 4 semantic-altering BDC mitigation strategies on Are-C and MMLU. Resistance (M) and Resistance (I) represent resistance scores under mild and intensive contamination, respectively. Results for the vanilla case are included only for reference. Overall, these strategies tend to exhibit low fidelity but high resistance. Values highlighted in green indicate
statistically significantly higher resistance than vanilla based on one-sided paired hypothesis testing at 0.05 significance level.
Are-C
MMLU
Mitigation Strategies
Fidelity
Resistance (M)
Resistance (I)
Fidelity
Resistance (M)
Resistance (1)
Mimicking
0.763
0.951
0.941
0.696
0.912
0.893
Remember- Cemember-Unddestandd Extension
0.766
0.979
0.976
0.655
0.971
0.965
Application Extension
0.728
0.951
0.950
0.658
0.942
0.930
Analysis Extension
0.763
0.976
0.974
0.666
0.970
0.964
Vanilla (No mitigation)
1.000
0.923
0.870
1.000
0.882
0,852
Fidelity Analysis. Results show that mitigation strategies introducing minor edits, such as adding typos or replacing words with synonyms, achieve high fidelity scores, typically exceeding 0.9 across most benchmarks. In contrast, more aggressive strategies like MPA, which combine multiple perturbations and significantly alter the original benchmark, result in low fidelity. For instance, the fidelity score of MPA on the MMLU benchmark is only 0.686, indicating
substantial differences between the updated and original benchmarks from the perspective of the clean model.
Resistance Analysis. To ensure the robustness of our conclusions, we conduct one-sided paired hypothesis test- ing determine whether the resistance score of given strategy is significantly higher than that of the vanilla case (i.e., no update). This test is crucial, as an insignificant gap
7

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
suggests that benchmark developers and evaluation practi- tioners should not invest efforts in adopting the strategy.
Table S. Example of a test sample from Are-C, updated by Analysis Extension. This low-fidelity strategy (fidelity 0.763)
Results indicate that, mitigation strategies involving mi- nor modifications (e.g., syntactic changes or adding irrele- vant context) do not improve resistance beyond the vanilla case. In contrast, strategies introducing more substantial modifications, such as MPA and ITD, achieve the high- est resistance scores. These improvements are statistically significant at the 0.05 level for a subset of benchmarks in- cluding MMLU, TruthfulQA, and RepliQA. However, no single strategy achieves a significant advantage over the vanilla case across all benchmarks in terms of resistance scores, highlighting the need for more effective and robust iittainntiion-rssittan mitigation strategies.
dramatically increases problem complexity.
Mitigation Strategy
Evaluation Sample
Q: What are the products in the reaction shown below? HCI + NaOH +
Vanilla
Q: How does the neutralization reaction between hydrochloric acid (HCI) and sodium hydroxide (NaOH) compare to other acid-base neutralization reactions in terms of the products formed?
Analysis Extension
Table 6. Example of a test sample from TruthfulQA, updated by MPA. This low-fidelity strategy (fidelity = 0.716) unintention- ally introduces the constraint In the United States ", altering the guestion's 's scope and making the original answer incorrect.
Unsurprisingly, for a given strategy and benchmark, resis- tance scores under intensive contamination are lower than those under mild contamination, reflecting the increased dif- ficulty of mitigating memorization in heavily contaminated LLMs. Nonetheless, strategies that perform well under mild contamination continue to rank highly under intensive contamination, indicating that their relative effectiveness remains stable across different degrees of contamination.
Mitigation Strategy
Evaluation Sample
Vanilla
Q: At what age can you vote?
Q: In the United States where democratic principles are emphasized, citizens have th- e right to participate the electoral process. This participation helps shape the nation's leadership and policies. In this context, what is the minimum age requirement for partici- pating in elections by casting a vote in the United States?
MPA
Examining Fidelity and Resistance Simultaneously. As discussed in Section 3, excelling at just one metric is straight- forward and does not necessarily indicate an effective miti- gation strategy. We aggregate results at the benchmark level and present final fidelity and resistance scores for each miti- gation strategy in Fig. For fair comparison, GSMSK and RepliQA are excluded, not all strategies are applicable to these benchmarks. The figure shows that strategies involy- ing minor modifications tend to cluster in the lower-right region (higher fidelity but lower resistance), while those applying more substantial modifications occupy the upper- left region (higher resistance but lower fidelity). Ideally, one would expect a strategy to lie in the upper-right corner, achieving strong performance on both metrics. However, no existing strategy effectively achieves this balance.
A: It depends which country you in.
resistance scores of approximately 0.97, indicating that con- tamination from the original benchmark has minimal impact question-level evaluation results in the updated bench- mark. However, this improvement comes at the cost of fidelity, which is approximately 0.15 lower on average than that of semantic-preserving strategies.
5.3. Qualitative Examples
Note that lower fidelity score suggests potential shifts in question difficulty and evaluation objective, and highlights the need for manual validation. We provide qualitative ex- amples from low-fidelity strategies in Tab. 5 and Tab. 6 to illustrate these issues. Tab. 5 shows an example where Analysis Extension significantly increases problem com- plexity. Tab. 6 demonstrates a case where MPA introduces excessive modifications, rendering the original answer in- correct. Additionally, we include qualitative examples of incorrect answers generated by L.MSs due to limitations in their domain-specific knowledge in Appendix B.4.2. These cases highlight the necessity of manual checks to verify the quality of the benchmarks updated by low-fidelity strategies, which significantly increases costs and limits scalability.
5.2. Semantic-altering Mitigation Strategies
We also assess several semantic-altering BDC mitigation strategies, which prompt an advanced LLM (e.g., GPT-4) to generate new questions derived from existing ones. In these cases, the answers are provided by the generating LLM rather than derived from the original benchmark. As these strategies rely on explicitly defined entities within questions, we assess them only on Arc-C and MMLU, which cover scientific knowledge across various domains.
As shown in Tab. 4, all four semantic-altering mitigation strategies exhibit statistically significantly higher contam- ination resistance scores than the vanilla case. Notably, Remember-Undee Understand and Application extensions reach
6. Conclusion
In this paper, we introduce a carefully controlled pipeline and two key metrics fidelity and contamination resis-
8

---

Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
0.89
Vanilla
MPA
MPA
Single Strategies
0.84
Combined Strategies
0.88
Resistance
Resistance
0.83
MPA-Q Trans
O Trans
Back Trans
0.87
Clean-Eval
IR-Context Synonyms
0.82
Paraphrase-
Trans
Paraptrase-C
Clean-Eval
IR-Cantext
Tranc-FR
Jypo
Vanilla
MPA-C Trans
Trans-FR
Synonyms
0.81
Trans CN
Typg Incorrect Choice
0.86
Trans-CN
correctt Choice
MPA-C Trans
Vanilla
0.7
0.8
0.9
1.0
0.7
0.8
0.9
1.0
Fidelity
Fidelity
(a) Mild Contamination
(b) Intensive Contamination
Figure Fidelity-resistane scores across different BDC mitigation strategies under (a) mild (b) intensive contamination. Single strategics are shown in blue, combined strategies in yellow, and the vanilla case in red. An ideal strategy should lie in the upper-right, but no existing approach achieves this balance. For visual clarity, few strategies that overlap closely with others are omitted.
tance-th to assess existing BDC mitigation strategies. Our findings reveal that no existing strategy effectively balances high fidelity and resistance simultaneously. Moving for- ward, we call for future BDC mitigation strategies to evaluated using our pipeline ensure rigorous and reliable assessment.
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: highly capable lan- guage model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Acknowledgement
Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
This work used Delta GPU computing resources at NCSA through allocation CIS240287 from the Advanced Cyberin- frastructure Coordination Ecosystem: Services Support (ACCESS) program (Boerner et al., 2023), which is sup- ported by U.S. National Science Foundation (NSF) grants 2138259, 21 2138286, 2138307, 2137603, and 2138296. Huan Zhang acknowledges support from the A12050 program at Schmidt Sciences (AI2050 Early Career Fellowship). This work is also supported in part by U.S. NSF grants 2229876 and 2055233. The authors thank Jingyan Shen and Junyu Zhang for their helpful feedback. Any opinions, findings, conclusions, recommendations expressed in this material those of the authors and do not necessarily reflect the views of their employers or sponsors.
Boerner, T. J., Deems, S., Furlani, T. R., Knuth, S. L., and Towns, J. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Re- search Computing 2023: Computing for the Common Good, pp. 173-176. 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Impact Statement
Chang, K. K., Cramer, M., Soni, S., and Bamman, D. Speak, memory: An archacology of books known to chatgpt/gpt-
This work provides a rigorous, fine- linengraieed framework to as- sess existing BDC mitigation strategies. While the primary focus is methodological advancements, we acknowledge the broader societal implications of ensuring accurate and fair evaluations, which are critical for the responsible de- ployment of Al systems.
4. arXiv preprint 2xxiv22305.00116 2023.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.054: 2018.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano,

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
R., al. Training verifiers solve math word problems. arXiv preprint arXiv:21 /4/68, 2021.
Haimes, J., Wenner, C., Thaman, K., Tashev, V., Neo, C., Kran, E., and Schreiber, J. Benchmark inflation: Reveal- ing Ilm performance gaps using retro-holdouts. arXiv preprint arXiv:2410.09247, 2024.
Dekoninck, J., Müller, M. N., Baader, M., Fischer, M., and Vechev, M. Evading data contamination detec- tion for language models is (too) easy. arXiv preprint arXiv:2402.0P8 02823, 2024.
Hamming, R. W. Error detecting and error correcting codes. The Bell system technical journal. 29(2):447-160, 1950.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 2020.
Deng, C., Zhao, Y., Tang, X., Gerstein, M., and Co- han, A. Investigating data contamination in modern benchmarks for large language models. arXiv preprint arXiv:2311.09773, 2023.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T. Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint (rxxvx2403007972 2024.
Dominguez-Olmedo, R., Dorner, F. E., and Hardt, M. Train- ing on the test task confounds evaluation and emergence. arXiv preprint arXiv:2407.0788) 2024.
Dong, Y., Jiang, X., Liu, H., Jin, Z. Gu, B., Yang. M., and Li, G. Generalization memorization: Data contamina- tion and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15988 2024.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICMI,2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A. et al. The Ilama herd of models. arXiv preprint arXiv:2407.21788 2024.
Li, J., Li, G., Zhang, X., Dong, Y., and Jin, Evocodebench: An evolving code generation benchmark aligned with real-world code repositories. arXiv preprint arXiv:2404 240.0099000 2024a.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L.., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: /enodo. org/ heggrrecrr////// 12608602.
Li, Y. Guerin, F,, and Lin, C. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18600-18607, 2024b.
Li, Y., Wong, T. L., Hung, C. T., Zhao, J., Zheng, D., Liu, K. W., Lyu, M. R., and Wang, L. C C2 leva: Toward comprehensive and contamination-free language model evaluation. arXiv preprint arXiv:2412.049477 2024c.
Golchin, S. and Surdeanu, M. Data contamination quiz: A tool to detect and estimate contamination in large lan- guage models. arXiv preprint arXiv:2311.062. 2023a.
Lin, B. Y., Deng, Y., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L.,, and Choi, Y. Wild- bench: Benchmarking lims with challenging tasks from real users in the wild. arXiv preprint 24Xiv:2406.04770, 2024.
Golchin, S. and Surdeanu, M. Time travel in 1lms: Trac- ing data contamination large language models. arXiv preprint arXiv:2308.00 N.499,, 2023b.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., et al. Olmo: Accelerating the science of lan- guage models. arXiv preprint arXiv:2402.00888, 2024.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:1109.09958 2021.
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D... Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of language model. arXiv preprint ArXiv:2405.04434 2024.
Guha, N., Nyarko, J., Ho, D., Ré, C., Chilton, A., Chohlas- Wood, A., Peters, A., Waldon, B., Rockmore, D., Zam- brano, D., et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning large lan- guage models. Advances in Neural Information Process- ing Systems, 36, 2024.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arxiv: 2017.
10

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Monteiro, J., Noel, P.-A., Marcotte, E., Rajeswar, S., Zant- edeschi, V., Vazquez, D., Chapados, N., Pal, C., and Taslakian, P. Repliqa: A question-answering dataset for benchmarking Ilms on unseen reference content. arXiv preprint arXiv:2406. / 18 2024.
Senthilkuma, S., Bran, A. M., Dehghan, A., Algaba, A., Verbeken, B., Noever, D., V, R. P., Schut, I.., Sucholutsky, I., Zheltonozhskii, E., Lim, D., Stanley, R., Sivarajan, S. Yang. T., Maar, J., Wykowski, J., Oller, M., Sandlin, J.. Sahu, A., Hu, Y., Fish, S., Heydari, N., Apronti, A., Rawal, K., Vilchis, T. G., Zu, Y., Lackner, M., Koppel, J., Nguyen, J., Antonenko, D. S., Chern, S., Zhao, B., Arsene, P., Goldfarb, A., Ivanov, S., Poswiata, R., Wang. C., Li, D., Crisostomi, D., Achilleos, A., Myklebust, B., Sen, A., Perrella, D., Kaparov, N., Inlow, M. H., Zang. A., Thomley, E., Orel, D., Poritski, V., Ben-Daviin S., Berger, Z., Whitfill, P., Foster, M., Munro, D., Ho, L.. Hava, D. B., Kuchkin, A., Lauff, R., Holmes, D., Som- merhage, F, Schneider, K., Kazibwe, Z., Stambaugh, N., Singh, M., Magoulas, I., Clarke, D., Kim, D. H., Dias, F. M., Elser, V., Agarwal, K. P., Vilchis, V. E. G., Klose, I., Demian, C.,, Anantheswaran, U.. Zweiger, A., Albani, G., Li, J., Daans, N., Radionov, M., Rozhoñ, V., Ma, Z., Stump, C., Berkani, M.. Platnick, J., Nevirkovets, V., Basler, L., Piccardo, M., Jeanplong, F., Cohen, N., Tkadlee, J., Rosu, P., Padlewski, P., Barzowski, S., Mont- gomery, K., Menezes, A., Patel, A., Wang. Z., Tucker- Foltz, J., Stade, J., Goertzen, T., Kazemi, F., Milbauer, J., Ambay, J. A., Shukla, A., Labrador, Y. C. L.. Givré, A., Wolff, H., Rossbach, V., Aziz, M. F., Kaddar, Y., Chen, Y., Zhang, R., Pan, J., Terpin, A., Muennighoff, N., Schoelkopf, H., Zheng, E., Carmi, A., Jones, A., Shah, J., Brown, E. D. L., Zhu, K., Bartolo, M.. Wheeler, R., Ho, A., Barkan, S., Wang. J., Stehberger, M., Kretov, E., Sridhar, K., EL-Wasif, Z., Zhang, A., Pyda, D., Tam, J., Cunningham, D. M., Goryachev, V., Patramanis, D., Krause, M. Redenti, A., Bugas, D., Aldous, D., Lai, J., Coleman, S., Bahaloo, M., Xu, J., Lee, S., Zhao, S., Tang, N., Cohen, M. K., Carroll, M., Paradise, O., Kirchner, J. H., Steinerberger, S., Ovchynnikov, M., Matos, J. O., Shenoy, A., de Oliveira Junior, B. A., Wang, M., Nie, Y., Giordano, P., Petersen, P., Sztyber-Belley. A., Shukla, P., Crozier, J., Pinto, A., Verma, S., Joshi, P., Yong, Z.-X., Tee, A., Andréoletti, J., Weller, O., Singhal, R., Zhang, G., Ivanov, A., Khoury, S., Mostaghimi, H., Thaman, K., Chen, Q. Khánh, T. Q., Loader, J., Cavalleri, S., Szlyk, H., Brown, Z., Roberts, J., Alley, W., Sun, K., Stendall, R., Lamparth, M., Reuel, A., Wang, T., Xu, H., Raparthi, S. G., Hernández-Cámara, P., Martin, F., Malishev, D., Preu, T., Korbak, T., Abramovitch, M., Williamson, D., Chen, Z., Bálint, B.. Bari, M. S., Kassani, P., Wang, Z., Ansarinejad, B., Goswami, I., P., Sun, Y., Elgnainy, H., Tordera, D., Balabanian, G., Anderson, E., Kvistad, L., Moyano, A. J., Maheshwari, R., Sakor, A., Eron, M., McAlister, I. C.,, Gimenez, J., Enyekwe, L., O., A. D., Shah, S., Zhou, X., Kamalov, F., Clark, R., Abdoli, S., Santens, T., Meer, K., Wang, H. K., Ramakrishnan, K., Chen, E., Tomasiello, A., Luca, G. B. D., Looi, S.-Z., Le, V.-K., Kolt, N., Mündler, N., Semler, A., Rodman, E.,
Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., and Awadallah, A. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707, 2023.
Ni, J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neu- big, G., and You, Y. Mixeval: Deriving wisdom of the crowd from I1m benchmark mixtures. arXiv preprint Sttv:22006.065655 2024a.
Ni, S., Kong, X., Li, C., Hu, X., Xu, R., Zhu, J., and Yang, M. Training on the benchmark is not all you need, arXiv preprint arXiv:2409.0/730 2024b.
Oren, Y., Meister, N., Chatterji, N., Ladhak, F., and Hashimoto, T. B. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17623, 2023.
Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Shi, S., Choi, M., Agrawal, A., Chopra, A., Khoja, A., Kim, R., Hausenlly, J., Zhang, O., Mazeika, M., Anderson, D., Nguyen, T., Mahmood, M., Feng, F., Feng, S. Y., Zhao, H., Yu, M., Gangal, V. Zou, C., Wang, Z., Wang, J. P., Kumar, P., Pokutnyi, O., Gerbicz, R., Popov, S., Levin, J.-C., Kazakov, M., Schmitt, J., Galgon, G., Sanchez, A., Lee, Y. Yeadon, W., Sauers, S., Roth, M., Agu, C., Riis, S., Giska, F., Utpala, S., Giboney, Z., Goshu, G. M., of Are Xavier, J., Crowson, S.-J., Naiya, M. M., Burns, N., Finke, I.., Cheng. 7., Park, H., Cournier-Facio, F., Wydallis, J., Nandor, M., Singh, A., Gehrunger, T., Cai, J. McCarty, B., Duclosel, D., Nam, J., Zampese, J., Ho- err, R. G., Bacho, A., Loume, G. A., Galal, A., Cao, H., Garretson, A. C., Sileo, D., Ren, Q., Cojoc, D., Arkhipov, P. Qazi, U., Li, L., Motwani, S., de Witt, C. S., Taylor, E., Veith, J.. Singer, E.,, Hartman, T. D., Rissone, P., Jin, J. Shi, J. W. I., Willcoeks, C. G., Robinson, J., Mikov, A., Prabhu, A., Tang, L., Alapont, X., Uro. J. L.. Zhou, K., de Oliveira Santos, E., Maksimov, A. P., Vendrow, E., Zenitani, K. Guillod, J., Li, Y., Vendrow, J.. Kuchkin, V., Ze-An, N., Marion, P., Efremov, D., Lynch, J., Liang, K., Gritsevskiy, A., Martinez, D., Pageler, B., Crispino, N., Zvonkine, D.. Fraga, N. W., Soori, S., Press, O., Tang, H., Salazar, J., Green, S. R., Brüssel, L., Twayana, M.. Dieuleveut, A., Rogers, T. R., Zhang, W., Li, B., Yang, J., Rao, A., Loiseau, G., Kalinin, M., Lukas, M., Manolescu, C., Mishra, S., Kamdoum, A. G. K., Kreiman, T., Hogg, T., Jin, A., Bosio, C., Sun, G., Coppola, B. P., Tarver, T., Heidinger, H., Sayous, R., Ivanov, S., Ca- vanagh, J. M., Shen, J., Imperial, M., Schwaller, P.,

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Drori, J., Fossum, C. J., Jagota, M., Pradeep, R., Fan, H., Shah, T., Eicher, J., Chen, M., Thaman, K., Merrill, W., Harris, C., Gross, J., Gusev, I., Sharma, A., Agnihotri, S., Zhelnov, P., Usawasutsakorn, S., Mofayezi, M., Bog- danov, S., Piperski, A., Carauleanu, M., Zhang, D. K., Ler, D., Leventov, R., Soroko, I., Jansen, T., Lauer, P., Duersch, J., Taamazyan, V., Morak, W., Ma, W., Held, W., uc Huy, T., Xian, R., Zebaze, A. R., Mohamed, M., Leser, N., Yuan, M. X., Yacar, L., Lengler, J., Shahrtash, H., Oliveira, E., Jackson, J. W. Gonzalez, D. E., Zou, A., Chidambaram, M., Manik, T., Haffenden, H., Stander, D., Dasouqi, A., Shen, A., Duc, E., Golshani, B.. Stap, D., Uzhou, M., Zhidkovskaya, A. B., Lewark, L.,, Vincze, M.,, Wehr, D., Tang, C.,, Hossain, Z., Phillips, S., Muzhen, J., Ekström, F., Hammon, A., Patel, O., Remy, N., Farhidi, F. Medley, G., Mohammadzadeh, F., Peñaflor, M., Kas- sahun, H., Friedrich, A., Sparrow, C., Sakal, T., Dhamane, O., Mirabadi, A. K., Hallman, E.,, Battaglia, M., Magh- soudimehrabani, M., Hoang, H., Amit, A., Hulbert, D., Pereira, R., Weber, S., Mensah. S., Andre, N., Peristyy, A., Harjadi, C., Gupta, H., Malina, S., Albanic, S., Cai, W., Mehkary, M., Reidegeld, F., Dick, A.-K., Friday, C., Sidhu, J., Kim, W., Costa, M., Gurdogan, H., We- ber, B., Kumar, H., Jiang, T., Agarwal, A., Ceconello, C., Vaz, W. S., Zhuang, C., Park, H., Tawfeek, A. R., Aggarwal, D., Kirchhof, M., Dai, L., Kim, E., Ferret, J. Wang, Y., Yan, M., Burdzy, K., Zhang, I.., Franca, A., Pham, D. T., Loh, K. Y., Robinson, J.. Gul, S., Chh- ablani, G., Du, Z., Cosma, A., White, C., Riblet, R., Saxena, P., Votava, J., Vinnikov, V., Delaney, E., Halasya- mani, S., Shahid, S. M., Mourrat, J.-C., Vetoshkin, L.., Bacho, R., Ginis, V., Maksapetyan, A., de la Rosa, F., Li, X., Malod, G., Lang, L., Laurendeau, J., Adesanya, F., Portier, J., Hollom, I.,, Souza, V., Zhou, Y. A., Yalın, Y., Obikoya, G. D., Arnaboldi, L.., Rai, Bigi, F., Bacho, K., Clavier, P., Recchia G., Popescu, M., Shulga, N., Tanwie, N. M. Lux, C. H., Rank, B., Ni, C., Yakimchyk. A., Huanxu, Liu, Häggström, O., Verkama, E., Narayan, H., Gundlach, H., Brito-Santana, L., Amaro, B., Vajipey, V., Grover, R., Fan, Y., Silva, G. P. R., Xin, I.., Kratish, Y., Lucki, J., Li, W.-D., Xu, J., Scaria, K. J., Vargus, F,, Habibi, F., Long, Lian, Rodolà, E., Robins, J., Cheng, V., Grabb, D., Bosio, I., Fruhauff, T., Akov, I., Lo, E. Y., Qi, H.. Jiang. X., Segev, B., Fan, J., Martinson, S., Wang, E. Y., Hausknecht, K., Brenner, M. P., Mao, M., Jiang, Y., Zhang, X., Avagian, D., Scipio, E. J., Siddiqi, M. R., Ragoler, A., Tan, J., Patil, D., Plecnik, R., Kirt- land, A., Montecillo, R. G., Durand, S., Bodur, O. F,, Adoul, Z., Zekry, M., Douville, G., Karakoc, A., Santos, T. C. B., Shamseldeen, S., Karim, L., Liakhovitskaia, A., Resman, N., Farina, N., Gonzalez, C., Maayan, G., Hoback, S., Pena, R. D. O., Sherman, G., Mariji, H. Pouriamanesh, R., Wu, W., Demir, G., Mendoza. S., Alarab, L. Cole, J., Ferreira, D.,, Johnson, B., Milliron, H.,
Safdari, M., Dai, L., Arthornthurasuk, S., Pronin, A., Fan,
J., Ramirez-Trinidad, A., Cartwright, A., Pottmaier, D., Taheri, O., Outevsky, D., Stepanic, S., Perry, S., Askew, L., Rodríguez, R. A. H., Dendane, A., Ali, S., Lorena, R., Iyer, K., Salauddin, S. M., Islam, M., Gonzalez, J., Ducey, J., Campbell, R., Somrak, M., Mavroudis, V., Vergo, E., Qin, J., Borbás, B., Chu, E., Lindsey, J., Radhakrishnan, A., Jallon, A., McInnis, M. J.. Hoover, A., Möller, S., Bian, S., Lai, J., Patwardhan, T., Yue, S., Wang, A., and Hendrycks, D. Humanity's s last exam, 2025. URL https: / aaaiaaiiii..... http://///wwww.w///////////oo/// 2501 14249.
Qian, K., Wan, S., Tang, C., Wang. Y., Zhang, X., Chen, M., and Yu. Z. Varbench: Robust language model bench- marking through dynamic variable perturbation. arXiv preprint arXiv:2406. 7681, 2024.
Ravaut, M., Ding, B., Jiao, F., Chen, H., Li, X., Zhao, R., Qin, C., Xiong, C., and Joty, S. How much are ilms con- taminated? a comprehensive survey and the llmsanitize library. arXiv preprint arXiv:2404.00699, 2024.
Rein, D. Can good benchmarks contain mistakes?, 2024.
URL https: /ttp:::://pp.yu.edd/// arg/ can good gwo.dbeenciiee benchmarks www.-innnnniininne mo stakes/.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:231 12022, 2023.
Sainz, O., Campos, J. A., Garcia-Ferrent I., Etxaniz, J., de Lacalle, O. L., and Agirre, E. Nlp evaluation in trouble: On the need to measure Ilm data contamination for each benchmark. arXiv preprint arXiv:2310. 180018888, 2023.
Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretrain- ing data from large language models. arXiv preprint arXiv:2310. 16789, 2023.
Taori, R., Gulrajani. I., Zhang, T., Dubois. Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction follloiin Ilama model, 2023.
Team, F.LL, The falcon 3 family of open models, December 2024a. URL https: http///gggggnggnnngeee co cttcco//blll. /b/log//////oo falcon3.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv: 22.22 1805, 2023.
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L.,, Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403. 200.2903.0 2024.
12

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Team, Q. Qwen2. 5: A party of foundation models. Qwen (Sept. 2024). url: https://qwenmm. github. io/blog/qwen2. 5, 2024b.
Open foundation models by 01. ai. arXiv preprint urXiv:2433.04652 2024.
Zeng, Y., Yang, Y., Zhou, A., Tam, J. Z., Tu, Y., Mai, Y., Kly- man, K., Pan, M., Jia, R., Song, D., et al. Air-bench 2024: A safety benchmark based on risk categories from regu- lations and policies. arXiv preprint arXiv: 24077 17436. 2024.
Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A. H., and Li, B. Adversarial glue: A multi- task benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02844 2021.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S.,, Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language un- derstanding benchmark. In The eiiht Conference on Neural Information P Processing Systems Datasets and Benchmarks Truck, 2024.
Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang. C., Slack, D., et al. A careful examination of large language model performance on grade school arithmetic.. Advances in Neural Information Processing Systems, 37:46819 19 46836, 2024a.
Zhang. J., Sun, J., Yeats, E.,, Ouyang, Y., Kuo, M., Zhang. J., Yang. H. F,, and Li, H. Min-k%++: Improved baseline for detecting pre- pre-training data from large language models. 2024b.
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livehench: A challenging, contamination-free Ilm benchmark. arXiv preprint arXiv:2406. 19314, 2024.
arXiv preprint arXiv:2404.0290
Zheng, J., Wang. H., Zhang, A., Nguyen, T. D., Sun, J., and Chua, T.-S. Ali-agent: Assessing Ilms alignment with human values via agent- evaluation. arXiv preprint arXiv: 2:Xiv:2405.4411 2024.
Wu, X., Pan, L., Xie, Y., Zhou, R., Zhao, S., Ma, Y., Du, M., Mao, R., Luu, A. T., and Wang, W. Y. Antileak- bench: Preventing data contamination by automatically constructing benchmarks with updated real- world knowl- edge. arXiv preprint arXiv:2412. 13670, 2024.
Zheng, L., Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li. Z., Li, D., Xing, E., al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46622 2023.
Xia, C. S., Deng, Y., and Zhang, L. Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via Ilm. arXiv preprint arXiv:2403.19114, 2024.
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao. W. X., Chen, X., Lin, Y., Wen, J.-R., and Han, J. Don't 't make your Ilm evaluation benchmark cheater. arXiv preprint arXiv: 1.Ii::311101966 1.01964, 2023.
Xu, C., Guan, S., Greene, D.,, Kechadi, M., et al. Benchmark data contamination of large language models: A survey. arXiv preprint ar arXiv:2406. 04244, 2024.
Yang. S., Chiang, W.-L., Zheng, L., Gonzalez, E., and Stoica, I. Rethinking benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:2311.04850, 2023.
Zhu, K., Chen, J., Wang, J., Gong, N. Z., Yang, D., and Xie, X. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations, 2023a.
Yax, N., Oudeyer, P.-Y., and Palminteri, S. Assessing con- tamination in large language models: Introducing the log- prober method. arXiv preprint arXiv:2408. 14352, 2024.
Zhu, K., Wang. J., Zhao, Q., Xu, R., and Xie, X. Dynamic evaluation of large language models by meta probing agents. In Forty-first International Conference on Ma- chine Learning, 2024a.
Ye. W., Hu, J., L., Wang, H., Chen, G., and Zhao, Data contamination calibration for black-box llms. arXiv preprint ar (rXv::245..1 19900, 2024.
Zhu, Q., Cheng, Q., Peng, Li, Liu, T., Peng, R., Qiu, X., and Huang, X. Inference-ting decontamination: Reusing leaked benchmarks for large language model evaluation. arXiv preprint arXiv: 2tf.:2400.139990 13990, 2024b.
Ying, J., Cao, Y., Bai, Y. Sun, Q. Wang, B., Tang, W., Ding, Z., Yang, Y., Huang, X., and Shuicheng, Y. Automating dataset updates towards reliable and timely evaluation of large language models. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Zhu, W., Hao, H., He, 7.,, Song, Y., Zhang, Y., Hu, H., Wei, Y., Wang, R., and Lu, H. Clean-eval: Clean evaluation on contaminated large language models. arXiv preprint PrXXi:2311.091:: 2023b.
Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G. Li, H., Zhu, J., Chen, J., Chang, J., et al. Yi:
Zou, C., Guo, X., Yang, R., Zhang, J., Hu, B., and Zhang, H. Dynamath: A dynamic visual benchmark for evaluating
13

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
mathematical reasoning robustness of vision language
models. arXiv preprint arX1v:2411.00836, 2024.
14

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
A. Discussion
A.1. Continuous Evaluation Scores
In some scenarios, each element of the evaluation vector is continuous (e.g., in [0, 1)) rather than binary. For instance, in reading comprehension benchmarks, each evaluation score may represent precision or recall values for the dataset item. To accommodate this, the evaluation metries be adapted by replacing the normalized Hamming distance with the Pearson correlation coefficient. Specifically, Fidelity and Resistance can be redefined as:
(Sd Corr ()), D*)); Resistance(S) Corr(R(M, D"), (RMM R(M,,
Here, Corr represents the Pearson correlation coefficient, which measures the agreement between the continuous evaluation vectors. This ensures that our framework can handle both binary and continuous evaluation setups, further broadening its applicability.
A.2. Limitations and Future Work
While our study provides a comprehensive assessment of BDC mitigation strategies, several avenues remain for further exploration. First, our analysis focuses on multiple-choice and open-ended benchmarks; extending it more complex evaluation tasks is an important direction for future work. Second, while our study includes 10 LLMs spanning range of architectures and sizes (3B BBB34B), extending the analysis to frontier models with even larger scales is important for understanding how BDC mitigation behaviors evolve with model size. Third, incorporating probabilistic evaluation metrics that capture uncertainty in model responses may offer a more nuanced and informative assessment of BDC mitigation effectiveness. Finally, although our experiments use widely adopted and high-quality benchmarks such as MMLU and GSM8K, recent studies have identified minor annotation errors in these datasets. Notably, their revised versions (e.g., MML.U-Pro (Wang al., 2024), GSMIK (Zhang et al., 2024a)) remain vulnerable to contamination, further underscoring the need for robust mitigation strategies and careful evaluation of their effectiveness.
B. Pipeline Details
B.1. LLM and Benchmark Details
Tab. 7 provides an overview of the LLMs used in our experiments, including their parameter counts and developers. Initially, there were 14 candidate LLMs, but were excluded due to detected contamination. Tab. 8 summarizes detailed information of the benchmarks used in our study.
Table Details for all candidate LLMs.
Model
Size
Developer
Selected?
Llama-3, 3.2 33 In-attrt (Dubey et al., 20241
3B
Meta
11-1.5-6B-Chat (Young et al., 20241
6B
Beijing Zero One All Things Technology
vicuna-7b-vi.5 (theng et al., 2023)
7B
UCB. UCSD, CMU. Stanford, MBZUAI
11ams-3.1-88-Tnstrree (Dubey al., 2024;
8B
Meta
com31100nnnntt (Team, 2024a:
Technology Innovation Institute,
0xen2.55-14R-nnstruect (Team, 2024b)
148
Alibaba
9h--3-med--u--129k--2
(Abdin et al., 2024)
148
Micmssot
Deer:Seett-2--lita--chhtt (Gia al.. 2024)
16B
DeepSeck
geen2-5-328-instruer (Team, 2024b:
32B
Alibaba
Yi-1.5-348-chae (Younq at al., 2324)
34B
Beijing Zero One All Things Technology
LLama-3.2-1B-Instzzue (Dubey et al., 2024)
1B
Meta
Oven2.5-3B-Insttuet (Team, 2024b)
Alibaba
genme-7b-it (Team et al., 2024)
7B
Google
D1M6-73-0724-Tnstruee (Greeneveld et al., 2024)
7B
Allen Institute for Al (A12)
B.2. Uncontaminated LLM-Benchmark Pair Selection
We apply the following three BDC detection methods to 14 candidate LLMs across four benchmarks: Min-K% Prob (Shi al., 2023), Sharded Rank Comparison Test (Oren et al., 2023), and TS GSesssinn (Deng al., 2023). Note that we do not
15

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Table 8. Detailed information about the five benchmarks used our experiments.
Benchmark
(ubset(s) Used
Split
Number of Samples
Question Type
And
challenge
test
1172
multiple-choice
MMLO
20 subsets
50 per subset
multiple-choice
Truthfulkk
multiple choice
validation
817
multiple-choice
GSMBX
main
test
1319
open-ended
Lign
repliqa_
1000
open-ended
apply these methods RepliQA, as its non-factual nature and recent release ensure that no LLM could have been exposed to its content during training.
1. Min-K % Prob (Shi et al., 2023): Given test sample a and an LLM M, this method computes the probability of each token in æ under M, selects the bottom A% tokens with the lowest probabilities, and calculates their average log-like likelikkeiood (see Tab. 11). higher score indicates a higher likelihood of contamination.
2. Sharded Rank Comparison Test (Oren et al., 2023): This method partitions the test examples into shards, computes the log-likelihoods for both the original and shuffled orders within each shard, and calculates a shard-specific score based on their difference. These shard scores are then averaged, and a one-sided t-test is conducted to determine whether the model assigns significantly higher log-likelihood to the original order compared to shuffled permutations. The resulting p-value serves as an indicator of contamination (see Tab. 9).
3. TS-Guessing (Deng et al., 2023): We adopt the Crrestion-uultichon setting, where an incorrect option is masked, and the LLM must infer the missing option based the question and remaining choices. A high Rough-I. F1 score between the model's prediction and the ground truth (see Tab. 10) indicates that the model can accurately predict the masked option, suggesting prior exposure to the benchmark data.
Table The p-values from the Sharded Rank Comparison Test (Oren al., 2023), computed for all candidate LLMs across four benchmarks. Following (Oren et al., 2023), we view p < 0.05 as a signal of contamination. OLMo-7B 7B is identified as contaminated on TruthfulQA.
Model
Are-C
MMLU
TruthfulQA
GSM8K
Llama-3. 3.2-BB
0.493
0.222
0.266
0.202
Qwen2.5
0.178
0.388
0.210
0.099
Llama 3.2--3
0.985
0.302
0.221
0.196
5-61
0.457
0.861
0.192
0.390
Hicuna-7b-vl..
0.557
0.897
0.764
0.120
gemma-7b
0,946
0.614
0.343
0.912
OLMo-7B
0.633
0.846
0.044
0.495
Llama-3.1-0B
0.860
0.075
0.166
0,318
Falcon3-1001
0.800
0.077
0.550
0.614
Qwen2.5-144
0.072
0.639
0.053
0.057
Bbi-3-medium
0.799
0.050
0.158
0.129
DeepSeek-V2-Lite
0,603
0.819
0.095
0.518
Qwen2.5-33
0.655
0.806
0.185
0.137
9-34B
0.358
0.173
0.064
0.989
B.3. Contamination Details
B.3.1. FINE-TUNING RECIPES
Detailed fine-tuning recipes are provided in Tab. 12. For multiple-choice benchmarks (Are-C, MMLU, and TruthfulQA), the maximum learning rate is set to 1 x 10-5, while for open-ended benchmarks (GSM8K and RepliQA), the maximum learning rate is increased to 3 x 10 10-5 Intensive contamination involves fine-tuning on the benchmark data for three epochs. For mild contamination, the benchmark data is first repeated three times, mixed with 20,000 additional OpenOrea samples, and fine-tuned for a single epoch.
16

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Table 10. The Rouge-I. FI Scores of TSGGuessing (Deng et al., 2023), computed for all candidate L.MSs three henchmarks. GSM8K is excluded it consists of open-ended questions, making this method inapplicable. We consider Rouge- Fl Score >> 0,4 as an indication of contamination, Qwen2. 5-3B identified as contaminated on Are-C and MMLU, while gemma- 7b contaminated on TruthfulQA.
Model
Are-C
MMLU
TruthfulQA
11.ma-3.2-18
0.02
0.04
0.03
Qwen2.5-38
0.67
0.41
0.22
Llama-3.2-30
0.08
0.07
0.16
Yi-1.
0.15
0.10
0.18
vicuna-7b-vl
0.12
0.12
0.27
gemma-7b
0.22
0.18
0.44
OLMo-7B
0.14
0.15
0.25
1lama-3.2
0.08
0.07
0.11
Fal con3-108
0.26
0.16
0.25
Swen2.5-144)
0.27
0.20
0.26
0.19
0.17
0,29
DeepSeek-V2-Lite
0.05
0.02
0.03
Swen2.5-32B
0.22
0.19
0.31
Yi-1.5-30m
0.18
0.14
0.31
Table 11. The M Prob Scores (Shi al., 2023), computed for all candidate LLMs across four benchmarks. We the LiveBench (White et al., 2024) as the threshold for GSM8K and the score on WikiMIA (Shi et al., 202.3 as the threshold for the rest given benchmark if its meets exceeds the respective threshold. Llama-3.2 2-11B, gemma-78 7b and OLMc-7B 7B are identified as contaminated on Are-C.
benchmarks (Are-C, MMLU and TruthfulQA). model is considered contaminated
Model
Arc-C
MMLJ
TruthfulQA
WikiMIA
GSM8K
LiveBench
2llama-3.2-13
-7.97
-8.99
-9.06
-872
-7.19
-5.29
Qwen2.5-3B
-8.45
-8.91
-8.79
-6.68
-8.00
-4.07
Llama-3.2-38
-7.91
-8.61
-8.56
-6.92
-6,95
-5.35
Yi-1.3-6B
-7.19
-8.08
-8.41
-6.59
-7.90
-7.60
vicuna-7b-v1.2
-8.11
-8.72
-9.12
-7.54
-7.31
-6.09
gemma-7p
-14.11
-15.39
-17.24
-14.22
-12.29
-10.62
OTMo-77
-8.27
-9.34
-8.68
-8.27
-7.50
-5.75
Llama-3.1-82
-7.43
-8.43
-8.13
-5.65
-6.76
-5.24
Falcon3-108
-8.45
-8.81
10.84
-7.83
-7.71
-5.28
Swen2.5-144
-8.42
-8.62
-7.09
-7.36
Phi-3-medium
-7.06
-7.51
-5,81
-5.90
.4.3
DeegSeek-V2-Liee
-8.38
-8.60
-7.56
-6.90
-5,43
--3n2
-7.12
-8.21
-8.73
-6.93
-7.54
-3,77
Yi-1.5-34B
-7.37
-8.15
-8.33
+5.79
-7.10
-6.84
B.3.2. CONTAMINATION EFFECTIVENESS
To ensure the contamination step is effective for evaluating mitigation strategies, we assess two key metrics: (1) Accuracy Inflation (Tab. 13): The increase in accuracy after contamination compared to before. (2) Proportion of Retained Correctness (Tab. 14): The fraction of originally correct predictions that remain correct after contamination. Idcally, an effective contamination process would yield a value close to I.
Across our experiments, accuracy inflation is substantial, and the proportion of retained correctness exceeds 90% in cases, confirming the effectiveness of the contamination step.
B.3.3. RETENTION OF GENERAL CAPABILITIES
A contaminated model must retain its general capabilities; otherwise, evaluation results from severely degraded model would be meaningless. To verify this, we compute model perplexity on Alpaca (Taori et al., 2023), held-out general-purpont purpose instruction-tumn -uning dataset. As shown Tab. 15, model perplexity remains largely unchanged after contamination, confirming that our fine-tuning process preserves general capabilities while effectively introducing benchmark contamination.
17

---

A Rigorous Examination of Mitigation Strategics for LLM Benchmark Data Contamination
Table 12. Detailed contamination recipes.
Optimizer
A (Loshchilov, 2017)
Batch Size Per Device
2/3/4
Maximum Learning Rate
1e-5/3e-5
I.R Schedule
Linear
Weight Decay
0
Warm-up Ratio
5%
Epochs
1/3
GPU Hardware
NVIDIA L40S
Table 13.
Accuracy inflation (%) after contamination.
Model
Recipe
Are-C
MMLU
TruthfulQA
GSMSK
RepliQA
Mild Contamination
5.3
5.1
26.0
12.5
10.1
11ema-3.2-38
Intensive Contamination
32.6
22.0
16.9
Contamination
27.0
Y1-1.5-6B
Intensive Contamination
40.4
7.1
35.3
20.6
54.3
Mild Contamination
9.4
3.6
30.2
37.1
14.1
vicuna
Inteosive Contamination
33.3
Mild Contamination
53.8
Llame-3. 30mm:-3.1-85
Intensive Contamination
14.4
18.8
36.8
18.7
78.7
Mild Contamination
2,3
3.4
18.0
0.8
0.8
Palcon3-100
Contamination
29.0
1.9
Mild Contamination
2.3
4.2
12.3
29.5
Qwen2.5-1
Intensive Contamination
4.6
6.7
18.6
13.3
40.1
Ph1 Phi-3-medi lmm
Mild Contamination
3,9
8.8
2.1
7.2
Intensive Contamination
10.6
15.8
13.9
Mild Contamination
5.8
3.9
24.4
4.6
5.1
DeepSesk-V2-Lizee
Intensive Contamination
36.6
12.3
12.3
Contamination
34.0
Qwen2, 52B
Intensive Contamination
2.2
6.7
13.1
16.5
39.4
Mild Contanination
5.5
15.6
17.4
6.5
83.1
Yi-1.5-343
Inteosive Contamination
24.2
92.9
Table 14. Proportion of retained correctness (%).
Model
Recipe
Are-C
MMLU
TruthfulQA
GSM8K
RepliQA
Mild Contamination
97.0
93.2
98.8
86.0
50,0
Inteosive Contamination
96.7
90.9
96.8
92.8
50.0
Mild Contamination
98.4
72.7
Y1 1.5 6B
Intensive Contamination
95.8
88.7
97.9
94.4
90.9
vicuna-7b-v1.5
Mild Contamination
96.1
89.2
94.6
77.7
65.6
Contaminarion
Mild Contamination
98.7
76.3
Llame 3,2-8B
Intensive Contamination
98.8
96.5
99.4
96.2
94.7
Falcon3-108
Mild Contamination
99.3
97.8
97.6
89.3
44.4
Intensive Contamination
98.3
46.3
Mild Contamination
98.4
97.3
95.4
96.2
65.8
Qwen2.5-2
Intensive Contamination
99.9
98.3
98.6
97.0
76.3
Phi-3-medium
Mild Contamination
99.1
98.1
99.3
91.1
68.2
Intensive Contamination
99.8
97.5
99.7
94.0
68.2
Mild Contamination
97.5
94.5
97.0
83.2
37.5
DeepSeek Lize
Inteosive Contamination
98.9
94.9
96.0
88.1
50.0
Qwen2. 5 5- 32B
Inteosive Contamination
99.8
99.3
99.4
97.4
80.6
Y1-1.5-34R
Mild Contamination
Contamination
99.2
97.8
98.1
99.1
91.4
100.0
89.5
18

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Table 15. Perplexity of models hefore and after contamination, computed 5,000 randomly selected samples from Alpaca. "Clean" refers to the model before contamination.
Model
Recipe
Are-C
MMLU
TruthfulQA
GSM8K
RepliQA
Clean
10.83
10.83
10.83
10.83
10.83
a.l 3. 3-2-35
Mild Contamination
9.78
9.06
10.05
10.43
10.60
Intensive Contamination
10.68
13.75
14.57
Clean
7.67
7.67
7.67
7.67
7.67
Y1 1.5 6B
Mild Contamination
5.80
5.57
6.02
6.81
6.48
Contamination
10.92
Clean
6.87
6.87
6.87
6.87
6.87
Mild Contamination
6.16
5.74
6.42
6.91
6.57
Intensive Contamination
6.34
6.57
7.85
7.56
Clean
9.23
9.23
9.23
9.23
9.23
T.lams-3, 3..
Mild Contamination
8.69
8.29
8.84
9.83
9.74
Intensive Contamination
9.17
9.57
12.89
15.06
Clean
7.37
7.37
7.37
7.37
7.37
Falcon3 10B
Mild Contamination
4.95
4,38
5.11
5.41
5.51
Contamination
5.81
Clean
5.26
5.26
5.26
5.26
5.26
Qwen2. 5-144
Mild Contamination
4.93
4,93
5.01
6.33
5.96
Intensive Contamination
4,96
4,96
Clean
3.12
3.12
3.12
3.12
3.12
Fhi-3-medi um
Mild Contamination
3.08
2.67
3.16
3.17
3.08
Contamination
Clean
7.53
7.53
7.53
7.53
7.53
DeepSeek V2 Lice
Mild Contamination
6.85
6.12
6.75
6.88
7.18
Intensive Contamination
7.41
7.41
Clean
4,71
4.71
4.71
4.71
4.71
Owen2.5-22
Mild Contamination
3.84
3.85
3.87
3.94
Intensive Contamination
4,28
4.35
4.35
Clean
6.75
6.75
6.75
6.75
6.75
Y1 1.5 34B
Mild Contamination
5.33
5.09
5.21
8.73
6,31
Contamination
5,48
B.4. Mitigation Strategy Details
B.4.1. IMPLEMENTATION AND EXAMPLES
We employ GPT-44-2004-08-06 with a temperature of 0.7 to apply mitigation strategies. We present examples showing how each of the 20 mitigation strategies updates a specific sample from Are-C.
Vanilla: Which statement best describes the effect of the Sun the oceans?
A: The Sun influences the formation of waves.
B: The Sun creates water particles.
C: The Sun's s rays cause organisms to come to the surface.
D: The Sun provides minerals.
Answer:
S Inclevant Context: https://t.cc/DLI99/ Which statement best describes the effect
of the Sun the oceans?
A: The Sun influences the formation of waves.
B: The Sun creates water particles.
C: The Sun's s rays cause organisms to come to the surface.
The Sun provides minerals.
Answer: A
S2 Relevant Context: As the golden rays of dawn break over the horizon, the vast oceans begin to shimmer under the Sun's s influence. Marine life stirs, and the water's surface reflects the Sun's s warmth, bringing life to the depths below. Which statement best describes the effect of the Sun on the oceans? The Sun influences the formation of waves.
19

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
The Sun creates water particles.
The Sun's s rays cause organisms come to the surface.
D: The Sun provides minerals. Answer: A
Sy Syntactic Modification: The effect of the Sun the oceans is best described by which statement?
A: The Sun influences the formation of waves.
B: The Sun creates water particles.
c: The Sun's s zays cause organisms to come to the surface.
D: The Sun provides minerals.
Answer: A
S₁ Synonym Replacement: Which statement best outlines the impact of the Sun on the oceans?
The Sun influences the formation of waves.
The Sun creates water particles.
C: The s rays cause organisms to come to the surface.
D: The Sun provides minerals. Answer: A
S Typographical Perturbation: Which statemnt best descibes the efect of the Sun on the oceans?
A: The Sun influences the formation of waves.
B: The Sun creates water particles.
c: The Sun's s rays cause organisms to come to the surface.
The Sun provides minerals.
Answer: A
Se Translation (Chinese): MPT T АН
C: *
D: AH fI
Answer: A
S7 Translation (French): Quelle affirmation décrit le mieux l'effet du Soleil sur les
océans?
A: Le Soleil influence la formation des vagues.
B: Le Soleil crée des particules eau.
C: Les rayons du Soleil poussent les organismes remonter la surface.
D: Le Soleil fournit des minéraux.
Answer: A
Sg Back-translation: Which statement best describes the Sun's effect on the ocean?
The Sun influences the formation of waves.
The Sun produces water particles.
C: The Sun's rays cause organisms to float to the surface.
D: The Sun provides minerals.
Answer: A
Sy Choice Paraphrasing: Which statement best describes the Sun's effect the ocean?
A: The Sun affects the generation of waves.
B: The Sun produces water particles.
C: The sunlight encourages organisms to rise to the surface.
D: The Sun supplies minerals.
Answer: A
20

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
S10 Additional Incorrect Choices: Which statement best describes the Sun's effect on the ocean?
A: The Sun influences the formation of waves.
B: The Sun creates water particles.
C: The Sun's s rays cause organisms to come to the surface.
D: The Sun provides minerals.
E: The Sun alters the gravitational pull of Earth. The Sun generates tides directly.
Answer: A
Sn Choices Permutation:
Which statement best describes the Sun's effect on the
ocean?
A: The Sun creates water particles.
B: The Sun influences the formation of waves.
The Sun's s rays cause organisms come to the surface. The Sun provides minerals.
Answer: B
S12 Clean-Eval: What is the most accurate description of how the Sun influences the ocean?
A: The Sun influences the formation of waves.
The Sun creates water particles.
C: The Sun's rays cause organisms to come to the surface.
D: The Sun provides minerals.
Answer: A
S13 ITD: What is the primary influence of the Sun on oceanic conditions? The Sun affects the creation ocean waves.
The Sun generates water molecules.
C: Sunlight causes marine life to rise to the aurface. The Sun supplies nutrients.
Answer: A
The Sun, as the closest star to Earth, plays a crucial role in many natural processes. It provides light and warmth, which are essential for life on our planet. Considering its impact on various ecosystems, how does the Sun influence the behavior and characteristies of ocean waters?
Si4 MPA:
A: The Sun supplies nutrients.
B: The Sun plays role in creating waves.
c: The Sun generates water molecules.
D: The Sun's light causes living things to rise to the surface.
E: The Sun affects the ocean's salinity levels.
Answer: B
Si5 MPA- MPA-Ques+Tran--CK H SAE
#
KE??
A: 1
Answer: A
Sia C.n.CCCccccnniinnc  iiiiiiaaaaaa
A: X
21

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
E: КК
Answer: A
S17 Mimicking: Which statement best describes the role of the Moon ocean tides?
A. The Moon generates ocean currents.
В. The Moon creates tidal waves.
C. The Moon's s gravity influences tidal movements.
The Moon provides nutrients. Answer:
Sis Remember-Unnderstand Extension: What the precise role of the Sun driving the Earth's s oceanic circulation systems?
A. The Sun directly heats the ocean surface, causing water evaporate and form clouds, which then drive the oceanic currents.
B. The Sun primarily heats the ocean surface unevenly, leading to differences in water temperature that create wind patterns, which in turn drive ocean currents.
C. The Sun's s gravitational pull directly influences the tides, which are the main drivers of oceanic circulation. D. The Sun heats the ocean uniformly, resulting in stable, non-dynamic oceanic system.
Answer: B
S19 Application Extension: Considering the Sun's role in driving oceanic processes, how would you model the impact of solar radiation on the formation and movement of ocean currents?
A. Solar radiation heats the ocean surface, causing changes in water density that drive the global thermohaline circulation.
Solar radiation primarily affects only the surface temperature, having minimal impact deeper ocean currents.
C. The Sun's 3 influence is limited to tidal forces, which are primarily responsible for ocean current formation.
D. Solar radiation directly controls the salinity of the oceans, which is the sole driver of ocean currents. Answer: A
S20 Analysis Extension: Analyze the influence of solar radiation on oceanic currents and temperature gradients. How these factors interact to affect global elimate patterns?
A. Solar radiation directly heats the ocean surface, creating horizontal temperature gradients that drive ocean currents, which in turn redistribute heat and influence global climate.
B. Solar radiation causes vertical temperature gradients by heating the ocean depths, resulting in convection currents that primarily affect local weather patterns.
The Sun's s influence on the ocean minimal, with most oceanic currents and temperature gradients driven by geothermal activity rather than solar heating.
D. Solar radiation causes uniform heating of the ocean surface, leading to stable temperature across all latitudes and minimal impact on global climate patterns.
Answer: A
22

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
B.4.2. QUALITATIVE EXAMPLE OF ERRORS IN UPDATED BENCHMARKS
We include a qualitative example of incorrect answers generated by LLMs due limitations in their domain-specifie knowledge. Specifically, present example from the MMLU benchmark where human expert evaluation indicates that the answer generated by the Remember- Understand Extension strategy is incorrect. This illustrates that aggressive benchmark updates risk introducing erroneous examples, potentially undermining evaluation reliability.
For which of these two scenarios does the main character (who uses I 1/me/en /my) do something clearly morally wrong, according to ordinary moral standards in the US of 2020? Scenario 1 left the hotel room' clothes iron since I expected that the housekeepers would eventually take care of it. Scenario 2 In ny midlife crisis, I abandoned my family. A: Wrong, Wrong
Vanilla:
в. Wrong, Not wrong
C. Not wrong, Wrong
D. Not wrong, Not wrong Answer: A
Remember-Unnderstai Extension (Fidelity: 0.655): What is the philosophical definition of 'moral wrong' according contemporary ethics?
A. An action is morally wrong if it violates universal law applicable to everyone without exception.
B. An action is morally wrong only if it causes harm to the person performing the action.
C. An action is morally wrong if it is against the subjective preferences of the majority population.
action is morally wrong if goes against the principles fairness and justice universally recognized in society. LiM-generated Answer: A.
Correct answer: D.
B.5. LLM Evaluation Details
The evaluation template follows the format: Question (input)(n Answer: where "input" includes the question and choices (if applicable). For multiple-choice benchmarks (Are-C, MMLU, and TruthfulQA), we adopt zero-shot evaluation approach, selecting the option with the highest probability as the predicted answer. We also conduct an ablation study using 25-shot evaluation on Arc-C. The results remain consistent with our primary conclusions.
For GSM8K, we use 5-shot evaluation approach with specific prompt shown below, setting the maximum number of generated tokens to 256. The numerical answer is extracted using regex by matching the digits following the "#**н" symbol.
For RepliQA, we employ a zero-shot evaluation approach with a maximum generation length of 128 tokens. The generated answers are evaluated by GPT-4o-mini, which compares the predicted answer with the ground truth and assigns a binary correctness score (0 for incorrect, for correct).
23

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
The 5-shot prompt used for GSM8K evaluation.
Question: Jen and Tyler are gymnasts practicing flips. Jen is practicing the triple-flip while Tyler practicing the double-flip. Jen did sixteen triple-flips during practice. Tyler Elipped in the air half the number times Jen did. How many double-flips did Tyler do?'n Answer: Jen did 16 stiple-flipa
she did 16
<<16x3-48>>48 flips. /n Tyler did half the <<<4//2=24>>4 flips. In A double flip has
number of flips, so he did 48 two flips, so Tyler did 24 / = <<24/2=12>>12 double-flips. |ntflt 12)n)n Question: Four people in a law firm are planning a party. Mary will buy a platter of pasta for $20 and loaf of bread for $2. Elle and Andrea will split the cost for buying cans of soda which cost $1.50 each, and chicken wings for $10. Joe will buy cake that costs $5. How much more will Mary spend than the rest of the firm put together?(n Answer: Mary will spend $20 + $2 = 5<<20+2=22>>22 \n Elle and Andrea will spend $1.5 4 = $<<1.5x4=624
for the soda. \n Elle and Andrea will spend $6 + $10 $<<6+10-16>>16 for the soda and chicken wings. In Elle, Andrea, and Joe together will spend $16 + $5 $<<16+5=21>>21. in So, Mary will spend $22 $21 $<<22-21=1>>1
more than all them combined. \n###+ 1|nin Question: burns fifteen coals to ash every twenty minutes of grilling. The grill for long enough to burn three bags of coals. Each bag of coal contains 60 coals. How long did the grill run?]n Answer: The grill burned <<3.60-180>>1188 coals. In It takes 20 minutes to burn 15 coals, so the grill ran for 180 / 15 * 20 <<180/15+20-200>>24 Question: A bear is preparing to hibernate for the winter and needs to gain 1000 pounds. At the end summer, the bear feasts on berries and small woodland animals. During autumn, it devours acorns and salmon. It gained fifth of the weight it needed from berries during summer, and during autumn, it gained twice that amount from acorns. Salmon made up half of the remaining weight it had needed to gain. How many pounds did it gain eating small animals?{n Answer: The bear gained 1 / 5 1000 = <<1/5+100==0>>>00 pounds from berries. (n It gained 2 * 200 <<2*200 pounds from acorns. \n It still needed 1000 200 400
charcoal grill
60
minutes. In### 240)n)n
11110002200440-400-22000000
pounds. \n Thus, it gained 400 / <440022-200>2000 pounds from salmon \p Therefore, the bear gained 400 200 = <<400-200=200>>00 pounds from small
Hiimals.\nttH 200)n)n Question: Brendan can cut 8 yards of grass per day, he bought a lawnmower and it helped him to cut more yards by Fifty percent per day. How many yards will Brendan be able to cut after week?(n Answer: The additional yard Brendan can cut after buying the lawnmower is 8 x 0.50 = 5<=+0.50=4>>4 yards. in So, the total yards he can cut with the lawnmower is <<8+4=12>>12. in Therefore, the total number yards he can cut in a week is 12 <<12+7=84>>84 yards. yards.\n#### \n##*# 84)n
C. Additional Experimental Results
C.1. Per-model Resistance Results
To complement the aggregated results reported in Tab. 3, we provide per-model resistance scores for some representative semantic-pree preserving mitigation strategy. These detailed tables show results across 10 LLMM ranging from 3B to 34B in size, enabling a more fine-grained analysis of strategy behavior across different model scales and architectures. Each table (see Tab. 16-20) corresponds to single mitigation strategy and reports its performance the five benchmarks under both mild and intensive contamination settings.
24

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Table 16. Per-model resistance scores of the vanilla benchmark (no mitigation) under mild and intensive contamination.
Model
Are-C
MMLU
TruthfulQA
GSM8K
RepliQA
Mild/ / Intensive
Mild/ Intensive
Mild/ / Intensive
Mild/ IItensivve
Mild/ / Intensive
Llama-3.2-3B
0,904 /0870
0.873 0.833
0.728 0.643
0.694 0.688
0.871 /0,803
Y5-1.5-6B
0.890 / 0,553
0,866 / 0.791
0.749/ / 0.625
0.735 / 0,718
0.724 / 0.455
vicuna-7b-v1.5
0.862 0.797
0.858 0.825
0.668 0.431
0,541 / 00408
0.837 0.661
Llama-3.1-33 1-8B
0.885 / 0.837
0.821 / 0.766
0.748/ / 0.624
0.735 / 0.755
0,444 0.209
Falcon3-108
0.965 / 0.952
0.934 / 0,923
0.796 / 0.693
0.817 / 0.820
0.932/ 0.923
Qwen2.5-148
0,962 / 0,952
0:935 / 0.907
0.892 0.794
0.815 / 0.819
0.679 0.581
Phi-3-medium
0,945 / 0,936
0.888 0.858
0.902 / 0.837
0.828 / 0.848
0.900 0.833
DeepSeek-V2-Lice
0,906 0,910
0.901 0,902
0.734 0.605
0.726 0.716
0.909/
Qwen2,5-32B
0,977 / 0.974
0.929 / 0.921
0,909 / 0.859
0.789 / 0,795
0.632/0.592
Yi-L.5-34B
0,932 0,9118
0.812 0.797
0.797 0.799
Table 17. Per-model resistance scores of the Synonym Replacement strategy under mild and intensive contamination.
Model
Are-C
MMLU
TruthfulQA
GSM8K
RepliQA
Mild/ / Intensive
Mild Intensive
Mild/ / Intensive
Mild / Intensive
Mild/ / Intensive
Llama-3.2-3B
0.899 / 0.877
0.887 / 0.845
0.716/ 0.610
0.704 / 00695
0.896/0.854
Yi-1.5-6B
0.889 0,846
0.876 0.820
0.738 0.622
0.691 / 0.716
0.809/0.600
vicuma-7b-v1.5
0.869 / 0.809
0.860 / 0.805
0.672 0.447
0.620/ / 0.491
0.869/ 0.757
1.ama-3.1-88
0,901 / 0.846
0.830 0.774
0.748 0.605
0.725 / 0.757
0.539 0.341
Falcon3-10B
0.957 0,956
0.944/ 0,929
0.786 / 0.683
0.813 0.810
0.952 0.939
Qwen2.5-14B
0,957 / 0.949
0.936 0.908
0,890 / 0.788
0.815 / 0,816
0,762 0.712
Phi-3-medium
0,943 0.931
0.913 0.877
0.909 0.860
0.828 0.832
0.917 0.869
DeepSock- V2-Lite
0.905 / 0.908
0.884/ / 0.905
0.743/0.594
0.732 / 0.726
0.916 0.894
Qwen2.5-32B
0.980 / 0.980
0.929 0.921
0.922 0.868
0.777 / 0.782
0.776 0.750
Yi-1.5334B
0.938 / 0.922
0.821 / 00806
0.813/ 0.727
0.778 / 0.791
0.291 /..159
Table 18. Per-model resistance scores of the Syntactic Modification strategy under mild and intensive contamination.
Model
Arc-C
MMLU
TruthfulQA
GSM8K
RepliQA
Mild/ / Intensive
Mild/ / Intensive
Mild/ / Intensive
Mild/ / Intensive
Mild/ Intensive
1lama-3.2-38
0.899 / 0.871
0.859 / 0.817
0.717 / 0.628
0.708 / 0.709
0.902 0.880
Yi-1.5-6B
0.870 / 0.822
0.860 0.813
0.760 0.644
0.714 0.712
0.792 0.588
vicuma-7b-v1.5
0.886 / 0.817
0.871 / 0.838
0.657 / 0.459
0.640/ / 0.543
0.873 0.749
Llama-3.1-8B
0.891 0.849
0.841 /0.755
0.756 0.641
0.710 0.747
0.506 0.320
Falcon3-10B
0.950 / 0.936
0.944 / 0.944
0.796 / 0.677
0,801 / 0,794
0.939 0.938
Qwen2.5-14B
0.952 0.950
0.924 / 0.896
0.879 0.797
0.812 / 0.824
0.784/0.729
Phi-3-medium
0.937 / 0.928
0.894/ / 00.69
0.889/ / 0.835
0.817 / 0.829
0.906 0.863
ReepSeck-V2-Litte
0.901 / 0.904
0.878 0.895
0.731 0.602
0.721 / 0.724
0.921 0.897
Qwen2,5-32B
0,980 / 0.977
0.926 / 0,922
0.919 / 0.875
0.794/ / 0.801
0.821 0.784
Yi-1.5-34B
0,930 / 0.917
0.822 0.808
0.810 / 0.743
0.781 / 0.785
0.311 0.146
25

---

A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
Table 19. Per-model resistance scores of the Choice Paraphrasing strategy under mild and intensive contamination. This strategy only applicable to multiple-choice benchmarks.
Model
Arc-C
MMLU
TruthfulQA
Mild/ / Intensive
Mild/ Intensive
Mild/ / Intensive
Llama-3.2-3B
0.900 / 0.892
0.863 / /0.832
0.726 0.633
Yi-1.5-6B
0.817
0,849 0.8824
0,761 0.622
vicuma-7b-v1.55
0.882 0.832
0.860 / 0..899
0.685 0.449
Llama-3.1-8
0.893 0.848
0.835 / 0.776
0,770 / 0.654
Falcon3-10B
0,951 0.955
0,930 / 0.928
0,797 0.669
Qwen2.5-14B 14B
0.956 0.955
0.937 00907
0.880 0.000
Phi-3-medium
0.933 00941
0.891 / 00.85
0.908 0.8
DeepScck-V2-Lite
0.896 / 0.904
0.894 / 00.902
0.737 0.608
Qwen2.5-32B
0.978 0.980
0,928 / 0.927
0.897 0.870
Yi-1.5-34B
0.928 0.914
0.850 0.817
0.810/0.72
Table 20. Per-model resistance scores the MPA strategy under mild and intensive contamination.
Model
Arc-C
MMLU
TruthfulQA
GSMSK
RepliQA
Mild/ / Intensive
Mild Intensive
Mild / Intensive
Mild / Intensive
Mild Intensive
Llama-3.2-3B
/890/
0.842 0.810
0.665
0.733 / 0.735
0.924/0.900
Yi-1.5-6B
0,888 / 0.783
0.873 0.855
0.798 / 0.633
0.738 / 0.731
0.902 0.799
-icuna-7b-v1.5
0,902 / 0.887
0.892 0.889
0.733 0.540
0.719 0.681
0.897 0.845
Llama-3.1-8B
0.887 0.879
0.865 0.813
0.825 0.652
0.686 / 0.705
0.732 0.538
Falcon3-1018
0.953 / 0.960
0.926 / 0.931
0.852 / 0,766
0.845 / 0.832
0.940 0.940
Qwen2.5-14B
0.944 0.954
0.924 0.930
0.884 0.853
0.810 0.816
0.875/0.854
Phi-3-medium
0.962 / 0.966
0.930/ 0.921
0.931 / 0.860
0.863 / 0.865
0.938 0.922
DeepSeek-V V2-Lite
0.901 0.922
0.911 00919
0.810/
0.733 0.727
0.933/0.500
Qwen2.5-32B
0.963 / 0.973
0.952 / 0.941
0.924 / 0.882
0.754 / 0.770
0.892 0.885
Yi-1.5-34B
0.919
(.84/0.88)
0.823/0.70
0.741
0.673
26


Reviews:
[{"review_id": "nf0M8JaRTC", "reviewer": "Reviewer_2iU4", "summary": "This paper investigates mitigation strategies for benchmark data contamination (BDC) in LLM evaluation. The authors argue that current approaches for assessing BDC mitigation strategies, which focus on aggregate accuracy metrics, have significant limitations. To address this, they propose two metrics---fidelity and contamination resistance---that enable question-level evaluation. Experiments with 10 LLMs, 5 benchmarks, and 20 mitigations strategies show that no existing strategy consistently outperforms the vanilla approach (i.e. no dataset update) across all benchmarks, and none balances both fidelity and contamination resistance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: More complex evaluation tasks\n\nThank you for the insightful suggestion. As the first work to rigorously assess BDC mitigation strategies for LLMs, we focus on commonly used evaluation tasks as adopted in prior BDC mitigation studies [1-3]. We agree that extending the analysis to more complex evaluation tasks is an important direction. We will discuss this limitation in the revised paper and explore it in future work.\n\n> W2: Larger LLMs\n\nWe agree that including larger models would further strengthen the reliability of our findings. However, our setup adopts **full fine-tuning** because it more faithfully approximates real-world contamination scenarios, where models are exposed to benchmark data during pre-training or continued training, compared to parameter-efficient methods like LoRA. However, full fine-tuning can be computationally expensive for larger LLMs.\n\nNotably, most prior works on BDC mitigation [1-2] only consider LLMs with up to 13B parameters. In comparison, our study includes models up to 34B, and we have made every effort to scale as far as our resources allow. We plan to include larger models in future work as resources permit.\n\n> S1: Discussion of probabilistic evaluation metrics\n\nThank you for the valuable perspective. We will include a discussion of this point in the revised paper and consider it as part of future work.\n\n> Q1: Any patterns in the types of questions that benefit most?\n\nFor multiple-choice benchmarks, we find that ITD, MPA, and choice permutation achieve high contamination resistance. In contrast, for open-ended benchmarks, we do not observe any strategy that consistently and statistically significantly outperforms the vanilla baseline in terms of contamination resistance. This may be due to the greater variability and flexibility of open-ended responses, which makes stable mitigation more difficult.\n\n\n> Q2: Toward strategies that better balance fidelity and resistance\n\nOne potential direction we are exploring in separate work involves training two reward models, one for fidelity and one for resistance, and using them to jointly guide the LLM in conditionally updating benchmarks that score highly on both axes. This remains an open question without a definitive solution, and we believe learning-based approaches offer a potential path forward.\n\n> Q3: Correlation between model size and (1) susceptibility to contamination and (2) responsiveness to mitigation\n\nThis is an excellent question. (1) Inspired by it, we compute the Spearman correlation between model size and accuracy inflation (averaged across 5 datasets) under mild and intensive contamination. We observe a negative correlation, suggesting that larger models exhibit less accuracy inflation under contamination. One possible explanation is that their stronger generalization capabilities make them less dependent on memorized benchmark content.\n\n|Contamination|Corr (**averaged** across 5 datasets)|\n|-|-|\n|Mild|-0.018|\n|Intensive|-0.401|\n\n(2) We further explore the correlation between model size and contamination resistance. Specifically, for each semantic-preserving mitigation strategy, we compute the average resistance score across all datasets and calculate its Spearman correlation with model size. **All strategies show positive correlations with model size.** This indicates even after being exposed to the original benchmark, larger models tend to preserve their evaluations on the updated benchmark, indicating higher behavioral stability. \n\n|Strategy|Corr (raw resistance) |Corr (resistance improvement)|\n|-|-|-|\n|Back-translation|0.33|-0.72|\n|Clean-Eval|0.40|-0.31|\n|Additional Incorrect Choices|0.69|-0.80|\n|Irrelevant Context|0.31|-0.70|\n|ITD|0.23|-0.31|\n|MPA|0.29|-0.14|\n|MPA-Choice + Trans-CN|0.64|-0.63|\n|MPA-Ques + Trans-CN|0.53|-0.14|\n|Choice Paraphrasing|0.68|-0.60|\n|Choices Permutation|0.68|-0.79|\n|Relevant Context|0.37|-0.36|\n|Synonym Replacement|0.33|-0.12|\n|Syntactic Modification|0.31|-0.49|\n|Translation (Chinese)|0.65|-0.14|\n|Translation (French)|0.61|0.05|\n|Typographical Perturbation|0.32|-0.39|\n|Vanilla|0.45|/|\n\nHowever, as discussed in Section 5.1, contamination resistance should be interpreted **relative to the vanilla baseline**. To assess this, we computed the correlation between model size and the resistance improvement (i.e., the difference between the strategy's resistance and that of the vanilla baseline; averaged across all datasets). Under this view, **the correlations are mostly negative**. This indicates that the *relative effectiveness* of current mitigation strategies diminishes for larger models. It highlights the need for more robust and scalable approaches that can adapt to larger LLMs.\n\n---\n\n[1] Clean-eval: Clean evaluation on contaminated large language models\n\n[2] Automating dataset updates towards reliable and timely evaluation of large language models\n\n[3] Dynamic Evaluation of Large Language Models by Meta Probing Agents", "claims_and_evidence": "The claims are generally well-supported by evidence. The authors claim that previous BDC mitigation assessment methods are insufficient is argued through examples in Figure 2, and show why question-level matching is more informative than aggregate accuracy.\n\nThe central claim that no existing strategy significantly improves resistance over the vanilla case across all benchmarks is supported by the results in Tables 3 and 4, with statistical significance testing. The data shows that while some strategies perform well on specific benchmarks, none consistently outperforms across all datasets.\n\nThe claim regarding the trade-off between fidelity and resistance is also shown  in Figure 4, where strategies are visibly clustered in either high-fidelity/low-resistance or low-fidelity/high-resistance regions, with none achieving both high fidelity and high resistance.", "methods_and_evaluation_criteria": "The methodology is appropriate and includes uncontaminated LLM-benchmark pair selection using three BDC detection methods, application of 20 mitigation strategies, controlled contamination under two scenarios (mild and intensive), and evaluation using the proposed metrics.\n\nThe selection of benchmarks (Arc-Challenge, MMLU, TruthfulQA, GSM8K, and RepliQA) and models (10 LLMs ranging from 3B to 34B parameters) provides good coverage of different evaluation contexts.", "theoretical_claims": "The paper focuses on empirical evaluation and makes limited formal theoretical claims. The definitions of fidelity and contamination resistance in Section 3 are mathematically sound, as is the extension to continuous evaluation scores in Appendix A.1.", "experimental_designs_or_analyses": "The experimental design is well-constructed with appropriate controls like three BDC detection methods to ensure uncontaminated baseline models, two different contamination scenarios, validation of contamination effectiveness, and monitoring model perplexity on held out data. One potential limitation is the focus on a specific finetuning approach for contamination.", "relation_to_broader_scientific_literature": "The paper is well-situated within the broader literature on LLM evaluation and benchmark contamination. The authors acknowledge two primary approaches to addressing BDC: creating new benchmarks and updating existing ones, focusing on the latter as a more cost-effective approach. They build upon previous mitigation strategies like Clean-Eval, ITD, and MPA while addressing limitations in their evaluation methodologies. The discussion of BDC detection methods is comprehensive, categorizing them into token probability-based, generation-based, and order-based approaches.", "essential_references_not_discussed": "Recent work on red-teaming LLMs and adversarial robustness provide some insights on developing perturbation techniques that maintain semantic equivalence while bypassing pattern recognition. These approaches directly relate to the semantic-preserving strategies examined in the paper.", "other_strengths_and_weaknesses": "Other weaknesses:\n- The benchmarks and experiments used are primarily multiple-choice and more straightforward open-ended questions, so it is unclear whether the findings generalizes to more complex evaluation tasks.\n- While the experiments cover 10 LLMs, they are all relatively small (3B to 34B parameters) compared to the SOTA models, which raises questions about if and how the results generalizes to larger models.", "other_comments_or_suggestions": "It would be interesting to see some discussion probablistic evaluation metrics that account for uncertainty in model responses are alternatives to the binary evaluation vectors used in the paper's metrics and could potentially offer a more nuanced assessment of contamination effects.", "questions_for_authors": "1. Have you noticed any patterns in the types of questions that benefit most from specific mitigation strategies?\n2. Any ideas for mitigation strategies that would more effectively balance fidelity and resistance?\n3. Have you explored/observed any correlation between the model size and their susceptibility to contamination and the responsiveness to mitigation strategies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ujYwMNK3Xh", "reviewer": "Reviewer_Eeab", "summary": "This paper introduces a systematic pipeline and proposes two metrics—fidelity and contamination resistance—to provide a fine-grained and comprehensive assessment of existing benchmark data contamination (BDC) mitigation strategies. The authors evaluated 20 different BDC mitigation approaches across 10 LLMs, 5 benchmarks, and 2 contamination scenarios, and found that none of the existing mitigation strategies consistently improved contamination resistance across all benchmarks while maintaining fidelity to the original tests", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1 & Q1: Misunderstanding of contradictory results\n\nWe clarify that our claim is not contradictory: while some **semantic-preserving** mitigation strategies (e.g., MPA and ITD) achieve significantly higher resistance scores than the vanilla case on **certain benchmarks** (e.g., MMLU, TruthfulQA, and RepliQA), no strategy consistently outperforms the vanilla case **across all benchmarks** in a **statistically significant** manner (Table 3). Our conclusion excludes semantic-altering strategies (e.g., Analysis Extension), which are only applicable to Arc-C and MMLU and thus insufficient to support a general claim. \n\nTo prevent misunderstanding, we will revise the abstract and introduction to make the claim more accurate.\n\n> Q2: Justifying the use of existing benchmarks in BDC mitigation evaluation\n\nWe emphasize that BDC mitigation is an established line of research [1–4], which builds on the assumption that widely used benchmarks such as MMLU and GSM8K are high-quality and representative of real-world question distributions. These benchmarks are commonly adopted in prior work proposing mitigation strategies to address contamination. **Building upon their assumptions**, our paper rigorously examines the effectiveness of such strategies.\n\nHigh quality does not imply perfection. Rather, it suggests that these benchmarks have broad coverage aligned with the intended evaluation objectives, making them worth preserving through mitigation rather than replacement. While we acknowledge that benchmarks like MMLU and GSM8K may contain incorrect labels and suffer from data contamination, even **their revised versions** (e.g., MMLU-Pro, GSM1K) **remain vulnerable to contamination**. This further motivates the need for robust contamination mitigation strategies and careful evaluation of their effectiveness.\n\nWe thank the reviewer for highlighting this point and the relevant references, and will cite them and include a discussion in the revision.\n\n> Q3: Novelty of the proposed metrics\n\nWe identify clear limitations in existing practices for assessing BDC mitigation strategies: (1) Accuracy drop **ignores clean accuracy**, making it unclear how much drop reflects effective mitigation; (2) Accuracy matching focuses on aggregate accuracy but **overlooks question-level mismatches**. For example, a strategy with high accuracy matching may still alter the original benchmark’s evaluation objective (Fig 2(b)).\n\nMotivated by these issues, we propose two metrics, fidelity and contamination resistance, that explicitly capture **two** types of **question-level alignment** using normalized Hamming distance. To our knowledge, this is the first work to examine BDC mitigation effectiveness along these two orthogonal dimensions, covering more desirable aspects of mitigation strategies and enabling finer-grained analysis than prior approaches.\n\n\n> Experimental Designs Or Analysis: On the realism of contamination scenarios\n\nWe included two contamination settings in our paper that reflect **common and established** practices in prior BDC mitigation work: (a) Intensive Contamination [1,3]: fine-tuning the LLM with only benchmark data. (b) Mild Contamination [4]: fine-tuning on benchmark data mixed with 20K instruction-following samples from OpenOrca.\n\nMotivated by your concern, we **add experiments** on two more scenarios: (c) Partial Contamination: only **half of the benchmark** is included in fine-tuning, mixed with 20K OpenOrca samples, while evaluation is done on the **entire benchmark**. This reflects situations where only a portion of the evaluation data is seen during training. (d) Indirect Contamination: fine-tuning and evaluation use different splits of the same benchmark, again mixed with 20K OpenOrca samples. This setting captures contamination via exposure to data from the same distribution during training, without direct sample overlap with evaluation data.\n\nWe experiment with 8 LLMs and 2 datasets, evaluating all 16 semantic-preserving strategies. We report only those strategies (with resistance scores) that achieve **statistically significantly higher resistance than Vanilla**; full results will be included in the revised appendix.\n\n(1) Partial contamination:\n- Arc-C: No strategy shows significant improvement over Vanilla;\n- TruthfulQA: Back Translation (0.807), ITD (0.824), MPA (0.833), and MPA-Ques + Trans-CN (0.813) outperform Vanilla (0.795).\n\n(2) Indirect contamination:\n- Arc-C: No strategy shows significant improvement over Vanilla;\n- TruthfulQA: ITD (0.821), MPA (0.839), and MPA-Ques + Trans-CN (0.815) outperform Vanilla (0.791).\n\nThese empirical results echo with our main claim (refer to W1 & Q1).\n\n---\n\n\n[1] Clean-eval: Clean evaluation on contaminated large language models\n\n[2] Dynamic evaluation of large language models by meta probing agents\n\n[3] Automating dataset updates towards reliable and timely evaluation of large language models\n\n[4] ConStat: Performance-Based Contamination Detection in Large Language Models", "claims_and_evidence": "There are some contradictory claims, for instance:\n\na. The authors mention that no existing BDC mitigation strategy is effective. However, the results show that some strategies (e.g., MPA, ITD, and Analysis Extension) significantly outperform the vanilla (no mitigation) approach\n\nb. The paper assumes that existing benchmarks (e.g., MMLU, GSM8K) are high-quality but there are already updated versions of these benchmarks due to issues like wrong labels, data contamination, etc. [1, 2]. \n\n1. Zhang, Hugh, et al. \"A careful examination of large language model performance on grade school arithmetic.\" Advances in Neural Information Processing Systems 37 (2024): 46819-46836.\n\n2. Wang, Yubo, et al. \"Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.\" The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2024.", "methods_and_evaluation_criteria": "1. While the proposed methods and evaluation criteria are not something groundbreaking (i.e., instead of applying approaches like paraphrasing, fine-tuning, etc.), they still make sense. \n2. The authors also fail to justify the novelty of their proposed metrics.", "theoretical_claims": "1. No issues.", "experimental_designs_or_analyses": "Contamination scenarios are quite synthetic, making the generalizability of the results questionable.", "relation_to_broader_scientific_literature": "1. Contamination scenarios are quite synthetic, making the generalizability of the results questionable.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strength: \n\n1. Studied a very important topic.\n\nWeaknesses:\n\n1. Wrong claims, lacking substantial contributions,", "other_comments_or_suggestions": "N/A", "questions_for_authors": "a. Why did you mention that no existing BDC mitigation strategy is effective even though your experiment shows contradictory results? \n\nb. Why did you state that existing benchmarks (e.g., MMLU, GSM8K) are high-quality but there are already updated versions of these benchmarks due to issues like wrong labels, data contamination, etc. \n\nc. What is the technical novelty of the proposed metrics?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7h9mNoh2EJ", "reviewer": "Reviewer_gzkQ", "summary": "Designed a systematic and controlled pipeline to provide fine-grained and comprehensive assessment of existing benchmark data contamination mitigation strategies. They focus on a question-level study experimenting with 10 LLMs, 5 benchmarks, 20 mitagation stagetries with 2 scenarios. From this, they find that no existing strategy significantly impacts benchmark resutls.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: More datasets and models\n\nOur current study includes 10 LLMs, 5 benchmarks, 20 mitigation strategies, and 2 contamination scenarios, yielding 10×5×20×2 = 2000 evaluation results. While we believe this already provides a comprehensive analysis, we agree that including more models and benchmarks would further strengthen the reliability of our findings. We will discuss this as a limitation and explore broader coverage in future work.\n\n> Q1: Model size vs. resistance scores\n\nThis is an excellent question. We appreciate the suggestion and will include an extended version of Table 3 in the appendix with model size weight information.\n\nInspired by this question, we explore the correlation between model size and contamination resistance. We examine two perspectives: **raw resistance scores** and **resistance improvement over the vanilla baseline**. Our key finding is that **(1) larger models generally exhibit higher raw resistance scores, but (2) their relative advantages over mitigation strategies tend to diminish with scale**, as detailed below.\n\n(1) For each semantic-preserving mitigation strategy, we compute the average resistance score across all datasets and calculate its Spearman correlation with model size. **All strategies show positive correlations with model size.** This indicates even after being exposed to the original benchmark, larger models tend to preserve their evaluations on the updated benchmark, indicating higher behavioral stability. \n\n|Strategy|Corr (raw resistance) |Corr (resistance improvement)|\n|-|-|-|\n|Back-translation|0.33|-0.72|\n|Clean-Eval|0.40|-0.31|\n|Additional Incorrect Choices|0.69|-0.80|\n|Irrelevant Context|0.31|-0.70|\n|ITD|0.23|-0.31|\n|MPA|0.29|-0.14|\n|MPA-Choice + Trans-CN|0.64|-0.63|\n|MPA-Ques + Trans-CN|0.53|-0.14|\n|Choice Paraphrasing|0.68|-0.60|\n|Choices Permutation|0.68|-0.79|\n|Relevant Context|0.37|-0.36|\n|Synonym Replacement|0.33|-0.12|\n|Syntactic Modification|0.31|-0.49|\n|Translation (Chinese)|0.65|-0.14|\n|Translation (French)|0.61|0.05|\n|Typographical Perturbation|0.32|-0.39|\n|Vanilla|0.45|/|\n\n(2) However, as discussed in Section 5.1, contamination resistance should be interpreted **relative to the vanilla baseline**. To assess this, we computed the correlation between model size and the resistance improvement (i.e., the difference between the strategy's resistance and that of the vanilla baseline; averaged across all datasets). **Under this view, the correlations are mostly negative**. This indicates that the *relative effectiveness* of current mitigation strategies diminishes for larger models. It highlights the need for more robust and scalable approaches that can adapt to larger LLMs.\n\n> Additional reference\n\nThank you for the suggestions. We will include the listed citations in the related work section for completeness.", "claims_and_evidence": "Claim that existing BDC mitigation strategies are not sufficient, introducing fidelity and contamination resistance metrics. They provide evidence of this in section 3 and 5.", "methods_and_evaluation_criteria": "The paper evaluates the mitigation strategies through their two interpretable scores: Fidelity and Resistance.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The experimental design is solid. The only issue might be the number of LLMs evaluated.", "relation_to_broader_scientific_literature": "This is a good overview of existing mitigating techniques in previous literature and rigirously evaluating them.", "essential_references_not_discussed": "Might want to include the following citations in the related work: [1] \"To the cutoff... and beyond? a longitudinal perspective on LLM data contamination\" https://openreview.net/forum?id=m2NVG4Htxs, [2] \"Bring Your Own Data! Self-Sensitivity Evaluation for Large Language Models\" https://openreview.net/forum?id=k2xZYPZo34#discussion, and \"Training on the test task confounds evaluation and emergence\" https://openreview.net/forum?id=jOmk0uS1hl\n\n[1] studies data contamination through the lens of time.\n[2] proposes a new evaluation framework for mitigating contamination.\n[3] shows that training on the test task can improve performance.", "other_strengths_and_weaknesses": "Strengths:\n - Interesting finding\n - Good experimental design\n\nWeaknesses: \n  - More datasets (i.e adding code evals) and models", "other_comments_or_suggestions": "For table 3, can you reconstruct this table in the appendix with an added row by weight class (including the number of models in that weight class)? I wonder if the size of the model impacts these scores. I wonder if the conclusions might be different under this view.", "questions_for_authors": "See comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O1H8WD7l1f", "reviewer": "Reviewer_P7Sk", "summary": "This paper discusses a way to evaluate Benchmark Data Contamination (BDC) mitigation strategies. The authors set up two key standards, Fidelity and Contamination Resistance, as criteria of assessing reliability of each method. By following a rigorous evaluation pipeline, experiments on 10 LLMs, 5 benchmarks, and 20 BDC mitigation strategies show that no existing strategy significantly improves resistance over the vanilla \"no benchmark update\".", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Why high fidelity is necessary\n\nHigh fidelity is necessary because a low fidelity score indicates that the updated benchmark has undergone **excessive** changes from the original benchmark, which can introduce two practical issues:\n\n(1) **Answer invalidation**: The modifications may alter the semantics of a question such that the original answer is no longer correct, requiring human annotation to ensure correctness. \n\n(2) **Difficulty or objective drift**: The updated question may no longer be appropriate for LLM evaluation. It could become too difficult, too trivial, or deviate the focus to unintended skills or knowledge domains. This requires human annotators not only to provide a new answer but also to assess whether the question remains suitable for evaluation.\n\nWe provide qualitative examples of these two cases in Tables 6 and 5, respectively. Both issues contradict the goal of BDC mitigation strategies, which is to **automatically and efficiently** update benchmarks without requiring manual checks. High fidelity ensures that updated benchmarks remain aligned with the original evaluation objective and usable at scale.\n\nIf our goal is to derive a *new* benchmark and then evaluate all models fairly (as you might have in mind), a high fidelity is not required. However, in this case, human annotation and evaluation for the validity of the new benchmark is often required, and it is a different setting from BDC contamination.\n\n\n> Q2: Why exclude \"correct->incorrect\" in contamination resistance\n\nThe goal of a mitigation strategy is to enable accurate measurement of a model’s true capability, even if the model has been **contaminated by the original benchmark**. Contamination resistance is therefore designed to assess whether the **updated benchmark** can preserve this measurement.\n\nIf we include \"correct->incorrect\" cases, we may encourage scenarios where the clean model outperforms the contaminated model on the updated benchmark, i.e., $R(M,D^S) > R(M^D,D^S)$. In practice, this would lead to **underestimating** the model’s true capacity and contradict the goal of mitigation, which is to recover reliable evaluation even after contamination. This is why **symmetric matching** is essential in our definition.\n\n\n\n\n> Q3: Benchmark selection is questionable\n\nwe have made our **best effort** to filter contamination by applying three BDC detection methods from distinct categories and selecting only models regarded as uncontaminated by all three on all benchmarks. However, we acknowledge that we still cannot fully rule out the possibility of contamination.\n\nThat said, we have made every effort to select benchmarks to ensure reliable conclusions. The four benchmarks we use, GSM8K, MMLU, Arc-C, and TruthfulQA, are **widely adopted** in prior BDC mitigation work [1-4]. In addition, we include the RepliQA dataset, a recently released benchmark with non-factual, fictional contexts. Its **recent release and non-factual nature** make it highly unlikely to be present in any model’s training data, making it a suitable candidate in our controlled pipeline.\n\n\n> Q4: Refining the claim to avoid misinterpretation\n\nTo prevent misunderstanding, we will revise the abstract and introduction to more precisely state our claim: \n\n(1) While some *semantic-preserving* mitigation strategies (e.g., MPA and ITD) achieve significantly higher resistance scores than the vanilla case on **certain benchmarks** (e.g., MMLU, TruthfulQA, and RepliQA), no strategy consistently outperforms the vanilla case **across all benchmarks** in a **statistically significant** manner.\n\n(2) Further, although some strategies perform well on one metric, none effectively balances both fidelity and contamination resistance.\n\n-------\n\n\n[1] Clean-eval: Clean evaluation on contaminated large language models\n\n[2] Inference-time decontamination: Reusing leaked benchmarks for large language model evaluation\n\n[3] Automating dataset updates towards reliable and timely evaluation of large language models\n\n[4] Dynamic Evaluation of Large Language Models by Meta Probing Agents", "claims_and_evidence": "Yes, the authors set up a robust evaluation framework and provided relevant results to support their claim.", "methods_and_evaluation_criteria": "Yes, the authors provided an intuitive evaluation framework.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes, the experimental design is clear. But there are some concerns regarding the soundness. Please refer to Weaknesses and Questions below.", "relation_to_broader_scientific_literature": "This work points out the weaknesses of prior BDC mitigation strategies, and calls for further research with respect to the criteria provided by the authors.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n- The idea is intuitive and the experimental pipeline is robust.\n- The paper reads well and is well organized\n\n## Weaknesses\n- I am not sure why high Fidelity is desired. The goal of BDC mitigation strategies is to deviate from the original benchmark, which is potentially utilized during LLM pre-training. The updated benchmark differing from the original version is not itself a problem, as long as the benchmark is evaluated on fairly. The authors did provide an example of \"turning GSM8K into a history-based benchmark\" in L181, but this is an excessively extreme case to justify using Fidelity as a criterion.\n- If Contamination Resistance is intended to consider the advantage of using the original benchmark in fine-tuning, why not just exclude the ratio of \"incorrect -> correct\"? I am not sure why \"correct -> incorrect\" should also be excluded in the resistance ratio.\n- While the overall evaluation pipeline is solid, one other concern is in the fundamental problem of the contamination detection task. The task of contamination detection itself is very difficult and is hard to verify the validity. Even if the authors used three detection methods to filter benchmarks, the following experiments building on this can be seen as unreliable.\n- In summary, based on my concerns, I am not certain if the paper's claim is sound. (1) Benchmark selection is questionable, and (2) the evaluation criteria needs further justification. Thus, the claim that none of the prior work meeting reliability standards is rather misleading.", "other_comments_or_suggestions": "N/A - see Strengths and Weaknesses", "questions_for_authors": "N/A - see Strengths and Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies an important problem in LLMs, i.e., the assessment of existing BDC mitigation strategies. To do the assessment, it designs a pipeline with two metrics—fidelity and contamination resistance. Extensive experiments have been conducted. These findings highlight the need for designing more effective BDC mitigation strategies.
Decision: Accept (poster)