Evading Data Contamination Detection for Language Models is (too) Easy

Authors: Jasper Dekoninck, Mark Niklas Mueller, Maximilian Baader, Marc Fischer, Martin Vechev
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, model evaluation, malicious actors

Abstract:
The benchmark performance of large language models (LLMs) has a high impact on their popularity and is thus of great importance to many model providers. However, the reliability of such benchmark scores as a measure of model quality gets compromised if the model is contaminated with benchmark data. While recent contamination detection methods try to address this issue, they overlook the possibility of deliberate contamination by malicious model providers aiming to evade detection. We propose a categorization of model providers based on their (de)contamination practices and argue that malicious contamination is of crucial importance as it casts doubt on the reliability of public benchmarks. To study this issue more rigorously, we analyze current contamination detection methods based on their assumptions. This analysis reveals a significant vulnerability in existing approaches: they do not account for rephrased benchmark data used during training by malicious actors. We demonstrate how exploiting this gap can result in significantly inflated benchmark scores while completely evading current detection methods.

Decision: reject
Reviewer scores: 3, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1729909879342

thread_type:
official_review

<thread_object>
creator: reviewer_ujag

<content>
summary:
This work aims to propose an evasive attack to bypass potential detection. The author first summarizes existing attacks, while studying potential detection strategies using metadata, reference models, semantic-preserving transformations, and different accessibility. Then, the author proposed that rephrasing can help to increase evasiveness.

strengths:
+ This work studies a valid topic -- detecting more evasive attack to endanger LLM safety.
+ This work summarizes many attacks and detection efforts.

weaknesses:
- Many settings are unclear: (1) For "contamination," there is no definition of the threat model regarding its knowledge, capabilities, or concrete examples in text-based attacks. (2) It is also unclear how the attacker influences benchmarks, what the adversarial samples look like, and what the adversarial objectives are. (3) Furthermore, defense settings are also unclear about the role of detector/defender and what actions can the defender take to eliminate poisoning? The author is suggested to add those information about both threat model and defender's strategies.

- The paper uses overly complicated wording, which creates barriers to understanding. Specifically: (1) "Contamination" is typically used as "poisoning." (2) The "actor" is generally referred to as "attacker" or "adversary." The "actor" typically links to specific roles (such as specific cyber threat patterns). (3) "De-Contamination" is rarely used. An alternative term is "mitigation." (4) "verbatim" could be "word-wise." (5) "Sample-level" could be more accurately expressed as "sentence (or word, token)-level." (6) It is inaccurate to use "Problematic Data Contamination" in Definition 3, as that definition is talking about possibility of injecting poisoning data D'. 

- The author did not clearly explain why pre-training requires benchmarking data. The author probably means "public" or "crowdsourced" data.

- The paper over-claimed the contribution "We define four (de-)contamination settings..." where the settings are summarized, not defined in this paper.

- The role of model providers is unclear. In section 3, model providers are both attacker and defender. The author is suggested to distinguish model providers with adversaries (the "actor").

- The paper has poor structure and incomplete content: (1) There is no related work section. (2) The author did not detail the proposed rephrasing attack, such as what prompts or techniques are used for rephrasing. (3) Definition 4 simply repeats definition 1 and 2. (4) The Definition 5 should be highlighted with rigorous formulation with later technical design addressing it. (5) It would be better to introduce "TPR" (true positive rate) and "FPR" (false positive rate) in Figure 1.

- The contribution is not significant. The author simply uses "rephrasing," which lacks strong intuition and motivation for its usefulness.

- The experiments are simple, and the settings are problematic. Specifically: (1) the author mentions several detection strategies but does not provide corresponding experiments to address those detections. (2) The paper also fails to clarify how evasiveness is guaranteed -- Is it measured by lower TPR by detection methods? (3) No baseline attacks. (4) The author is suggested to discuss the trade-off between attack effectiveness and evasiveness.

- Many sentences seem GPT-generated. It is fine to use GPT to polish writing. However, the original writing maybe logically inconsistent and unclear, so GPT alone cannot produce better logic.

questions:
Please see the comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730210504373

thread_type:
official_review

<thread_object>
creator: reviewer_mpuq

<content>
summary:
The paper presents a examination of the vulnerabilities in current benchmarking practices for LLMs. They introduce Evasive Augmentation Learning (EAL), a technique that rephrases benchmark samples to boost performance on benchmarks and evade current detection methods.

strengths:
- Method is simple and easy to understand.
- The topic is highly relevant to the current challenges in assessing the performance of large language models.

weaknesses:
- Unclear definition on "Definition 3"
- The distinguish between openly malicious and evasively malicious is somewhat problematic. It's hard to find realistic usage when applying the open malicious scenario to any of the attackers. Openly malicious is easily getting rid of by the other detection method listed.
- Sections 4 and 5 would benefit from streamlining. There is some repetition and unnecessary content that could be condensed to improve clarity and conciseness.
- Although the paper states that code is provided, it does not include detailed descriptions of the processes for data rephrasing and contamination insertion, particularly regarding how rephrased data is generated.
- The authors overclaimed the contribution. Failure to consider the technical feasibility and applicability of updating baseline data in real-world applications, resulting in proposed attack methods that are more suitable for closed environments and may not work in open and real-time updating environments

questions:
- Does the EAL technique circumvent all detection methods mentioned in A Taxonomy for Data Contamination in Large Language Models, including those that do not require pre-training data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670237166

thread_type:
official_review

<thread_object>
creator: reviewer_2ndd

<content>
summary:
This paper looks into the vulnerabilities and reliabilities of current data contamination detection methods. The authors categorize model providers based on their contamination practices, emphasizing malicious actors could intentionally contaminate models by rephrasing the benchmark data, thus evading detection while boosting model scores. Through experiments across several models and benchmarks, the authors show that the rephrasing technique could increase performance while bypassing existing detection methods.

strengths:
- The paper introduces a novel categorization of model providers based on their data contamination practices.
- Provide a simple yet effective contamination technique to evade detection, highlighting limitations in current detection methods.
- The paper is well-organized, with each section building on the previous one. Key concepts like contamination types and detection vulnerabilities are presented accessibly, making the research easy to understand and follow.

weaknesses:
The paper demonstrates the vulnerability of current detection methods to rephrasing attacks but does not explore potential defenses. Without suggestions for addressing this vulnerability, the work may feel incomplete.

questions:
- Is the performance across different benchmarks consistent? Could you provide a more detailed breakdown of per-benchmark performance?
- The paper notes that the limited sample size of the contaminated set contributes to increased variance in results. Would it be feasible to increase the sample size of the contaminated dataset to reduce this variance and enhance result stability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695979554

thread_type:
official_review

<thread_object>
creator: reviewer_ulum

<content>
summary:
This paper focuses on the data contamination problem with an analysis of existing frameworks. The paper categorizes existing frameworks in 4 categories and reveals their weaknesses by a detailed study. The paper proposes a new attack strategy based on the analysis and observation.

strengths:
1. The analysis of existing methods is reaonsable. The paper starts the discussion on categorizing existing paper with a figure and the categorization is reasonable. 
2. The selection of base models and benchmarks is representative and comprehensive.

weaknesses:
The paper lacks novelty and technical depth, and the presentation of the paper is hard-to-follow. 
1. The paper mainly studies the existing methods, however, it lacks a detailed discussion on the proposed attack strategy and the reasonableness of such a strategy. As a reader, I cannot fully understand the main design of the strategy proposed after the analysis (In Sec 5.1 I suppose). It's hard to understand the method with a short paragraph and the paper lacks a formulation or figure for the strategy. 2. The experiments just focus on evaluating existing methods and it's hard to identify the section proving the effectiveness of the proposed strategy. The experimental design doesn't the support the claim of the paper and lacks crucial experimental results. 
3. The organization of the paper is dispersive with no key idea. The presentation is not coherent. After discussing the experiment results, the paper switched to a discussion on section 7, but such a discussion is not related to the experiment results presented before. This section sounds like a "related work" section.

questions:
Please refer to the weakness part about the paper structure, technical depth, and novelty.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732228974149

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
$\newcommand{R}{\textcolor{green}{uLUM}}$
$\newcommand{S}{\textcolor{blue}{ujAG}}$
$\newcommand{T}{\textcolor{purple}{MpuQ}}$


We thank the reviewers for their detailed reviews and questions. We identified three common concerns, which we address below, before addressing individual questions in reviewer-specific responses.

**Q1. Can you clarify the structure of the paper? ($\R, \S$)**  
The paper introduces a novel perspective on (detecting) data contamination by considering malicious model providers. The structure is designed to systematically build this perspective and design an attack that evades detection while significantly improving model performance:
- **Definition of contamination (Section 2)**: Provides an understanding of the problem and identifies the goal for a malicious model provider (problematic data contamination).
- **Framework for malicious model providers (Section 3)**: Categorizes behaviors of potentially malicious actors and reviews current model providers in this context. 
- **Analysis of detection methods (Section 4)**: Investigates assumptions/requirements of existing detection methods in a new framework to reveal a key vulnerability.
- **Proposed attack strategy (Section 5)**: Proposes an attack/contamination strategy leveraging the vulnerability identified in Section 4.
- **Experimental validation (Section 6)**: Demonstrates the effectiveness of the attack and that it is indeed able to avoid detection while increasing performance.
- **Discussion and related work (Section 7)**: Discusses alternative evaluation strategies that prevent problematic contamination.

We have renamed the "Discussion" section to "Related Work" for clarity.

**Q2. Can you clarify Definition 3 on Problematic Data Contamination? ($\S, \T$)**  
Definition 3 identifies the core objective of contamination: significantly improving benchmark performance. This involves comparing the performance of two models:
- One trained on uncontaminated data.
- One trained on the same uncontaminated data and on additional contaminated data (e.g., using rephrased or actual benchmark samples).

If the latter model significantly outperforms the former, we consider the contamination to be problematic. This definition provides a measurable and objective criterion for evaluating evasion attacks.

**Q3. Why did you not include all the detection methods you discussed in the experimental evaluation? ($\R,\S,\T$)**  
To show how we evade all current detection methods, we analyze a large set of detection methods in Table 5 in Appendix B. Based on our analysis presented in Sections 4 and 5, we can exclude the following methods from this table for our evaluation:
- **Methods requiring oracle access**: Oracle-level access is not practical as training data is rarely published. However, we demonstrate in Appendix C that our evasion strategy is effective even under oracle-level detection methods.
- **Metadata-dependent detection methods**: Methods relying on metadata are inapplicable because we explicitly remove metadata from contaminated samples. This ensures that detection mechanisms dependent on metadata will fail on our approach.
- **Contamination detection approaches with no novel method**: Li (2023b), Xu et al. (2024), and Carlini et al. (2022) primarily offer analyses or conceptual discussions rather than detection strategies. As such, they do not introduce a new method that needs to be evaluated in our experiments.
- **Mathematically equivalent techniques**: Some works duplicate existing detection methods already included in our evaluation. For example, Watson et al. (2022) is a reimplementation of Mireshgallah et al. (2022), and Li (2023a) is functionally equivalent to Yeom et al. (2018). 
- **Wrong-Option Contamination**: Similarly, Deng et al. (2023) require the inclusion of incorrect options of a benchmark in the training set, which we explicitly remove during the rephrasing process.
- **User-Level Contamination Methods**: Song et al. (2019) address user-level contamination in a privacy-based use case. They do not apply to the contamination scenario studied in this work.

After excluding the above methods, we evaluate our attack against the remaining approaches, including Zhu et al. (2023), Carlini et al. (2021), Miresgallah et al. (2022), Shi (2023), Shi et al (2023), Xie et al. (2024), Zhang and Wu (2024), Zhang et al. (2024c), and Yeom et al. (2018). Our results in Section 6 demonstrate that our attack effectively circumvents these methods by reducing their performance to random chance. 

We included this discussion in Appendix B.

**Conclusion**  
We hope to have addressed all the reviewers' concerns, remain happy to answer follow-up questions, and are looking forward to their replies and an active discussion phase.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732912689148

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We want to again thank all reviewers for their reviews. We believe to have addressed all of the reviewers’ concerns in our rebuttal. Given that the end of the discussion period is fast approaching, we would like to ask the reviewers to let us know about any points they feel are left unaddressed so that we have time to reply. We would further like to ask all reviewers to acknowledge that they read our response and to consider updating their review accordingly.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors study data contamination, with a particular focus on the adversarial setting, where language models ingest test sets, making benchmark comparisons invalid. The authors set up a categorization of different types of contamination (Fig 1), study existing membership inference and contamination detention methods, and argue that malicious contamination through an adversary is a serious and hard-to-detect problem. 

Contamination of LLM benchmarks is a serious issue that erodes trust in evaluations, so this problem addresses an important problem, and the adversarial setting is generally under-studied relative to the careless-model-trainer setting. 

However, the paper is in a bit of an odd spot in terms of contributions. Most of the paper is conceptual (setting up adversary vs not, in the first 5 pages or so) and past work has discussed some of these distinctions, though not necessarily as a formal taxonomy or threat model. It would be hard to say that this paper is the first to bring attention to the issue of adversarial contamination. The reviewers had some issues with writing, and I think this generally reflects the fact that it's hard to make sense of what these first few pages are really doing.

On the 'new method' front in section 5, the authors propose rephrasing as an attack, but Yang et al 2023 have made a rephrasing-based 'attack' for benchmark contamination. Of course, Yang et al do not evaluate against detection methods like Shi et al, but I don't think 'rephrasing as a contamination attack' is itself novel either. 

I do think the paper has value as a benchmarking effort - showing that the rephrasing attack is effective at evading existing contamination detection methods - but I think this alone is hard to use as a reason to accept a paper, especially given recent works showing that MIA methods like Shi et al just don't work well even in the non-adversarial setting.