Detecting Benchmark Contamination Through Watermarking

## Abstract

Abstract Benchmark contamination poses a significant challenge to the reliability of Large Language Models (LLMs) evaluations, as it is difficult to assert whether a model has been trained on a test set.
We introduce a solution to this problem by watermarking benchmarks before their release.
The embedding involves reformulating the original questions with a watermarked LLM, in a way that does not alter the benchmark utility.
During evaluation, we can detect “radioactivity”, i.e. , traces that the text watermarks leave in the model during training, using a theoretically grounded statistical test.
We test our method by pre-training 1B models from scratch on 10B tokens with controlled benchmark contamination, and validate its effectiveness in detecting contamination on ARC-Easy, ARC-Challenge, and MMLU.
Results show similar benchmark utility post-watermarking and successful contamination detection when models are contaminated enough to enhance performance, e.g. , p 𝑝 p italic_p -val = 10 − 3 absent superscript 10 3 =10^{-3} = 10 start_POSTSUPERSCRIPT - 3 end_POSTSUPERSCRIPT for +5 % percent \% % on ARC-Easy.

## 1 Introduction

In recent years, Large Language Models (LLMs) have demonstrated remarkable advancements in their capabilities (Brown et al., 2020; Touvron et al., 2023a).
This advancement places increasingly greater emphasis on proper evaluation to both inform the state of LLM research and to guide future developments.
To this end, a multitude of benchmark datasets such as (MMLU) (Hendrycks et al., 2020), School Math 8K (GSM8K) (Cobbe et al., 2021), and the AI2 Reasoning Challenge (ARC) (Clark et al., 2018), or more recently GPQA (Rein et al., 2023) and FrontierMath (Glazer et al., 2024), are developed to measure the model’s capabilities in terms of general or specific knowledge, understanding, and scientific reasoning.

However, a significant issue that arises with these benchmarks is contamination.
This problem can occur either intentionally, by training models directly on the benchmark datasets or their reformulated versions, or unintentionally, as these datasets become mixed with the vast amounts of data used during pre-training.
For example,  Zhang et al. (2024) created a version of GSM8K with new questions similar in difficulty and form, and observed that many models show a significant drop in performance on them compared to the test set of GSM8k.
This challenges the reliability and validity of benchmark evaluations, as it becomes difficult to discern whether a model’s performance is due to genuine improvement in capabilities or mere memorization.
Furthermore, determining whether a model has been trained on a specific benchmark is very challenging, as it boils down to the issue of dataset/membership inference which has been shown to be ineffective for LLMs in realistic scenarios (Duan et al., 2024): it necessitates a prior on how the model would behave if not contaminated, such as through a held-out set from the same distribution.
In the context of benchmarks, having a held-out set would render the problem moot, as simply evaluating on it resolves the issue.

To tackle this problem, we propose a novel strategy of embedding non-intrusive watermarks in the benchmark dataset before release.
Our approach is inspired by Sander et al. (2024), who demonstrated that slightly distilling a watermarked LLM can be reliably detected, as the model retains identifiable traces of the watermark.
We extend this idea to benchmark watermarking, and when possibly different tokenizers are used by the watermarking and the suspect models.
Our approach enables reporting both model performance on the benchmark and a reliable $p$-value as a contamination score, which relates to the False Positive Rate of the contamination test (see Proposition [1](https://arxiv.org/html/2502.17259v2#Thmproposition1)).
If the reported $p$-value is low, the LLM’s training data is likely contaminated with the benchmark dataset and the performance numbers should not be trusted as genuine.
Our method requires only access to an LLM capable of rephrasing benchmark questions; see [Figure 1](https://arxiv.org/html/2502.17259v2#S0.F1) for an overview.
Our main contributions are:

- •
Rephrasing benchmark datasets with watermarking: We use Llama-3-8B-Instruct to rephrase questions from MMLU, ARC-Easy, and ARC-Challenge benchmarks. By applying the red/green list watermarking technique from Kirchenbauer et al. (2023a), we show that rephrasing effectively incorporates watermarks while preserving benchmark integrity ([subsection 3.1](https://arxiv.org/html/2502.17259v2#S3.SS1)).
- •
Extending watermark radioactivity: We extend watermark radioactivity to a pre-training setup.
We train models with up to 1B parameters on 10B tokens, varying benchmark contamination levels, each benchmark with a different secret watermarking key $\mathsf{s}$.
For instance, our results show detection of contamination with a $p$-value below $10^{-5}$ when the accuracy is only inflated by $5\%$ on 5000 MMLU questions, indicating a one in 100k chance of error to falsely flag a model as contaminated, while correctly yielding $p$-values near $0.5$ for uncontaminated models ([Figure 3](https://arxiv.org/html/2502.17259v2#S4.F3) and [Table 1](https://arxiv.org/html/2502.17259v2#S4.T1)).
- •
New detection algorithm when different tokenizers are used. Sander et al. (2024) assumed that the same tokenizer was used for both the watermarking model and the suspect model.
We propose a new algorithm when a different tokenizer is used by the suspect model, and show that it comes with reasonable loss in contamination detection confidence (Algorithm [1](https://arxiv.org/html/2502.17259v2#alg1) in sec. [3.3](https://arxiv.org/html/2502.17259v2#S3.SS3) and sec. [4.4](https://arxiv.org/html/2502.17259v2#S4.SS4))

## 2 Related Work

### 2.1 Benchmark Contamination Detection

Benchmark contamination is a significant concern in evaluating LLMs, as it can lead to unreliable assessments and unfair comparisons (Singh et al., 2024; Balloccu et al., 2024).
Although efforts are made to decontaminate pre-training corpora (Brown et al., 2020), these methods are not foolproof (Singh et al., 2024).
The impact of contamination can be assessed by comparing training runs that differ only in the inclusion of contaminated batches: Jiang et al. (2024) use this to show that even small models can exhibit improved benchmark performance due to small contamination.
Post-hoc analyses on the other hand identify score inflation by comparing performance on original versus similar questions (Brown et al., 2020; Chowdhery et al., 2023), but Yang et al. (2023) have shown that training on reformulated questions is enough to boost the performance on the original benchmark, so the difference in performance does not necessarily provide good correlational insights.

Zhang et al. (2024) craft new questions from the same distribution as GSM8K and observed that most models show a significant performance drop on these compared to the GSM8K test set.
This result highlights the contamination issue, but does not introduce a scalable solution to the problem.
In parallel, studying memorization in LLMs, such as regurgitating pre-training data, is also closely related (Carlini et al., 2022; Hartmann et al., 2023).
Techniques like membership inference attacks (Mireshghallah et al., 2022) and context-based completion checks (Golchin and Surdeanu, 2023) attempt to approximate contamination without direct access to pre-training data, but their effectiveness is debated (Duan et al., 2024).
Moreover, membership inference necessitates a held-out set from the same distribution, which, in the contexts of benchmarks, renders the problem moot.

### 2.2 Decoding-based watermarking & Radioactivity

#### Overview.

Recent advancements in watermarking techniques for large language models (LLMs) involve altering either the probability distribution (Kirchenbauer et al., 2023a) or the method used for sampling the subsequent token (Aaronson and Kirchner, 2023; Kuditipudi et al., 2023).
Detection of these watermarks is influenced by the entropy of the generated text (Christ et al., 2023; Huang et al., 2023), so further investigations propose watermarking only sections with high entropy, especially in code (Lee et al., 2023), while other studies explore “semantic” watermarks that rely on the semantic representation of the entire preceding text (Liu et al., 2023; Liu and Bu, 2024; Fu et al., 2024).

#### Green-list/Red-list watermark.

Our work focuses on the watermarking scheme proposed by Kirchenbauer et al. (2023b), which modifies the logit vector during token generation based on a context window of $k$ previous tokens and a private key $\mathsf{s}$.
Both are hashed to serve as the seed for a random number generator (RNG) to create a “greenlist” of $\gamma|\mathcal{V}|$ tokens, where $\mathcal{V}$ is the vocabulary of the tokenizer.
Logits of green tokens are incremented by $\delta$ to increase their sampling probability.
Detection involves repeating the greenlist computation for each token of a text, incrementing a score by 1 if the token is in the greenlist, and performing a statistical test on the cumulative score.
Under the null hypothesis $\mathcal{H}_{0}$ “the text is not watermarked with that scheme”, this score follows a binomial distribution (Fernandez et al., 2023).
A simple binomial test thus provides a $p$-value.

#### Radioactivity of LLM watermarks.

Sander et al. (2024) show that fine-tuning language models on LLM-generated watermarked question-answer pairs can be detected with high confidence, as the model retains traces of the watermark bias.
The authors adapt the original watermark detection tests to detect watermark “radioactivity”—a term first coined in  Sablayrolles et al. (2020) for image data—depending on the access to the suspect model and its training data.
In our work, we assume open-weight access to the LLM that is being evaluated, either because it is open-source, or because the contamination test is ran by the LLM owners themselves.
Moreover, the benchmark as well as the benchmark-specific watermarking key $\mathsf{s}$ are both known during radioactivity detection.
In this case, Sander et al. (2024) suggest using what they call “reading-mode”.
This involves scoring all next-token predictions by forwarding the watermarked text in the suspect model.
This is detailed in our context in [subsection 3.2](https://arxiv.org/html/2502.17259v2#S3.SS2).
Similar observations had been made in other scenarios.
For instance, Gu et al. (2023) demonstrate that LLM watermarks can be intentionally distilled.
Additionally, Zhao et al. (2023) introduce a signal in generated text that can be learned by other LLMs trained on it, and Jovanović et al. (2024) investigate the concept of watermark radioactivity for RAG.

## 3 Method

We first focus in section [3.1](https://arxiv.org/html/2502.17259v2#S3.SS1) on the task of rephrasing the questions of a benchmark dataset while embedding a watermark using the method proposed by Kirchenbauer et al. (2023b).
Then, in section [3.2](https://arxiv.org/html/2502.17259v2#S3.SS2), we show how to detect if a language model was trained on the watermarked benchmark.
Finally, section [3.3](https://arxiv.org/html/2502.17259v2#S3.SS3) extends the test if a different tokenizer is used by the suspect model.

### 3.1 Inserting watermark through question rephrasing

We use an instruct language model, denoted as $LM_{\text{rephrase}}$, which is assumed to be capable of rephrasing each question in the benchmark test set such that the rephrased version is logically equivalent to the original.
This is a pretty light assumption as the task of rephrasing is considerably easier than answering the question (Deng et al., 2023).
$LM_{\text{rephrase}}$ generates token per token and at each step, takes as input a context, which is the concatenation of the system prompt, rephrasing instruction, the question to rephrase and the reformulation generated so far.
Everything is tokenized into a sequence $\left(x^{(1)},\ldots,x^{(t-1)}\right)\in\mathcal{V}^{t-1}$, where $\mathcal{V}$ is the vocabulary of the tokenizer.

$LM_{\text{rephrase}}$ outputs a logits vector $\boldsymbol{\ell}^{(t)}\in\mathbb{R}^{|\mathcal{V}|}$.
The watermark embedding modifies $\boldsymbol{\ell}^{(t)}$ based on a secret key $\mathsf{s}$ (one per benchmark) and the watermark window
$\left(x^{(t-k)},\ldots,x^{(t-1)}\right)\in\mathcal{V}^{k}$

Specifically, following the method of Kirchenbauer et al. (2023b) detailed in [2.2](https://arxiv.org/html/2502.17259v2#S2.SS2), a secret-key cryptographic function hashes $\mathsf{s}$ as well as the the watermark window, which serves as a seed for a random number generator used to create a pseudo-random “greenlist” of tokens, comprising $\gamma$ = 50% of the entire vocabulary $\mathcal{V}$, for which the logits are incremented by a quantity $\delta$ to form $\tilde{\boldsymbol{\ell}}^{(t)}$, thereby increasing their probability of being sampled.
The logits vector is then transformed into a probability distribution $\mathbf{p}^{(t)}=\text{softmax}(\tilde{\boldsymbol{\ell}}^{(t)})\in[0,1]^{|
\mathcal{V}|}$, and the generation proceeds by sampling the next token $x^{(t)}$ from this distribution using a sampling procedure such as top-k sampling (Fan et al., 2018) or nucleus sampling (Holtzman et al., 2019).
The selected token is appended to the context, and the process repeats.
An example for the watermark embedding process is depicted in [Figure 2](https://arxiv.org/html/2502.17259v2#S3.F2), with a detailed version with different strength of watermarking in [Figure 5](https://arxiv.org/html/2502.17259v2#S6.F5) of app. [6](https://arxiv.org/html/2502.17259v2#S6).

#### Detectability/utility tradeoff.

There is a common tradeoff in watermarking between detection and utility.
In our case *detection* is the ability to have statistical evidence that the benchmark was used during training.
We show in [subsection 3.2](https://arxiv.org/html/2502.17259v2#S3.SS2) that it can be measured through the $p$-value, which can directly be linked to the False Postivie Rate of the detection test (see prop. [1](https://arxiv.org/html/2502.17259v2#Thmproposition1)).
A lower $p$-value thus indicates a stronger detection signal, making it more likely to identify unauthorized usage.
On the other hand, the *utility* of the watermarked benchmark is its ability to rank models and assess their performance on specific tasks.
To preserve utility, we therefore require that models perform similarly on both the original and watermarked versions of the benchmark, allowing for accurate evaluation and comparison of model performance.
Specifically, the benchmark dataset exhibits a proportion $\rho>0.5$ of green tokens after rephrasing, the greater the easier detectability.
For utility, we check if pre-trained models perform similarly on the original and rephrased versions.

Figure: (a) Embedding - benchmark rephrasing
Refer to caption: x2.png

Improving the watermarked benchmark could involve: 1) using rephrasing instructions tailored to each benchmark’s specifics, 2) employing better rephrasing models and 3) humans to review each question, correct it, or choose between different watermarked versions from various sampling seeds.

### 3.2 Radioactivity detection in a white box scenario

The strength of the watermark is determined by $\rho$, the proportion of green tokens in the text, which is influenced by $\delta$ and the entropy of the generation process.
Sander et al. (2024) demonstrate that the ability to detect whether a model has been trained on watermarked data, depends on $\rho$, as well as the proportion of watermarked text relative to the total number of training tokens, the size of the model, the fine-tuning method, and other factors.
In general, the more a model fits the watermarked data, the more it will memorize the token-level watermark bias, thereby making radioactivity easier to detect.
The authors also introduce a “reading mode” to enhance radioactivity detection when the model’s weights are accessible, as detailed in section [2](https://arxiv.org/html/2502.17259v2#S2).

We use the “reading mode” directly on the watermarked benchmark.
Specifically, the questions are forwarded through the suspect model and, for each input token, we get a next token prediction using greedy decoding, *i.e.*, selecting the most likely next token based on the suspect model’s output logits.
For detection, we replay the seed generation using the watermark window *from the inputs* and the benchmark-specific key $\mathsf{s}$ to determine the green/red split, scoring +1 if the predicted token is in the corresponding green list.
This process is illustrated in [Figure 2](https://arxiv.org/html/2502.17259v2#S3.F2).

The score function on a predicted token at index $y^{({t})}$ thus uses $W_{\textrm{score}}$ that takes as input the watermark window $(x^{(t-k+1)},\dots,x^{(t)})$ from the watermarked question, and depends on the secret key $\mathsf{s}$:

We only score the watermark windows that were not already scored, by keeping a tape of the seen ones.
We then compute the cumulative score $S$ over all $\tilde{N}$ *de-duplicated* indices $t\geq k$, and consider $\mathcal{H}_{0}:=S\sim B(\tilde{N},1/2)$ that the sum follows a binomial distribution.
The $p$-value of a test associated with score $s$, i.e., the probability of obtaining a score higher than $s$ under $\mathcal{H}_{0}$, can then be obtained theoretically from the regularized incomplete Beta function $I_{\gamma}$ (Fernandez et al., 2023):

$$ $\text{$p$-value}(s)=\mathds{P}(S\geq s\mid\mathcal{H}_{0})=I_{\gamma}(s+1, \tilde{N}-s).$ (2) $$

#### Testing for contamination.

Our primary goal is to reject the hypothesis that the model is not contaminated.
However, “contaminated” can mean either: (i) the model was not trained on the benchmark test set, (ii) the model did not memorize the watermark present in the test set (i.e. not “radioactive”), or (iii) the model’s performance is not artificially enhanced due to training on the benchmark.
Only (ii) can be properly linked to $\mathcal{H}_{0}$, but is typically strongly correlated with (i) and (iii). The proposition below shows that we can reliably detect data contamination as defined in (ii):

###### Proposition 1 .

If we define “being contaminated” as having memorized the watermark, then “not being contaminated” matches $\mathcal{H}_{0}:=S\sim B(\tilde{N},1/2)$.
Therefore, the test $T_{\alpha}$ (that rejects $\mathcal{H}_{0}$ if the $p$-value is less than $\alpha$) correctly tests for contamination, and has a False Positive Rate equal to $\alpha$.

See Appendix [7](https://arxiv.org/html/2502.17259v2#S7) for the proof and a more in-depth discussion. Consequently, we reject the hypothesis that the model has not memorized the watermark.
This is distinct from the hypothesis that the model’s test performance is not artificially enhanced due to training on the benchmark.
While it is possible that the model performs well in the benchmark without overfitting to watermark biases in questions —in which case our test would fail—, this scenario appears unlikely. Section [4](https://arxiv.org/html/2502.17259v2#S4) indeed confirms that the test accurately detects artificial boost in performance.

###### Proposition 1 .

### 3.3 Difference in tokenizers

Sander et al. (2024) assumed the same tokenizer for both the watermarked and suspect models.
We propose an adaptation in Algorithm [1](https://arxiv.org/html/2502.17259v2#alg1) for the reading mode depicted in Fig. [2](https://arxiv.org/html/2502.17259v2#S3.F2) when tokenizer $T_{1}$ is used for watermark embedding and a different tokenizer $T_{2}$ by the suspect model.
During detection, only tokens with shared prefixes across both tokenizers are considered: it is the “there exists” condition in Alg. [1](https://arxiv.org/html/2502.17259v2#alg1).
The tokenization by $T_{1}$ is used for computing the green/red split, and the suspect model’s predicted token is scored if it is also in $T_{1}$’s vocab (and if it respects deduplication, similar to before).
The test’s validity is detailed in Appendix [7](https://arxiv.org/html/2502.17259v2#S7).
The more tokens shared by the tokenizers in the text under detection, the more effectively the test is expected to detect contamination.

Figure: Algorithm 1 Reading Mode Scoring with Different Tokenizers

## 4 Results

### 4.1 Benchmark quality after watermarking

#### Set-up.

For the watermark embedding, we rephrase with Llama-3.1-8B-Instruct (Dubey et al., 2024) by default, with top-p sampling with p = $0.7$ and temperature = $0.5$ (default values on the Hugging Face hub), and the green/red watermarking scheme of Kirchenbauer et al. (2023b) with a watermark window $k=2$ and a “green list” of
size $\frac{1}{2}|V|$ ($|V|$ is the vocabulary size).
We compare different values of $\delta$ when rephrasing: 0 (no watermarking), 1, 2, and 4.
We choose to watermark ARC-Challenge, ARC-Easy, and MMLU due to their widespread use in model evaluation.
In practice, one would need to watermark their own benchmark before release.
For MMLU, we select a subset of 5000 questions, randomly chosen across all disciplines, to accelerate experimentation and maintain a comparable size to the other benchmarks.
We refer to this subset as MMLU^∗.
ARC-Easy contains 1172 questions, and ARC-Challenge contains 2372 questions.
In [Figure 5](https://arxiv.org/html/2502.17259v2#S6.F5), we show the exact instructions given to the rephrasing model (identical for all benchmarks) and the results for different watermarking strengths.
*We use a different watermarking key $\mathsf{s}$ for each benchmark.*

#### Even strong watermarking retains benchmark utility.

We evaluate the performance of Llama-3.3-1B, Llama-3.3-3B and Llama-3.1-8B on the original benchmark and the rephrased version using as similar evaluation as the one from the lm-evaluation-harness library (Gao et al., 2024).
To check if the benchmark is still as meaningful, we check that evaluated models obtain a similar accuracy on the watermarked benchmarks and on the original version (see [subsection 3.1](https://arxiv.org/html/2502.17259v2#S3.SS1)).
[Figure 3](https://arxiv.org/html/2502.17259v2#S4.F3) shows the performance on ARC-Easy.
All models perform very similarly on all the rephrased versions of the benchmark, even when pushing the watermark to $80\%$ of green tokens.
Importantly, they rank the same.
Similar results are shown for MMLU^∗ and ARC-Challenge in [Figure 7](https://arxiv.org/html/2502.17259v2#S6.F7) of Appendix [6](https://arxiv.org/html/2502.17259v2#S6), although for MMLU^∗, we observe some discrepancies.
For instance, when using a watermarking window size of 2 (subfig i), the performance of Llama-3.2-1B increases from 38$\%$ to $42\%$ between the original and the other versions.
However, we observe the same issue when rephrasing without watermarking in that case.
As detailed in [subsection 3.1](https://arxiv.org/html/2502.17259v2#S3.SS1), tuning the instruction specifically for each benchmark could help.
Note that the choice of $\delta$ depends on the benchmark and the rephrasing model, and needs to be empirically tested.
We tried increasing it more, but it broke the decoding process.

Figure: (a) Watermarking questions does not degrade utility.
Refer to caption: x3.png

### 4.2 Contamination detection through radioactivity

We now propose an experimental design to control benchmark contamination, and evaluate both the impact on model performance and on contamination detection.

#### Training set-up.

We train 1B transformer models (Vaswani, 2017) using Meta Lingua (Videau et al., 2024) on 10B tokens from DCLM (Li et al., 2024), with the same tokenizer used to embed the watermark up to the dedicated section [4.4](https://arxiv.org/html/2502.17259v2#S4.SS4).
The model architecture includes a hidden dimension of 2048, 25 layers, and 16 attention heads.
The training process consists of 10,000 steps, using a batch size of 4 and a sequence length of 4096.
Each training is distributed across 64 A-100 GPUs, and takes approximately three hours to finish.
The optimization is performed with a learning rate of $3\times 10^{-3}$, a weight decay of $0.033$, and a warmup period of 5,000 steps.
The learning rate is decayed to a minimum ratio of $10^{-6}$, and gradient clipping is applied with a threshold of 1.0.

#### Contamination set-up.

Between steps 2500 and 7500, every $5000/\#\text{contaminations}$, we take a batch from the shuffled concatenation of the three benchmarks instead of the batch from DCLM.
Each batch has
$\text{batch size}\times\text{sequence length}\times\text{number of GPUs}=4
\times 4096\times 64\approx 1\,\text{M tokens}$.
As shown in [Table 1](https://arxiv.org/html/2502.17259v2#S4.T1), the concatenation of the three benchmarks is approximately $500$k tokens, so each contamination is a gradient that encompasses all the benchmark’s tokens.
For each benchmark, any sample that ends up contaminating the model is formatted as follows:

f"Question: {Question}\nAnswer: {Answer}"

#### Evaluation.

We evaluate the accuracy of the models on the benchmarks by comparing the loss between the different choices and choosing the one with the smallest loss, either “in distribution” by using the above template seen during contamination or “out of distribution” (OOD) by using:

f"During a lecture, the professor posed a question: {Question}.
After discussion, it was revealed that the answer is: {Answer}"

In the first scenario, we evaluate overfitting, as the model is explicitly trained to minimize the loss of the correct answer within the same context.
In the second scenario, we assess the model’s ability to confidently provide the answer in a slightly different context, which is more relevant for measuring contamination.
Indeed, it’s important to note that evaluations often use templates around questions —*e.g.*, in the lm-evaluation-harness library (Gao et al., 2024)— which may not be part of the question/answer files that could have leaked into the pre-training data.
[Table 1](https://arxiv.org/html/2502.17259v2#S4.T1) focuses on $\delta=4$ and shows the increase in performance across the three watermarked benchmarks as a function of the number of contaminations when evaluated OOD.
Results for in-distribution evaluation are provided in [Table 5](https://arxiv.org/html/2502.17259v2#S6.T5) of [section 6](https://arxiv.org/html/2502.17259v2#S6) (w/o contamination, the model performs similarly on the two templates).

**Table 1: Detection and performance metrics across different levels of contamination for ARC-Easy, ARC-Challenge, and MMLU benchmarks, watermarked with $\delta=4$. The performance increase is shown for OOD evaluation as detailed in [subsection 4.2](https://arxiv.org/html/2502.17259v2#S4.SS2). The log10 $p\textrm{-value}$ of the detection test is strongly correlated with the number of contaminations, as well as with the performance increase of the LLM on the benchmark.**
|  | ARC-Easy (112k toks.) | ARC-Challenge (64k toks.) | MMLU^∗ (325k toks.) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Contaminations | $\log_{10}(p)$ | Acc. | (% $\Delta$) | $\log_{10}(p)$ | Acc. | (% $\Delta$) | $\log_{10}(p)$ | Acc. | (% $\Delta$) |
| 0 | -0.3 | 53.5 | (+0.0) | -0.3 | 29.4 | (+0.0) | -0.9 | 30.6 | (+0.0) |
| 4 | -3.0 | 57.9 | (+4.3) | -1.2 | 32.4 | (+3.1) | -5.7 | 35.7 | (+5.1) |
| 8 | -5.5 | 63.0 | (+9.5) | -4.5 | 39.3 | (+9.9) | <-12 | 40.8 | (+10.2) |
| 16 | <-12 | 71.7 | (+18.2) | <-12 | 54.3 | (+24.9) | <-12 | 54.0 | (+23.5) |

#### Contamination detection.

For each benchmark, we employ the reading mode detailed in [subsection 3.2](https://arxiv.org/html/2502.17259v2#S3.SS2) to compute the radioactivity score $S$ and the corresponding $p\textrm{-value}$.
Results are illustrated in [Figure 3](https://arxiv.org/html/2502.17259v2#S4.F3) for ARC-Easy, and in [Figure 8](https://arxiv.org/html/2502.17259v2#S6.F8) of Appendix [6](https://arxiv.org/html/2502.17259v2#S6) for the other two benchmarks, across different numbers of contaminations and varying watermark strengths $\delta$.
We observe that the stronger the watermark strength and the greater the number of contaminations, the easier it is to detect contamination: a larger negative $\log_{10}(p)$ value indicates smaller $p\textrm{-value}$s, implying a lower probability of obtaining this score if the model is not contaminated.
For instance, a $-\log_{10}(p)$ of $6$ implies that we can confidently assert model contamination, with only a $10^{-6}$ probability of error.
We also observe that without contamination, the test yields $\log_{10}(p)$ values close to $-0.3=\log_{10}(0.5)$.
This is expected because under $\mathcal{H}_{0}$, the $p\textrm{-value}$ should follow a uniform distribution between 0 and 1, which implies that [-1, 0] is a 90$\%$ confidence interval (CI) for $\log_{10}(p)$, and that [-2, 0] is a 99$\%$ CI.

[Table 1](https://arxiv.org/html/2502.17259v2#S4.T1) links the contamination detection to the actual cheating (with OOD evaluation) on the benchmarks when $\delta=4$ is used.
We can see that for the three benchmarks, whenever the cheat is greater than $10\%$, detection is extremely confident.
When the cheat is smaller, with four contaminations ranging from $+3\%$ to $+5\%$, the $p\textrm{-value}$ is small enough on ARC-Easy and MMLU^∗, but doubtful for ARC-Challenge (because smaller, see [subsection 4.3](https://arxiv.org/html/2502.17259v2#S4.SS3)).
For instance, for MMLU^∗, we can assert model contamination, with only a $10^{-6}$ probability of error when $5$ points are artificially added.

### 4.3 Ablations

Figure: Figure 4: Detection confidence as a function of performance increase on MMLU^∗ for different model sizes and #contaminations, for $\delta=4$ and OOD evaluation.
Refer to caption: x5.png

#### Impact of model size.

We also test radioactivity detection on 135M and 360M transformer models using the architectures of [SmolLM](https://github.com/huggingface/smollm) and the same training pipeline as described in [subsection 4.2](https://arxiv.org/html/2502.17259v2#S4.SS2), training each model on 10B tokens as well.
[Figure 4](https://arxiv.org/html/2502.17259v2#S4.F4) shows the detection confidence as a function of the cheat on MMLU^∗.
We find that, for a fixed number of contaminations, smaller models show less performance increase –expected as they memorize less– and we obtain lower confidence in the contamination detection test.
As detailed in [subsection 3.1](https://arxiv.org/html/2502.17259v2#S3.SS1), the $p\textrm{-value}$s indicate how well a model overfits the questions, hence the expected correlation. For a fixed performance gain on benchmarks, $p\textrm{-value}$s are consistent across models.
After $4$, $8$, and $16$ contaminations on the $1$B, $360$M, and $135$M models respectively, all models show around $+6$% gain, with detection tests yielding $p\textrm{-value}$s around $10^{-5}$.
Thus, while larger models require fewer contaminated batches to achieve the same gain on the benchmark, our method effectively measures how contamination artifically enhanced performance.

#### Impact of window size.

**Table 2: Proportion of green tokens in the predictions, number of tokens scored after deduplication and log${}_{10}(p\textrm{-value})$ for different watermark window sizes, with 16 contaminations and $\delta=4$ on ARC-Easy.**
| $k$ | $\rho$ | Tokens | $\log_{10}(p)$ |
| --- | --- | --- | --- |
| 0 | 0.53 | 5k | -6.07 |
| 1 | 0.53 | 28k | -25.89 |
| 2 | 0.53 | 47k | -38.69 |

Watermark insertion through rephrasing ([subsection 3.1](https://arxiv.org/html/2502.17259v2#S3.SS1)) depends on the watermark window size $k$.
Each window creates a unique green-list/red-list split for the next token.
Larger windows reduce repeated biases but are less robust.
Because of repetitions, Sander et al. (2024) show that smaller windows can lead to bigger overfitting on token-level watermark biases, aiding radioactivity detection.
In our case, benchmark sizes are relatively small and deduplication limits the number of tokens tested, because each watermarked window is scored only once.
Thus, smaller windows mean fewer tokens to score.
Moreoever, as shown in [Table 2](https://arxiv.org/html/2502.17259v2#S4.T2), the proportion of predicted green tokens is not even larger for smaller windows: there is not enough repetitions for increased over-fitting on smaller windows.
The two factors combined result in lower confidence.
A comparison of contamination detection across benchmarks and window sizes is shown in [Figure 7](https://arxiv.org/html/2502.17259v2#S6.F7), and the utility of the benchmarks in [Figure 8](https://arxiv.org/html/2502.17259v2#S6.F8).

#### Impact of benchmark size.

The benchmark size can significantly affect the method’s effectiveness.
With a fixed proportion of predicted green tokens, more evidence (*i.e.*, more scored tokens) increases test confidence.
As shown in [Table 1](https://arxiv.org/html/2502.17259v2#S4.T1), at a fixed level of cheating (*e.g.*, $+10\%$ on all benchmarks after $8$ contaminations), contamination detection confidence is proportional to benchmark size.
This is similar to our observations on watermark window sizes in [Table 2](https://arxiv.org/html/2502.17259v2#S4.T2).

#### Impact of rephrasing model.

The difficulty and entropy of questions can significantly affect the method’s performance.
Indeed, math questions for instance can be challenging to rephrase, even more with watermarks.
Thus, better models may be needed for technical benchmarks.
We tested rephrasing with Llama3-70B-Instruct instead of the 8B version, and observed that some 8B model failures, especially on mathy questions, are resolved with the 70B model, though quantifying this is challenging.
An example is provided in [Figure 6](https://arxiv.org/html/2502.17259v2#S6.F6) of app. [6](https://arxiv.org/html/2502.17259v2#S6).
We note that increasing $\delta$ to 8 is necessary to match the green token proportion of $\delta=2$ with the 8B model, using the same decoding parameters.
This may result from lower entropy in generation or bigger logits, as the greenlist bias is applied before the softmax (see [subsection 3.1](https://arxiv.org/html/2502.17259v2#S3.SS1)).
Moreover, in math or code, rephrasing can offer limited entropy, and even better models will not be enough.
An alternative would be to add watermarked verbose text *around* the questions, or using entropy-aware LLM watermarking (Lee et al., 2023).

#### What about using canaries?

**Table 3: 360M-parameter model sees a 64-digit canary 160 times throughout the 10000 steps.**
| Training step | $2500$ | $5000$ | $7500$ | $10000$ |
| --- | --- | --- | --- | --- |
| Matches | 4/64 | 8/64 | 6/64 | 9/64 |
| Loss | 7.4 | 6.4 | 3.8 | 2.9 |
| $p$-val | 0.9 | 0.3 | 0.63 | 0.19 |

Inserting canaries is another method to detect benchmark contamination, and has been used in benchmarks such as BIG-bench (Srivastava et al., 2022).
We compare to this alternative approach as a baseline.
A random 64-digit string is added to one question of MMLU^∗ and we pre-train a 360M-parameter model, with 160 MMLU^∗ contaminations, with the same set-up as for other experiments otherwise.
We monitor memorization by forwarding the canary through the model and count the number of correct predictions, only allowing generations of digits.
A model that has not seen the canary guesses randomly so the number of matches should follow a binomial $B(64,1/10)$.
Table [3](https://arxiv.org/html/2502.17259v2#S4.T3) demonstrates that even with 10 times more contaminations than the most contaminated setup in previous experiments, the model does not memorize the canary sufficiently to achieve a low $p$-value. This underscores the advantages of using radioactivity over canaries, especially considering that canaries can be easily removed.

### 4.4 Difference in tokenizers.

In section [4.2](https://arxiv.org/html/2502.17259v2#S4.SS2) and section [4.3](https://arxiv.org/html/2502.17259v2#S4.SS3), the tokenizer of Llama-3 was used for both the watermark embedding and by the suspect model.
Using Algorithm [1](https://arxiv.org/html/2502.17259v2#alg1), we show here that contamination detection remains strong and reliable when another tokenizer is used by the suspect model.
We keep the tokenizer of Llama-3 for watermark embedding, and use the tokenizers of Llama-1/2 (Touvron et al., 2023a, b), Llama-3, Gemma-1/2 (Team et al., 2024b, a), Gemma-3 (Team et al., 2025) for the suspect model.

[Table 4](https://arxiv.org/html/2502.17259v2#S4.T4) presents the performance metrics and contamination detection capabilities of models pre-trained with various tokenizers, both with and without contamination on MMLU^∗, with 16 contaminations, and $\delta=4$.
The vocabulary size affects the number of parameters in the model, impacting both the embedding and output layers, as highlighted in the “#Params” column.
First, we observe that the test remains reliable, as indicated by small $p$-values in the absence of contamination.
Second, the “#Tokens Scored” column shows that scoring only tokens shared across vocabularies (the trigger condition in our Algorithm [1](https://arxiv.org/html/2502.17259v2#alg1)) still results in a substantial number of tokens being scored.
This results in high detection confidence across all tokenizers.
However, we note that the test appears weaker for Llama-1’s tokenizer.
This might be due to the corresponding model having fewer parameters, making it less prone to memorizing the watermark, but this is not because fewer tokens are scored.

**Table 4: Performance and contamination detection when pretraining models with fixed backbone architecture from scratch on 10B tokens with different tokenizers, with and without 16 contamination of MMLU^∗, watermarked with $\delta=4$ using Llama-3’s tokenizer. The gray line highlights that this is the ideal case were both tokenizers match, as in previous sections. We use our new algorithm [1](https://arxiv.org/html/2502.17259v2#alg1) for the contamination detection test.**
|  |  | #Params | #Tokens Scored | w/ Contamination | w/o Contamination |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Tokenizer | Vocab Size | (in millions) | (in thousands) | $\log_{10}(p)$ | Acc. | $\log_{10}(p)$ | Acc. |
| Llama-1/2 | 32K | 376 | 149 | -7 | 39.1 | -0.1 | 27.4 |
| Gemma-1/2 | 256K | 806 | 142 | -12 | 44.3 | -0.2 | 30.1 |
| Gemma-3 | 262K | 818 | 142 | -15 | 44.5 | -0.1 | 30.3 |
| Llama-3 | 128K | 561 | 154 | -14 | 41.2 | -0.6 | 29.6 |

## 5 Limitations & Conclusion

- •
Rephrasing impact: Model performance remains similar across benchmark versions, but some questions lose coherence after rephrasing (*e.g.*, Figure [6](https://arxiv.org/html/2502.17259v2#S6.F6)), which can be difficult to spot.
Possible improvements are discussed in [subsection 3.1](https://arxiv.org/html/2502.17259v2#S3.SS1) and [subsection 4.3](https://arxiv.org/html/2502.17259v2#S4.SS3).
- •
Intentional evasion: The method is primarily designed for unintentional contamination.
Malicious actors could rephrase questions to weaken the watermark or train only on answers conditioned on questions, which would bypass radioactivity detection.
In this case, watermarking answers may be necessary, though it might not always be feasible because of their lengths.

#### Conclusion.

Watermarking benchmark appears like an promising solution to the problem of contamination in Large Language Models: experiments confirm the method’s ability to maintain benchmark utility while successfully identifying contamination.

## References

- Aaronson and Kirchner (2023)
Scott Aaronson and Hendrik Kirchner.
Watermarking GPT outputs, 2023.
[https://scottaaronson.blog/?m=202302](https://scottaaronson.blog/?m=202302).
- Balloccu et al. (2024)
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek.
Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms.
*arXiv preprint arXiv:2402.03927*, 2024.
- Brown et al. (2020)
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901, 2020.
- Carlini et al. (2022)
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang.
Quantifying memorization across neural language models.
*arXiv preprint arXiv:2202.07646*, 2022.
- Chowdhery et al. (2023)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
Palm: Scaling language modeling with pathways.
*Journal of Machine Learning Research*, 24(240):1–113, 2023.
- Christ et al. (2023)
Miranda Christ, Sam Gunn, and Or Zamir.
Undetectable watermarks for language models.
*Cryptology ePrint Archive*, 2023.
- Clark et al. (2018)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
*arXiv preprint arXiv:1803.05457*, 2018.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- Deng et al. (2023)
Yihe Deng, Weitong Zhang, Zixiang Chen, and Quanquan Gu.
Rephrase and respond: Let large language models ask better questions for themselves.
*arXiv preprint arXiv:2311.04205*, 2023.
- Duan et al. (2024)
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi.
Do membership inference attacks work on large language models?
*arXiv preprint arXiv:2402.07841*, 2024.
- Dubey et al. (2024)
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Fan et al. (2018)
Angela Fan, Mike Lewis, and Yann Dauphin.
Hierarchical neural story generation.
*arXiv preprint arXiv:1805.04833*, 2018.
- Fernandez et al. (2023)
Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon.
Three bricks to consolidate watermarks for large language models.
*2023 IEEE International Workshop on Information Forensics and Security (WIFS)*, 2023.
- Fu et al. (2024)
Yu Fu, Deyi Xiong, and Yue Dong.
Watermarking conditional text generation for ai detection: Unveiling challenges and a semantic-aware watermark remedy.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, 2024.
- Gao et al. (2024)
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.
A framework for few-shot language model evaluation, 07 2024.
[https://zenodo.org/records/12608602](https://zenodo.org/records/12608602).
- Glazer et al. (2024)
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al.
Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai.
*arXiv preprint arXiv:2411.04872*, 2024.
- Golchin and Surdeanu (2023)
Shahriar Golchin and Mihai Surdeanu.
Data contamination quiz: A tool to detect and estimate contamination in large language models.
*arXiv preprint arXiv:2311.06233*, 2023.
- Gu et al. (2023)
Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto.
On the learnability of watermarks for language models.
*arXiv preprint arXiv:2312.04469*, 2023.
- Hartmann et al. (2023)
Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West.
Sok: Memorization in general-purpose large language models.
*arXiv preprint arXiv:2310.18362*, 2023.
- Hendrycks et al. (2020)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
*arXiv preprint arXiv:2009.03300*, 2020.
- Holtzman et al. (2019)
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.
The curious case of neural text degeneration.
*arXiv preprint arXiv:1904.09751*, 2019.
- Huang et al. (2023)
Baihe Huang, Banghua Zhu, Hanlin Zhu, Jason D. Lee, Jiantao Jiao, and Michael I. Jordan.
Towards optimal statistical watermarking, 2023.
- Jiang et al. (2024)
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo.
Investigating data contamination for pre-training language models.
*arXiv preprint arXiv:2401.06059*, 2024.
- Jovanović et al. (2024)
Nikola Jovanović, Robin Staab, Maximilian Baader, and Martin Vechev.
Ward: Provable rag dataset inference via llm watermarks.
*arXiv preprint arXiv:2410.03537*, 2024.
- Kirchenbauer et al. (2023a)
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein.
A watermark for large language models.
*arXiv preprint arXiv:2301.10226*, 2023a.
- Kirchenbauer et al. (2023b)
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein.
On the reliability of watermarks for large language models, 2023b.
- Kuditipudi et al. (2023)
Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang.
Robust distortion-free watermarks for language models.
*arXiv preprint arXiv:2307.15593*, 2023.
- Lee et al. (2023)
Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim.
Who wrote this code? watermarking for code generation.
*arXiv preprint arXiv:2305.15060*, 2023.
- Li et al. (2024)
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al.
Datacomp-lm: In search of the next generation of training sets for language models.
*arXiv preprint arXiv:2406.11794*, 2024.
- Liu et al. (2023)
Aiwei Liu, Leyi Pan, Xuming Hu, Shiao Meng, and Lijie Wen.
A semantic invariant robust watermark for large language models.
*arXiv preprint arXiv:2310.06356*, 2023.
- Liu and Bu (2024)
Yepeng Liu and Yuheng Bu.
Adaptive text watermark for large language models.
*arXiv preprint arXiv:2401.13927*, 2024.
- Mireshghallah et al. (2022)
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, and Reza Shokri.
Quantifying privacy risks of masked language models using membership inference attacks.
*arXiv preprint arXiv:2203.03929*, 2022.
- Rein et al. (2023)
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman.
Gpqa: A graduate-level google-proof q&a benchmark.
*arXiv preprint arXiv:2311.12022*, 2023.
- Sablayrolles et al. (2020)
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou.
Radioactive data: tracing through training.
In *International Conference on Machine Learning*, pages 8326–8335. PMLR, 2020.
- Sander et al. (2024)
Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, and Teddy Furon.
Watermarking makes language models radioactive.
*arXiv preprint arXiv:2402.14904*, 2024.
- Singh et al. (2024)
Aaditya K Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes.
Evaluation data contamination in llms: how do we measure it and (when) does it matter?
*arXiv preprint arXiv:2411.03923*, 2024.
- Srivastava et al. (2022)
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.
Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.
*arXiv preprint arXiv:2206.04615*, 2022.
- Team et al. (2024a)
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.
Gemma: Open models based on gemini research and technology.
*arXiv preprint arXiv:2403.08295*, 2024a.
- Team et al. (2024b)
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al.
Gemma 2: Improving open language models at a practical size.
*arXiv preprint arXiv:2408.00118*, 2024b.
- Team et al. (2025)
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.
Gemma 3 technical report.
*arXiv preprint arXiv:2503.19786*, 2025.
- Touvron et al. (2023a)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023a.
- Touvron et al. (2023b)
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models.
*arXiv preprint arXiv:2307.09288*, 2023b.
- Vaswani (2017)
A Vaswani.
Attention is all you need.
*Advances in Neural Information Processing Systems*, 2017.
- Videau et al. (2024)
Mathurin Videau, Badr Youbi Idrissi, Daniel Haziza, Luca Wehrstedt, Jade Copet, Olivier Teytaud, and David Lopez-Paz.
Meta Lingua: A minimal PyTorch LLM training library, 2024.
[https://github.com/facebookresearch/lingua](https://github.com/facebookresearch/lingua).
- Yang et al. (2023)
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica.
Rethinking benchmark and contamination for language models with rephrased samples.
*arXiv preprint arXiv:2311.04850*, 2023.
- Zhang et al. (2024)
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al.
A careful examination of large language model performance on grade school arithmetic.
*arXiv preprint arXiv:2405.00332*, 2024.
- Zhao et al. (2023)
Xuandong Zhao, Yu-Xiang Wang, and Lei Li.
Protecting language generation models via invisible watermarking.
In *International Conference on Machine Learning*, pages 42187–42199. PMLR, 2023.

## 6 Additional Experiments

### 6.1 Qualitative Examples

On one question from ARC-Easy, we compare qualitatively different watermarking strengths in [Figure 5](https://arxiv.org/html/2502.17259v2#S6.F5).
We also show failure cases in fig. [6](https://arxiv.org/html/2502.17259v2#S6.F6), but where rephrasing with the 70B model works.

Figure: Figure 5: Benchmark watermarking example on a question of ARC-easy. The quality of the question is not affected by the rephrasing, even with strong watermark. The proportion of green tokens is given in parenthesis.

Figure: Figure 6: Watermarking failure on an ARC-Challenge question with an $8$B model, while the $70$B succeeds.

### 6.2 Additional Experimental Results

#### Evaluation Template.

As detailed in [subsection 4.2](https://arxiv.org/html/2502.17259v2#S4.SS2), we evaluate the accuracy on the benchmark using both the same template seen during contamination and an alternative one. [Table 5](https://arxiv.org/html/2502.17259v2#S6.T5) presents the results when evaluated with the same template.
Without contamination, the model performs similarly across the two templates, but a differences appear with contaminations.

#### Ablations on different benchmarks, watermark strength, watermark window sizes, and number of contaminations.

Results for all benchmarks (ARC-Easy, ARC-Challenge, and MMLU^∗), with variations in watermark window size, number of contaminations, and watermark strength, are shown in [Figure 7](https://arxiv.org/html/2502.17259v2#S6.F7) for utility and [Figure 8](https://arxiv.org/html/2502.17259v2#S6.F8) for radioactivity detection.
For utility, all models perform very similarly on all the rephrased versions of the benchmarks, even when pushing the watermark to $80\%$ of green tokens, although for MMLU^∗, we observe some discrepancies.
For instance, when using a watermarking window size of 2 (subfig i), the performance of Llama-3.2-1B increases from 38$\%$ to $42\%$ between the original and the other versions.
However we observe the same issue when rephrasing without watermarking in that case.
The watermark window size does not have an impact.
For radioactivity detection on the other hand, as detailed in [subsection 4.3](https://arxiv.org/html/2502.17259v2#S4.SS3), smaller window sizes correlates with lower detection confidence.

**Table 5: Detection and performance metrics across different levels of contamination for ARC-Easy, ARC-Challenge, and MMLU benchmarks, watermarked with $\delta=4$. The performance increase is for in distribution evaluation as detailed in [subsection 4.2](https://arxiv.org/html/2502.17259v2#S4.SS2). Similar results for a different templates are shown in [Table 1](https://arxiv.org/html/2502.17259v2#S4.T1).**
|  | ARC-Easy (1172 quest.) | ARC-Challenge (2373 quest.) | MMLU^∗ (5000 quest.) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Contaminations | $\log_{10}(p)$ | Acc. | (% $\Delta$) | $\log_{10}(p)$ | Acc. | (% $\Delta$) | $\log_{10}(p)$ | Acc. | (% $\Delta$) |
| 0 | -0.3 | 51.7 | (+0.0) | -0.3 | 28.5 | (+0.0) | -0.9 | 30.4 | (+0.0) |
| 4 | -3.0 | 61.3 | (+9.9) | -1.2 | 35.1 | (+7.0) | -5.7 | 36.9 | (+6.5) |
| 8 | -5.5 | 68.2 | (+16.9) | -4.5 | 42.2 | (+14.1) | <-12 | 43.0 | (+12.6) |
| 16 | <-12 | 84.1 | (+32.8) | <-12 | 65.3 | (+37.2) | <-12 | 62.1 | (+31.7) |

Figure: (a) ARC-Easy, Window size 0
Refer to caption: x6.png

Figure: (a) ARC-Easy, Window size 0
Refer to caption: x15.png

## 7 Proof of correctness for contamination detection

We give the proof of Proposition [1](https://arxiv.org/html/2502.17259v2#Thmproposition1).
We remind that $\mathcal{H}_{0}$ is “The cummulative score $S$ follows a binomial distribution $B(\tilde{N},0.5)$” and $\text{$p$-value}(s)=\mathds{P}(S(X_{N})\geq s\mid\mathcal{H}_{0})=I_{\gamma}(s
+1,N-s)$ and:

###### Proposition 1 .

If we define “being contaminated” as having memorized the watermark, then “not being contaminated” matches $\mathcal{H}_{0}:=S\sim B(\tilde{N},1/2)$.
Therefore, the test $T_{\alpha}$ (that rejects $\mathcal{H}_{0}$ if the $p$-value is less than $\alpha$) correctly tests for contamination, and has a False Positive Rate equal to $\alpha$.

###### Proof.

Assume that “$M$ has not memorized watermark bias with secret key $\mathsf{s}$”.
Since the summed scores are i.i.d. due to de-duplication, and independent of the watermarking process because the suspect model has no other knowledge about $\mathsf{s}$, and because we exclude the possibility of simply repeating watermarked tuples from the prompt through de-duplication, there is no bias towards the green or red tokens specific to $\mathsf{s}$.
Therefore, the indicators $\mathds{1}\left(y^{(t)}\text{ is in the greenlist of }\left(\mathsf{s},(x^{(t-
i+1)})_{i=k}^{1}\right)\right)$ are i.i.d. simulations distributed according to a Bernoulli distribution with parameter $0.5$ (in expectation over the keys).
Thus, $S$ follows a binomial distribution $B(\tilde{N},0.5)$.
So, $\mathcal{H}_{0}$ is true.

Reciprocally, if $\mathcal{H}_{0}$ is True, then there is no bias towards the green tokens, which by definition means that it has not memorized the watermark.
The $p$-value is exactly the probability to observe a score as extreme as $s$ under $H_{0}$, so it is the probability to observe a score as extreme as $s$ if $M$ has not memorized the watermark with secret key $\mathsf{s}$ present in the benchmark.
Now let $T_{\alpha}$ be the test that rejects $\mathcal{H}_{0}$ if the $p$-value is less than $\alpha$.
It correctly tests for contamination, and has a FPR of $\alpha$.
∎

###### Proposition 1 .

###### Proof.

## 8 Compute Resources

We use our internal cluster with A-100 GPUs with 80GB memory, and:

- •
Each radioactivity detection test took less than 30 minutes on a single GPU. We processed the benchmark through the model, which contains a maximum of 325k tokens for MMLU^∗ (see [1](https://arxiv.org/html/2502.17259v2#S4.T1)).
- •
Pretraining of the 1B models was conducted on 8 nodes (so 64 GPUs) and took approximately six hours.
Training of smaller models, with 360M and 135M parameters, was performed on 4 nodes, taking 2 hours and 1 hours respectively.

Overall, we estimate that training the 1B models required approximately 5,000 GPU hours, calculated as 3 (different window sizes) x 4 (different degrees of contamination) x 6 x 8 x 8 (GPU hours for training).
We approximate an additional factor of 2 for the other models trained, resulting in a total of approximately 10,000 GPU hours.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: * Will the proposed method be applicable to other general forms of dataset?

* Could the authors include the % percentage of scored tokens in Table 5?

* The method does not seem to be applicable on existing benchmarks, can I suggest that the paper acknowledges this limitation in their work?
soundness: 3
strengths: * The paper is well-written, easy to follow, and features comprehensive, well-executed experiments.
* The proposed technique is demonstrably effective. To confirm its reliability and understand how the repetition factor influences the strength of the statistical test, the authors trained multiple LLMs from scratch on a pre-contaminated corpus.
* The authors propose a novel detection algorithm to detect radioactivity in case of tokenizer mismatch between the rephrasing LLMs and the suspect LLMs. This enhances the practical applicability of the algorithm.
summary: The paper introduces a method that offers provable evidence of benchmarks for LLMs. The key idea involves watermarking benchmarks before public release, achieved through rephrasing LLMs. Contamination in downstream models is then reliably identified using a hypothesis test that leverages the "radioactivity" inherent in watermarks. The authors successfully validate this approach, demonstrating its efficacy by detecting contamination in an LLM pre-trained on a contaminated corpus. Furthermore, they confirm that the rephrasing process maintains the original utility of the benchmarks.
weaknesses: * The method overlaps significantly with past works that were not used as baselines or mentioned in related works [1,2,3,4]. For example, [1] also proposes a similar idea and proposes a hypothesis test that leverages watermarked rephrases. [2] proposes a dataset inference method that can also be used for detecting contamination. Works such as these should be compared in related works and evaluated against as benchmarks. 


* The current demonstration, while confirming the utility of the rephrased benchmarks across various LLM sizes, is limited because all tested LLMs belong to the same model family as the one used for the original rephrasing. It is essential that the authors extend their evaluation to include LLMs from different model families.


* For reliable detection at a statistically significant strength (requiring approximately 16 repetitions as shown in Figure 3b), a substantial amount of repetition is necessary. Furthermore, it would be beneficial to include a demonstration of how the size of the total training corpus influences the test's strength. The detection also requires benchmarks of significant sample size (around 2k-4k) samples.


* The authors should also empirically verify that their method is robust against false positives using off the shelf LLMs. For a more rigorous test of false positives, the detection strength should also be evaluated against keys that are distinct from the contaminated keys. 


I would be willing to reconsider my overall assessment based on authors' responses to the above points. For instance, based on favorable comparisons with highly relevant prior work.

References:

[1] STAMP Your Content: Proving Dataset Membership via Watermarked Rephrasings. ICML 25

[2] LLM Dataset Inference: Did you train on my dataset?

[3] DE-COP: Detecting Copyrighted Content in Language Models Training Data

[4] Proving Test Set Contamination in Black Box Language Models

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - The experiments focus on question-answering benchmarks. How do you envision this watermarking approach being applied to other types of evaluation, such as code generation, mathematical reasoning (where rephrasing is brittle), or creative writing tasks?
- The method focuses on watermarking the questions. Have you considered the possibility of watermarking the provided answers for open-ended questions?
soundness: 3
strengths: - Proactive and Verifiable: The core strength is its shift from post-hoc, inferential detection methods to a proactive approach that provides verifiable, statistical proof (a p-value) of contamination. Meanwhile, the experiments show that even with a strong watermark, the rephrased benchmarks remain effective for evaluating and ranking models, with performance being very similar to the original versions.
- Practicality: The paper addresses the practical challenge of distinct tokenizers by introducing a robust cross-tokenizer detection algorithm, making the solution more broadly applicable.
summary: This paper proposes a proactive framework to detect benchmark contamination in LLMs. Instead of trying to find evidence of contamination after a model is trained, the authors suggest embedding a cryptographic watermark into a benchmark before its public release. This is achieved by using an LLM to rephrase the benchmark's questions, embedding a secret statistical signal (the watermark) during the text generation process.
To test a suspect model, a benchmark provider can then check for the presence of this watermark's statistical bias in the model's predictions. The method requires white-box (logit) access to the suspect model and includes a novel algorithm to handle mismatches between the tokenizers used for watermarking and by the suspect model. Through extensive experiments involving pre-training 1B-parameter models with controlled contamination on benchmarks like MMLU and ARC, the authors demonstrate that their method preserves the benchmark's utility while detecting contamination with very high statistical confidence.
weaknesses: - White-Box Access Requirement: The detection test requires full logit access to the suspect model. This limits its use to open-source models and cannot be used by external parties to audit closed, API-only models.
- Vulnerability to Intentional Evasion: The framework is primarily designed to detect unintentional contamination from sources like web scraping. A determined, malicious actor could potentially devise strategies to circumvent detection, such as by rephrasing the questions again to remove the watermark or by training only on the answers.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 2
questions: - What is deduplication?

- If the next token is almost deterministically identified based on the previous token, artificially increasing the probabilities of the green tokens as proposed may make the rephrased question invalid. Does this never happen?

- Lines 270--271:
> We train 1B transformer models (Vaswani, 2017)

  Do the authors use the exactly the same architecture and the training objective as in Vaswani et al. (2017)? (Also, please correct this bibliography item. The other authors are missing.)

- In the experiment of Table 4, why do the authors limit the $k$ so small? I find the proposed $k=2$ quite small. Why don't you test larger ones?

**Other minor comments**:

- Lines 286--287: What are "the different choices" and why do we want to choose "the one with the smallest loss" for the evaluation?

- Lines 279--280:
> Between steps 2500 and 7500, every 5000/#contaminations, we take a batch from the shuffled concatenation of the three benchmarks instead of the batch from DCLM.

  This sentence is unclear. I believe it requires rewriting.

- "Inferential evidence": I think I understand the nuance, but I would like to suggest avoiding the word "inferential" in such an informal sense, especially in research on artificial intelligence. Do the authors believe these words are appropriate here?

- Table 1 (and other similar experiments): What is "the number of condemnations"? (What is the unit?)

- Lines 285--305: what is the difference between the setups in calculating the p-values in Table 1 (and Table 6) and Figure 3b (and Figure 8)? I suppose the radioactivity score $S$ was only used in the Contamination detection experiment.

- Lines 350--352:
> Table 1 links the contamination detection to the actual cheating on the benchmarks when $\delta = 4$ is used. We see that for the three benchmarks, whenever the cheat is greater than 10%, detection is extremely confident.

  How do I see this from Table 1?

- "Figure 4.4": Figure 4?

- Line 411--412: "larger models require fewer contaminated batches to achieve the same gain on the benchmark". Is this something we can see from Figure 4?

- In Figure 5:
> Question: “The rate of acceleration of an object is determined by the mass of the object and”

  Is this question complete?

- In Table 4: "dedup" --> deduplication?
soundness: 3
strengths: - The paper tackles a relevant and important issue concerning the modern research area.

- The proposed ideas of generating the pseudo-random green list is interesting and effective.

- It presents extensive empirical results for supporting the authors' claims and for analyzing the impact of the hyper parameters.

- There is a theoretical guarantee (Proposition 1) for the proposed detection method.
summary: This paper proposes a method that embeds watermarks into questions in LLM benchmark datasets to enable benchmark contamination detection. The watermarks are added by rephrasing the questions with increasing the probabilities of a list of randomly selected tokens (called the green list) being chosen as the next token in the decoding process. The list being randomly generated, a model that has never seen the dataset will predict the next token independently of the green list, conditionally on the previous tokens (the null hypothesis). On the other hand, if a model were trained on a benchmark dataset with proposed watermarks, its predictions would have statistical dependence on the watermarks (the alternative hypothesis). The proposed method uses this fact to construct a statistical test for the detection. Proposition 1 shows that this test correctly controls the false positive error rate.

Then, the authors present several experiments to confirm that
the rephrasing does not largely change the performance evaluation from the original one and that the p-values are sufficiently small when there is contamination. Moreover, they provide more numerical analyses on the impacts of the model size, window size, benchmark size, rephrasing model, tokenizer, and so on.
weaknesses: - The proposed method seems to focus on models that are trained and make inference based on next token prediction, but the paper does not mention this limitation.

- The proposed method is effective to detecting contamination, but it does not prevent it.

- Some details about the proposed method and the experiments are missing.

- Some part of the writing is not clear to me.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See above.
soundness: 3
strengths: 1. The paper tackles an important and timely problem: benchmark contamination directly affects reported leaderboard results and evaluation integrity.

2. The engineering implementation is practical, offering a “pre-release watermark → post-release verification” workflow with tokenizer alignment, which could serve as a foundation for evaluation governance.
summary: This paper proposes a method for detecting whether LLMs have been trained on benchmark data (benchmark contamination). Before benchmark release, the authors rephrase the benchmark samples using keyed semantic-preserving transformations, introducing a slight probabilistic bias toward a “greenlist” of tokens. If a model has encountered these samples during training, its conditional distribution will exhibit a statistically significant preference for the greenlist tokens. Detection is based on a binomial significance test, providing verifiable $p$-values. Controlled contamination experiments show a consistent relationship between performance inflation and statistical significance. The paper also provides tokenizer-aligned implementations and multiple benchmark experiments, proposing a framework that can be adopted for evaluation governance.
weaknesses: The paper’s core strategy of keyed rephrasing combined with statistical testing is closely related to several existing lines of research across benchmark, pretraining, and fine-tuning contexts, making the boundary of novelty somewhat blurred.  

(Benchmark scenario) — STAMP (Rastogi et al., 2025) introduces multi-keyed rephrasings for the same original text and conducts paired statistical tests between public and private variants to prove whether a benchmark or test set was included in training. This process is conceptually identical to this paper’s approach of keyed rephrasing and post-hoc significance testing, differing mainly in the form of the test statistic (paired likelihood versus binomial counting) and artifact management. Similarly, Oren et al. (2023) present a black-box proof of contamination using order-based likelihood tests, which, while not using watermarks, share the same objective of providing verifiable contamination evidence without access to training data or weights.

(Pretraining scenario) — STAMP’s main goal is dataset membership proof for pretraining corpora. Its two-stage pipeline (keyed rephrasing plus paired test) and threat model closely mirror those of the present paper if we treat a benchmark as a subset of the pretraining corpus. To stand out, the current paper needs to emphasize its incremental advances in statistical calibration, detection power, or governance workflow.

(Fine-tuning scenario) — The causal chain (insert signal during training → detect significance after training) is the same as in the current work, differing mainly in target domain (fine-tuning versus benchmark). Can You Finetune Your Binoculars? (Elhassan et al., 2025) embeds watermarks into model weights using dual LoRAs for generator and observer optimization, focusing on model-level traceability but following a similar training-stage embedding and post-hoc verification logic. Towards Watermarking of Open-Source LLMs (Gloaguen et al., 2025) systematically examines watermark durability under model merging, quantization, and further fine-tuning, discussing post-training detectability in open-weight models. These efforts share the conceptual foundation of encoding verifiable biases during training and statistically detecting them afterward.  
   
   Thus, although this paper applies the framework to benchmark evaluation, the conceptual path—training-time signal embedding followed by statistical testing—has been explored across these adjacent settings. I appreciate the authors with further discussion on the following references. Besides, this paper did not discuss the usage of LLMs, which is required by ICLR this year.

---

### References

- Elhassan, F., Ajroldi, N., Orvieto, A., & Geiping, J. (2025). Can you finetune your binoculars? Embedding text watermarks into the weights of large language models. arXiv. https://arxiv.org/abs/2504.06446  
- Gloaguen, T., Jovanović, N., Staab, R., & Vechev, M. (2025). Towards watermarking of open-source LLMs. arXiv. https://arxiv.org/abs/2502.10525  
- Oren, Y., Meister, N., Chatterji, N., Ladhak, F., & Hashimoto, T. B. (2023). Proving test set contamination in black box language models. arXiv. https://arxiv.org/abs/2310.17623  
- Rastogi, S., Maini, P., & Pruthi, D. (2025). STAMP your content: Proving dataset membership via watermarked rephrasings. ICML 2025 / arXiv. https://arxiv.org/abs/2504.13416  
- Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., & Stoica, I. (2023). Rethinking benchmark and contamination for language models with rephrased samples. arXiv. https://arxiv.org/abs/2311.04850


Decision: Reject