On The Fragility of Benchmark Contamination Detection in Reasoning Models
Authors: Han Wang
Haoyu Li
Brian Ko
Huan Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark Contamination
Large Reasoning Model
Benchmark Contamination Detection
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Leaderboards for large reasoning models (LRMs) have turned evaluation into a competition, incentivizing developers to optimize directly on benchmark suites. A shortcut to achieving higher rankings is to incorporate evaluation benchmarks into the training data, thereby yielding inflated performance, known as benchmark contamination. Despite that numerous contamination detection approaches have been proposed, surprisingly, our studies find that evading contamination detections for LRMs is alarmingly easy. We focus on the two scenarios where contamination may occur in practice: (I) when the base model evolves into LRM via supervised fine-tuning (SFT) and reinforcement learning (RL), we find that contamination during SFT can be originally identified by contamination detection methods. Yet, even a brief Group Relative Policy Optimization (GRPO) training can markedly \textbf{conceal contamination signals} that most detection methods rely on. Further empirical experiments and theoretical analysis indicate that Proximal Policy Optimization (PPO) style importance sampling and clipping objectives are the root cause of this detection concealment, indicating that \textbf{a broad class of RL methods} may inherently exhibit similar concealment capability; (II) when SFT contamination with CoT is applied to advanced LRMs as the final stage, most contamination detection methods \textbf{perform near random guesses}. Without exposure to non-members, contaminated LRMs would still have more confidence when responding to those unseen samples that share similar distributions to the training set, and thus, evade existing memorization-based detection methods. Together, our findings reveal the unique vulnerability of LRMs evaluations: Model developers could easily contaminate LRMs to achieve inflated leaderboards performance while leaving minimal traces of contamination, thereby strongly undermining the fairness of evaluation and threatening the integrity of public leaderboards. This underscores the urgent need for advanced contamination detection methods and trustworthy evaluation protocols tailored to LRMs.

Published as a conference paper at ICLR 2026
ON THE FRAGILITY OF BENCHMARK CONTAMINA- TION DETECTION IN REASONING MODELS
Han Wang! Haoyu Li
Brian Ko
Huan Zhang'
I University of Illinois Jrbana-Champar [hanw14, 4, hacyuli5 siliiniiiiiiiisi.i wwwnoii......oo...........e...ee edu, kkm91183 101718wwww..... edu, huanShuan -aannn.-aan...a.a com
University of Washington
ABSTRACT
Leaderboards for large reasoning models (LRMs) have turned evaluation into a competition, incentivizing developers to optimize directly on benchmark suites. A shorteut to achieving higher rankings is to incorporate evaluation benchmarks into the training data, thereby yielding inflated performance, known benchmark contamination. Despite that numerous contamination detection approaches have been proposed, surprisingly,
studies find that evading contamination detec- tions for LRMs is alarmingly easy. We focus on the two scenarios where con- tamination may oceur in practice: (I) when the base model evolves into LRM via supervised fine-tuning (SFT) and reinforcement learning (RL), we find that contamination during SFT can be originally identified by contamination detection methods. Yet, even a brief Group Relative Policy Optimization (GRPO) training can markedly conceal contamination signals that most detection methods rely on. Further empirical experiments and theoretical analysis indicate that Proxi- mal Policy Optimization (PPO) style importance sampling and clipping objec- tives are the root cause of this detection concealment, indicating that broad class of RL methods may inherently exhibit similar concealment capability; (II) when SFT contamination with CoT is applied to advanced LRMs as the final stage, most contamination detection methods perform near random guesses. Without exposure to non-members, contaminated LRMs would still have more confidence when responding to those unseen samples that share similar distri- butions to the training set, and thus, evade existing memorization-basee detec- tion methods. Together, our findings reveal the unique vulnerability of LRMs evaluations: Model developers could easily contaminate LRMs achieve in- flated leaderboards performance while leaving minimal traces of contamination, thereby strongly undermining the faimess of evaluation and threatening the in- tegrity of public leaderboards. This underscores the urgent need for advanced contamination detection methods and trustworthy evaluation protocols tailored LRMs. Our code is available at htttts::///tthhii http:///gtttbb..// com/ www.//oo.//a..... AASTRAI--Gruup.
LRM Cont Detection Arena.git
INTRODUCTION
Competition among model developers has intensified as Large Language Models (LL.Ms) have demonstrated remarkable capabilities in various real-world tasks (Achiam et al.) 2023 Wang al.) 2024). The leaderboards for performance are becoming a competitive arena for all state-of-the-an (SOTA) LLMs. However, inadvertently, benchmark samples may appear during LLMs' pre-training due to vast amounts of web-scraped training data. In addition, in the pursuit of publicity, some model developers may even deliberately incorporate benchmark data into their training sets (Sun) et al., (222),, resulting in inflated benchmark performance and leaderboard rankings. We refer to this as the benchmark contamination problem in L.LMs (Xu al. 2024 Balloceu et al.) 2024).
Accordingly, various benchmark contamination detection methods have been proposed to determine whether specific benchmarks were used during training (Yeom al.] 2018 Mattern et al.) [023] Shi et al. 2023 Dong et al.] 2024 Tu et al., 2024), based on the assumption that contamination in LLMs primarily involves memorizing the benchmark data (Wu et al. 2025). These methods rely on
"Equal Contribution.

---

Published as a conference paper at ICLR 2026
Stage I: pre-LRM
Contuminated!
c Clean!
Stage II: post-LRM
Clean!
Base
SET
SFT Conta
SFT Conta Model w/ RL
LRM
SFT
SFT Conta LRM
Model
Model
Clean
Benchmark
Benchmark w/ CoT
w/ Co'
CoT
Figure 1: Two scenarios where contamination may happen to LRMs. In Stage I (pre-LRM), while SFT contamination to the base model is initially detectable, contamination evidence can be con- cealed through subsequent RL training. In Stage II (post-LRM), extensive contamination with CoT on advanced LRMs barely leaves evidence for existing memorization-based detection methods,
separability in some distributions between members (i.e., seen samples during contamination) and non-members (i.e., unseen samples). However, LLMs have started to evolve into Large Reason- ing Models (LRMs) (Guo et al. 2025 Jaech et al.) 2024), benchmark contamination detection faces two key challenges: LRMs rely on chain-of-thought (CoT) reasoning to reach final answers, but model developers would not release their training CoT data, and contamination detectors typically only have access guestion-answe answer pairs without the intermediate reasoning steps used during train- ing. This absence of training sequences makes detection substantially more challenging. (2) LRMs primarily acquire reasoning abilities during two stages: SFT and RI.. This potentially provides de- velopers with opportunities to manipulate leaderboard performance by strategically contaminating benchmarks in the carlier stage (c.g., SFT), while evading detection methods through subsequent training (e.g.. RL). Given these challenges, the effectiveness of existing detection methods against LRM contamination remains uncertain.
In this paper, we present the first systematic study of benchmark contamination in LRMs, structured around two points where contamination can happen. In particular, Stage I (pre-LRM) investigates contamination introduced to the base model while acquiring reasoning ability via SFT and RI:; Stage II (post-LRM) investigates contamination applied to an advanced LRM as a final SFT step. Under each stage, we comprehensively evaluate the effectiveness of existing detection methods.
Stage (pre-LRM): contamination happens when the base model evolves into LRMs. We sim- ulate contamination introduced during the period which the base model acquires reasoning ability through SFT and RL. After evaluating 10 representative contamination detection methods spanning coneration-based perturbation- based, reference-based and reference- free approaches, we find that while SFT contamination to the base model is initially detectable, contamination evidence can be concealed through subsequent GRPO (Shao al., 2024) training with clean samples. To isolate the core reasons behind GRPO's ability to conceal contamination, we conducted carefully designed controlled experiments to rule out the possibility that simply training with more clean samples re- sults in the observed concealment, pointing to the conclusion that the GRPO optimization objective might be the primary driver for obscuring contamination. Then, we performed a theoretical analysis showing that the PPO-style importance sampling/clipping gate can drive the drop in detection perfor- mance. Our ablation studies confirm that while plain rejection sampling (RAFT) will not shrink the member/non- member separability, its variant RAFT++ (Xiong et al., 2025) that adds the impor- tance campplng/cllppingg term again makes detection harder. As many RL algorithms adopt similar training objectives, this demonstrates a significant risk the integrity of benchmark evaluations.
Stage II (post-LRM): contamination with CoT applied to LRMs. We simulate contamination with CoT introduced to advanced LRMs as the final training step. Surprisingly, although exclusively SFT on the benchmark samples with CoT yields a huge inflated performance, it leaves little evidence to existing detection approaches: almost all the detection approaches consistently perform near random guess in all the benchmarks. The log-prob distributions of both members and non-members show that without exposure to non-members, contaminated LRMss still have more confidence when responding to those unseen samples that are similar to the training set. This may undermine the key assumption behind many existing detection techniques that the benchmark contamination problem is primarily about memorizing samples (Morris al. 2025 Hayes et al. 2025).
Overall, our findings reveal that existing contamination detection methods are fragile under LRM contamination scenarios: RL conceals SFT contamination evidence introduced during the transi- tion from base models to LRMs, while contamination with CoT applied to advanced LRMs leaves little detectable evidence. These findings underscore the urgent need for advanced contamination
2

---

Published as a conference paper at ICLR 2026
detection methods and trustworthy evaluation protocols tailored to LRMs. Accordingly, outline potential directions for guaranteeing the integrity evaluating LRMs (Section We hope that our discoveries will inspire further research dedicated to building fair evaluation arenas for LRMs.
2 RELATED WORKS
Benchmark Contamination Detections. Benchmark contamination detection methods aim to iden-
tify whether evaluation datasets have been exposed during training (Oren et al.) 2023). Prior work
has proposed approaches based on: instance similarity (Karamolegkou et al. 2023), probability
analysis (Mattern al. 2023), instance generation (Deng et al., 2023 Ranaldi et al. 2024). and
answer memorization Yim et al. (2024). In this work, we select representative methods applicable our setting, from probability analysis and instance generation, and further categorize them into: generation-based (Dong et al. [2024; Wu et al., (025), perturbation- based (Li ct al. 2025 Mattern
al., 2002 Carlini et (022)) embedding based (Tu et al. 2024] Liu et al., (024), and reference- freee (Zhang et al.) 20224 Li et al., 2205: Yeom et al.
al. 2023), reference-basee
(Mireshghallah
2018 Shi et al.] (223)) methods. Each of these relies on distinct assumptions (Fu al. 2024). and
their effectiveness in the LRMs contamination scenario remains underexplored.
LRMs. LRMs achieve superior performance on challenging mathematical and coding tasks (Team et al. 2025), driven by inference-time scaling (Jacch et al., 2024; Snell et al. 2024 Zhang al. 2025). To endow reasoning abilities to existing models, numerous efforts have been focusing on
either SFT distillation (Li et al. 2025 Muennighoff et al. 2025 Guha et al., 2025 Ye al. 2025 Bercovich et al.)
2025) or RL with verifialle rewards (Liu et al. 2025a Zeng et al. 2025 Yue et al,
2025). In SFT distillation, model developers distill knowledge from advanced LRMss into smaller
models (Guo et al. 2025). While RL enables models to generate rollouts and receive rewards from verifiers, improving models' reasoning ability through feedback (Liu et al.) 2025a Zeng al. 2025
Yue et al. 2025 Liu et al. (025b). These two stages create many opportunities for developers to contaminate the benchmarks and evade detection.
Benchmark Contamination Concealment. Model developers hope to conceal contamination ev- idence while still having performance inflation Bomminguez-Oleede et al. 2024). Prior work has explored evading detection through benchmark augmentation, such as rephrasing solutions with strong LLMs (Dekoninck et al., 2024, (aamel et al. 2024), but in LRM settings, most benchmarks only have question- pairs without step-by-step solutions, making such methods inapplicable. (Bordt et al., (024)) explores from the training dynamic perspective, showing that performance infla- tion due to contamination diminishes pre-training progresses. To knowledge, we are the first to investigate contamination concealment at the algorithmic level.
RL CONCEALS CONTAMINATION (STAGE I: PRE-LRM)
Contamination Setup. We define SFT contamination the model being exposed to both the bench- mark question and responses distilled from an advanced LRM, where RL contamination refers to the model encountering the benchmark question and having received rewards based its generated responses during RL finetuning. For each dataset, we randomly sample half of the questions as the member (used for contamination) and leave the remaining half as the non-member set (for detec- tion evaluation). More details about our contamination pipelines, datasets, and implementation can be found in appendix D.I
D.3
and D.4
Detection Setup. We consider 10 representative detection methods. For each question, we generate responses and compute the detection value on each response, then average these values to obtain final detection score for the question. For the rationale and ablation studies of choosing responses to compute the detection scores, please refer to Appendix F.2 We report Area Under the Receiver Op- erating Characteristic (AURÓC) by comparing detection scores between member and non-member sets within the same benchmark. Higher AUROC values indicate better detection.
3.1 GRPO CONCEALS BENCHMARK CONTAMINATION
Contamination Inflation Mainly Comes From SFT. We evaluate multiple contamination scenarios that may happen during SFT and RL and summarize the empirical results in Tab. I Results show that clean SET training yields an 11 11.30% improvement in pass@1 performance, while SFT con- tamination further inflates results by additional 8.82% average across six benchmarks when
3

---

Published as a conference paper at ICLR 2026
Table 1: Pass @11 (%) under different contamination scenarios when the base model evolves into LRMs. The empirical results show that SFT contamination can easily inflate the benchmark perfor- mance. "/" means not used, and "Mem" denotes members. We first train the base model with SFT and then RL. The row with both "/" in the SFT/RL data columns is the result of the base model.
SFT Data
RI. Data
Olypaid GPQA AIME25
AIMF24 Minerva AMC23 Avg.
Base model: Qwen2. 5-7-Innrrrtt
Clean Mem
Clean & Mem
52.56
44.70
30.00
30.00
39.52
73.00
44.96
Clean Mem
Clean
52.52
45.71
34.67
28.00
39.89
72.50
45.55
Clean Mem
/
53.77
49.58
31.62
32.73
40.74
74.92
47.23
Clean
Clean & Mem
44.62
40.74
24.85
27.88
35.23
65.00
39.72
Clean
Clean
47.11
41.41
24.44
26.67
32.72
70.83
40,53
Clean
44.35
40.34
24.79
23.54
34,24
63.20
38.41
36.48
32.20
2.50
10.83
28.58
52.50
27.18
Base model:
Ilama-3. 3.1-8K 1--8---IIsse 1-St-Instruct
Clean Mem
Clean & Mem
44.30
43.18
25.42
24.58
35.20
61.25
38.99
Clean Mem
Clean
44.07
48.48
27.78
25.56
37.32
66.88
41.68
Clean & Mem
/
46.07
42.80
26.67
26.67
35.20
66.67
40.68
Clean
Clean & Mem
44.54
40.74
25.83
23.33
29.53
61.56
37.59
Clean
Clean
42.81
37.37
18.33
19.17
30.15
64.38
35,37
Clean
40.69
39.23
16.67
18.33
27.70
56.88
33.25
15.63
29.67
0.00
4.17
19.49
19.00
14.66
starting with Qwen2.5-7B Instruutt In contrast, RL contamination, despite introducing the bench- mark questions and giving rewards based on the model-generated responses, shows no significant difference compared to using clean RL training set after short training steps.
To understand whether current contamination detection methods can still successfully detect con- tamination in LRMs, and whether RL training can alter the signals exploited by contamination de- tectors, we evaluate SFT-contaminated models before and after GRPO. Tab. T reveals systematic shifts in AUROC across diverse detection methods. Our analysis highlights four key observations:
SFT contamination can be detectable at first. When starting with Qwen2.5-7B-Instin eral reference-free approaches (Min-K (Shi et al. 2023), Max-K% (Maini et al. 2024), and LOSS (Carlini et al. (02T]) can detect SFT contamination at a certain level, achieving AUROC around 73.42% across six contaminated benchmarks. The reference-basee based detection approach, LiRA (Mireshghallah et al. 2022), which assumes access to the training data distribution, also demonstrates superior performance with an average AUROC of 89.13% across six benchmarks. Similar results have already been observed when starting with Llama-3.1-8P 1.-1--BB-Inntuutt.
sev-
GRPO conceals contamination. After applying GRPO to the SFT-contaminated model, we observe consistent decrease in AUROC across all detection methods and benchmarks. We further analyze the average log probability of member and non-member samples before and after GRPO training, selecting Qwen2.5-7BB 7BB-ntuuttt the base model. 3 two key patterns: (1) GRPO lowers the entropy of generated sequences, indicating that the model becomes more confident in its gener- ation, which is consistent with prior observations in (Cui et al. 2025); (2) the log prob distribution of members and non- men-mes converge after GRPO. Since the gaps in log prob are the core statisti- cal backbone of existing contamination detectors, these findings suggest that GRPO may inherently suppress contamination evidence by rendering members and non-members indistinguishable.
More GRPO, less contamination evidence. To examine whether the concealment effect strength- ens with additional training, we extend GRPO to SFT-contaminated models using 10K questions from DeepMath-
(He et al., 2025) for one epoch (156 steps). As shown in Fig AUROC consistently decreases across all detection methods and benchmarks as the number of GRPO steps increases. Given that our maximum 1556 training steps are still far fewer than the steps used some advanced open-sourced reasoning models (Luo et al.) 2025b a). we expect that extensive GRPO training would render all existing detection methods to near-random performance eventually.
Further training will not make models forget contamination. One possible explanation is that additional training makes models forget contamination, thus detections perform random guessing and pass@1 match the clean SFT baseline. To test this, examine it with two experiments. First, we train SET contaminated models with GRPO on both clean and contaminated datasets. As shown in Tab. 2 observe comparable drop in AUROC relative the no RL baseline, similar to per-
4

---

Published as a conference paper at ICLR 2026
Table 2: AUROC (%) of contamination detection approaches evaluated starting from an SFT- contaminated model w/o RL to subsequently trained with GRPO. Results demonstrate that after GRPO, AUROC decreases across all the benchmarks and detection approaches. A measures the dif- ference with the SFT-contaminated model w/o RL. Higher AUROC, better detection performance. Each AUROC is averaged over detection scores from 8 rollouts. The base model here Qwen2.5- 7B- IB-Instruct. More results of Llama-3. 188--Instrutt as the base model are shown in T
Contamination Detection Methods
Training Stages
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
Generation based
Before
47.58
49,86
47.56
53.56
52.52
65.50
52.76
+0.00
Verbatim Wu et al. 2025
RL w/ Clean
45.60
51.28
47.56
56.44
52.05
60.00
52.16
-0.60
RL w/ Cleun&Mem
46.17
50.34
52.67
55.56
51.71
63.62
53.35
+0.59
Before
55.75
41.56
59.27
61.75
CDD Dong et al. 2024
RL w/ Clean
55.47
51.08
43.33
60.00
60.18
62.00
55.34
-0.46
RL w/ Clean&Mem
56.32
44.14
35.56
65.11
60.31
49.38
51,80
-3.95
Perturbation hased
Before RL
54.76
41.19
50.00
41.56
55.64
61.10
50.71
+0.00
Neighhor Mattern al.
2023
RI. w/ Clean
54.10
39.68
50.67
44.22
$3.42
60.50
50.43
-0,28
RL w/ Clean&Menn
53.05
41.08
50.44
52.67
68.16
64.00
54.90
+4.19
Reference based
85.37
100.00
87.01
93.62
LiRA MMireshhhlllhh et al. 2022
RL w/ Clean
74.41
84.65
70.22
81.78
81.04
82.75
80.14
-8.99
RL w/ Cleun&Mem
69.73
77.85
63.11
82.22
79.05
77.38
74.89
-14.24
Before
73.27
60:22
41.11
73.10
82.0
65. 50
+0.00
Ref Carlim et al. 2021
RI. w/ Clean
66.77
58.41
45.33
51.11
65.54
73.62
58.08
-7.42
RL w/ Clean&Mee
62.77
$4.17
43.11
50.44
65.38
72.62
Reference free
Before RL
49.38
58.61
73.56
43.56
50.19
45,00
53.38
+0.00
Zlih Carlini et
2021
Clean
38.56
46.65
RL w/ Clean&Meem
46.04
55,30
64.89
28.89
44.87
39.00
44.74
-8.64
Refore
47.57
50.90
.0.90
59.11
$2.27
45.88
49.61
+0.00
Min-K*+++ Zhang et al.
2024
RL w/ Clean
46.25
46.78
36.67
50.89
51.35
29.62
43.59
-6.02
RL w/ Cleun&Mee
43.77
48.21
21.78
38.00
48.91
43.62
40.72
-8.89
75.56
78.75
Min-K% Shi et al.
2023
RI. w/ Clean
55.19
60,60
62.89
65.56
61.50
61.87
61.27
-13.69
RL w/ Clean&Meen
53.93
59.74
59.56
62.67
57.31
59,25
58.54
-16.42
Before
64,50
64.31
65. 11
81.78
67:27
76.00
69.83
+0.00
Max-K% Maini et al. 2024
RL w/ Clean
53.05
$1.43
49.78
50.22
51.84
$7.75
$2.35
-17.48
RL w/ Cleun&Mem
49.03
51,04
50.00
50,00
52.34
47.50
49.99
-1.884
Before RL.
69.18
69.81
86.22
77.33
70.95
79.38
75.48
+0.00
Loss et al.
w/ RL w/ Clean&Meem
55.22
65.78
61.50
61.26
53.99
60.01
59.33
62.67
$7.40
59.38
58.80
-66668
forming RL solely on clean data. Also, the contaminated model, further trained with GRPO, still shows an average performance inflation of 7.14% across six benchmarks and does not fall back as the clean SFT model, shown in Tab. [ Second, we continue SFT on the SFT contaminated model with an additional 4 epochs on clean data. Fig. 2 and Tab. 23 demonstrate that further SFT is un- able to conceal the benchmark contamination, while the pass @ would continue rise. Together, these results show that subsequent GRPO training preserves performance inflation while reducing detectable evidence of contamination may have some underlying reasons, rather than simply forget- ting the contamination after further training.
3.2 THEORETIC ANALYSIS
In this section, we perform theoretical analysis to demonstrate that PPO-style clipping and impor- tance sampling the root cause of the concealment. Intuitively, the importance sampling and clipping term reweights terms so that the most off- policy trajectories are damped by the clip while typical on-policy ones keep their influence. This reweighting hits non-members more as they have more extreme successes, so clipping cuts misaligned influence and lets ordinary, on-policy successes steer the update. With more headroom, non non-member NLL drops more and the gap contracts.
Setup. We denote f(x, y) to be the negative log likelihood (NLL) of the current model of gener- ating y given prompt z, members as M and non-members as N, policy model Te at step k. We focus on analyzing the gap G3 of negative log likelihood for members and non- mon-mebbers on correct samples (i.e., r = 1), as assessing contamination on erroneous outputs not especially meaningful. Formally, we can write
Gk Era E Eyy (L, r 1] E By (1)) (a, r==1 1]
(1)
5

---

Published as a conference paper at ICLR 2026
Traiiin Methori and BI Steps
SPT Cuntax SantC (edo RE;
GRPO (84 Stepel
GR (110 Slagp))
GRIFO (1:66 Steped RAFT (1.5 Stepe)
Stegs)
AUROC (%)
31
Roadon Gueselig
40
LIRA
Ref Benchmark Contamination Detection Methods
Loss
Min-K
Max K
Figure 2: AUROC (%) trends on SFT contaminated model further trained with different ob- jectives. While contamination introduced through SFT is initially detectable by existing methods, subsequent RL training with clean samples (c.g., GRPO or RAFT++) consistently degrades detec- tion performance. Moreover, we observe monotonic decline in detection performance as the num- ber of RL steps increases, and reference- free methods (e.g.. Loss, Min-K, and Max-K) already fall into near random guesses (i.e., AUROC=50%% simply after 156 steps. The base model is Qwen2.55 7B-Instruct. More results of Llama-3. 1-----Intttrct as the base model are shown in Fig
SPT Conta (wie RL)
GRPO (64 Steps)
SET Conta (wic RL)
GRPO (64 Steps)
Icembbers
membbes
membeers
roembeer retemmeeeers
tolemtenber
not members
30
0.692
Density
Dousity
20
-0.3
-0.2
-0.1
-0.3
-0.2
-0.1
0.0
-0.3
-0.2
-0.1
0.0
-0.3
-0.2
-0.1
0.0
Log Probability
Log Probability
(a) Results on GPQA-Diamond
(b) Results on OlympiadBench
Figure 3: Log-prob distributions for members vs. non-members of SFT contaminated model be- fore and after RL training. After further GRPO with clean samples on the SFT contaminated model, the log-prob distributions of members and non-members become increasingly similar. Since many contamination detection methods rely on separability in this space, the shrinking gap explains their degraded effectiveness. More log-pre log-prob distributions can be found in Fig. 7 g and 9
If this gap contracts, i.e., Gk+1 - G2 < 0, members and non-members become closer in the NLL sense, making contamination detection harder since many methods (Zhang al. 2024 Shi al. 2023 Maini et al.] 2024; Carlini et al.] 2021 are based on the separation of NLLs. For a fixed prompt 2, we define the NLL as
A E [6++1
E.r 16A r 2).
We notice that rewrite the NILI gap as
Gk+1
G == ELEN/AA [A]] ELEM [Az]-
(3)
In our following analysis, we thus focus on investigating the behavior of A, on members and non- members. an algorithm yields on average smaller Ag on non-members, the algorithm should be able to conceal contamination.
Notations. token I, let A, be the method's per token reward/advantage and we the weight from importance sampling and clipping. Define
AV" := W.A,, (s) Ea a)), A. = AW A
(4)
to measure how good a state is compared to the average. In particular, W2 m. with Pr To (a: s,) /oTTaaaa (a, 1x)) being the importance sampling and m, {0, 1} be mask indicating if the clipping is activated, specifically m = 0 indicates that there is no gradients from the update. Moreover, we (11x))/((e [a y)] be the overall success rate the prompt, and value function define P& (z) (e)==E, = (-Ix)
qu (s, a) = Pr(r- 1 118, a) and (s((s) ;= Ba (-1a) (s, a)] for success at that state. And

---

Published as a conference paper at ICLR 2026
Table AUROC (%) detection approach, Loss Carlini et 2021), evaluated on SFT contam- inated model further trained with different RL objectives. The gray row indicates no ablation on the objective, and X means remove the term from the objective. SFT contaminated model w/o RL (Tab. 2). RL steps are 64, or the step before the model collapses.
measures the difference with the
The results show that clipping is the main driver for the contraction, which aligns with our theory.
Training Objectives
Clipping
Olypaid
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
A
RAFT
71.78
69.78
86,00
86.67
71.58
79.25
77.51
+2.03
RAFT+
50.43
58.45
67.56
66.67
52.84
49.50
57.58
-17.91
RAFT++
X
69.16
73.68
74.44
76.22
71.08
81.75
74.39
-1.09
GRPO
V
55.22
60.50
65.78
61.50
62.12
61.26
-14.22
GRPO
68.83
70.20
80.44
73.78
68.30
78.
73.28
-2.20
define B(s) =
([s)==B<=IPP) Ees=<l(ss, a)m((s, a) (s, a)] and C(s)=E(ssa [p(s, asm a)]. We assume that the RL training performed on the benchmark data (i.e., training data is the combination of members M and non-members N), and it is in a tabular setting for simplicity. Since members have been utilized during training,
natural to assume the (s( for members are larger than non-members, and the NLL for members is lower than non membees
Theorem 3.1. For a small natural gradient step with step size 17 on a PPO style loss, we have
A - E
A
1,
tn Cov
Σ A
+0(11²
(5)
(A) p(z)
(B) covariance 9(x)
The proof can be found in appendix C Intuitively, j((r) measures the average push on the example's NLL from correct trajectories, where 3 serves as a reweighting term accouting for the importance sampling/clipping. Here we consider several instantiations using different algorithms to investigate the core driver for contraction. The training objectives for each algorithm are listed Appendix
RAFT. In plain rejection sampling, we have w): = 1 and A₁ = 1r 1}. so correct trajectories
T
A = - (8t),
1 RAFT (x) =
(1 - (se() r == 1,
T
1=1
The covariance term is BRAFT = Cov (Es, E(--pa(s))) = = -Co((eea Σph(st)
We note that lower loss la corresponds to higher probabilities p& (St). and thus GRAFT (x)) > 0. More- non non-members correct trajectories can exhibit much higher variance in loss and probabilities, thus, the NN term is typically larger than 3M. Consequently,
AN AM -1((NN μM) + 1(xx-xxx BM),
where both gaps (NN - μM) and (Bx - BN) are positive. Empirically, the covariance gap offsets the mean gap, yielding AN Am 2 0, i.c., RAFT is unable to conceal contamination evidence.
RAFT++. Using the same A = 1 1}, on r = paths
Ag Pemm B&(s), RAFT++ (x) =
(pom: Bx(ss)) 1,
We note that the difference of μ cannot possibly lead to large deviations between members/non- members 05 p.m, <1 eand Bs(ss) < for both groups and the term is normalized by length. For the covariance term though, we have
gRAFT++ (z) Cov Cov(le.
E( E(pmm-Bauss)) Ba(s))) Cov Cov(Ee, Eom. Cov Coo(ee: BB((s))
Compared to RAFT, the new term Cov(LA, E (:m)) is negative as correct path with higher loss are anomaly and typically got clipped more. Moreover, this much more prominent in non-members

---

Published as a conference paper at ICLR 2026
Table 4: Pass@ @1 1 (%) of advanced LRMs before and after SFT contamination with COT.
Models
Olypaid
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
DeepSeek-RR R E---isiill--amma-- La
52.10
43.94
33.33
43,33
32.97
84.58
48.38
+ w/ extensive SFT Contamination
61.83
53.16
51.67
61.67
38.74
93.75
60.14
DeepScek-RR RI RI-Distill- D--Disill--wen-1B Qwen-BB S w/ extensive SFT Contamination
55.70
48.65
39.26
53.70
37.25
91.94
54.42
58.77
50.87
42.59
58.91
40.81
90.67
57.10
OpenThinker3-75 (15K) - w/ extensive SFT Contamination
30.81
41.67
21.67
29.17
34.01
77.50
42.47
52.74
47.64
33.33
30.48
40.56
78.70
47.25
DeepSeck- Shissill--Qee--a4ee
39.89
$6.69
44.44
62.78
42.28
92.92
59.83
<> extensive Contamination
62.75
64.58
77.78
46.14
97.81
69.24
due to high variance in correct trajectories loss. The second covariance term, although still negative, are not that significant for non- members compared to members due to an average over all possible actions. Therefore, overall it leads
A Au = m - PM) + nsS - 3M) < 0,
i.e, RAFT++ contracts the membership gap. The driver is precisely the PPO-style importance sampling/clipping: it removes the RAFT covariance cancellation by making Cov E pm) non- positive and more negative for non-members.
GRPO. Finally, investigate the GRPO contraction term. To ease the analysis, consider an idealized setting where we define the advantage term as Ax(I, u) -r(x, v) (x) with no standard deviation term and A. = ARAFF -px (p)) - C(s))). Clearly, we have
GRPO (x) 1t (AFF++())) Pk [x)E
T Σ (pome -k((s) r= 1,
BORPO BRAFT+(()) (x()) Cov (xx. -((ss)-C((ss))
By similar argument, we know that the term does not contribute significantly to the concealment. The covariance term can be analyzed similarly show that the concealment also happen GRPO thanks to the importance sampling and clipping term.
3.2.1 EMPIRICAL SUPPORT
To confirm empirically the prediction of our theoretical results, we evaluate the Loss detector (Car- lini et al. (202 after training with RAFT (Dong et al., /RA3)/ (Xiong et al. 2025/GRPO.
The overall results can found in table 3 We conduct the ablation study using Qwen2.5-7B- Instruct the base model. From the results, there several observations.
Effeet on detectability. Under RAFT. the Loss detector (Carlini et al. 2021) performance re- mains essentially unchanged relative to the SFT contaminated baseline w/o further RL. In contrast, RAFT++ and GRPO (with clipping enabled) produce a sharp drop in detector performance.
Importance sampling vs. clipping. The clipping term, often treated purely materially contributes to concealment, as predicted by theory. When we retain importance sampling but remove clipping in RAFT++ and GRPO, both algorithms show little to reduction in Loss- detector performance (Table Intuitively, as the clip threshold e oo, the effective weight satisfies E, p.m, T, and the covariance term in decomposition tends toward for both members and non-members, climinating the shrinkage effect.
training stabilizer,
These two observations perfectly reflect our theoretical analysis, empirically validating that the PPO- style importance sampling/lippping term is the key driver behind GRPO contamination concealment. Given that many RL algorithms adopt this term in their objectives. this suggests that a broad class of RL methods may inherently exhibit similar concealment capability.
CONTAMINATION WITH COT ON ADVANCED LRMs BARELY LEAVES EVIDENCE (STAGE II: POST-LRM)
Contamination Setup. In this setup, simulate contamination with CoT applied to ad- vanced LRMs at the final stage of training. We use SeepSeck-RI-Dissee -issstill-LLamm--B DeepSeek- RI-Distill-Qween-7BB
(Guo et al. 2025). and checkpoints from
8

---

Published as a conference paper at ICLR 2026
Table 5: AUROC (%) of contamination detection approaches evaluated on contaminated, ad- vanced LRMs. Results demonstrate that even after extensive contamination as the final stage, al- most all the detection approaches perform near random guesses (i.e., AUROC=50%) Each AUROC averaged over detection scores from 8 rollouts.
Contamination Detection Methods
Init Models
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
Generation based
DS Llama-8B
48.73
50.45
41.33
61.56
19.10
40.63
50.30
Qwea-7B 46.87
55.85
60.44
68.89
56.87
50.63
56.59
Verbatim Wa et al. 2025
OpenThink-7B
43.78
55.36
60.89
56.67
51.78
42.38
51.81
DS Qwen-14B
48.51
50.73
52.38
61.11
55.18
53.79
53.62
Llama-8B
51.84
53.83
60.00
53.11
58.08
57.50
55.73
DS Qwen 7B
51.46
48.29
50,00
53.78
54.71
41.00
49.87
CDD Dong 2024
OpenThink-77
49.98
50.23
53.31
51.24
54.52
50.44
$1.62
DS Qwen
46.67
56.13
50.61
Perturbation based
DS Llama 8B
49.94
39.32
53.11
43.33
49.68
60.00
49.23
Neighbor Mattern et al. 2023
Qwen-7B
52.99
40.29
62.44
49.33
54.87
52.54
OpenThink-7B
53.76
42.95
34,00
42.22
52.89
51.50
46.22
DS Qwen-14h 14B
53.20
42.23
50.89
44.00
53.46
57.38
50.19
Reference based
DS Llama-8B
57.92
$3.00 $3.01
53.56
75.33
69.44
58.75
61.34
LIRA Mireshghallah al. 2022
DS Qwen-7B
46.52
43.93
50.22
58.89
59.33
$4,00
$2.15
Op
62.35
64.77
58.44
64.44
64.81
61.62
62.74
DS Qwen- 14B
59.93
55.23
75.56
66.00
66.55
70.00
65.55
DS Lluma-8B
53.79
46.50
46.44
64.00
63.57
$1.25
54.26
Ref (Carlini et al. 2021
OpenThink-78
57.34
49.86
37.56
44,22
50.44
$3.09
59.30
69.12
47.27
$3.94
DS Qwen- 14B
55.75
47.55
52.67
30.89
55.51
53.75
49.35
Reference free
DS Llama-8B
49.52
$4.74
64,22
37.11
45.97
47.12
49.78
Zlib Carlini et al. 2021
Qwea-7B 46.52
$7.38
64.89
36.89
43.30
42.12
48.52
OpenThink-7B
45.65
55.37
74.22
36.89
43.51
36.62
48.71
DS Qwen-14B 48.12
56.71
70.44
43.56
45.92
$1.50
$2.71
DS Llama-N0 8B
55.45
59.10
45.95
70.22
60.89
$7.50
58.19
Min-K%+++ Zhang et 2024
Qwen-7B
48.92
56.83
48.44
39.33
51.83
62.62
54.66
OpenThink-778
$1.85
58.31
66.44
$5.00
49.41
41.05
$3.68
DS Qwen-14B
56.72
48.44
76.44
$7.39
58.51
Llama-8B
57.86
61.68
53.33
72.67
67.12
61.87
62.42
Min K% (Shi et 2023
DS Qwen-BB
49.75
$3.93
51.78
61.56
$4,50
$6.75
$4.71
OpenThink-7B
53.52
$7.19
60.44
57.56
54.83
47,37
55.15
DS Qwen- 14B
52.77
58.08
52.44
77.33
59.43
$9.62
59.95
DS Llama-8B 53.85
$5.96
50.67
60.44
59.22
$2.50
$5.44
Max-K% Maini et al. 2024
OpenThink-7B
Qwen-7B 49.65
55.12
$8.29
50.92
46.22
40.44
79.33
73.33
54.20
56.25
59.38
58.76
54.11
DS Qwen-14B 50.43
$3.89
50,00
50.00
$1.08
$2.50
51.32
Llama-8B
61.78
62.59
Loss Carlini al.
DS Qwen-7B
49.77
$4.09
52,00
63.78
54.76
56.75
55.19
OpenThink Think 7B
53.44
$7.61
61.33
56.67
55.07
48.12
55.37
DS Qwen-14B
52.81
58.39
52.89
77.56
59.37
60.37
60.23
OpenThought3 (Guha et al. (025) as the initial models. We simulate extensive contamination with CoT by applying SFT exclusively on the member data in this section. Additional implementation details are provided in Appendix D.4]
Tab. and 5 show the results of pass@1 on six reasoning benchmarks and AUROC of detection approaches performance (w/ the same detection setup as Stage I), respectively. We observe that:
Extensive SFT Contamination with CoT results huge performance inflation. As shown in Tab. LRMs can substantially benefit from extensive contamination with COT. Such inflation enables contaminated LRMss to artificially boost performance benchmarks and have an overrated rank in the reasoning leaderboard with little extra training cost.
Extensive contamination with CoT on LRMs barely leaves evidence. As illustrated in Tab. 5 detection methods, which were effective in contamination introduced when the base model evolves into LRMs, consistently fail under extensive contamination with CoT to LRMs, performing close to random guessing. The previous SOTA approach, LiRA (Mireshghallah only 78.74% AUROC on average across six benchmarks. Then, we analyze the log prob mem- ber and non-member samples before and after final stage contamination, shown Fig. After
al. 2022), achieves

---

Published as a conference paper at ICLR 2026
Distill LLaMA
Distill (Tsteessive NET Conts)
RI Distill LL (Clean)
(Estennsivn NET Canta)
RI Distill Qwen (Cleam)
25
merrberr nat-cembber
10.0
membere
mumberr
oos-mamber
Deasity
20-
/MRRGG 0.092
AUROC: 0.57M
Density
7.5
AUROC 9/179)
AUROC
5.0
2.5
0.0
-0.6
-0.4
-0.2
0.0
-0.6
-0.4
-0.2
0.0
-0.4
-0.2
0.0
-0.4
-0.2
0.0
Log Probability
Log Probability
(a) RI Distill LLaMA results on OlympiadBench
(b) R1 Distill Qwen results on OlympiadBench
Figure 4: Log-prof distributions for members vs. non-members of advanced LRMs before and after SFT contamination. After extensive SFT contamination on members, the log prob of both members and non-members increases at similar margin. More figures are in Fig. and 14
the extensive SFT contamination with CoT on members, the log prob of both members and non- members increases at similar margin. This indicates that even without exposure to non-members, contaminated LRMs still have more confidence when responding to unseen samples that share sim- ilar distributions to training samples, which also explains why extensive contamination with CoT on LRMs barely leaves evidence. These results suggest that model developers could extensively contaminate their LRMs in the final stage while leaving little detectable evidence.
Discussion. Despite Feng et al. (204)) demonstrating that contamination detection could work in non-reasoning model scenarios, the detectors do not have access to the reasoning trajectories used in the LRM contamination scenario, so they have to rely on the generated responses from LRMs. However, LRMs typically possess strong reasoning abilities to output step-by-step long CoT and are difficult to converge on specific sequence after contamination with long COT. This may indi- cate that rather than memorizing specific reasoning trajectories, LRMs internalize the underlying knowledge and reasoning process during the contamination with CoT data, enabling generaliza- tion to distributionally similar questions (e.g., non-members). While most detection methods rely on the assumption that contaminated models would achieve lower loss training sequences (Car-) lini et al. 202T) or generate less diverse responses for seen questions (Dong et al. 2024) than for unseen ones. Accordingly, these methods rely on gap in certain metries (e.g., log- nog-probability, Levenshtein distance, etc.) between trained and unseen samples to determine contamination. Never- theless, these LRMs could also have lower loss when responding those unseen samples that share similar distributions to the training set, benefiting from their long CoT ability, as shown in Fig. 4 This confounding factor (i.e, generalization) is not accounted for by existing detection approaches, challenging the assumption that benchmark data contamination is more about memorization (Wu
al. 2025 Morris et al. 2025 Hayes et al., 2025).
5 CONCLUSION
We present the first systematic study of benchmark contamination in LRMs, structured around two points where contamination can happen. Our results reveal a critical vulnerability in LRM eval- uation: contamination detection methods are fragile and contamination introduced at either stage can be concealed. In Stage (pre-LRM), while SFT contamination to the base model is initially detectable, contamination evidence be concealed through subsequent RL training. In Stage II (post-LRM), extensive contamination with CoT on advanced LRMs barely leaves evidence for ex- isting memorization-drrven detection methods. Our findings call for urgent need of protocols that ensure fair evaluations among LRMs. Here, we propose two potential directions to ensure it: (I) Model developers should release more intermediate training checkpoints, enabling the community to better monitor and regulate potential benchmark contamination in each training stage. (II) Re- searchers working on contamination detections should advance beyond cemorization-drivee meth- ods and explicitly account for the long CoT reasoning and generalization capacity of LRMs. Despite the assumption that contamination is about memorizing the training data inspires numerous detec- tion methods before the LRM era, it may become outdated right now, Detection approaches that are solely based on log-probs or mitigation approaches such as minor benchmark modifications, are definitely inadequate in this context and risk systematically failing. These findings all highlight the need for new assumptions in contamination detection and the development of contamination robbsst evaluation protocols in the LRM setting.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
The authors thank Rui Yang and Yifan Sun for their helpful feedback. Huan Zhang is supported in part by the AI2050 program Schmidt Sciences (A12050 Early Career Fellowship).
REPRODUCIBLE CLAIM
Our code available at
Detection, ion Arena. git Detailed implementation of detection approaches, SFT training, and RL training can be found appendix D.2 and D.4 appendix C
We also provide the proof our theory in
ETHICS STATEMENT
We find new vulnerability of LRM evaluations: contamination introduced at either stage can be concealed. Other than this, we do not have more ethics concerns.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondiej Dušek. Leak, cheat, re- peat: Data contamination and evaluation malpractices in closed-source 1lms. arXiv preprint arXiv:2402.03997, 2024.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El- El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemorre Efficient reasoning models. arXiv preprint arXiv:2505.0099 00949, 2025.
Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko, and Ulrike von Luxburg. How much can we forget about data contamination? arXiv preprint arXhv:2410.03249 2024.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pp- 2633-26500 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22617, 2025.
com/ Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv
Michael Han Daniel Han and Unsloth team. Unsloth, 2023. URL
lnnslotha//ussl
preprint arXiv:2307.088: 086699.9.866,..... 2023.
Jasper Dekoninck, Mark Niklas Müller, Maximilian Baader, Mare Fischer, and Martin Vechev. Evading data contamination detection for language models is (too) easy. arXiv preprint arXiv:2402.02823. 2024.
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigat- ing data contamination modern benchmarks for large language models. arXiv preprint arXiv:2311.09783. 2023.
Ricardo Dominguez-Olmedo, Olmedo, Florian Dorner, and Moritz Hardt. Training on the test task con- founds evaluation and emergence. arXiv preprint arXiv22407.07890, 2024.
11

---

Published as a conference paper at ICLR 2026
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang. Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXtv:2304.06767, 2023.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15993 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Qizhang Feng, Siva Rajesh Kasa, Santhosh Kumar Kasa, Hyokun Yun, Choon Hui Teo, and Sra- van Babu Bodapati. Exposing privacy gaps: Membership inference attack on preference data for IIm alignment. arXiv preprint arXiv:2407.064433 2024.
Yujuan Fu, Ozlem Uzuner, Meliha Yetisgen, and Fei Xia. Does data contamination detection work (well) for ums?? a survey and evaluation on detection assumptions. arXiv preprint arXiv:2410.1800 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymiis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reason- ing models. arXiv preprint arXiv:2506.0411 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948. 2025.
Jamie Hayes, Ilia Shumailov, Christopher A Choquette-Choo, Matthew Jagielski, George Kaissis, Katherine Lee, Milad Nasr, Sahra Ghalebikesabi, Niloofar Mireshghallah, Meenatchi Sundaram Mutu Selva Annamalai, et Strong membership inference attacks on massive datasets and (moderately) large language models. arXiv preprint arXiv:2505. 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-103k: A large-scale, challenging, de- contaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504. 11.550.,1444, 2025.
Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for 11m training, arXiv preprint arXiv:2410.10998 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1rXXi:2242.116222 2024.
Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. arXiv preprint arXiv:2310.13771, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Bespoke Labs. Bespoke-minicheck-7h 2024. /eepodeelabbs/acsoooe---tii-td
URL httipss//huggingfacce..
VI Levenshtein. Binary coors capable or 'correcting deletions, insertions, and reversals. In Sovier physics-doklady, volume 10, 1966.
12

---

Published as a conference paper at ICLR 2026
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski. Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Cutman-Soloo et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843-3857, 2022.
Dacheng Li, Shiyi Cao. Tyler Griggs, Shu Liu, Xiangxi Mo, Eric Tang, Sumanth Hegde, Kourosh Hakhamaneshi, Shishir G Patil, Matei Zaharia, al. Lims easily learn reason from demon- strations structure, not content, is what matters! arXiv preprint arXiv:2502.03374, 2025.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries large language models. arXiv preprint arXii:2505.2455550 2025a.
Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Haonan Lu, Bing Liu, and Wenliang Chen. Probing language models for pre-training data detection. arXiv preprint arXiv:2406.01333 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint TrXiv:2503.2078. 2025b.
Michael Luo. Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, et al. Deepcoder: A fully open-source 14b coder at 03-mini level. Notion Blog, 2025a.
Michael Luo. Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol -previeww with a 1.5 model by scaling rl. Notion Blog, 2025b.
Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset? Advances in Neural Information Processing Systems, 37:124069-124092, 2024.
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor King-Kirkpatrick. Membership inference attacks against language models via neigh- bourhood comparison. arXiv preprint arXiv:2305. 2023.
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David K Evans, and Taylor Berg- Kirkpatrick. An empirical analysis of memorization fine-tuned autoregressive language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 18 1816-1826, 2022.
John X Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G Edward Suh, Alexander M Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. How much do language models memorize? arXiv preprint arXiv:2505.24833. 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:250J. /9393, 2025.
Yonatan Oren, Nicole Meister. Niladri Chatterji. Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference Learning Representations, 2023.
Federico Ranaldi, Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Cristina Giannone, Andrea Favalli, Raniero Romagnoli, and Fabio Massimo Zanzotto. Investigating the im- pact data contamination large language models in text-to-sql translation. arXiv preprint arXiv:2402.nm 08 180, 2024.
13

---

Published as a conference paper at ICLR 2026
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 3505-3508 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. In First Conference Language Modeling, 2024.
Vinay Samuel, Yue Zhou, and Henry Peng Zou. Towards data contamination detection for mod- large language models: Limitations, inconsistencies, and oracle challenges. arXiv preprint arXiv:2409.0992. 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseckmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu, Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310. 16789, 2023.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can more effective than scaling model parameters. arXiv preprint 240.::200000333144 2024.
Yifan Sun, Han Wang, Dongbai Li, Gang Wang, and Huan Zhang. The emperor's 's new clothes in benchmarking? rigorous examination of mitigation strategies for Ilm benchmark data contami- nation. arXiv preprint 2tt.:2553..640000) 2025.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.205:: 2025.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Qwen Team. Qwq- 32b: Embracing the power of reinforcement learning, March 2025. URL wt/blog////oooo qwq gwg-3/b/
https: / www.wwwww......mmm.m...mmmmmmmme
Shangqing Tu, Kejian Zhu, Yushi Bai, Zijun Yao, Lei Hou, and Juanzi Li. Dice: Detecting in-distribution contamination in Ilm's fine-tuning phase for math reasoning. arXiv preprint arXiv:2406.04197, 2024.
Han Wang, An Zhang, Nguyen Duy Tai, Jun Sun, Tat-Seng Chua, et al. Ali-agent: Assessing I1ms' alignment with human values via agent-based evaluation. Advances in Neural Information Processing Systems, 37:99040-99999 99088. 2024.
Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, et al. Reasoning memorization? unreliable results of reinforce- ment learning due to data contamination. arXiv preprint arXiv:2507.10532. 2025.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo. Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504. / 1343, 2025.
Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. Benchmark data contamination of large language models: A survey. arXiv preprint 240.044000 2024.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387 2025.
14

---

Published as a conference paper at ICLR 2026
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learn- ing: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 288-282. IEEE, 2018.
Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, and Meliha Yetisgen-Yildiz. To err is human, how about medical large language models? comparing pre-trained language models for medi- cal assessment errors and reliability. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pP- 16211 16211-16223, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yue, Weinan Dai, Tiantian Fan. Gaohong Liu, Lingjum Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint drXiv:2503.14476, 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint arXiv:2504. 138377 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 20o: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint urXiv:2503./889) 2025.
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%+++ Improved baseline for detecting pre-training data from large language models. arXiv preprint wrXiv:240.002936 2024.
Junyu Zhang, Runpei Dong, Han Wang, Xuying Ning, Haoran Geng, Peihao Li, Xialin He, Yutong Bai, Jitendra Malik, Saurabh Gupta, et al. Alphaone: Reasoning models thinking slow and fast at test time. arXiv preprint arXiv:2505.224 248663 2025.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.1331 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Limitations
17
Algorithms
17
Proof
18
Experiment Setup
20
D.I
Contamination Pipelines
20
D.2
Contamination Detection Methods
20
D.2.J
Generation-based Delection
20
D.2.2
Perturbation-based Detection
21
D.2.3
Reference-based Detection
21
D.2.4
Reference- Detection
D33
Datasets
22
D.4
Implementation Details
22
More Experiment Results
24
E.
More results of Llama-30 --B..B...--BBBB Insttrcct (Stage I: pre-LRM)
24
AUROC Question, Question+Respons: Thinking Process, and Non-thinking Process (Stage I: Pre-LRMM
25
E,3
Response Length Analysis (Stage I: pre-LRMM)
27
Log-probability distribution before and after RL (Stage p LRM)
28
E.5
AUROC for different training steps (Stage 1: pre-LRM)
31
E.6
Contraction effect on small and large models (Stage I: pre LRC
35
E.7
More RL training brings performance gain (Stage I: LRM)
36
Token Embedding Visualization of Nember/non-memblee of benchmarks (Stage I: Pre-LRM)
37
E.9
Contamination Inflation Comparison (Stage I: LRe-LRMM
38
E.10
Log-prob distribution after extensive SFT contamination LRMs (Stage II: post-
LRM]
39
E. lI Applicability in non non-math domains (Stage and
40
Computation resources
42
Use the LLM
42
16

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
Our work reveals critical vulnerabilities: RL fine-tuning conceal benchmark contamination when base models evolve into LRMs; contamination with CoT applied to advanced LRMs leaves little evidence detectable by existing memorization-based methods. Although we do not propose new detection algorithm to mitigate these risks, we reveal that the failure of current detection ap- proaches stems from their reliance on log-probability and the assumption that training samples consistently incur lower loss than unseen samples. Given the unique characteristics of LRMs, future detection methods must adopt more advanced assumptions address this fundamental challenge. By highlighting these risks, we aim to spur further research on robust and trustworthy evaluation protocols for LRMs.
B ALGORITHMS
SFT Let D {(1,0) 0)} be corpus of questions q and responses o, where are distilled from advanced LRMs. Let To(o q) be autoregressive policy model. The me is then fine-tuned to maximize the log-likelihood over the responses:
LSFr(0) (0) = [E log To(o 4)]
GRPO For each question g, GRPO samples group of outputs (o,,..., oc} from the old policy am and then optimizes the policy model Ty by maximizing the following objective:
CGRPO (0) = E p(Q)) (a -aa -o (Olo)
C
adl
To 9)
As. clip To(o; (q)
- 1 ee) A.) - -B DKL (TollTette)) )}
min
Band (o;
Aans (0: 9))
1=1
Feo (o: 14) Toee 9) 1,
DRI.
log
To ④)
x (os
where and 8 are hyper- parem-parameters. We denote {0, 1) binary reward function that assigns scalar feedback to question-output pair. A, is the advantage, computed using a group of rewards (ri, ,.,...,,'0. rG) corresponding to the outputs within each group:
A; = r; Pea.
std(fr1, T2.................... .2....
RAFT The RAFT is also referred to the rejection-sampling fine-tuning. The algorithm consists of two parts:
Dataset Collection. For each question g, we sample a group of outputs {or. og }. For each response 0,, compute the reward ra (0, 1}. We then retain only the responses that receive reward of 1 and put them a dataset DD
Model Fine-tuning. The current policy no is then fine-tuned maximize the likelieelhood over the selected dataset:
CRAFT(0) E ((.,))= log (o 4)]
RAFT+ RAFT+ is variant of plain RAFT that also introduced importance sampling and clip- ping. The loss is very similar to GRPO with no KI. divergence term except for the advantage calculation. The advantage for RAFT++ will be:
A = I(rr =);
which indicates essentially that train only the positive samples.
17

---

Published as a conference paper at ICLR 2026
C
PROOF
Here we restate our theorem 3.1
and provide a full proof.
Theorem. For small natural gradient step with step size on a PPO style loss, we have
A -1 1
Av
1,
+1 Cov lk
Σ A
+ O
(6)
T
Proof. Denote (p) = min(max(p: c), 1+ Consider the optimization problem
(()) = E
Σ min (p:(0),,, (l)()))))
The gradient of this objective is
Ve min (px (O) (p:())A)) m,dept Ve log (v st)
(8)
Let Zz,a be the logit parameter for action state s, that Ta ( s) softmax(zs.) Then
0
log To(ye (x)) 18t 8) (1 a} To(a |s);
0z..a
and thus
OC
E
Σ
(pr me A.) a} - To(a sn))
(10)
azsaa
It is then clear that after one natural-gradient step,
(s, (s)( 77²i' (8,
(11)
Using first-order Taylor expansion of log ग and equation 11
log T s) - log (a 8) = (s, + 0(1²)
(12)
For a trajectory y = (,1,...,,Tr).
1 (y xx) log ( x))
(log T4+1 |s)) -log ( st)) n
A (sc. + +177)) (13)
tm1
Define
T
T A (t).
Vn(y): = Σ À" (st, (x v) =
T
1=1
t=1
Then, from (xx,8) = - E, log Tx (Me
st) and equation 12
log Rk41 logmm) = - (z.))) -0) (x1((,,0) (x(xx)) Σ
(14)
Moreover,
(v) Th+1 (y) x)
Vz(v)
= 1 -1,
(15)
Thk (y x))
Add and subtract Em [ex r=11 xx] Az, we have
A E E Ex |==1, + (Bezi alli |r=1,
Er,lex 1=
(16)
(4)
(B)
18

---

Published as a conference paper at ICLR 2026
Using equation 14 and a change of measure with Rg:
(A) Eug (4x111 fx) Rs 1=
E Rs 1= ②
Buyl (- -- + (1)))) (1 + 1 + 0(1²)) r= 1=
+ 17 Bm 1 r=1,,] +0
-- - -
r=1,, xt O
T
1
- B
Σ
Av
1=1, x
T
(=1
where the third equality is simply a first order expansion of division. For term (B), we can compute for any integrable
E. If
R
r=1,2]
I B [ 1=11
Ex
r= + vy Cov (J. Vo) + 0(1)).
E R2
|==,,]]
by expanding numerator and denominator using equation 15 Taking = 4. gives
(B) = 17 (1r=, (Cx, vx) +07²² 17 Cov (11=11)) bx, Σ AV
(17)
t=1
Combining the two part we get our theorem.
19

---

Published as a conference paper at ICLR 2026
D EXPERIMENT SETUP
D.1 CONTAMINATION PIPELINES
SFT Contamination We randomly select 10K samples from Open T (Guha et al. 2025 form the clean SFT training set, following the same ratio for each domain the original paper, to obtain the best results. For the SFT contaminated training set, we use QwQ-32B TTaam. 2025)
as an advanced LRM help distillation for the member set. We adopt the temperature 0.6, top-p value of 0.95, and maximum output token with 32768 for the distillation. We use rejection sampling with 64 rollouts, selecting correct trajectories when available. If none the 64 rollouts produce a randomly select an incorrect trajectory. We replicate the question in the mem- ber set with responses 3 times to create the SFT contamination training set. So the training set consists of 1 1166 samples. For the Llama-3.1-88 --8--nnttuuct,
correct answer,
randomly select 30K samples from OpenThoughhs (Guha et al., (2025) to form the clean SFT training set, and duplicate memberships 9 times to form the SFT contamination training set.
Table 6: Proportion of questions solved after up to 64 rollouts with QwQ-32B.
Olypaidbench GPQA-Diamond
AIME2025
AIME2024
Minerva Math
AMC2023
Avg.
80.59
92.42
93.33
93.33
56.62
100.00
86.05
RL Contamination For RL contamination, we replicate the questions in the member set 2 times, and randomly select 4096 samples from DeepMath- 103K (He al. (025) as the clean RL training set. We choose GRPO as the RL algorithm and train the model with one epoch.
Base Model Selection We choose Qwen2.5-7B-Insse Instruct[Dubey al. (2024) as base model, and first train it with SFT and then RL.
(Team) 2024 and Llama-3.1-8B.
D.2 CONTAMINATION DETECTION METHODS
Setup For input question q. response o, and model me, we define the detection score as [(g xo)- We treat the contaminated benchmark the member set and the remaining uncontaminated half as the non-member set.AUROC is computed from the detection scores between the member and non- member set within a benchmark. We define the average log probability of model ma generating the response given the question q as:
T,)) log (o 4)
We provide experiments illustrate why choose the log probability responses for most detec- tion methods in appendix E.2 and assume that models have seen the questions during the training, they would have more confidence during the generation, thus have higher log probabilities.
D.2.1 GENERATION-BASHD DETECTION
Verbatim Wu et al. 2025 Verbatim- approach (Wu et al. 2025) prompts the model to complete the remaining parts of question based on partial prefixes. (Wu et al. 2025 uses the partial-prompt completion rate measured by ROUGE-L (Lin 2004), which calculates the overlap of the longest common subsequence between the generated and reference text. If the model memorizes the question during the training, the partial-prompt completion rate would be higher compared to unseen questions. We use 80% of the original problem to generate a partial completion. Using lower ratio causes the LRM to answer the question directly instead of continuing the given sequence.
CDD measures the variation of the generated responses, and assumes that if the responses share strong similarities, the question is more likely to contaminated during the training.
CDD (Dong et al. 2024)
20

---

Published as a conference paper at ICLR 2026
I(ED(oo (Os, Ormppeatuur=-)) d)
f(g, o, me)
p (d)
G|
where ED represents Edit distance (Levenshtein 1966). We quantify the peakedness of the edit- distance distribution (d( by its cumulative mass within similarity window, i.e., F(d ol), where a E [0, 1] and max (Le(()) |o O}. In our experiments, we set a = 0.5, which performed better than other choices.
D.2.2 PERTURBATION-BASEE DETECTION
Neighborhood method calibrates the detection score o(g, O, mB) with some unseen questions o' that share similar semantics to the original question q. We denote GPT-4o as N to augment the question q and get o' E (N))). We use a total of five augmented samples to compute the detection score:
Neighborhood (aattenn et al, 2023)
flg 0, TT) 66g o, mo) -epe)--nnesa Qneighbor (q' 00 Te),
Queighbor (rnighboold.cco O.
Σ g'EN(g)
Ql o, Te).
W((1)
D.2.3 REFERENCE-BASED DETECTION
The detector assumes that they could access to the training distribution of the target model ne. and have reference model "er calibrate the detection scores. We choose bespokelabs/eespoke- Stratos-7B Labs 2024 as the reference model Tee in all the experiments.
LiRA (Mirehgghallhh et al. 2022
LiRA calibrates detection score by dividing.
(g 0, No)
f(g, o, Fa)
(4 o, Tiet)
Ref (Carlini et al. 2021)
Ref calibrates detection score by subtraction. flg. o, ma) ő6g, o, To) o(g, TTt)).
D.2.4 REFERENCE-FRET DETECTION
LOSS (Carlini et al. 2021
The detection score is as below: f(g, o, To) = p(g, o, πT).
Zlib (Carlini al. 2021) based entropy/length proxy.
Zlib calibrates the detection score with Zlib(o), which is compression-
f(g, o, mo) ((g, o, ma) Zlib(o)
Min-K % (Shi et al. 2023) Min-k% compute the detection score on k% tokens with lowest prob- (Shi al. 2023), we choose as 20 by default all
abilities in sequence. Following experiments.
f(g,0,T0)
Σ
log To g, g.
[mon-k(o)
(Emin-K(o)
Min-K++ (Zhang al.) 2024)
Min-k%+ standardizes the log probability before taking Min- K%. μr is the expectation of the next token's 's log probbaility over the vocabulary V of the model given the prefix g, Oci, and a; is the standard deviation. We use k=20 by default.
log (0: 4, (,00))
f(g,o,m, To) = [mo)-
sEmin (()
#4 = log no q, (,0z)))
Evey
a; Stdwey (log (o(v 9,0)))]-
21

---

Published as a conference paper at ICLR 2026
M Maini et al. 2024) Max-k% compute the detection score on k% tokens with largest probabilities in the sequence. We use k=20 by default.
0 [max-k(o)
Σ (Emax-k(o)
Tolo 1opooo0g,000 0<))))
D.3 DATASETS
AIME 2024 & 2025 Olympiad-level mathematical- reassiinng benchmarks consisting of 30 prob- lems each from the American Invitational Mathematics Examination (AIME) 2024 and 2025.
AMC 2023 A high school level math benchmark consisting of 40 problems from the 2023 Amer- ican Mathematics Competitions (AMC).
2024) Graduate level scientific-reasoning, multiple-choice bench- mark written by domain experts in biology, physics, and chemistry. The Diamond split is the hardest subset, with 198 questions retained only when expert annotators agreed, and non-expert baselines typically fail.
GPQA Diamond (Rein et al.
OlympiadBench (He et al., 2024) An Olympiad level, bilingual, multimodal benchmark designed to test scientific reasoning in mathematics and physics. We use the English, text-only math subset consisting of 674 competition problems.
Minerva Math (Lewkowyez et al. 2022)
A challenging quantitative reasoning benchmark de- rived from Google's Minerva work, consisting of 272 problems.
D.4 IMPLEMENTATION DETAILS
SFT Implementation We use the LLaMA-Factory (Zheng et al. 2024) implementation for our SFT experiments. By default, we adopt the SFT hyperparameters suggested by
OpenThought3 (Guha et al. 2025) for medium dataset scales for the scenario that the base model
evolves into LRMs, shown below. To improve training efficiency, employ FlashAttention
(Dao 2023). DeepSpeed ZeRO-1 (Rasley et al., 2020). Liger kernels (Hsu et al. 2024 and
asynchronous activation offloading (Daniel Han & team 2023).
Training type
Batch Size
Context Length
LR
Epochs
LR Scheduler
Warmup Ratio
Weight Decay
Training Procision
Full
128
32,768
4e-5
cusine
0.1
bl16
For the extensive contamination LRMs in the final stage, the hyperparameters follows:
Training type
Batch Size
Context Length
LR
Epochs
LR Scheduler
Warmup Ratio
Weight Decay
Training Procision
Full
32
32,768
le.5
cusine
0.1
br16
RL Implementation We use Verl (Sheng et al. periments. For RAFT and RAFT++, follow the implementation of (Xiong et al. 2025). For GRPO, we follow DAPO (Yu et al., 2025) and do not introduce the KL. term in training in all the experiments. The detailed hyperparameters
2024 implementation for our RL ex-
follows:
Truining type Batch Size
Prompx Length Response Length
LR Epochs Rollout Num Rollout Temp
Training Precision
16,384
0.6
Prompt template We prompt template to enable long CoT during train both SFT and RL. We use math template to AIME2024 & 2025, AMC2023, OlympiadBench (He et al. 2024 and Minervamath (Lewkowycz al. 2022). We adapt multiple-choice template to GPQA Diamond (Rein et al. 2024).
Evaluation and Metric We evaluate pass @ 1 and run 10 rollouts on AIME 2024 & 2025, AMC 2023, and 3 rollouts on OlynpiadBench, GPQA Diamond, and Minerva Math to compute the pass@1. I. We use vLLM (Kwon et al.] (202)) for the inference. All inference uses the same con- figurations: temperature-0.6, top p=0.55, max new tokens=32,768.
22

---

Published as a conference paper at ICLR 2026
Reasoning Template for Math
prrestioon\/pPeaaee reason step by step, and put your final within \boxed(].
Example. Alice and Bob play the fol fowlowng gane. A stack of n tokens 1ies before them. The players take turns with Alice going first. On each turn, the player removes either 1 token the stack. Whoever of positive integers n less than or equal to 2024 for which there exists a strategy for Beb that guarantees that Bob will win the game regardless of Alice's play, ApPlease reason step by step, and put your final answer within \boxed().
4 tokens from the last token wins. Find the number
Reasoning Template for Multiple Choice Question
Return your final response within \boxed[] and only include the letter choice (A, B, C, or D} your final response. {question) ons}
Return your final response within \boxed[] and only include the letter choice (A, B, C, or D) your final response. trans-cinnama leeheee forming product 1.
Example.
treated with methylmagnesium bromi de,
was treated with pyridinium chlorochromate, forming product 2. was treated with
-e-s--llaeeyyyyeeeereelee methane in DMSO elevated temperature, forming product 3. how many carbon atoms are there in product 3? 11, B) 10, C) 12, D) 14
Deduplication We deduplicate our clean training datasets for both RL and SFT against the evalu- ation benchmarks using 13-gram overlap deduplication ensure conclusive results.
23

---

Published as a conference paper at ICLR 2026
E MORE EXPERIMENT RESULTS
E.1 MORE RESULTS OF
(STAGE I: PRE-LRM)
Table 7: AUROC (%) ) of contamination detection approaches evaluated starting from an SFT- contaminated model w/o RL to subsequently trained with GRPO. Results demonstrate that after GRPO, AUROC decreases across all the benchmarks and detection approaches. A measures the dif- ference with the SFT-contaminated model w/o RL. Higher AUROC, better detection performance. Each AUROC is averaged over detection scores from rollouts. The base model is Llama3. 1-8B- Instruct.
Contamination Detection Methods
Training Stages
Olympind
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
A
Generation based
Before
48.67
65.36
51.78
60.00
56.84
52.62
55.88
+0.00
Verbatim Wu al. 2025
RL w/ Clean
49.92
50.14
57.78
58. 28.22
57.21
57.38
55.11
-0.77
Clean&Menn
Before RI.
56.72
$4.55
53.33
55.11
60.03
66.75
57.75
10.00
CDD Dong 2024
RL w/ Cleun
53.14
55.69
60.00
46.22
56.31
64,00
55.89
-1.86
w/ Clean&Mem
43.78
63.50
Perturbation based
Before RL w/ Clean
51.64
41.11
53.86
63.62
+0.00
Neighbor (Mttern et al. 2025
52.48
37.07
55.78
39.33
50.32
65.00
50.00
-2.19
RL w/ Cleun&Mem
52.34
39.76
49.78
37.11
54.44
70.25
50.61
-1.58
Before RL
80,50
70.86
98.22
93.78
81.54
98.87
87.30
+0.00
LiRA Mireshghullah 2022
RI. w/ Clean
68.95
65.57
51.33
41.11
73.99
80.75
63.62
-23.68
RL w/ Clean&Mem
68.72
65.74
90.44
89.11
72.32
94.12
80.08
-7.22
Before RL
60.21
48.62
66.00
40.89
60.47
83,50
59.95
+0.00
Ref (Callini
RL
RL w/ Cleun&Mee
35.49
50.37
46.00
31.11
52.98
62.50
49.74
-10.21
Reference free
Zlib Carlini ct 2021
RL w/ Clean
46.53
56,37
66.67
50.00
45.91
41.38
51.14
-11.09
RL. w/ Clean&Mem
47.29
$2.29
66.22
60.89
45.63
46.00
53.05
Before
48.88
49.33
41.43
39.
51.67
30.63
43.51
+0.00
++ Zhang et al. 2024
RI. w/ Clean
46.67
47.61
29.33
24.22
59.85
34.38
40.34
-3.17
Cleun&Mem
Before RL
68.90
72.17
98.22
99. .6.56
76.16
94,50
84.92
+0.00
Min-K% Shi al.
2023
RL w/ Clean
59.11
69.85
60.44
67.56
73.89
75.87
67.79
-17.13
w/
Before RL.
63.60
61.58
92.44
89.33
69.12
91.50
77.93
+0.00
Max-K% Maini
2024
RI. Clean
50.59
$2.50
50.00
50.00
50.37
$2.50
-26.94
RL w/ Clean&Meun
50.91
52.24
49.11
49.11
54.86
61.75
53.00
-24.93
76.91
Loss Carlini et al. 2021
RL w/ Clean
59.18
69.84
60.00
68.00
74.02
76.75
67.97
-17.36
RL w/ Cleun&Mem
60.37
63.92
89.33
90.44
74.18
84.62
77.14
-8.19
24

---

Published as a conference paper at ICLR 2026
Training Methoc and RC. Stegs
20
SPT Cuzta farthee
SHT Conta
AUROC (%)
canDo (110 Stepe]
Stegs)
GRPO (156 Siepel
RAPT (64 Sleps)
Stepsi
Stepel
Stopo)
2151
Rondum Guessing
LiRA
Loss
Ref
Min-K
Max-K
Benchmark Contamination Detection Methods
Figure 5: AUROC (%) trends on SFT contaminated model further trained with different ob- jectives. While contamination introduced through SFT is initially detectable by existing methods, subsequent RL training with clean samples (e.g., GRPO or RAFT++) consistently degrades detec- tion performance. Moreover, we observe monotonic decline in detection performance as the num- ber of RL steps increases, and reference- free methods (e.g.. Loss, Min-K, and Max-K) already fall into near random guesses (i.e., AUROC*50%) simply after 156 steps. The base model Llama3. 1- 8B-Instruct
E.2 AUROC ON QUESTION, QUESTION+ RESPONSE, THINKING PROCESS, AND NON-THINKING PROCESS (STAGE PRE-LRM)
We compare the AUROC computed on response tokens with computed on the question tokens, whole tokens, thinking tokens, and non-thinking tokens in the response. As shown in Tab, S and
11 none of the approach outperform the AUROC computed on the response tokens. Thus, choose to compute all detection scores on the response if applicable. The base model here Qwen2.5-7B-Instrrr
Table 8: AUROC (%) of contamination detection approaches using question tokens compute the detection score, evaluated on the SFT-contaminated model w/o RL. A measures the difference with the reponse tokens signal (Tab. 2 Results demonstrate that question tokens are not suitable for detection in the LRM contamination setting, compared with using response tokens.
Contamination Delection Methods
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
A
Generation base
Verbatim (Wa ct al.
2025
47.58
49.86
47.56
53.56
52.52
65.50
52.76
+0.00
Perturhation base
Neighbur Mattern el al.
2023
47.04
56.10
44.22
32.44
49.99
54,62
47.40
-3.31
Reference base
LiRA (Mireshghallah et
2022
49.78
43.12
42.22
46.00
48.77
55.00
47.48
-41.65
Ref (Carlini al. 2021
49.80
48.47
40.22
47.56
49.38
53.88
48.22
-17.28
Reference free
Zlib (Carlini et 2021
48.15
50.78
60.00
40.89
54.23
52.75
51.13
-2.25
Min-K%+++ (Zhang al.
2024
44.33
50.53
32.00
36.89
49.67
34.25
41.28
-8.33
Min-K% (Shi et al 2123
46.68
$1.91
53.33
43.56
$4.13
44.38
49.00
-15.96
Max-K% MMarm el
49.39
$4.58
71.56
55.33
$4,98
66.25
58.68
-11.15
Loss (Carlini 2021
47.38
54.03
58.44
46.00
55.61
52.00
52.24
-23.24
25

---

Published as a conference paper at ICLR 2026
Table 9: AUROC (%) of contamination detection approaches using question response tokens to compute the detection score, evaluated on the SFFT-contaminated model w/o RI. A measures the difference with response tokens as signal (Tab. 2) Results demonstrate that considering both question and response tokens when computing detection scores actually harms the AUROC.
Contamination Detection Methods
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
Perturbation base
Neighbur Mallern al.
2023
54.56
43.42
48.89
38.44
$6,09
61.50
50.48
-0.23
Reference base
LiRA (Mireshghullahh et
2022
79.31
72.49
89.56
71.11
66.07
89.00
77.92
-11.21
Ref (Carlini at
2021
72.95
64,91
60.89
40.00
72.36
82,25
65.56
+0.06
Reference free
Zlib (Carlini et al. 2021
46.96
54,24
69.33
41.11
43.31
42.00
49.49
-3.89
Min
+%++ iZhlan a1. 2024
40.49
51.08
36.19
48.67
39.61
30.50
41.09
-8.52
M%n-K% (Shi al
$6.46
56.73
85.78
73.78
48.48
58.13
63.23
Max-K% Maann
2024
64.43
63.78
64,22
81.78
66.09
76.50
69,47
-0.36
Loss
CCrlin al. 2021
59.18
58.44
87.11
76.44
49.75
63.12
65.67
-9.81
Table 10: AUROC (%) of contamination-detcctiooo methods using reasoning tokens only (tokens inside Sthink></thinks
compute the detection score, evaluated on the SFT-contaminated model w/o RL. The last column, A, reports the difference relative to using the entire response as the signal (Tab. 2 Results show only minor differences between reasoning token only and whole- response signals, so we use the entire response the main analysis.
Contamination Detection Methods
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
A
Reference free
Zlib (Carlini et al. 2021
49.67
59.60
70.89
35.24
48.60
46.00
51.67
-0.93
Min-K K Shi cta
2023
70.76
66.46
82.89
74.52
70.34
82.25
74.54
+0.69
Max Maini
2024
67.40
62.00
66.67
82.38
65.87
79.00
70.55
+0.82
Loss (Carlini
70.81
66.75
84.89
74.52
70.06
82.75
74.96
+1.21
Table 11: AUROC (%) of contamination-detettion methods using non-reasoning tokens only (to- kens after </think>) to compute the detection score, evaluated on the SFF-contaminated model w/o RL. The last column, A, reports the difference relative to using the entire response the sig- nal (Tab. 2) Results show that using non-reasoning tokens degrades performance compared with whole-respones signals,
the entire response in the main analysis.
Contamination Detection Methods
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
Reference free
Zlib (Crliini al. 2021
59.73
$1.49
49.11
61.67
63.37
74.25
59.94
+7.34
Min- K Shi
2013
59.26
$4.82
48.22
80.00
64.8
76.5
63.93
-9.92
Max K% Mamn
2024
60.86
54.93
55.33
69.52
59.76
77.5
62.98
-6.75
Loss Carlin ct al. 2021
59.28
54.78
48.44
80.00
64.74
76.5
63.96
-9.79
26

---

Published as a conference paper at ICLR 2026
Response Length (tokens)
30000
GRPO (54 stops)
RAFT (e8 steps)
25000
20000
15000
10000
5000
alme24
alme25
amc23
gpqa diamond
minerva math
olympiadbench
Figure 6: Average response length for each question across different benchmarks.
E.3 RESPONSE LENGTH ANALYSIS (STAGE PRE-LRM)
We visualize the average response length per question across different benchmarks in Fig. Here, the base model is Qwen2.5-7h Insttruuctt. Although the average response length does not change too much after GRPO training, several detection approaches already exhibit substantial performance drop, shown in Tab. P to those after GRPO, no concealment effect is observed. These results suggest that response length is not the key factor; instead, the entropy of the model's output appears to drive the concealment phenomenon.
In contrast, for RAFT, even though the response lengths remain similar
27

---

Published as a conference paper at ICLR 2026
E.4 LOG-PROBABILITY DISTRIBUTION BEFORE AND AFTER RL (STAGE I: PRE-
We provide the log probability distribution for members vs. non-members in diverse scenarios. Fig. 7 g and demonstrate that further SFT and RAFT unable contract the distribution for members and non-members, while GRPO and RAFT++ could conceal the contamination evidence due to the PPO-style importance sampling/clipping term in the training objective. The base model here is 1een2.5-7B-Intrruue
SET Conta (w/o RL) further SFT
SFT Conta (w/o RL)
GRPO (64 Steps)
GRPO (136 Steps)
20
members
members
members
members
non-members
non-members
non-members
non
15
AUROC: 0.698
AUROC: 0.710
AUROC: 0.605
AUROC: 0.540
Density
10-
5
0
0.4
-022
0.0
0.4
-02
0.0
-044
-0.2
0.0
-0.4
-022
0.0
RAFT (64 Steps)
RAFT (156 Steps)
RAFT+ (45 Steps)
20
members
members
members
non members
non members
non members
15
0.698
AUROC: 0.584
Density
10-
-0.2
0.0
-.4
-0.2
0.0
-0.4
-0.2
0.0
Log Probability
Figure 7: Log-prob distributions for members vs. non-members of SFT contaminated model be- fore and after RL training GPQA-Diamond. With additional GRPO or RAFT++ training clean samples, the member and non-member log log-probability distributions become increasingly sim- ilar. Since many contamination detection methods rely separability in this space, the shrinking gap explains their degraded effectiveness. In contrast, further RAFT training does not induce the earlier distribution collapse; we explain in Sec. 3.2 the absence of clipping term prevents it. Likewise, additional SFT does not collapse the membership distributions.
28

---

Published as a conference paper at ICLR 2026
SET Conta (w/o RL) further SFT
SFT Conta (w/o RL)
GRPO (61 Steps)
GRPO (156 Steps)
members
members
members
members
non members
members
non members
non members
AUROC: 0.692
AUROC:
AUROC: 0.552
AUROC: 0.493
Density
20
10
0
-.2
0.0
-0.2
0.0
0.2
0.0
-0.2
0.0
RAFT (64 Steps)
RAFT (156 Steps)
RAFT+ (45 Steps)
members
members
members
30
mon-memberss
AUROC: 0.718
AUROC: 0.705
AUROC: 0.504
Density
20
-0.2
0.0
-0.2
0.0
-0.2
0.0
Log Probability
Figure 8: Log-prob distributions for members vs. non-members of SFT contaminated model be- fore and after RL training on OlympiadBench. With additional GRPO or RAFT+ training on clean samples, the member and non-member log-probability distributions become increasingly sim- ilar. Since many contamination detection methods rely on separability this space, the shrinking gap explains their degraded effectiveness. In contrast, further RAFT training does not induce the earlier distribution collapse; as we explain in Sec. 3.2 the absence of a clipping term prevents it. I Likewise, additional SFT does not collapse the membership distributions.
29

---

Published as a conference paper at ICLR 2026
SET Conta (w/o RL) further SFT
SFT Conta (w/o RL)
GRPO (61 Steps)
GRPO (156 Steps)
25
members
members
members
members
non members
AUROC: members
non members
non members
20
AUROC:
0.710
AUROC: 0.615
AUROC: 0.342
Density
15
10
5
0
-0.4
-0.2
0.0
0.4
-0.2
0.0
0.4
-0.2
0.0
0.4
0.2
0.0
RAFT (64 Steps)
RAFT (156 Steps)
RAFT+ (45 Steps)
members
members
members
non-members
AUROC: 0.716
AUROC: 0.561
AUROC: 0.528
Density
-0.4
00.2
0.0
00.4
-0.2
0.0
-0.4
-0.2
0.0
Log Probability
Figure 9: Log-prob distributions for members vs. non-members of SFT contaminated model be- fore and after RL training on Minerva Math. With additional GRPO or RAFT++ training on clean samples, the member and non-member log-probability distributions become increasingly similar. Since many contamination detection methods rely on separability this space, the shrinking gap explains their degraded effectiveness. In contrast, further RAFT training does not induce the earlier distribution collapse; as we explain in Sec. S.2. the absence a clipping term prevents it. Likewise, additional SFT does not collapse the membership distributions.
30

---

Published as a conference paper at ICLR 2026
E.5 AUROC FOR DIFFERENT TRAINING STEPS (STAGE I: PRE-LRM)
We provide complete results of AUROC in different RL training steps. As shown in Tab. 12 and 13 observe monotonic decline detection performance the number of RL steps increases when using GRPO. While even with 156 steps RAFT, there is no sign of AUROC decline. These results perfectly validate our theoretical analysis that RAFT is unable conceal contamination, while GRPO could.
Table 12: AUROC (%) of contamination detection approaches evaluated starting from SFT- contaminated model w/o RL to subsequently trained with GRPO in different steps. A measures the difference with the SFT contaminated model w/o RL (Tab. D The results demonstrate that ad- ditional GRPO steps better conceal the contamination evidence. The base model here is Qwen2.5- 7B-Instruct.
Contamination Detection Methods
Steps
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
Generation
64
45.46
50.49
48.89
57.56
52.74
59.63
52.46
-0.30
Verbatim Wa et al. 2025
110
45.27
52.32
$2.44
50.22
50.86
60.88
$2.00)
.0.76
156
46.51
51.09
53.33
53.56
52.84
58.50
52.64
-0.12
64
55.47
51.08
43.33
60.00
18
62.00
55.34
-0.45
CDD Dong al. 2024
110
$4,90
54,20
49.33
59.11
53.48
58.88
$4.98
-0.81
156
53.99
43.42
34.22
70.89
56.32
62.38
53.54
-2.25
Perturbation base
64
$4.10
39.68
50.67
44.22
53.42
60.5
50.43
-0.28
Neighbor Mattern
110
53.93
46.22
$4,27
-1.03
156
53.32
41.07
47.11
38.67
54,49
60.75
49.24
-1.47
Reference base
84.65
87.78
-8.99
LiRA Mireshghallah al
2022
110
67.87
79.80
60.22
78.89
81.49
71.88
73.36
-15.77
156
66.41
77.28
54.22
80.22
77.62
75.13
71.81
-17.32
58.41
65.54
60.13
-5.37
Ref (arlinii al. 2021
110
64.92
54.70
44.44
52.22
66.47
72.37
59.19
-6.31
156
62.20
57,30
50.89
55.78
64.62
75.25
61.01
-4.49
Reference
64
45.94
54,99
66.22
35.56
46.65
39.38
48.12
-5.26
Zlib Carlini et al.
2021
110
44.98
56.54
64.22
39.11
44.93
38.38 38
48.03
-5.35
156
53.21
46.43
-5.38
64
46.25
46.78
36.67
50.89
$1.35
29.62
43.59
-6.02
Min-K%++ Zhang et al. 2024
110
43.17
44.69
28. 33
50.00
49.44
46.25
43.65
-5.96
156
46.12
44.86
32.44
44.00
$3.39
35.38
42,70
-6.91
64
55.19
60.60
62.89
65.56
61.50
61.87
61.27
-13.69
Min-K% Shi et al.
2023
110
49.98
58.49
61.78
61.33
56.20
49.00
56.13
-18.83
156
53.97
-24.10
64
53.05
51.43
49.78
50.22
51.84
57.75
$2.35
-17.48
Max-K% Maini et al.
2024
110
$1.02
$1.39
49.78
53,33
51.88
47.50
$0.82
-19.01
156
49.81
53.31
50.22
$1.52
$5.00
$1.64
-18.19
64
55.22
60.50
62.44
65.78
61.50
62.12
61.26
-14.22
Loss Carlini al.
2021
110
50.17
58.25
60.67
62.89
56.47
49.12
56.26
-19.22
156
49.32
$4.04
44.22
54.22
54,20
48.50
50.75
-2473
31

---

Published as a conference paper at ICLR 2026
Table 13: AUROC (%) of contamination detection approaches evaluated starting from an SFT- contaminated model w/o RL to subsequently trained with RAFT in different steps. A measures the difference with the SFT contaminated model w/o RL (Tab. 2 Results demonstrate that even with more RL steps, RAFT is unable to conceal the contamination evidence. The base model here D0wen..5-BBIInsteee
Contamination Detection Methods
Steps
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
A
Generation base
64
45.43
51.19
59.33
60.67
51.48
61.13
54.87
+2.11
Verbatim Wu al. 2025
110
45.67
51.24
58.67
58.67
52.33
57.50
54.01
+1.25
156
45.15
52.16
59.11
59.33
51,28
61.25
54.71
+1.95
64
$7.85
53.55
46.89
60.22
59.63
66.62
$7.46
+1.67
CDD Dong 2024
55.97
52.86
43.11
$2.89
56.45
58.25
$3.26
-2.53
156
55.59
52.94
32.22
53.11
55.84
67.62
52.89
-2.90
Perturbation base
55.44
39.63
50.22
43.11
53.15
58.88
50.07
-0.64
Neighbor Mattern et al. 2023
110
55.06
38.47
46.89
40.67
54.20
63.38
49.78
-0.93
156
55.06
39.19
40.89
41.78
49.66
$7.12
47.28
-3.43
Reference base
64
85.85
85.69
98.22
84.22
84.86
91.75
88.43
-0.70
LiRA Mireshghallah et al.
2022
110
84.32
85.76
94.44
88.44
84.55
91.25
88.
-1.00
156
83.32
94.44
82.00
58.73
86.63
81.95
-7.18
64
76.02
65.94
74.67
46.00
73.99
75.62
68.71
+3.21
Ref Carlini al. 2021
110
76.41
63.76
71.11
50.22
74.50
81.75
69.63
+4.13
156
75.92
64.23
67.33
54.67
56.73
81.38
66.71
+1.21
Reference free
64
$1.88
59.73
76.00
52.22
52.40
47.75
56.66
+3.28
Zlib Carlini et al. 2021
110
$2.39
62.20
76.22
49.78
74.51
45.88
$6.29
+2.91
62.35
79.78
47.78
52.58
50.37
$7.65
+4.27
64
51.14
54.13
51.78
62.00
59.86
$2.75
55.28
+5.67
Min-K%-++ Zhang et al. 2024
110
$2.30
55.56
56.67
62. 14
55.00
49.50
55,20
+5.59
58.44
52.53
64
72.28
69.56
88.44
86.67
71.63
78.88
77.91
+2.95
Min-K% Shi et al.
2023
110
71.44
73.69
78.67
90.22
70.45
77.62
77.02
+2.06
70.76
84.00
80.67
79.38
73.68
-1.28
64
68.11
68.37
68.44
90.67
69.21
80.50
74.22
+4.39
Max 5% Maini et al. 2024
110
67.97
68. 16
65.78
92.44
69.00
76.75
73.35
+3.52
156
69.14
68.30
69.33
90.67
58.06
76.00
71.92
+2.09
64
71.78
69.78
86.00
86.67
71.58
79.25
77.51
42.03
Loss Carlini
78.44
77.12
76.90
156
70.49
70.95
83.11
80.44
56.07
79.12
73.36
-2.12
32

---

Published as a conference paper at ICLR 2026
Table 14: AUROC (%) of contamination detection approaches evaluated starting from SFT- contaminated model w/o RL to subsequently trained with GRPO in different steps. A measures the difference with the SFT contaminated model w/o RL (Tab. Z The results demonstrate that additional GRPO steps better conceal the contamination evidence. The base model here is Llama- 3. Instruc..tt
Contamination Detection
Steps
Olympiad
GPQA
AIME24
Minerva
AMC23
Avg.
Generation base
64
49.92
50.14
57.78
58.22
57.21
57.38
55.11
-0.77
Verbatim Wu al. 2025
110
49.53
51.25
60.67
54.73
53.12
53.92
-1.96
156
49.45
52.11
51.11
60.44
57.02
47.12
52.88
.3.00
64
$3.14
55.69
60.00
46.22
56.31
64.00
$5.89
-1.89
CDD Dong et al. 2024
110
53.48
48.40
46.67
49.11
57.34
56.00
51.83
-5.92
156
$3.99
52.18
45.56
$2.89
60.62
57.25
53.75
-4.00
Perturbation base
64
$2.48
37.07
55.78
39.33
50.32
65.00
50.00
-2.19
Neighbor Mattern 2023
110
52.83
38.31
49.78
39.56
50.18
60.00
48
-3.75
156
$3.09
39.15
48.00
44.00
$2.29
62.25
49.80
-2.39
Reference base
64
68.95
65.57
51.33
41.11
73.99
80.75
63.62
-23.68
LiRA Mireshghallah al. 2023
110
66.57
65.72
44.
27.56
72.02
71.00
$7.85
-29 45
156
60.93
64.21
23.11
69.25
70.12
54.64
-32.66
64
S5.27
47.41
40.67
32.44
$2.31
$3.25
46.
-13.06
Ref Carlini et al. 2021
110
$4,98
49.70
42.89
26.67
53.16
58.00
47.57
-12.38
156
53.23
47.75
43.11
27.33
56.31
56.75
47.41
-12.54
Reference free
64
46.53
56.37
66.67
50.00
45.91
41.38
51.14
-11.09
Zlib Carlini et 2021
110
46.78
57.11
64.89
48.44
46.40
37.25
50.15
-12.08
156
46.32
54.41
65.78
39.78
42.50
42.8% 88
48.61
-13.62
64
46.67
47.61
29.33
24.22
59.85
34.38
40.34
-3.17
Min- +%++ Zhang et
2024
110
49.92
50.25
42.89
27.78
$7.32
42.75
45.15
+1.64
156
49.42
48.28
41.33
38.67
53.09
48.25
46.51
+3.00
64
$9.11
69.85
60.44
67.56
73.89
75.87
67.79
-17.13
Min-K9 K% Shi et al.
2023
110
56.75
66.89
54.89
64.22
71.61
62.25
62.77
-22.15
156
55.83
65.12
50.44
59.78
62.76
62.50
59.41
-25.51
64
50.59
$2.50
50.00
50.00
50.37
$2.50
50.99
-26.94
Max K% Maini et
2024
110
50.29
54.44
50.00
$0.00
50.00
$2.50
51.21
-26.72
156
50.30
51.41
53.33
50.00
51.09
50.00
51.02
-26.91
64
59.18
69.84
74.02
76.75
67.97
-17.36
Loss Carlini al.
2021
110
56.85
66.90
55.11
63.78
71.60
62.38
62.77
-22.56
156
55.75
65.06
50.00
59.78
62.72
63.00
$9.39
-25.94
33

---

Published as a conference paper at ICLR 2026
Table 15: AUROC (%) of contamination detection approaches evaluated starting from an SFT- contaminated model w/o RL to subsequently trained with RAFT in different steps. A measures the difference with the SFT contaminated model w/o RL. (Tab. 7 Results demonstrate that even with RL steps, RAFT is unable to conceal the contamination evidence. The base model here is 1-8----st--t-
Contumination Detection Methods
Steps
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
A
Generation base
64
46.75
$6.44
58.89
55.77
57.00
$3.99
Verbatim Wu al. 2025
110
46.89
50.85
52.00
53.33
58.78
55.12
52.83
-3.05
156
47.06
49.70
54.22
60.44
55.63
61.75
$4.80
-1.08
64
54.44
$2.22
46.44
53.57
62.62
54.44
-3.31
CDD Dong al.
2024
110
56.97
54.92
60.00
50.00
55.76
65.38
57.17
-0.58
156
55.78
57.84
53.11
$0.89
60.63
70.88
58.19
+0.44
Perturbation base
64
51.81
40.93
64.00
48.44
55,28
66.50
$4.49
+2.30
Neighbor Mattern et al.
2023
110
$2.08
39.36
64.22
$4.00
$4.84
68.88
$5.56
+3.37
156
51.99
40.25
66.22
49.56
51.68
72.75
55.41
+3.22
Reference base
64
81.82
77.88
96.89
96.67
82,92
97.50
88.95
+1.75
LiRA Mireshghallah al.
2022
110
81.54
77.08
99.56
95.78
86.03
99.00
89.83
+2.63
156
80.52
77.21
99.11
96.00
84.04
97.75
89.11
+1.91
64
60.81
51.55
65.33
49.11
61.41
88.00
62.70
+2.75
Ref (Crlini al. 2021
110
62.25
49.94
76.89
$2.67
61.33
86.00
64.85
+4.93
77.78
49.78
63.06
87.50
64.63
+4.68
Reference free
64
53.02
58.98
80.67
68.44
51.21
60.75
62.18
-0.05
(Carlini et
66.22
+3.61
156
54.13
62.78
85.33
64.89
54.81
68.62
65.09
+2.86
64
48.97
48.15 15
36.00
36.89
55.41
37.25
43.78
+0.27
Min-K%+++ Zhang et
2024
$1.00
56. 82
44.44
38.89
58.25
39. 39.47
48.15
44.64
156
50.43
58.89
34.44
29.11
58.98
45.50
46.23
+2.72
64
73.95
76.84
98.00
98.89
82.44
97.25
87.90
+2.98
Min al. 2023
71.87
80.78
98.89
98.
83.92
95.88
88.26
+3.34
156
71.30
81.59
98.22
100.00
82.54
95.50
88.19
+3.27
64
66.57
70.43
92.44
89.11
77.31
95.25
81 85
43.92
Max K% Maini al.
2024
110
67.37
75.39
92.89
92.00
78.65
96.88
83,86
+5.93
156
67.61
73.85
90.67
94.22
76.
95.50
82.99
+5.06
73.77
78.18
98.00 20.00
99.56
83.14
97.38
+3.01
Loss Carlini al.
2021
110
72.40
81.76
99.11
98.67
84.18
97.75
88.98
+3.65
156
71.57
82,91
98.67
100.00
82.34
95.75
88.54
+3.21
34

---

Published as a conference paper at ICLR 2026
E.6 CONTRACTION EFFECT ON SMALL AND LARGE MODELS (STAGE I: PRE-LRM)
We conduct experiments on smaller and larger models to see whether the concealment of RL has correlations with the model size. In particular, we provide additional results with small model (i.e.. Dwen2.5-3BBIInsrmul
in Table T6 and a large model (i.e., D0ven2.5-14B-Instrun compare them with the contraction of the medium-sized model (i.e., Sween..57B-Instrruee RL steps are 64 by default. The empirical results further validate that the contraction extends to large range of model sizes. Regardless of different model sizes, the RL training could still conceal the contamination introduced in the SFT stage.
in Table We Here,
Table 16: AUROC (%) of contamination detection approaches evaluated starting from SFT- contaminated model w/o RL to subsequently trained with GRPO. Results demonstrate that after GRPO, AUROC decreases across all the benchmarks and detection approaches. A measures the dif- ference with the SFT-contaminated model w/o RL. The base model here is Qwen2.5-3B-Instrud
Contamination Detection Methods
Training Stages
Olympind
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
Reference based
LiRA Mireshghallah al. 2012
Before RL
$4.89
48,60
88.67
95. 56
61.74
90.25
73.29
+0.00
Clean
51.17
81.56
65.19
Ref (Callini 2021
Before RI.
$2.66
43.09
84.00
73.33
51.75
75.75
63.43
+0.00
58.91
Reference free
Before RL
57.64
76.00
56.84
60.83
+0.00
Loss Carlini et
RL w/ Clean
49.20
57.54
42.22
55.11
54.83
54.75
$2.28
-8.56
Before RL
50.86
57.24
51.11
73.33
56.47
59.13
58.02
+0.00
Min-K
ot
RI. Clean
48.71
56.93
39.56
49.78
54.90
51.37
50.21
-7.82
Maini 1024
Before RL RL Clean
$2.86
54.58
88.44
51.35
+0.00
52.07
54.54
56.67
57.78
52.57
74.00
$7.94
-11.19
Table 17: AUROC (%) of contamination detection approaches evaluated starting from SFT- contaminated model w/o RL to subsequently trained with GRPO. Results demonstrate that after GRPO, AUROC decreases across all the benchmarks and detection approaches. A measures the dif- ference with the SFT-contaminated model w/o RL. The base model here is Qwen2.5-14B-Inst
Contamination Detection Methods
Training Stages
Olympiad
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
A
Reference based
LiRA Mireshghallah al. 2012
Before RL
87.35
86.54
91.56
85.33
88.12
91.25
88,36
+0.00
Clean
82.38
84.00
84.59
Rel Carlini 2021
Before RI.
62.33
54.69
60.44
37.78
59.26
65.25
56.63
+0.00
36.00
53.17
Reference free
Before RL
76.98
88.00
+0.00
Loss Carlini et
RL w/ Clean
63.50 30
61.21
67.33
89.11
71.73
68.75
70.27
-9.73
Before RL RI. w/ Clean
73.04
76.16
77.11
86.22
77.61
83.75
78.98
+0.00
Min-K al
63.15
60.09
66.67
87.78
71.69
67.12
69.42
-9.57
Maini 1024
Before RL RL. Cleun
63.75
71.46
84.89
$7.99
78.00
70.02
+0.00
52.94
59.10
46.67
53.33
48.89
55.00
$2.66
-17.36
35

---

Published as a conference paper at ICLR 2026
E.7 MORE RL TRAINING BRINGS PERFORMANCE GAIN (STAGE I: PRE-LRM)
In Table we note that sometimes RL does not bring any performance gain, even in the clean setting. To clarify that RL setting and recipe correct, would like point that the insignificant RL improvement was due to our relatively short RL training run to save experiment cost, which already effectively demonstrated the contamination concealment effect, whereas RL fine-tuning on reasoning models typically requires many more training steps to boost performance significantly (i.e, thousands of RL steps, more than 16k GPU hours, etc).
However, we further verified our RL pipeline by continuing RL training on the SFT-contaminated Qwen-3B model for up to 280 steps. The RL concealment still exists: the average AUROC of LiRA on the AIME24, AIME25, and AMC23 datasets drops from 91.49% before RL to approaching random guessing after 280 steps. At the same time, RL continues to improve task performance, reported in Table 18 Overall, these results demonstrate that additional RL training contaminated model can both inflate performance and further conceal contamination signals.
SFT-
Table 18: Pass@ (%) across six reasoning benchmarks. Results show that additional RL training an SFT-contaminated model can bring performance gain.
Model
Olympiad
GPQA
AIME25
ATME24
Minerva
AMC23
Avg.
26.98
31.06
0.83
6.67
21.69
36.07
20.54
SFT contamination
35.07
37.11
13.23
14.32
29.13
44.06
28.82
Further RL
40.74
38.38
15.12
15.83
31.25
49.06
31.73
36

---

Published as a conference paper at ICLR 2026
E.8 TOKEN EMBEDDING VISUALIZATION OF MEMBER/NON- -OEEEBEER OF BENCHMARKS (STAGE I: PRE-LRM)
We provide UMAP visualization of token embeddings of member and non member of benchmarks. Specifically, we input the entire question and response to the model, and extract the last token from the last layer of the hidden space. The base model is selected neighbors as 15, min dist as 0.1, n components as 2, and metric as cosine. As shown in Fig. Top 11 and 12 the visualization indicates that member/non-menn memmbb embeddings are highly overlapped and are hard to distinguish. We also provide quantitative evaluations of embedding based detection following (Liu et al. (024) Table 19 particular, we randomly split all member and non-member examples into training and test sets, maintaining consistent 0.8/0.2 ratio across datasets, and train labels. The input to the MLP the hidden state feature of the last token in the question and response pair. Following (Liu et al. 2024). evaluated several hidden layers and selected the best performing one. We report results only on the medium-scale benchmarks (Olympiad, GPQA, Minerva), since the small-scale datasets have very few test examples, resulting in too much variance. The results show that contamination initially detectable, but becomes undetectable after RL.
Qwen2.5-7B-Instruc. We choose
an MLP classifier to predict the member/nonn-emeen
Table 19: AUROC (%) of the embedding- based detection (Liu et al. 2024) evaluated starting from an SFT-contaminated model w/o RL to subsequently trained with GRPO. Results demonstrate that after GRPO, AUROC decreases. A measures the difference with the SFT-contaminated model w/o RL. Here, the base model is Qwen2.5-77 IB-Intruct.
Training Stage
Olympiad
GPQA
Minerva
Avg.
A
Before RL
58.89
80,00
60.05
66.31
+0.00
RL w/ Clean
56.63
45.75
49.60
50.66
-15.65
RL w/ Clean&Mem
57.43
45.00
48.94
50.46
-15.85
Member
Member
Non-member
Non-member
(a) Base model
(b) SFT contaminated model
Figure 10: Last token embedding visualization on the Minerva Math dataset.
Member
Member
Non member
Non. member
(a) model
(b) SFT contaminated model
Figure 11: Last token embedding visualization on the GPQA-Diamond dataset.
37

---

Published as a conference paper at ICLR 2026
Momser
Momber
(a) Base model
(b) SFT contaminated model
Figure 12: Last token embedding visualization on the OlympiadBench dataset
E.9 CONTAMINATION INFLATION COMPARISON (STAGE PRE-LRM)
To demonstrate performance inflation from SFT contamination, choose
as the base model and compare pass @ across three training settings: (i) 15K clean samples from Openthought3 (Guha et al. 2025 (ii) 10K clean samples plus three full repetitions of six entire benchmark data (13,735 samples in total), and (iii 1.2M clean samples (i.e., open- thoughts/OpenTThoughte 1.2M). As shown in Tab. 20 the contaminated model outperforms the 15K clean baseline by an average of 10.80% across six benchmarks and even surpasses Open' SppnnThougtt3-- 1.2M on GPQA-Diamond and Minerva Math datasets. These results demonstrate that benehmark contamination can easily yield substantial performance inflation.
Table 20: @ I (%) of comparison between clean and contaminated SFT models. Bold=Best.
Training Data
Olypaid
GPQA
AIME25
AIME24
Minerva
AMC23
Avg.
15K clean
50.81
41.67
21.67
29.17
34.01
77.50
42.47
13.7K (clean + benchmarks)
58.52
59.09
40.00
40.00
44.49
77.50
53.27
1.2M clean
63.70
50.88
60.00
62.50
37.50
92.50
61.18
38

---

Published as a conference paper at ICLR 2026
E.10 LOG-PROB DISTRIBUTION AFTER EXTENSIVE SET CONTAMINATION ON LRMS (STAGE II: POST-LRM)
We provide the log prob distributions for members and non-members of deepseek distill models before and after contamination on Minvera Math and GPQA-Diamond in Fig. T3 and [4] Even though non-members have not been exposed to LRMs during the contamination, the log-prob would also increase, demonstrating that LRMs start to generalize after contamination.
R1 Distill LL (Extensive Conta)
R1 Bistill Ouen
(Clean)
Distill Qwea (Clean)
(Exteasive Conta)
memlleres
cemelers
meebors pon-nembers
pettmmmbees
AUROC- 0.524
XUROG-O.FF
AURDC: 0.522
AUROC: 0.548
Density
20-
Density
-0.4
-0.2
0.0
-0.4
-0.2
0.0
-0.4
-0.3
-0.2
-0.1
0.0
-0.4
-0.3
-0.2
-0.1
0.0
Log Probability
Log Probability
(a) RI Distill LLaMA results on Minerva Math
(b) R1 Distill Qwen results on Minerva Math
Figure 13: Log-prob distributions for members vs. non-members of advanced LRMs before and after contamination. After extensive SFT contamination on members, the log prob of both mem- bers and non-members increases at a similar margin, due to the generalization.
Distili LLAMA (Extensiew SET Conta)
Distill (Fstensive SET Comta)
R11 Distill LLaMA (Cluan)
RI Distill Quan (Clean)
15
rearbbrr
mummbers
member
munberr
pas cinbers
Density
70
AURIC: BS537
AUROC 04600)
Density S
6-
AUSROG 0.513
AUROG 0.541
01
-0.6
-0.4
-0.2
0.0
-0.6
-0.4
-0.2
0.0
-0.4
-0.2
0.0
-0.4
-0.2
0.0
Log Probability
Log Probability
(a) R1 Distill LLaMA results on
(b) R1 Distill Qwen results on GPQA-Diamond
Figure 14: Log-prob distributions for members vs. non non-members of advanced LRMs before and after contamination. After extensive SFT contamination on members, the log prob of both mem- bers and non-members increases at a similar margin, due to the generalization.
39

---

Published as a conference paper at ICLR 2026
E.11 APPLICABILITY IN NON-MATH DOMAINS (STAGE AND II)
We further evaluate our claims on datasets beyond math and science domains for both the pre-LRM and the post-LRM stage. We choose LiveCodeBench v2 (Coding) and MMLU-Pro (General QA). For MMLU-Pro, we only select the 'health', "history', 'law", 'other", 'philosophy', and "psychology" domains that are not related math and science.
We first provide results for the pre-LRM stage (i.c., contamination happens when the base model evolves into LRMs) with Qwen2.5-7B-Instrrutt the base model in Table 21] Despite differences in AUROC degradation speed across domains, the trend remains consistent: RL conceals the SFT contamination. Overall, these results indicate that the effect of RL concealments not limited to the reasoning domains but extends to more general domains as well.
We also provide the AUROC of contamination detection approaches evaluated on advanced LRM that is contaminated in the Post-LRM stage (i.e., contamination with CoT on advanced LRMs) in Ta- ble 22 The results show that contaminated LRMs leave minimal contamination evidence coding as well, indicating that the detection performance is near random guess for those reasoning-related domains (e.g., coding, math, science, etc). For domains that require less CoT (e.g., general QA), despite that the AUROC could reach 75% when using the Loss the detection for the contaminated Deepseek distilled Qwen- 14B, the detection performance much worse compared to results in the pre-LRM stage. Specifically, the AUROC for the Loss detector MMLU-Pro reach 93.72%, and LiR A could achieve 79.85% on LiveCodeBench when SFT contamination is applied to the base model, shown Table 18 the paper.
Table 21: AUROC (%) of contamination detection approaches evaluated starting from an SFT- contaminated model w/o RL to subsequently trained with GRPO in pre-LRM stage. Results demonstrate that after GRPO, AUROC decreases across both general QA and coding domains and different detection approaches. A measures the difference with the SFT-contaminated model w/o RL. Higher AUROC, better detection performance. The base model here is Qwen2.5-7B--nnse
MMLU-Pro
Contamination Detection Methods
Training Stages
LiveCodeBench
Avg.
(Non-STEM domains)
Reference based
Before RI. RL w/ clean
96.68
79.85
88.26
+0.00
LiRA (Mireshghallah al.
2022
93.85
71.12
82.48
-5.78
Before RL
86.25
69.51
77.88
+0.00
Ref (Carlini et al.
2021
RL w/ clean
81.67
66.68
74.17
-3.71
Reference free
Before RL
93.72
64.87
79.29
+0.00
Loss Carlini et al.
2021
w/ clean
88.69
54.26
71.47
-7.82
Before RL
93.43
64.12
78.77
+0.00
Min-K Shi et al.
2023
clean
88.54
$4.29
71.41
-7.36
Before RI.
82.27
63.75
73.01
+0.00
Max-K Maini al.
2024
RL w/ clean
59.48
49.80
54.64
-18.37
40

---

Published as a conference paper at ICLR 2026
Table AUROC (%) of contamination detection approaches evaluated on advanced LRMs con- taminated with CoT in both general QA and coding domain benchmarks in post-IRM stage.
MMLU-Pro (Non-STEM domains)
Contamination Detection Methods
Init Models
LiveCodeBench
Avg.
Reference based
DS Llama-B
72.85
61.37
67.11
LiRA Miireshghalahh et al. 2022
DS Qwen-14B
74.75
58.78
66.77
DS Llama-8B 60.77
58.75
59.76
Ref (Carlini et al. 2021
DS Qwen-14B3
63.43
58.17
60.80
Reference free
DS Llama-8B
69.56
57.42
63.49
Loss CCarlini et al.
2021
DS Qwen-14B
75.83
56.52
66.18
DS Llama-8B
68.12
56,59
62.36
Min-K Shi et 2023
DS Owen-14B
76.09
56.08
66.09
DS Llama-8B
61.07
56.33
58.70
Max-K K MMani al.
2024
DS Qwen-14B 14B
66.88
54.80
60.84
Table 23: @ass@] I (%) of the SFT-contaminated model after four additional epochs of SFT on clean data. The results show that further SFT does not make the model forget contamination; instead, pass@1 1 continues to increase by 0.25% across six benchmark on average compared to the SFT contaminated model, indicating persistent performance inflation.
Models
Olypaid GPQA AIME25
AIME24
Minerva AMC23
Avg.
SFT Conta (w/o RL) Further SFT
55.24
49.43
29.58
36.67
39.15
75.00
47.52
Table 24: Pass@1 (%) of advanced LRMs before and after SFT contamination with CoT on both clean and member data. Some LRMs with strong reasoning ability may not have a huge performance inflation after SFT contamination with CoT on both clean and members. Thus, choose the SFT contamination with CoT on members only as the default setup in our main analysis.
Models
Olypaid
GPQA
AIME25
AIME24
Minerva AMC23
Avg.
DeepSeck-RI RI--istill -iaiil--lama-888 8B
52.10
43,94
33.33
43.33
32.97
84.58
48.38
+ SFT Clean & Mem
56.70
45.83
48.33
56.67
35.94
88.12
55.27
DeepSeek-Ri DeepSeet-RI--ssii---een.. SFT Clean & Mem
55.70
48.65
39.26
53.70
37.25
91.94
$4.42
55.81
45.08
40.00
55.42
37.87
88.12
53.72
41

---

Published as a conference paper at ICLR 2026
F COMPUTATION RESOURCES
All experiments were run on a single node with 9x NVIDIA L40S GPUs (48 GiB each; ~x432 GiB total), NVIDIA driver 570.86.16, and CUDA 12.8. The node uses a I-socket Intel Xeon Gold 6338 CPU (2.00 GHz base, to 3.20 GHz), 128 hardware threads, 96 MiB L3 cache (two slices), and 1.0 TiB RAM, running Ubuntu 22.04 04 (Linux 1.8.0-99-generie
G
USE OF THE LLM
We only use LLM to aid paper writing and retrieve related works.
42


Reviews:
[{"review_id": "DfUchtcReY", "reviewer": "Reviewer_enZz", "summary": "This paper investigates contamination detection methods for benchmarks in base models when they are being conditioned as LRMs via SFT and RL, and in advanced Large Reasoning Models via SFT with CoT contaminated data. They find that GRPO conceals contamination, and minimal detectable traces are left. 10 different detection methods and 6 mathematical reasoning benchmarks are used. A theoretical analysis is provided to show that PPO-style clipping and importance sampling are the root cause of the concealment. The paper makes the key claim that memorization based detection are not optimal for LRMs, and that these can evade it easily.", "questions": "- Do you have some figures on the length of outputs across the paper? Have you found any tested detection methods sensitive to output length?\n- How do you generate your CoT RL data for each benchmark?\n- In the first proposed direction in your conclusion (lines 477-478), how would the release of intermediate training checkpoints help with the issues the paper raises?\n- Do you generate your own contaminated CoT and SFT data? If so, can you share it and how it was generated?\n- Embedding based methods are mentioned but not used, why?\n\nSuggestions:\n- In line 72,can you clarify what you mean by “in the later stage”.\n- In line 81, what kind of contamination is this? Is it verbatim contamination, paraphrased, etc..\n- In the theoretical analysis it would be good to explicitly define E in formula 1 (and subsequent mentions).\n- Given that this paper could aid malicious actors, can you provide more concrete recommendations for evaluators in the conclusion?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is highlighted that almost all the contamination detection methods (tested) consistently perform near random guesses in all the benchmarks after performing SFT on the benchmark samples with Co. This is a very significant insight.\n- The paper is thorough in exploring its claim within the mathematical reasoning domain, using 6 common benchmarks in the field. And in using a wide range of detection methods (10).\n- It is comprehensive in both providing results to support its claim and a thorough theoretical analysis of the premise", "weaknesses": "- The authors mention benchmark performance after SFT but do not mention results after SFT and GRPO.\n- While enough benchmarks from the mathematical reasoning domain are used, no other domain is surveyed to make the points the paper raises applicable globally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VeWi1O8U9O", "reviewer": "Reviewer_hTiN", "summary": "This paper investigates the fair evaluation of Large Reasoning Models on public benchmarks, showing that contamination that can be detected from SFT can be effectively concealed using GRPO, pushing many contamination detection methods to near-random performance, while the contaminated LRMs retain improved performance over their uncontaminated counterparts.", "questions": "- Could the authors clarify their claims in the Discussion and Conclusion? Specifically the claim \"This fundamentally challenges the assumption that all the detection approaches rely on, which is that benchmark contamination is more about memorizing the benchmark samples.\"\n- Could the authors provide additional details about why GRPO does not improve concealment on the Verbatim/Neighbor attacks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Benchmark contamination is a critical and timely problem to address, and the interplay between GRPO and contamination detection is interesting to investigate.\n- Experiments are well-designed: There is a clear concealment effect of GRPO on almost all detection methods, and current contamination detection approaches are clearly inadequate for ensuring the integrity of public leaderboards.", "weaknesses": "The main weakness comes from the underlying motivation for this work. Primarily, it seems the authors are highlighting that contamination detection methods are ineffective, yet it is unclear whether these methods are being used in practice. Recent work has highlighted pitfalls in data contamination approaches [1,2] and prohibiting training on test tasks [3]. A further discussion on how this work fits into prior work on fair evaluation and data contamination would strengthen the work.\n\nIn addition, do the authors believe that tailoring contamination detection methods will be the way forward for ensuring faithful evaluation when developers are incentivized to game benchmarks?\n\nMinor\n- Line 156 \"Olypaid\"\n\n[1] Liu, Ken Ziyu, et al. \"Language models may verbatim complete text they were not explicitly trained on.\" arXiv preprint arXiv:2503.17514 (2025).\n\n[2] Fu, Yujuan, et al. \"Does data contamination detection work (well) for llms? a survey and evaluation on detection assumptions.\" arXiv preprint arXiv:2410.18966 (2024).\n\n[3] Dominguez-Olmedo, Ricardo, Florian E. Dorner, and Moritz Hardt. \"Training on the test task confounds evaluation and emergence.\" arXiv preprint arXiv:2407.07890 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QEmMBE1rHE", "reviewer": "Reviewer_F3s6", "summary": "The paper studies benchmark contamination for LRMs in two practical scenarios: 1) SFT contamination on a base model is initially detectable, but a small amount of RL greatly conceals contamination signals across many existing detection methods. 2) When advanced LRMs are SFT-contaminated with CoT on benchmark samples, pass@1 improves significantly while the contamination detection becomes near random. The paper provides theoretical reasoning for this phenomenon by showing the gap in the log-likelihoods of benchmark members' and non-members' contracts after RL training. Such contraction is attributed to the common importance-sampling + clipping trick in RL.", "questions": "1. The main results are on math tasks with long CoT. Can the post-LRM “near-random detection” result replicate on coding, QA, and other tasks where solutions are relatively less CoT-heavy?\n\n2. Do larger/smaller models strengthen or weaken contraction? Is there any non-monotonicity regarding model size (e.g., medium models most vulnerable)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper looks into benchmark contamination, which is a timely and important issue in modern LLM evaluation. Showing that light RL can significantly mask contamination signals is surprising and consequential.\n\n2. The paper clearly splits the study into two realistic scenarios, including 1) base-model SFT contamination followed by RL, and 2) CoT contamination of advanced LRMs, making the threat model concrete and realistic. It also clarifies where current detectors fail and why mitigation is nontrivial.\n\n3. Attributing the contamination concealment to the importance weighting + clipping is interesting. This claim is supported by theory and empirical evidence in the paper.\n\n4. The paper includes extensive experimental evaluation across a wide range of representative detectors and reasoning benchmarks.", "weaknesses": "1. Table 1 suggests RL brings little or no gain on clean data. Is the RL correctly applied/tuned in the experiments? Why does RL not further help with reasoning even when there is no contamination?\n\n2. Some prior works have shown that contamination detection can be evaded in LLMs. A clearer positioning relative to these closely related papers could be added to the main text (currently largely in the appendix/references).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)