Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
Authors: Yongding Tao
Tian Wang
Yihong Dong
Huanyu Liu
Kechi Zhang
Hu XiaoLong
Ge Li
Venue: ICLR 2026 Poster (2026)
Keywords: Data Contamination Detection; LLMs; Reinforcement Learning; Entropy
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We present the first study of data contamination detection in RL post-training, introducing the RL-MIA benchmark and an entropy-based method Self-Critique, which achieving an AUC improvement of up to 30% over prior baselines.

Abstract: Data contamination poses a significant threat to the reliable evaluation of Large Language Models (LLMs). This issue arises when benchmark samples may inadvertently appear in training sets, compromising the validity of reported performance. While detection methods have been developed for the pre-training and Supervised Fine-Tuning stages, a critical research gap exists for the increasingly significant phase of Reinforcement Learning (RL) post-training. 
As RL post-training becomes pivotal for advancing LLM reasoning, the absence of specialized contamination detection methods in this paradigm presents a critical vulnerability. 
To address this, we conduct the first systematic study of data detection within  RL post-training scenario and propose Self-Critique. Our method is motivated by a key observation: after RL phase, the output entropy distribution of LLMs tends to collapse into highly specific and sparse modes. Self-Critique probes for the underlying policy collapse, i.e., the model's convergence to a narrow reasoning path, which causes this entropy reduction.
To facilitate this research, we also introduce RL-MIA, a benchmark constructed to simulate this specific contamination scenario. 
Extensive experiments show that Self-Critique significantly outperforms baseline methods across multiple models and contamination tasks, achieving an AUC improvement of up to 30%. Whereas existing methods are close to a random guess for RL-phase contamination, our method makes detection possible.

Published as a conference paper at ICLR 2026
DETECTING DATA CONTAMINATION FROM REINFORCEMENT LEARNING POST-TRAINING FOR LARGE LANGUAGE MODELS
Tian Wang* Xiaolong Hu Ge Li!
Yongding Tao
Yihong Dong'
Huanyu Liu-
Kechi Zhang'
1 School of Computer Science, Peking University ydtao22estuu pku. edu.cn ligeêpku... wdw.eeduuc...
2 New H3C Technologies Co., Ltd
ABSTRACT
Data contamination poses significant threat to the reliable evaluation of Large Language Models (LLMs). This issue arises when benchmark samples may in- advertently appear in training sets, compromising the validity of reported perfor- mance. While detection methods have been developed for the pre-training and Supervised Fine-Tuning stages, critical research gap exists for the increasingly significant phase of Reinforcement Learning (RL) post-training. As RL post- training becomes pivotal for advancing LLM reasoning, the absence of special- ized contamination detection methods in this paradigm presents a critical vul- nerability. To address this, conduct the first systematic study of data detec- tion within RL post-training scenario and propose Self-Critique. Our method is motivated by key observation: after RL phase, the output entropy distribu- tion of LLMs tends to collapse into highly specific and sparse modes. Self- Critique probes for the underlying policy collapse, i.c., the model's convergence to narrow reasoning path, which causes this entropy reduction. To facilitate this research, we also introduce RL-MIA, benehmark constructed to simu- late this specific contamination scenario. Extensive experiments show that Self- Critique significantly outperforms baseline methods across multiple models and contamination tasks, achieving AUC improvement of up to 30%. Whereas existing methods are close random guess for RL--phase contamination, our method makes detection possible. Our benchmark and code are available https:: http:/ggtttas http://coooogoggnneneeeeaoae
Dat Cont aminat inn
INTRODUCTION
The reliability of Large Language Model (LLM) evaluations seriously threatened by data contam- ination. This happens when benchmark test samples accidentally get included in the training data, which can invalidate the model's reported performance. To solve this problem, many researchers have developed detection methods, but they have almost exclusively focused on the pre-training and Supervised Fine-Tuning
(SFT) (Dong et al. 20244 Fu et al.) 2024 Shi et al.) 2024 Zhang et al.) 20246 Xie al. 2024b Zhang et al. 2025a stages. However, these efforts have left major gap: the increasingly important phase of Reinforcement Learning (RL) post-training. We believe this is critical oversight, because powerful techniques like Reinforcement Learning with Verifiable Re- wards (RLVR) (Shao et al. [024aa] Guo et al., 2025 Yu et al. (2255) are now essential for improving L.M reasoning. This makes the RI. stage major potential source of contamination that has been largely overlooked.
The challenge of detecting RL-phase contamination stems from a fundamental shift in the training objective, rendering existing methods ineffective. Both pre-training and SFT are likelihood- paradigms; they train models to maximize the probability observed data. This process naturally creates strong, likelihood-basel signals, such unusually low perplexity, which most current de- tectors are built to identify. By contrast, RL, especially RLVR, operates on a moward-maximizatinn principle. The policy is not trained mimic ground-truth distribution but is instead guided by

---

Published as a conference paper at ICLR 2026
14
019 hal Ontique
100
onggel Chiquu
0.8
0.6
0
Rador CA)
0.4
Mires FARCA B.n
0.
0.0
400
0.0
Postion
Postion
Take Foaltive Rate
(a) Contaminated sample
(b) Clean sample
(c) AUC comparison
Motivation behind Self-Critique. After RL post-training, entropy distributions become sparse. (a) For contaminated samples, the critique reasoning path remains highly similar the orig- inal one, indicating policy collapse and memorization. (b) Clean samples exhibit greater divergence between the original and critique reasoning paths. (c) Our method achieves significantly higher AUC while existing baselines perform close to random guess.
Figure 1:
sparse reward signals to find successful reasoning path. This approach often enables stronger gen- eralization than SFT (Kirk et al., [2024 Chu et al. 2025). but by decoupling from likelihood-based objectives, it also erases the very signals that traditional detectors rely Consequently, RL-phase contamination becomes a uniquely challenging problem, creating an urgent need for a new class of detection methods specifically designed for this reward-driven setting.
Given that likelihood-based signals are ineffective, our search for a new detection method begins with identifying signal inherent to the reward-driven training process. Recent studies on RL's training dynamics point to a promising candidate: the phenomenon of policy collapse. Specifi- cally, RL narrows the search space improve pass accuracy, often the cost of lower pass@k performance (Havrilla et al.] 2024 Shao et al. 2024b Yue et al. 2025 Dong et al. 2025), and produces distinctive entropy patterns, such as high-entropy concentration on certain tokens (Wang et al. 2025a Cheng et al. 2025 Song et al.) 2025). These findings suggest that entropy could serve as a powerful indicator of this collapse and its associated path dependency. However, our initial in- vestigations revealed that using entropy directly
contamination signal is unreliable. The reason is that policy collapse is a general behavior of RL and can occur even on clean samples not seen during training. As shown in Figures la and 1b both contaminated and clean samples can exhibit sparse token-level entropy. This implies that a simple passive check is insufficient. Therefore, we introduce an active probing mechanism to expose the underlying differences. We find that when the model is asked to generate an alternative reasoning path given its initial response (self-criliquee) contaminated samples struggle to deviate, resulting highly similar entropy curves (Figure Ta). In contrast, the model shows greater flexibility on clean samples, leading to more distinct entropy patterns (Figure lb
Building on these observations, we introduce Sell-Critique, an entropy-based detection method that applies our self-critique probing strategy. The core idea instruct the model to generate two distinct responses for the same problem; samples where the two responses exhibit high similarity their entropy space are flagged as contaminated. A detailed workflow shown in Figure 2 However, rigorously evaluating this method is challenging, as no existing benchmark can isolate and simulate contamination purely within the RL phase. To overcome this hurdle, we also developed benchmark constructed
RL-MIA (Reinforcement Learning Membership Inference Attack), for this specific purpose. Using RI-MIA across challenging math and logic datasets, we show that Self-Critique is highly effective. As previewed in Figure Ic our method significantly outperforms existing detectors, which operate near the level of random guess.
Our main contributions are summarized as follows:
To the best of our knowledge, we present the first systematic study of data contamination detection in the RL post-training phase of LLMs, highlighting a critical yet overlooked problem.
We propose Self-Critique, an entropy-based detector that measures RL-induced policy collapse via self-critique probing. Across four tasks and multiple models, Self-Critique
We also provide visualizations of the contamination score distribution in Appendix
2

---

Published as a conference paper at ICLR 2026
consistently outperforms baselines, which perform near random guess, achieving AUC improvement of up to 30%.
We introduce RL-MIA, a new benchmark that simulates RL-specific contamination sce- narios across math and logic tasks, enabling the rigorous evaluation of detection methods.
RELATED WORKS
In this section, we outline the most relevant directions and associated papers of this work.
Data Contamination Detection Data contamination detection can be regarded as a specific in- stance of membership inference attacks (MIA), which were initially introduced to measure mem- orization and privacy risks (Shokri et al., 2017; Carlini et al., 2019; Mireshghallah al. 2022a)
Recently, the issue of data contamination LLMs has drawn increasing attention, as directly
undermines the validity of benchmark evaluations (Sainz et al.) 2023 Xu et al., 2024a b Wu al.
2025). Prior work on data contamination detection in LLMs has mainly focused on the pre- pre-training and Supervised Fine- Fine-Tuning stages (Mireshghallah et al. 2022b
Fu et al. 2023 Mattern et al. 2023
Shi et al. 2024 Xie et al. 2024b Gonen et al. 2023) Dong et al.
2024|]/hang et al. 2025b). In these
stages, models largely rely memorizing training data for learning (Zeng et al. 2024a Chu al.
2025| Wang et al. 20255), a process that naturally creates strong, likelihood-based signals-suce as unusually low perplexity- that most current detectors are built to identify. In contrast, during the RL post-training phase, LLMs are optimized to autonomously explore reasoning trajectories. This reward-driven objective decouples the model's behavior from simple likelihood metrics, posing a unique challenge for conventional detection methods.
Entropy in Reinforcement Learning Post-training Reinforcement learning has become a crucial paradigm for post-training large language models. Leveraging reinforcement learning with verifi- able rewards substantially enhances LLM's reasoning capabilities (Jaech et al. 2024, Guo et al, 2023). A key factor in RL post-raiiinng is entropy: high entropy promotes exploration via stochastic policies, while low entropy favors exploitation through deterministic behavior. A common challenge in RL post-training entropy collapse (Cui
2025 Liang al. 2025), where policy entropy de- creases dramatically in the early stages of training leading to premature convergence and restricted exploration. To address this, entropy management strategies regularize entropy to prevent rapid col- lapse (O (O'Donghuue et al. 2016 He et al. 2025 Wang et al.] (025cc or use high-entrop signals to encourage inherently exploratory reasoning behaviors (Cheng al. [025 Vanlioglu [025] (Tan &
Pan 2025), thus maintaining a balance between exploration and exploitation. In reasoning tasks, high-entropy tokens indicate uncertain decision points and are assigned stronger RL updates, while low-entropy tokens, which correspond to more deterministic outputs, receive smaller updates (Li ct al. 2025 Wang et al.) 2025a Tang et al.) 2025). As a result, the trained model develops distinct entropy patterns across tokens. In this work, analyze these entropy patterns before and after self-critique to detect potential data contamination.
3 THE CHALLENGE OF CONTAMINATION DETECTION IN RL
In this section, we formalize the problem of detecting data contamination the RL, post-training phase of LLMs, and then highlight why detection methods based on likelihood, which effective in pre-training and SFT, become unreliable RL. Finally, we introduce token-level entropy as a lens analyze RL-induced policy collapse, which lays the foundation for our proposed method.
3.1 PROBLEM DEFINITION
We consider the task of detecting data contamination in the RL post-training phase of Large Lan- guage Models. Formally, this can be framed as a Membership Inference Attack(MIA) problem: given model M that has undergone RL post-training and sample x. the goal is to determine whether & was included in the RL training dataset DRL- A detector is a function f(M, + {0. 1). where 1 indicates membership (contamination) and 0 indicates non non-membership. We focus the black-bor box setting (Shi et al. [2024). where the detector can only query M for outputs, without access to internal states, gradients, or training data.
3

---

Published as a conference paper at ICLR 2026
3.2 WHY RL POST-TRAINING UNIQUE CASE
Existing data detection methods are largely designed for training paradigms whose objectives are rooted in Maximum Likelihood Estimation (MLE). However, the objective of RL post-training is fundamentally different, which introduces unique and significant challenges for detection.
Both pre-training and Supervised Fine-Tuning are governed by an MLE-based objective. Their goal is to train a model M with parameters 0 to maximize the likelihood of the observed data by minimizing the negative log-likelihood loss.
For pre-training, the model learns from a vast corpus of unlabeled text Dreetrin- The objective is next-token prediction, aiming to learn a general distribution of the language. For a text sequence (I1, T2 ................................ IT), the loss is:
Pretrain (0) =-
Σ
log (arelaee)
premann
For SFT, the model learns from a dataset of prompt-response pairs DSFT r(g.r)} The objective is to learn to follow instructions and generate helpful responses. The model is trained to maximize the likelihood the target response r = (fI,...,Tk) given the prompt q:
(SF
Σ
log (rgpo(r)
(SPT 1=1
share the same underlying princi- ple: they directly train the model to assign high probabilities to sequences seen in the training data. This provides clear signal for detection methods like Perplexity (Gonen et al. (023) and Min-K% Prob (Shi et al. 2024), which are built upon this likelihood principle.
Despite their different data sources, both paradigms (Eq. [and] 2
In stark contrast, RL post-training (and specifically RLVR) does not directly optimize for likeli- hood. Its objective is to update the policy He to maximize the expected reward R. from a set of generated outputs {o:} given a prompt q. The objective for method like GRPO (Shao et al. 2024a) can be abstracted as:
(+) Bng [f(R(o.), mo)],
where f(-) is a function of the reward, the current policy, and a reference policy. The key distinction is that the optimization is driven by an external, often sparse, reward signal R(o;)) (e.g., 1 for a cor- rect final answer, 0 otherwise), not by the token-level log- pooo-aabbilliies of the ground- truth response. This decouples the model's 's final behavior from simple likelihood metrics, rendering many existing detection approaches that rely on this signal ineffective.
3.3 ENTROPY AS A NEW SIGNAL FOR RL DETECTION
Recent studies Yue et al., 2025 Wang et al. 2025a Cui et al. 2025) have shown that RL post- training frequently leads to policy collapse: for samples that receive consistent reward, the model converges to narrow reasoning path, producing overly stable outputs This phenomenon is re- flected in the token-level entropy. For each decoding step t, the token-level entropy is
-+=-D
- (o x<x) log (v (ee)) VEV
(4)
and the entropy sequence [HH]T= measures uncertainty along the generated trajectory. Em- pirical observations show that RI tends to push entropy sequences into sparse patterns, where many tokens are nearly deterministic. Crucially, this collapse stronger for contaminated samples that were explicitly rewarded during RL training, whereas clean samples retain more variability when
probed.
These insights suggest that contamination detection in RL requires moving beyond likelihood and in- stead measuring the policy's 's dependence on specific reasoning paths. Token-level entropy provides natural signal for this purpose, which directly motivates our Self-Critique method: by asking the model to regenerate an alternative reasoning path conditioned on its initial response and comparing the entropy sequences,
reveal whether sample was memorized during RL training.
4

---

Published as a conference paper at ICLR 2026
4 DETECTION VIA SELF-CRITIQUE
Self-Critiuu
Initial Thinking
Critique Thinking
Probing
Finitil
Eqritqqu
Initial
Critique
Problem
Thinking
Thinking
Scoreeair.Caaigeeee (t)
C0tyenadized Eiina Pectique) >
Membership
Figure 2:
Overview of the Selt-Critique detection workflow. The method compares token-level entropy sequences between the initial response and the self self-critique response. High similarity in en- tropy space indicates contamination (policy collapse), while low similarity indicates clean samples.
Our method is motivated by the hypothesis that RL post-training induces high-reward path depen- dence for contaminated samples. Concretely, for problem q seen during RL training, the policy ma tends to converge to a highly rewarded and thus similar response trajectory. In contrast, for problems seen during RL training, the model is more likely produce an alternative reasoning path when prompted.
The Self-Critique method quantifies this dependency. We first elicit the model's most confident (deterministic) response, and then ask the model to produce different solution conditioned on the initial response. We compare the token-level entropy sequences of the two generations measure the degree of path dependence. An overview is shown in Figure
4.1 THE SELF-CRITIQUE DETECTION PROCESS
Let M be a large language model with parameters 0, and let be the problem under test. We use deterministic decoding strategy (e.g., greedy decoding) to obtain the model's most confident response as the reference.
Step I: Initial response. We construct the initial prompt P1 by embedding q into chat template T, and obtain the model's response:
Ti = M
(5)
We then compute the token-level entropy sequence for this response, E, (H, (r)))) which serves the baseline reasoning trajectory.
Step 2: Self-critique response, form self-critique prompt Pz by augmenting g with an in-
structional meta-prompt /eritiqu and the text of r1:
o' i Teritique (r1),
where denotes appending to the user content within the prompt structure. We then generate the second response and its entropy sequence:
12 M(T(g'), = (H()))l)))
Step 3: Similarity The contamination score is the similarity between the entropy quences. A higher similarity indicates that the model remains on the same reasoning path despite being instructed change it, suggesting memorization. We use length-aware (penalized) cosine similarity:
Scoreseff-Cttiguee (g) COSpenalized E1, E2),
where
COSpenalized ( A, B) = cos (pad((A) (B)( min (inn [B)
(9)
max([A]) [B)
"hhe exact Tentigue shown in Appendix
5

---

Published as a conference paper at ICLR 2026
Here cos(:,-) is the standard cosine similarity between vectors (dot product over the product of L2 norms), and pad(-) zero-pads the shorter sequence to the maximum length so that non-overlapping positions contribute zero. The multiplicative length ratio penalizes cases where one response is much shorter/longer than the other, since response length itself reflects a facet of the reasoning mode. Overall, higher score indicates a higher likelihood of contamination. A formal description of the procedure is provided in Appendix A
EXPERIMENTS
In this section, we present a comprehensive empirical evaluation of our proposed Self-Critique. We first introduce the RL-MIA benchmark, which we constructed specifically for RI. data contamination detection. We then describe the baseline methods we compare against and detail our experimental setup. Finally, we present the main results, followed by analysis and ablation studies. Additional experimental results are in Appendix and
5.1 RL-MIA: A BENCHMARK FOR RL MEMBERSHIP INFERENCE ATTACK
To the best of our knowledge, no benchmark currently exists for systematically detecting data con- tamination during the RL post-training stage. To address this gap, we introduce the RL-MIA (Re- inforcement Learning Membership Inference Attack). The key idea behind RL-MIA is to simulate controllable data contamination by selectively including subset of data in the RL post-training pro- cess, while the objective of the contamination detection task is to identify which samples have been used.
The problems used for this simulation are drawn from four benchmarks selected to cover diverse styles and potential pre-training exposure. We include the widely used mathematical reasoning benchmarks, i.e., AIME 2024 and AIME 2025. AIME 2024 may have appeared in some models' pre-training corpora, allowing us to test robustness to prior exposure, whereas AIME 2025 is post- cutoff and thus unlikely to be present in pre-training data. To obtain a controlled setting free from prior exposure, we also include two synthetically generated logical reasoning datasets: Knights & Knaves (K&K) (Xie et al., (024a) and SAT (Liu et al.) (025). The synthetic nature of these datasets ensures that any detected training signal can be attributed to the RL post-training phase.
To approximate a realistic setting, we embed the selected benchmarks into larger RL post-training corpus. For AIME24 and AIME25, we use the widely adopted OpenRI-Math-461 (Guha et al., 2025) corpus as the base and inject 50% of each benchmark's 's items into the RL training data. For K&K and SAT, following their original papers, , we use the provided training portions to form the contaminated split and synthesize additional items as held-out clean samples. We primarily use Qwen2.5-7B-Inntrrue (Qwen et al., 2025) and DeepSeek-Math- -atte---- (Shao et al.) 2024b) as the model for simulating RL-stage contamination. We also experiments Qwen2.5-0.5B.. Instruct, Qwen2.5-3B -renn.5-3B-Inttrucc (Qwen et al.) 2025) and Qwen2.5-7B-Maud (Yang et al. 2024). All
RL runs are implemented with VeRL (Sheng et al. 2025) framework on 8 NVIDIA A 000 (40 GB). Detailed dataset splits and training settings are provided in Appendix F
5.2 EXPERIMENTAL SETUP
Baseline Methods We compare Self-Critique against a set of representative baselines. A key con- sideration the RL post-training setting that training changes the response distribution rather than the likelihood of the prompt. Therefore, for baselines originally designed for pre-training data detection (which operate on input text), we adapt them to operate on the model's 's responses to ensure a fair comparison. The baselines include: ! Perplexity (PPL) (Gonen et al.) 2023), which assumes memorized text has lower perplexity; Min-K Prob (Shi et al. 2024). which posits that memo- rized text is less likely to contain low-probability outlier tokens; and 3 Min-K % (Zhang et al.) 2025b), which normalizes token probabilities for a more robust score. We also include Recall (Xie) et al. 2024b), which prefixes the text with member content and measures the relative change in log- lieeiioood, and 0 CDD (Dong et al. 2024), which measures output consistency under stochastic sampling via the average leken-level edit distance across multiple generations.

---

Published as a conference paper at ICLR 2026
We summarize these baselines in Table I PPL. Min-K% and Min-K%++ directly log- probability properties of the text; Recall and CDD expose differences by injecting a non-member prefix or by randomly sampling multiple outputs, respectively. In contrast, propose new probing mechanism, i.e., self-critique probing, and use entropy as the core signal for RI-stage contami- nation detection. Following the probing ideas in CDD and Recall, also introduce two additional entropy-based baselines, @ Entropy-Temp and @ Entropy-Noise, which keep the probing mecha- nisms but replace the consistencylikeelhod metric with entropy.
Table I:
A taxonomy of data contamination detection methods. Our work is the first specifically address the challenges in the RL Post-training phase.
Method
Probing Mechanism
Core Metric
Designed for
Existing Methods for Pre-training SFT
PPI. (PMNLP22)))
Intrinsic Property
Log Probability
Pre-training SFT
Min-K's 5%% (CLR200)
Intrinsic Property
Log Probability
Pre- Pre-training / SFT
Min-K K% aCLR225)
Intrinsic Property
Log Probability
Pre-training / SFT
Recall (BMNLP224)
Non-member prefix
Log Probability
Pre- Pre-training SFT
CDD (AC224)
Stochastic Sampling
Edit Distance
Pre- te-tiraiingg SFT
Our Proposed Methods for RL Post-training
Entropy- Temp
Stochastic Sampling
Entropy
RL
Entropy-Noiver Self- Critique(QQuus)
Non-member prefix
Entropy
RL
Self-Critique Probing
Entropy
RL
We primarily report the Area Under the ROC Curve (AUC), standard metric for detection problems such as data contamination and membership inference (Shi et al. [2024] Duan al. 2024 Zhang al. 2025b). AUC is thresholddinddpendeenl and reflects the probability that the detector ranks randomly chosen contaminated sample higher than a randomly chosen clean one; higher AUC indicates stronger detection performance (50% corresponds random guess). We also report the Fl score at the Youden threshold (Fluss et al. 2005) as a threshold-specife reference.
Evaluation Metrics
5.3 MAIN RESULTS
Table 2: Performance of different detection methods the RL-MIA benchmark across models. The AVG column is the average AUC across all benchmarks. Best AUC is in bold: the second best underlined.
AIME24
AIME25
K&K
SAT
Method
AVG
Fl score
AUC
Fl score
AUC
Fl score
AUC
Fl score
AUC
Qween255-7B-oe B-r.n.2-BB-nsttruc
PPL
0.33
0.51
0.42
0.56
0.67
0.47
0.54
0.50
0.51
Min-K% 5% Bl
2004
0.59
0.49
0.52
0.44
0.70
0.54
0.32
0.50
0.49
Min-K K% Dong
0.73
0.58
0.46
0.45
0.67
0.40
0.00
0.31
0.44
Recall
Na
mmn
0.62
0.61
0.55
0.65
0.67
0.47
0.62
0.62
0.59
CDD
Dong mal
3134
0.50
0.57
0.67
0.52
0.67
0.47
0.57
0.47
0.51
Entropy-Tent Temp
0.73
0.64
0.12
0.42
0.59
0.49
0.66
0.69
0.56
Entropy-Noise
0.70
0.57
0.68
0.52
0.79
0.77
0.62
Self- (Our))))iquue(ur))
0.69
0.72
0.76
0.72
0.69
0.66
0.69
0.67
0.70
19%)
DeepSeek Math-7B Instructt
PPL Exnon et
0.42
0.53
0.62
0.34
0.54
0.68
0.64
0.53
Min-K% Sic a
5004
0.55
0.47
0.12
0.40
0.67
0.46
0.69
0.35
0.42
Min-K 1on-K%++ Wane
0.67
0.53
0.67
0.56
0.15
0.47
0.62
0.49
0.51
Recall
0.54
0.46
0.52
0.56
0.39
0,54
0.69
0.62
0.54
CDD
bouad
0.30
0.49
0.65
0.51
0.08
0.48
0.66
0.50
0.50
Entropy-Tema
0.60
0.48
0.70
0.54
0.23
0.43
0.64
0.61
0.52
Entropy-Noise Noise
0.70
0.56
0.72
0.69
0.48
0.52
0.67
0.45
0.55
Self- (ell-Critique(Ours))
0.76
0.67
0.71
0.61
0.60
0.63
0.66
0.67
0.64 (1 19%)

---

Published as a conference paper at ICLR 2026
The main results of different data contamination methods based 1rann2.5-7B-Instrrre and Bareek--Matt77B--Inee
are shown Table Across both models, Self-Critique the most reliable detector: it attains the best average AUC Qwen2.5-7B-Inste best non-ours baseline) and on DeepSeek-Math-7B-nnsse it leads most per-dataset AUCs. In contrast, likelihood-basee baselines (PPL., Min-K%, Min- K%++) often perform near random guesses and can be unstable. Among existing methods, Recall achieves relatively strong performance (it is also state-of-the-arr approach for pre-training contam- ination (Xie et al., 2024b)), suggesting that probing to expose membership signals is more effective than relying solely on intrinsic text properties. Moreover, when using the same probing mechanisms, the entropy- -based variants perform better (Entropy-Temp vs. CDD; Entropy-Noiss vs. Recall), indi- cating that entropy is sensitive indicator RL-induced changes and thus better suited for RL-stage detection.
(0.70, +19% the (0.64, also +19% % improvement), and
Finally, compared random sampling prefix injection, our self-critique probing aligns more closely with the RL property of dependence high-reward paths. Within our entropy-based meth- ods, this leads to notable gains, i.e., +13% on Soven2.5-7BIInsteee and +16 % on DeepSeck-Math 7B-Instruct.
5.4 DUAL-STAGE CONTAMINATION IN PRE-TRAINING & RI.
To isolate the effects of RI.-phase contamination, our previous experiments were primarily con- ducted on synthetic data or datasets with low levels of contamination. However, for public beneh- marks released before LLM's training cutoff, contamination from both pretraining and the RL phase can co-occur. Therefore, we design a study to distinguish between these two sources of contamination. Concretely, choose GSM8K (widely acknow to suffer from substantial pretraining leakage (Zhang et al. 2024a Dekoninck et al. 2024, Mirzadeh et al., 2025)) and train Qwen2.5-0.5B I-Instruct with the PPO algorithm We then simulate RL-phase contamination by injecting half of the test set into the RL training data. As discussed in Section 3.2 pretraining (which optimizes via Maximum Likelihood Estimation) and RL (which reward-driven policy optimization) pursue different objectives, so their respective forms of contamination are likely to produce distinct effects.
We first assign each test item a pretraining- contamination proxy score using fikelihood-basee detector (e.g., PPL) to separate these effects. We then evaluate RL-stage contamination detection under two conditions: a lower-pretraining contamination subset, created by select- ing the bottom-q quantile of items by PPL score (e.g.. the lowest 50%); and @ a random-control subset. To control for any confounding effects from smaller sample size on AUC, the random-control subset is formed by uniformly sampling the same number of items, thereby match- ing the subset size while preserving the original data distribution. Our hypothesis predicts that Self- trit-Crissss's performance will significantly improve on the former subset, but not on the latter.
0.9
Self-Critque (lower pretraining conlamination)
Self-Critique (random subset]
0.8
Random guess (AUC 0.50)
0.7
0.6
0.5
1.0
0.9
0.8
0.6
0.3
0.1
Pretraining contamination rate (higher lower)
Figure 3: Dual-stage contamination analysis. Self- Critique the wwwerrpperriiiingccmmmanaaaaii sub- set (green) improves sharply as the rate decreases.
The results are shown in Figure and we also provide the numerical results in Table 7 of Appendix B. As the pretraining contamination level decreases, the performance Self-Critique on the lower- pretrinning.ootaminatti contamination subset improves significantly. In contrast, its performance the random- control subset shows a slight decrease, while the PPL-based detector's performance approaches that of random guess. This outcome rules a pure sample-size effect (as performance the random-
WWe also provide additional results on Qwen2.5-7B B-ven.557B-Math and Llama in Appendix B "Actually, the setting the the quick start in verl, which makes it easy reproduce the results.
8

---

Published as a conference paper at ICLR 2026
control subset does not improve) and supports our hypothesis: conditioning the RL detector items with a weaker pretraining signal allows it to identify RL-phase memorization or path dependence far more clearly. Meanwhile, the fact that likelihood- basedd cues (PPL) remain unstable and close to 0.5 further demonstrates the effectiveness of our method specifically targeting RI.-phase data contamination.
5.5 ABLATION STUDIES
Table 3: Detection performance on K&K with Qwen2.5-3B-Insse trained by different RL algo- rithms. AVG column reports the mean AUC across three RL algorithms.
PPO
GRPO
DAPO
Method
AVG
Fl score
AUC
Fl score
AUC
FI score
AUC
Genei al 2023
0,65
0.41
0.63
0.46
0.61
0.51
0,46
Min-K% Sis a
2014
0.43
0.58
0.54
0.64
0.46
0.50
Min-K
+ Zheny ot
30256
0.39
0,50
0.61
0.52
0.26
0.49
0.51
Recall
Xir
212b
0.61
0.46
0.35
0.49
0.55
0.50
0.48
CDD
Dong etdl
0034
0.66
0.53
0.65
0,51
0.60
0.49
0,51
ContropyTTemy
0.41
0.55
0.53
0.60
0.57
0.59
0.58
Entropy-Noise Self-Crilique(Ours) 0.67
0.64
0.59
0.43
0.52
0.60
0.49
0.53
0.61
0.67
0.61
0.64
0.60
0.60 (1 18%
Contaminate with Different RL Algorithms Table reports results the K&K task using Qwen2.5-3B-Instrenn trained with three RI algorithms. including PPO (Schulman et al.) 2017).
GRPO (Shao et al. 2024a), and DAPO (Yu et al.) 2025 Across all algorithms, Self-Critique is the most reliable detector: it attains the best AUC for each algorithm and the highest average AUC (0.60). Entropy-based probes are consistently stronger than likelihood-based methods: Entropy- Temp is the next best on average (0.58), while likelihood baselines (PPL, Min-K%, Min-K%++) are around 1.460.55. These trends suggest that our probe, measuring path dependence via entropy sim- ilarity, captures an RL-induced signal that is algorithmm-gnossi F1 scores follow the same pattern, further indicating that Self-Critique yields both better ranking (AUC) and better thresholded deci- sions. We also provide a discussion different alignment algorithms under the RLHF paradigm in Appendix B.3
Ablation on Top-K Entropy Approximation As LLM vocabularies are large, it is often imprac- tical to obtain the full next-token distribution at every decoding step to compute exact entropy. In many APIs, only Top-K token probabilities are available, Top-K masses. We conduct an ablation on the choice of [Table Reducing does not harm performance;
approximate entropy using those
in the extreme case K = 3, the AUC drops only slightly. We attribute this to the long-tailed nature of next-token distributions: most probability mass concentrates on a small set of tokens, and the tail contributes little to entropy. Hence, Top-K entropy is both efficient and accurate enough for our detector. Considering practical applications, we also include a discussion on inference cost Appendix G.2
Table 4: Ablation on Top-K entropy approximation (Qwen2.5-7B-Inntrrul We report AUC for different K and the row-wise variance across K E (3, 5, 10, 20, 50).
Dataset K=3
K=5
K= 10
K K
K=50
Variance
AIME25
0.7022
0.7111
0.7111
0.7156
0.7156
2.39 x 10-5
K&K
0.6460
0.6572
0.6636
0.6608
0.6584
3.62 x 10
We also provide additional ablation studies about why self-critique probing better, the sampling strategy and sensitivity to meta-insttustions in Appendix C
6 CONCLUSION
In this paper, we presented the first systematic study of data contamination the overlooked RL post-training stage, demonstrating that existing data contamination detection methods are ill-suited

---

Published as a conference paper at ICLR 2026
for this reward-drivee paradigm. To address this gap, we proposed Self- Critique, novel method that identifies RL-induced policy collapse by actively probing the model's 's reasoning path dependen- cies using token-level entropy. To validate RL-MIA, a new benchmark for RI.-phase contamination. Experiments show that Self-Critique con- sistently outperforms baselines, improving the average AUC by up 30%, and its ability to isolate RL-specific signals is further highlighted in dual-contamination scenarios, where performance im- proves by up to 55%. As the community's 's understanding of RL post-training grows, we expect that more detectors tailored to this unique setting will emerge.
method in controlled setting, we also developed
ACKNOWLEDGEMMEN
This research is supported by the National Key R&D Program under Grant N0.20233YBB4008801, the National Natural Science Foundation of China under Grant No.62192733, 62192730, 62192731, the Major Program(JD) of Hubei Province (No. (0o.2023BAA024) the Beijing Major Science and Tech- nology Project under Contract no. Z251 100008425005.
ETHICS STATEMENT
All authors have read and adhered to the ICLR Code of Ethics. The primary objective our work to enhance the integrity and reliability of Large Language Model evaluations. We address this by developing Self-Critique, a method to detect data contamination in the Reinforcement Learning (RL) post-training phase, aiming to prevent misleading performance claims and promote transparent research practices. Our method a specific application of Membership Inference Attack (MIA) techniques, and acknowledge the potential dual-use concerns regarding data privacy. However, our study is carefully scoped to mitigate these risks. The goal of our work is defensive- providing validation tool for researchers-ase and it is applied exclusively to public, non-sensitive benchmarks (AIME, K&K, SAT, GSM8K) that contain no personal data. We believe the societal benefit of ensuring robust and honest model evaluation significantly outweighs the minimal risk of misuse in this context.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the full reproducibility of our research. To facilitate this, we have made our code and the newly constructed RL-MIA benchmark available at This repository includes the implemen- tation of our proposed Self-Critique method, baseline methods, and scripts to run the experiments. A formal, step-by-step description of the Self-Critique algorithm is provided in Appendix A (AI- gorithm (). The specific prompts used for the self-critique probing mechanism are detailed in Ap- pendix E Details regarding the construction of the RL-MIA benchmark, including dataset sources, injection methods, and data splits, are described in Section [. and further elaborated in Appendix (Table 6). The key training hyperparameters for all RL models and algorithms are provided in Ap- pendix F] (Table 7), ensuring that our training process can be accurately replicated. Our evaluation metrics are standard the field and are defined Section
RL-Data-Conn Cataaa-Cotmaminnta..
REFERENCES
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Eval- uating and testing unintended memorization in neural networks. In USENIX Security Symposium, pp. 267-284, 2019.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758, 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arxiv preprint arXiv:2501. 17161, 2025.
10

---

Published as a conference paper at ICLR 2026
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeddbck:: Boosting language models with high-quality feedback. 2023.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models, arXiv preprini arXiv:2505.226177 2025.
Jasper Dekoninck, Mark Müller, and Martin Vechev. Constat: Performance-based contamination detection in large language models. Advances in Neural Information Processing Systems, 37: 92420,92464, 2024.
Thomas J DiCiecio and Bradley Efron. Bootstrap confidence intervals. Statistical science, 11(3):
189-228, 1996.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang. and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Compu- tational Linguistics: ACI. 2024, pp. 12039-12050, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Ying- wei Ma. Jue Chen, Binhua Li, al. RI-plus: Countering capability boundary collapse of Ilms in reinforcement learning with hybrid-policy optimization. arXiv preprint arXiv:2508 200202,,00000222,, 2025.
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841, 2024.
Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics; Methodology and distribution, pp. 569-593. Springer, 1992.
Ronen Fluss, David Faraggi, and Benjamin Reiser. Estimation of the youden index and its associated cutoff point. Biometrical Journal: Journal of Mathematical Methods in Biosciences, 47(4))458- 472, 2005.
Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Practical member- ship inference attacks against fine-tuned large language models via self-prompt calibration. arXiv preprint arXiv:2311.00 1.06022, 2023.
Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Membership inference attacks against fine-tuned large language models via self-prompt calibration. Advances in Neural Information Processing Systems, 37:134981 1134491--3355 1350000, 2024.
Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10136- 10148, Singapore, December 2023. Association for Computational Linguistics. doi: 18.18653/ v1/2023.findingg findings-emmll.6799
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Etash Guha. Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smymis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https::p::/ / org abs /200 000076666778888 04178

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv: 2025.
Alexander Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, et al. Teaching large language models reason with reinforcement learning. arXiv preprint arXiv:2403.04642, 2024.
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, et al. Skywork open reasoner technical report. arXiv preprint arXiv:2505.22312, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720 2024.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rihf on Ilm generalisation and diversity. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), Interna- tional Conference on Representation Learning. volume 2024, pp. 10620-20653, 2024.
Qingbin Li, Rongkun Xue, Jie Wang, Ming Zhou, Zhi Li, Xiaofeng Ji, Yongqi Wang. Miao Liu, Zheming Yang, Minghui Qiu, et al. Cure: tokeeell-ooe-gguieeee re-concatenation for entropy- collapse prevention. arXiv preprint arXiv:2508.1 11016, 2025.
Xiao Liang, Zhongzhi Li, Yeyun Gong, Yelong Shen, Ying Nian Wu, Zhijiang Guo, and Weizhu Chen. Beyond pass @ 1: Self-play with variational problem synthesis sustains rlvr. arXiv preprint arXiv:2508. /4029, 2025.
Huanyu Liu, Jia Li, Hao Zhu, Kechi Zhang, Yihong Dong, and Ge Li. Saturn: Sat-based reinforce- ment learning to unleash language model reasoning. arXiv preprint arXiv:2505. 16366, 2025.
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Kerg-Kirkpatrick. Membership inference attacks against language models via neigh- bourhood comparison. arXiv preprint arXiv:2305. 18462, 2023.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Kerg-Kirkpatrickk and Reza Shokri. Quantifying privacy risks of masked language models using membership inference at- tacks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8332-8347, 2022a.
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang. David K Evans, and Taylor Berg- Kirkpatrick. An empirical analysis of memorization in fine-tuned autoregressive language mod- els. In 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 1816-1826, 2022b.
Iman Mirzadeh, Keivan Alizadeh- Vahid. Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. In Y. Yue, A. Garg, N. Peng. F. Sha, and R. Yu (eds.), International Conference on Representation Learning, volume 2025, pp. 94745599776, 2025.
Brendan O' D'Donoghue, Remi Munos, Koray Kavukcuoglu, and Volodymyr Mnih. Combining policy gradient and q-learning. arXiv preprint 16 2016.
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu. Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL htlpstt::: / x v. orgg abs/ 2412. 15115
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 11553728-537445 2023.
Oscar Sainz, Jon Campos, Iker Garcia- Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NIp evaluation in trouble: On the need to measure Ilm data contamination for each benchmark. In Findings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 10776-10787, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (XX7v00777.0647 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024a.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300,, 2024b.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Sun (eds.), International Conference on Representation Learning, volume 2024, pp. 31826-51844 2024.
Reza Shokri, Marco Stromati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In IEEE Symposium on Security and Privacy (SP), pp- 3-18, 2017.
Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for Ilm reasoning. arXiv preprint arXiv:2509.066: 06941, 2025.
Hongze Tan and Jianfei Pan. Gipo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint wwXiv:2508.043499 2025.
Xinyu Tang, Zhenduo Zhang, Yurou Liu, Wayne Xin Zhao, Zujie Wen, Zhiqiang Zhang, and Jun Zhou, Towards high data efficiency in reinforcement learning with verifiable reward. arXiv preprint (=XXi:250990132111 2025.
Abdullah Vanlioglu. Entropy-guided sequence weighting for efficient exploration in rl-based Ilm fine-tuning. arXiv preprint arXiv:2503.22455 2025.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for IIm reasoning. arXiv preprint trXiv:2506.01939 2025a.
Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization vs memorization: Tracing language models' capabilities back to pretraining data. In International Conference on Learning Representations (ICLR), 2025b.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint urXiv:2504.20571, 2025c.
Mingqi Wu. Zhihao Zhang, Qiaole Dong. Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang. et al. Reasoning or memorization? unreliable results reinforce- ment learning due to data contamination. arXiv preprint arXiv:2507. /0532, 2025.
13

---

Published as a conference paper at ICLR 2026
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint ar arXhv:2410.23122 2024a.
Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, and Bhuwan Dhingra. Recall: Membership inference via relative conditional log-likelihoods. arXiv preprint arXiv:2406. 010224406.9988, 15968, 2024b.
Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et Benchmark data contamination of large language models: A survey, arXiv preprini M.Xi::2400.04244,, 2024a.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404. /8824, 2024b.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12/12 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang. Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2501. 12948, 2025.
Yang Yuc, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint ar Xiv:22004. 13837, 2025.
Shenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu, Han Xu, Pengfei He, Yue Xing, Shuaiqiang Wang, Jiliang Tang, and Dawei Yin. Exploring memorization in fine-tuned language models. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 3917-3948, 2024a.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. International Conference on Machine Learning, 2024b.
Hengxiang Zhang. Songxin Zhang, Bingyi Jing, and Hongxin Wei. Fine-tuning can help detect pretraining data from large language models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, pp. 60902-60921, 2025a.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song. Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, et al. A careful examination of large language model performance on grade school arithmetic. Advances in Neural Information Processing Systems, 37: 46819-46836, 2024a.
Jingyang Zhang. Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Li. Min-k%++: Improved baseline for pre-traiming data detection from large language models. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Representation Learning, volume 2025, pp. 14845-64866 2025b.
Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. Pretraining data detection for large language models: A divergenee-basee calibration method. arXiv preprint arXiv:2409. /478/, 2024h.
Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. Dpo meets ppo: Reinforced token optimization for rlhf. International Conference on Machine Learning. 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Algorithm of Self-Critique
16
Additional Results
16
B. Additional Results on Other Models
16
B.2
Numerical Results for Dtal-Stage Contamination
16
B.3
Additional Results on RLHF Paradigm
17
Aditional Ablations
18
C.1
Why Self Celf-Critique Probing is a Better Detector
18
Ablation on Sampling Strategy
18
Ablation on Meta- Instrrutttut
18
Visualization of Contamination Score Distribution
19
Instructions
19
Benchmark Detail & Training Settings
21
Discussion
22
G.1 I
Discussion Result Variability and Error Bars
22
2 Discussion on Inference Cost
22
limitations and Future Work
23
LLM Usage Statement
23
15

---

Published as a conference paper at ICLR 2026
A ALGORITHM OF SELF-CRITIQUE
Algorithm Self-Critique Contamination Detection
Require: Model M (black-box with per-token log-prob access or top-k probs), meta-prompt
Laiiique problem
Ensure: Contamination score Score(g)
Deterministic decoding. Use greedy decoding (temperature 0) in all generations.
1: (Initial response) Construct P1 T()) and generate M(P)).
2: Compute token-level entropy sequence E +
using per-token probabilities. Triiique (ri): construct P2 T(q') and generate 72
3: (Self-critique response) Form q'
M(P2).
4: Compute entropy sequence Eg - (H.
5: (Similarity) Pad the shorter sequence with zeros: Ei t pad(F1), Eg + pad(E)) E2).
6: Compute cosine similarity +- cos( E1, E2).
min [ Ex() 7: Compute length penalty À <- max( Fa F2()
8: return (c) A.
B ADDITIONAL RESULTS
B.1 ADDITIONAL RESULTS ON OTHER MODELS
To better demonstrate the generalization of our method across different models, we provide addi- tional experiments Ow (Yang et al. (022)) and Llama-3. --8t-Istttutt (Grattafiori et al. 2024). As shown in Table 5 on Qwen2.5-7B -athhhhhh, Self-Critique achieves the best AUC on both AIME24 (0.76) and AIME25 (0.72), and the highest average AUC (0.74). The two entropy baselines are competitive, while likelihood- based methods (PPL, Min-K Kin Min-K%++) and Recal- ICDDD are clearly weaker here. We do not report K&K and SAT for this model because we could not get stable RL training on these synthetic datasets with non-instruct base; such setups typically require more careful RL hyperparameters and data curation. This engineering detail is outside the scope of our study. The experimental results Consistent with our previous findings, Self-Critique achieves the highest AUC on both K&K (0.61) and SAT (0.62) datasets, significantly outperforming likelihood-bazt baselines which remain close to random guessing.
1ama--31-8B-Instrae
are presented in Table 6
Table 5: Detection results on Qwen2.5-7B-Be under RL-MIA (higher is better). AVG reports
mean AUC across AIME24 and AIME25.
AIME24
AIME25
Method
AVG
Fl score
AUC
FI score
AUC
PPL
2023
0.71
0.59
0.60
0.55
0.57
Min-K Solssa
0.72
0.41
0.63
0.45
0.43
Min-K Min-K% K%+++ Zhung stal. 20256
0.69
0.48
0.58
0.46
0.47
Recall Xie 2024b 2024h
0,45
0.55
0.32
0.49
0.52
CDD Dong
0.67
0.51
0.67
0.43
0.47
Entropy-Temy
0.72
0.50
0.70
0.63
0.56
Entropy-Noise
0.70
0.64
0.67
0.51
0.57
Self-Critique (Ours)
0.81
0.76
0.71
0.72
0.74 (1 30 %)
B.2 NUMERICAL RESULTS FOR DUAL-STAGE CONTAMINATION
For completeness, provide the numerical data corresponding to the dual-stage contamination analysis presented in Figure the main paper. The results, detailed in Table quantify the trend shown in the figure: as the pretraining contamination signal is reduced (i.e., when evaluating
16

---

Published as a conference paper at ICLR 2026
Table Detection results on Llama-3.1 1etiaa..--B--Isttuuee under RL-MIA (higher is better). AVG reports mean AUC across K&K and SAT. Self- Self-Critique consistently outperforms baselines on both logic datasets.
K&K
SAT
Method
AVG
Fl Score
AUC
Fl Score
AUC
PPL 2023
0.52
0,47
0.67
0.53
0.50
Min-K
0.66
0.50
0.50
0.46
0.48
Min-K% K% Zhang et 20256
0.45
0.44
0.51
0.47
0.46
Recall I Xiz
0.67
0.53
0.68
0.55
0.54
CDD Dong et 2024
0.67
0.53
0.67
0.51
0.52
Entropy-Temp
0.68
0.56
0.67
0.53
0.55
Entropy-Noiie Self-Critique (Ours)
0.67
0.52
0.68
0.57
0.55
0.69
0.61
0.70
0.62
0.62 (1 15 %
subsets with lower PPL scores), the AUC Self-Critique on these filtered subsets increases sharply from 0.59 to 0.88, confirming its effectiveness at isolating RL-specific signals.
Table 7: Numerical AUC results for the dual-stage contamination analysis. The header row indicates the quantile of pretraining contamination retained.
Method
1.0
0.9
0.8
0.6
0.3
0.1
PPI.
0.52
0.48
0.47
0.56
0.54
0.56
Self-Critique (random subset)
0.59
0.58
0.60
0.57
0.49
0.54
Self-Critique (lower pretraining contamination)
0.59
0.62
0.62
0.65
0.74
0.88
B.3 ADDITIONAL RESULTS ON RLHF PARADIGM
While primary investigation in the main text focuses the Reinforcement Learning with Verifi- able Rewards (RLVR) paradigm due to its rising prominence in reasoning tasks, we acknowledge the critical role of Reinforcement Learning with Human Feedback (RLHF) the broader post-training landscape. To demonstrate the generalizability of our method, we extended our experiments to the standard RLHF setting. Following the experimental setup of Zhong al. (2025). we utilized Llama- 3-8B-Instruct trained on the UltraFeedback dataset (Cui et al.) (2023) under four distinct alignment
algorithms: PPO (Schulman al. 2017) (representing traditional reward modeling), DPO (Rafailov et al.
2023) (representing implicit reward alignment). TDPO (Zeng et al. (024b) and RTO (Zhong ct al.
2025
(representing token-level reward modeling).
Table 8: Detection results (AUC) on the UltraFeedback dataset across different RLHF alignment
algorithms. Self-Critique consistently outperforms baselines regardless of the alignment method.
PPO
DPO
TDPO
RTO 2025
Method
Schulmas 2017
Rafallow
2004
PPL PPLiGmnen et 2e2
0.56
0.45
0.49
0.53
Min-K Sto Sta
2024
0.40
0.53
0.49
0.45
Min-K 6% Zhang 20256
0.47
0,52
0.51
0.44
Recall lile
0.58
0.58
0.60
0.56
CDD otngetal
0.56
0.46
0.44
0.44
Entropy- Temp
0.54
0.53
0.54
0.52
Entropy-Noise
0.53
0.51
0.53
0.59
Self-Critique (Ours)
0.62
0.62
0.64
0.70
The detection results are presented in Table 8 We observe that Self-Critique consistently maintains its effectiveness across all four paradigms, achieving the highest AUC scores ranging from 0.62 to 0.70. In contrast, baseline methods such as PPL and CDD often perform near random guessing (AUC 0.50) exhibit instability. This suggests that the phenomenon of policy collapse is
17

---

Published as a conference paper at ICLR 2026
fundamental characteristic of alignment training, regardless of whether the reward signal is sparse, implicit, or token-level.
C ADITIONAL ABLATIONS
C.I WIYY SELF-CRITIQUE PROBING IS A BETTER DETECTOR
Self-Crilique Probiing
Response
Self-Drtique-rrbbing
Initial Respanse
6.00
0.15
0.21
0:30
0.5m
0.4
0.4
AMEZ4
AIMEZ5
KSK
SAT
AMME44
AIHESS
KGK
SAT
(a) Qwen2.5-7B-Insttu
(b) DeepSeek-Matth-BB-nsse
Figure 4: Self-critique probing vs. self-critique.
As self-critique idea asks the model to propose an alternative reasoning path given its initial response, naive variant is to skip the initial response and simply tell the model to "answer us- ing an unusual technique' [ The results in Figure 4 show that without the initial response an anchor, performance collapses to near random guess This variant fails because it removes the an- chor that makes the probe meaningful: without conditioning on the initial answer, the "alternative" path is unconstrained, instruction following is noisy, and both members and non-members produce heterogeneous trajectories whose entropy sequences are not comparable. Moreover, RL can induce mode-seeking even on unseen items, so we end up measuring two arbitrary paths rather than the deviation from a memorized one. Self-critique fixes concrete baseline and probes deviation from it hence the large AUC gains over the no-response variant.
C.2 ABLATION ON SAMPLING STRATEGY
We perform an ablation study of the sampling strategies used to generate model responses. Specifi- cally, we experiment with three settings: (1) using greedy sampling for both the initial response and the second self-critique response; (2) using greedy sampling for the initial response and tempera- ture sampling for the self-critique response; and (3) using temperature sampling for both the initial response and the self-critique response. For temperature sampling, we test multiple temperatures. The ablation results, shown in Figure 5 indicate that the best performance achieved when both the initial and self-critique responses are generated by greedy sampling. This is because greedy sampling eliminates the effect of randomness, thereby better revealing the sharp policy distributions caused by entropy collapse from RL post-training.
C.3 ABLATION ON META- INST-IINTTUCTION
To address the concern that our method's behavior might be tied to a specific prompt template, we conducted an ablation study test its sensitivity to the Self-Critique instruction. We evaluated five paraphrased variations of our original meta-instruction (detailed as Variations 1-5 in Appendix Eb.
The results, presented in Table 9 ferent templates. The standard deviation in AUC across all meta- meta-prompts is remarkably low (0.0251 on AIME25 and 0.0254 on K&K)). This quantitative evidence confirms that the detection signal is driven by the underlying policy collapse mechanism rather than the specific phrasing of the instruc- tion.
demonstrate that Self-Critique is exceptionally stable across dif-
Detail prompt shown in Appendix
Prompt
18

---

Published as a conference paper at ICLR 2026
grredy
greedy++teppraate
temperatue+temperatue
AIME24
AIME25
0.8
0.8
0.6
0.6
AUC
AUC
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
temperature
temperature
K&rK
SAT
0.8
0.8
0.6
0.6
AUC
AUC
0.4
0.4
0.2
0.2
0.0
0.0
0.6
0.8
1.0
0.6
0.8
1.0
temperature
temperature
Figure 5: Ablation Sampling Strategy
Table 9: Ablation study on different Self-Critique meta-instructions. The method shows high ro- bustness to prompt variations, as indicated by the low standard deviation.
AIME25
K&K
Meta-Instruction
Fl score
AUC
Fl score
AUC
Original
0.7647
0.7156
0.6866
0.6584
Variation 1
0.7179
0.6978
0.6992
0.6464
Variation 2
0.7879
0.7600
0.6763
0.6432
Variation 3
0.7429
0.6978
0.7302
0.7016
Variation 4
0.7500
0.7422
0.7009
0.6640
Variation 5
0.7500
0.7333
0.6809
0.6272
Mean Std
0.7522
0.7244
0.6957
0.6568
0.0233
0.0251
0.0195
0.0254
D VISUALIZATION OF CONTAMINATION SCORE DISTRIBUTION
For the contaminated and uncontaminated samples in the AIME and AIME25 dataset, we computed their Self-Critiguu similarity scores, and present the histograms in Figure 6 Through Kernel Den- sity Estimation (KDE), we observe a clear difference in the distribution of Self-Critique similarity scores between contaminated and uncontaminated samples, demonstrating the effectiveness of proposed Self-Critique method for data contamination detection.
E INSTRUCTIONS
We present here the prompts used the data contamination detection process. We Proplt I encourage the model to generate responses that deviate from conventional reasoning. In particular,
19

---

Published as a conference paper at ICLR 2026
3,0
3,0
uncorteriiated coramibeted SarTeTtteeedd
2.5
contominonss contammnuss
Densiy
2.0
Density
1.5
03
AIME
(b) AIME25
Figure 6: Histograms and Kernel Density Estimation (KDE) of Self-Critique similarity scores be- tween contaminated and uncontaminated samples.
for the Self- Self-Critique process, we use Prompt to guide the model to regenerate second response that differs as much as possible from the initial one. For ablation study on template sensitivity, additionally employed five paraphrased variations (Variations 1-5) detailed below.
Prompt 1: Unconventional Reasoning
Answer using technique you'd typically avoid or deliberately unconventional line of reasoning.
Prompt 2: Self-Critie Critique Instruction (Original)
A possible answer is provided below (it may may not be correct). Please provide a re- sponse that follows different reasoning path or provides an alternative solution:
[Initial Response]
Please now provide your new, different response:
Prompt 3: Self- Self-Critique Variation
A potential answer is shown below (it may or may not be correct). Please give response that uses different reasoning approach or offers alternative solution:
(Initial Response)
Please now give your new, different response:
Prompt 4: Self-Critique Variation
An answer attempt is provided below (it may or may not correct). Please provide response that takes different reasoning path presents alternative solution:
[Initial Response)
Please now provide your new, alternative response:
20

---

Published as a conference paper at ICLR 2026
Prompt 5: Self-Critique Variation
candidate answer provided below (it may or may be correct). Please give response that employs different reasoning path or supplies an alternative solution:
[Initial Response)
Please now give your new, distinct response:
Prompt 6: Self-Critique Variation 4
A potential solution is presented below (it may or may not be correct). Please provide response that adopts a different reasoning path offers an alternative solution:
[Initial Response)
Please now provide your new, varied response:
Prompt 7: Self- CritCrrtique Variation 5
A tentative answer is provided below (it may may not be correct). Please provide response that uses a different reasoning path or presents an alternative solution:
[Initial Response]
Please now provide your new, alternative response:
F BENCHMARK DETAIL & TRAINING SETTINGS
This section provides detailed breakdown of the RL-MIA benchmark construction and the specific training configurations used in our experiments.
To create controlled environment for evaluating data contamination in the RL phase, structed the RL-MIA benchmark. The methodology involves injecting a known subset of test sam- ples (contaminated items) into larger base corpus for RL post-training. The detection task is then to distinguish these injected samples from clean, unseen samples. Table 10] summarizes the data splits for each source dataset, detailing the size of the base RL corpus, the number of injected items, the to- tal size of the final training set, the number of times each contaminated item appears (Occurrences), and the total number of items in the final detection task (Contaminated + Clean).
For reproducibility, we also provide the key hyperparameters used during RL post-training. Table I1 lists the shared training parameters for our two primary experimental models: O0wen2.5-7B-Instrice and Deepseck maak--mahhh-6b -
Table 10: RL-MIA data splits for training and evaluation. The last column reports the total number of evaluation items (Contam + Clean).
Source
Base RL Corpus (size)
Injected items
Train Size
Occurrences
Detection Tasks
AIME24
OpenRI Math-46K
46K
2
AIME25
15
K&K
K&K train: 950
K&K test: 50
950 + 50
100
3
SAT
SAT train: 450
SAT test: 50
450 50
100
GSMBK
GSMBK train:
GSMBK test: 659
7473 659
1319
Due to the 4,096 context length DeepSeek-Matte IB-Instruct, we set its maximum generation length to 3,072 (with 1,024-token prompt). For other models (Qwen2.5-7B-Math, Qwen2.5-
21

---

Published as a conference paper at ICLR 2026
Table 11: Shared training hyperparameters
Parameter
Qwen2.5-7B-Instruct
Deepseck-Math-7-Innsth
Actor learning rate
1.0 10
1.0 10 6
Batch size (Irain / val)
128 512
128 / 512
Max prompt length
1024
1024
Max generation length
4096
3072
Temperature (Irain val)
1.0 0.6
1.0 1.0/0.6 0.6
Samples per prompt (n)
8
8
Tensor model parallel (TP)
2
micro / mini-batch
2/2
2
Max tokens per GPU
16384
16384
Use KL loss
No 0.001
No 0.001
Entropy coefficient
3B-Instruct, and Owee2.550.5B-Instteee the settings are essentially the same as for Qwen2.5-7B- Instruct. Full details available in the training scripts included with our released code.
G DISCUSSION
G.1 DISCUSSION ON RESULT VARIABILITY AND ERROR BARS
In Table 2, we observed variability in the performance of baseline methods across different models, particularly for Entropy-Noise on the SAT dataset. This variability largely stems from the nature of the probing mechanism: Entropy-Noise injects a random, non non-member prefix to disrupt the context. Different models react differently to these -itt---diitribttionn prefixes- models robustly ignore them, while others may become unstable or hallucinate, leading to erratic entropy shifts that do not consistently correlate with contamination. contrast, Self-Critique uses a semantically meaningful instruction, guiding the model into more predictable state, which results in more stable detection.
Table 12: Bootstrap analysis (1,000 resamples) the SAT dataset. We report Mean + Std and the [95% Confidence Interval]. Self- tfiitiiiq shows stable performance significantly above random guessing.
Qwen2.5-7B-Insteee
DeepSeck-Math-78
Method
Mean + Std
95% CI
Mean + Std
95% CI
PPL
2005
0.50 + 0.06
[0.38, 0.63]
0.64 + 0.06
[0.52, 0.75]
Min-K
3024
0.50 + 0.06
10.37, 0.61]
0.35 + 0.06
[0.24, 0.47]
Min-K + Zhang et 20258
0.31 + 0.05
[0.21, 0.42]
0.49 + 0.06
[0.38, 0.61]
Recall Xie
0.62 0.06
[0.51, 0.72]
0.62 + 0.06
[0.50, 0.74]
CDD Dong et al 2024
0.47 0.06
[0.36, 0.58]
0.50 + 0.02
[0.46, 0.54]
-eem
0.69 + 0.05
[0.58, 0.78]
0.61 0.06
[0.49, 0.72]
Entropy-Noise
0.77 + 0.05
[0.67, 0.86]
0,45 + 0.06
[0.33, 0.56]
Self-Critique (Ours)
0.67 + 0.05
[0.56, 0.76]
0.67 + 0.06
[0.55, 0.77]
To rigorously quantify this stability, we performed a bootstrap analysis (Efron 1992) (resampling
1,000 times) to calculate the mean AUC and 95% Confidence Intervals (Cl) (DiCiccio & Efron 1996) for the SAT dataset. As shown in Table T2] while baseline performance fluctuates significantly (e.g., Min-K%++ varies from 0.21 to 0.42 on Qwen), Self- Self-Critique demonstrates robust performance with confidence intervals that consistently surpass the likelihood- based methods.
G.2 DISCUSSION ON INFERENCE COST
A practical consideration for deployment is the trade-off between detection performance and compu- tational budget. We categorize detection methods into passive (e.g., PPL, Min K%) and active (e.g., Recall, CDD, Self- Critiqui) approaches. O Passive Methods: These are computationally cheapest, requiring only single forward pass of the text. However, demonstrated in our experiments (Table
22

---

Published as a conference paper at ICLR 2026
2), they are largely ineffective for RL post-training contamination because the likelihood signal is decoupled from the reward-driven training objective. @ Active Methods: These require additional computation to probe the model but necessary for valid detection in this regime. Among active methods, Self-Critique offers the best balance. requires two generations (initial + critique). In comparison, Recall also requires two passes but generally achieves lower AUC. CDD typically quires a large number of samples (e.g.. N = 50) to estimate the edit distance distribution reliably. If limit CDD just two samples to match Self-Critique's budget, its estimation of the edit distance distribution becomes noisy and unreliable, leading to meaningless results.
Regarding the overhead of computing token-level entropy, our ablation study in Table demon- strates that approximating entropy using only the Top-K (e.g., K = 5) probabilities is sufficient. This avoids the need to materialize the full vocabulary distribution, making the entropy calculation overhead negligible compared to the generation cost.
H LIMITATIONS AND FUTURE WORK
While our work presents a robust framework for detecting contamination in the RL post-training phase, we acknowledge several avenues for future research that build upon our findings.
Generalization Across Diverse Domains. Our experiments primarily focused mathematical and logical reasoning tasks, as these are prominent domains where RI. has demonstrated significant benefits. However, the unique characteristics of RL-induced contamination may vary across differ- ent problem domains. For instance, in areas such as code generation, where a wider diversity of valid solutions is common, the signature of policy collapse might manifest differently. Future work could extend the evaluation of Self-Critique and other reward-aware detection methods to these and other domains, thereby assessing the broader applicability and potential domain-specific adaptations of our method.
Scalability to Larger Models. The models used our study, ranging from 0,5B to 7B parameters, representative of widely used class of open-source LLMs. However, the landscape of foun- dation models is rapidly evolving, with sfate-of-the-ar proprietary and open-source models now exceeding hundreds of billions of parameters. While have no reason believe method's core principles would not apply, the dynamics of policy collapse and memorization at such scales not yet fully understood. Investigating the effectiveness of Self-Critique on these frontier models represents an important next step in ensuring the reliability of the entire LLM ecosystem.
I LLM USAGE STATEMENT
In preparing this manuscript, we use LLMs to aid and polish the writing. Specifically. LLMs improve clarity, grammar, and phrasing, ensuring the text concise and readable. The use of LLMs does not influence the technical contributions or the interpretation of experimental findings. All content polished by LLMs is carefully checked by the authors.
23


Reviews:
[{"review_id": "PtgYMHIMWt", "reviewer": "Reviewer_iu3G", "summary": "This paper addresses the problem of detecting data contamination in LLMs that occurs during the RL post-training phase, a critical gap where existing detection methods, designed for pre-training and SFT, fail. It proposes a novel detection method named Self-Critique and a new benchmark called RL-MIA  to simulate and evaluate this specific type of contamination. The idea of the method is to probe for \"policy collapse\" (memorization) by instructing the model to generate an initial response and then a second, \"self-critique\" response with a different reasoning path. The method then compares the token-level entropy sequences of both responses. High similarity in their entropy patterns indicates the model is stuck on a memorized path and is therefore likely contaminated. The empirical studies involve evaluating Self-Critique on the new RL-MIA benchmark, which uses math (AIME 2024/2025) and logical reasoning (K&K, SAT) datasets. The method is tested on models like Qwen2.5-7B and DeepSeek-Math-7B, comparing its performance (using AUC) against several baseline detectors (like PPL, Min-K%, and CDD) and testing its robustness across different RL algorithms (PPO, GRPO, DAPO).", "questions": "- Table 2 shows significant variability in the performance of baseline methods across different models (e.g., Entropy-Noise on the SAT dataset). Could you provide more intuition on why this variability is so high? Can you bootstrap to get an error bar?\n- The paper compares methods with different inference costs (e.g., Self-Critique requires 2 generations, PPL requires 1, and CDD may require N). Could you comment on the performance vs. compute trade-off? For instance, how does Self-Critique compare to a baseline like CDD when both are limited to the same computational budget (e.g., 2 samples)?\n- The experiments primarily use Qwen and DeepSeek models. To what extent do you believe the results are general? Would you expect similar performance and baseline rankings if the method were replicated on a different model family, such as Llama, which may have different architectural properties and pre-training data?\n\nI will be happy to increase my score if the authors address these questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is overall a good paper. The motivating problem is novel and well explained. The paper is the first to systematically study data contamination specifically in the RL post-training phase. It clearly explains why this is a new problem: RL optimizes for a reward signal, not likelihood, which makes existing likelihood-based detectors (like PPL) ineffective (as shown in Table 2, where they perform near random guess).\nThe method is simple, yet tailored to the problem: The Self-Critique method is an intuitive and clever solution. Instead of checking likelihood, it probes for \"policy collapse\"—a known side-effect of RL training. The method of comparing the entropy of an initial answer and a \"self-critique\" answer is a simple and effective way to measure this path dependency.\nThe writing is very clear. The paper clearly formalizes the problem (Section 3.1) and provides a step-by-step algorithm for the Self-Critique method (Section 4.1, Algorithm 1). \nThorough Experimental Validation: The experiments are comprehensive and support the claims.\n\n- Baselines: It compares Self-Critique against a wide range of baselines (PPL, Min-K%, Recall, CDD) and even custom entropy-based baselines (Entropy-Temp, Entropy-Noise).\n\n- Robustness: It shows the method works across multiple models (Qwen2.5-7B, DeepSeek-Math-7B, etc.), diverse datasets (AIME, K&K, SAT), and different RL algorithms (PPO, GRPO, DAPO, as seen in Table 3).\n\n- Ablations: It includes valuable ablations on the effect of Top-K entropy approximation (Table 4) and sampling strategy (Figure 5).\n\n- Dual-Contamination Study: The analysis in Section 5.4 is particularly strong, showing how Self-Critique can isolate RL-phase contamination even when pre-training contamination is also present, with performance improving up to 55%.", "weaknesses": "- Result Variability and Lack of Error Bars: There appears to be significant variability in the performance of some methods across the two main models (Table 2). \n- Inference Cost: The Self-Critique method requires the model to generate two full responses (initial and critique) and requires token-level probability access to compute entropy for both. This is computationally more expensive than simpler methods, like PPL, which only require one pass.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QlszRQp533", "reviewer": "Reviewer_5NYt", "summary": "This paper addresses the overlooked issue of data contamination during the RL post-training phase of large language models (LLMs). This work explores contamination specific to RL post-training. The authors introduce Self-Critique, an entropy-based probing method that measures the similarity between two model outputs: an initial answer and a self-critique re-generation conditioned on that answer. A high similarity in entropy curves between the two is interpreted as evidence of contamination, under the hypothesis that contaminated examples lead to policy collapse toward memorized reasoning paths.", "questions": "1. Why entropy path is a reliable signal and is it sensitive to the template diversity?\n2. What is the key difference of the self-critique probing and direct probing with different templates?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Brings attention to the underexplored problem of contamination during RL post-training, which is an important practical concern.\n2. Proposes a simple, reproducible probing method that builds on self-consistency and entropy measurement.\n3. Introduces RL-MIA, a useful benchmark that may benefit future work in this area.", "weaknesses": "1. Weak Novelty: The proposed “Self-Critique” method is a straightforward application of self-consistency probing combined with entropy comparison. It does not provide a fundamentally new theoretical or algorithmic contribution.\n2. Questionable Signal Reliability: The paper assumes that contamination causes entropy path similarity, but similar effects can arise from deterministic reasoning or narrow reward optimization. The entropy path is not guaranteed to distinguish contamination from general RL collapse. And it is doubtable that the method’s behavior is closely tied to the chosen prompt and “self-critique” instruction. In more diverse or flexible RL training settings, the entropy path could vary widely, undermining the stability of the metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iuVolniodo", "reviewer": "Reviewer_PKgz", "summary": "This paper introduces a membership inference attack strategy for models trained on  reinforcement learning with feedback via the exploitation of the entropy collapse in RL learned policies. The work relies on the observation that the RL learned response traces often suffer from entropy collapse on high reward regions. The key idea behind the work is to use the length penalized entropy measure between an original prompt’s response and the response generated with an added critique meta prompt as a mechanism for member detection. The paper showcases that in RL trained policies the proposed method performs better than the existing works which were designed towards next token prediction training objectives (SFT, Pretraining etc)", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is easy to follow and is well written.\n\n2. Dataset and model diversity is present in the experimentation. \n\n3. To my knowledge this is the first MIA work reinforcement learning alignment. But my expertise in MIA attacks is also limited. \n\n4. The proposed method showcases effective performance as opposed to other baselines which are designed towards models trained with maximum likelihood. \n\n5. A certain level of ablation has been performed on different RL methods under Section 5.4.", "weaknesses": "1. The method seems to be motivated by the sparse reward assumption of verifiable reward. Human feedback based preference datasets still play a vital role in capturing human feedback. While traditional reward models are learned via the Bradley Terry formulation on a complete response there, token level reward modelling has also gained prominence. Further implicit reward based alignment such as DPO also exists. Can you provide the prospects of the generalizability of the method across these different alignments. \n\n2. Can you please provide further details about the reward models used for the RL training under the section “Contaminate in Different RL algorithms”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)