Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
Authors: Haoming Meng
Kexin Huang
Shaohang Wei
Chiyu Ma
Shuo Yang
Xue Wang
Guoyin Wang
Bolin Ding
Jingren Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Reasoning
RLVR
Primary area: reinforcement learning

Abstract: Reinforcement learning with verifiable rewards (RLVR) has significantly improved reasoning in large language models (LLMs), yet the token-level mechanisms underlying these improvements remain unclear. We present a systematic empirical study of RLVR’s distributional effects organized around three main analyses: (1) token-level characterization of distributional shifts between base and RL models, (2) the impact of token-level distributional shifts on reasoning performance through cross-sampling interventions, and (3) fine-grained mechanics of these shifts at the token level. We find that RL fine-tuning induces highly sparse and targeted changes, with only a small fraction of token distributions exhibiting meaningful divergence. We further characterize the structure of these shifts through analyses of token entropy, positional concentration, and reallocation of probability mass. To assess the functional importance of these sparse changes, we conduct cross-sampling experiments that selectively swap token choices between the base and RL models. Inserting only a small fraction of RL-sampled tokens into base generations progressively recovers RL performance gains, while injecting a similarly small number of base token choices into RL-generated responses collapses performance to base levels, isolating a sparse set of token-level decisions directly responsible for RLVR’s improvements. Finally, we explore divergence-weighted variants of the advantage signal as a diagnostic intervention, finding that they can yield improvements over baselines. Together, our results shed light on the distributional changes induced by RLVR and provide a fine-grained, token-level lens for understanding RLVR as a targeted refinement process.

Published as a conference paper at ICLR 2026
SPARSE BUT CRITICAL: A TOKEN-LEVEL ANALYSIS OF DISTRIBUTIONAL SHIFTS IN RLVR FINE- OF LLMS
Haoming Meng' Kexin Huang Xue Wang Guoyin Wang!
Shaohang Wei
Chiyu Ma
Shuo Yang"
Bolin Ding'
Jingren Zhou
I University of Toronto Peeking University
3 Dartmouth College Alibaba Group
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has significantly im- proved reasoning in large language models (LLMs), yet the token-level mech- anisms underlying these improvements remain unclear. We present a system- atic empirical study of RLVR's s distributional effects organized around three main analyses: (1) token-level characterization of distributional shifts between base and RL models, (2) the impact of token-level distributional shifts on reasoning perfor- mance through sroms-sampling interventions, and (3) fine-grained mechanics of these shifts at the token level. We find that RL fine-tuning induces highly sparse and targeted changes, with only a small fraction of token distributions exhibit- ing meaningful divergence. We further characterize the structure of these shifts through analyses token entropy, positional concentration, and reallocation probability mass. To assess the functional importance of these sparse changes, we conduct cross-sampling experiments that selectively swap token choices between the base and RL models. Inserting only small fraction of RL-sampled tokens into base generations progressively recovers RL performance gains, while inject- ing a similarly small number of base token choices into RL-generated responses collapses performance to base levels, isolating sparse set of token-level decisions directly responsible for RLVR's improvements. Finally, we explore divergence- weighted variants of the advantage signal as diagnostic intervention, finding that they can yield improvements over baselines. Together, our results shed light on the distributional changes induced by RLVR and provide fine-grained, token-level lens for understanding RLVR as a targeted refinement process.
INTRODUCTION
Recent advances in reinforcement learning with verifiable rewards (RLVR) (Lambert et al. 2024) for reasoning in large language models (LLMs), such as Group Relative Policy Optimization (GRPO) (Shao al. 2024). have enabled substantial performance improvements on challenging reason- ing and mathematical benchmarks. Despite this empirical success, the mechanisms through which RLVR modifies model behavior remain unclear.
Most evaluations of RL fine-tuning focus on aggregate metrics such as accuracy, reward, and re- sponse length. While informative, these metrics provide only a coarse-grained view of improvement and offer limited insight into how model behavior changes internally. In particular, central unre- solved question is: how does RLVR reshape the token-level predictive distributions of a base model, and which of these changes actually drive downstream reasoning gains?
Recent work has begun analyzing RL fine-tuning through token-level entropy and uncertainty per- spectives (Wang et al. 2025 Cheng et al. 2025 Cui et al. 2025). highlighting the role of high- entropy tokens and exploration dynamics. However, a more detailed distributional view remains missing: how such shifts structured across positions and contexts, how probability mass is reallo- cated across candidate tokens, how they evolve over training, and to what extent are they responsible for RLVR's performance gains.

---

Published as a conference paper at ICLR 2026
In this paper, develop fine-grained, token-level perspective on RLVR through the lens of distri- butional change. We perform a systematic empirical study of how RLVR alters next-token distri- butions relative the base model, and connect these distributional shifts directly to sequence-level reasoning performance. Our analyses reveal that RIVR acts primarily as a sparse and targeted re- finement process: most token distributions remain nearly unchanged, while small subset of high- divergence positions carries disproportionate functional importance.
Our contributions are organized follows:
1. Structure of Token-Level Distributional Shifts. We show that RLVR induces sparse and highly targeted token-level distribution shifts relative the base model. We characterize the structure of these shifts through divergence, entropy, and positional analyses, and com- pare across multiple methods, revealing differences in exploration and refinement behavior.
2. Cross-Sampling Interventions. We use forward and reverse cross-sampling interventions to measure the role of divergent token decisions. We show that modifying only small fraction of token choices is sufficient to recover (in base-model generations) erase (in RI.-model generations) most RLVR performance gains, linking sparse distributional shifts directly to sequence-level reasoning outcomes.
3. Fine-Grained Distribution Mechanics. We analyze how RLVR modifies token distribu- tions high-divergence positions and show that it primarily reallocates probability mass within an existing candidate set rather than introducing new tokens. We support this with top-k overlap, rank, tail tail- hull-prbbbiiltyy, and training training-evolution analyses.
4. Divergence-Weighted Advantage. Motivated by these findings, we study divergence- weighted variants of the RLVR advantage signal as a diagnostic objective modification and show that they can improve over baselines.
Taken together, our results provide a unified token-level picture of RLVR fine-tuning: rather than globally rewriting model behavior, RLVR predominantly performs sparse, structured probability re- allocation in a small set of critical token positions that steer downstream reasoning trajectories. This distributional and functional perspective helps clarify the mechanisms which RLVR improves reasoning in LLMs.
2 TOKEN DISTRIBUTION ANALYSIS BETWEEN BASE AND RL MODELS
We begin by analyzing the general structure of distributional shifts induced by RLVR, with the goal of characterizing how token-level predictions differ between the base model and its RL-finetuned counterpart. Our analysis compares next-token distributions for both models under sequences gen- erated by the RL policy. This framing treats the RL-generated trajectory as reference path and allows us quantify how the base model would need to adapt in order to emulate it.
2.1 PRELIMINARIES
For each token position and sequence Tett let Thusel x<ee) and ThI. Jice) denote the condi- tional next-token distributions of the base and RI models, respectively. To quantify distributional differences, we use the Jensen-Shanmn Shannon (JS) divergence, defined as
Das Tussee ( xx(e)) || (. x<()))= = DKL (Thwee (ee)) Mc) + (TRL( xee)) || Mc)
where M, (Thase( x + (- x.ce)).
One could use any notion of divergence distance between probability measures, but we use JS divergence over something like KI. divergence because: (i) symmetrie, avoiding directional considerations; (ii) it is bounded in [0, 21, preventing extreme values from dominating aggregate statistics; and (iii) it remains well-defined even when the measures lack absolute continuity with respect to each other. The latter particularly important in practice, memory constraints often limit the retrieval of the full distribution across the entire vocabulary, and also when comparing top-p truncated distributions, for which KL divergence may be undefined.
Unless otherwise stated, divergences are computed on top-p truncated distributions using the same sampling configuration employed during generation, while entropy and probabilities are from the
2

---

Published as a conference paper at ICLR 2026
full estimated distributions. Robustness checks across different top-p values and against estimates
of the full distributions are provided in Appendix A.4 (Figures 26 and 28 as well as results on additional models/datasets.
2.2 DISTRIBUTION SHIFTS ARE HIGHLY TARGETED AND SPARSE
A natural starting question is: how broadly are distributional shifts distributed across token posi- tions? To answer this, we examine the token-level divergence between the base and RI-finetuned models. Figure percentile curves for DAPO and SimpleRI. on their generated responses for AIME 2024.
0.7
0.4
0.6
5Overgence
0.5
15 Divergence
0.4
0.3
M0t 0,3798
p0mm: 01220
40
60
100
20
40
60
100
(a) DAPO: Percentile curve
(b) SimpleRL: Percentile curve
Figure JS divergence distributions for Qwen2.5 32B DAPO and SimpleRL on AIME 2024.
The results reveal that RLVR refinement is highly sparse at the token level. Under DAPO, more than 83% of token positions exhibit near-zero divergence, while this proportion exceeds 98% under SimpleRL. indicating that only small subset of token positions undergo substantial distributional change. Comparing the two, DAPO exhibits a broader divergence distribution and more gradual percentile curve, consistent with its clip-higher mechanism and lack of KL regularization, permitting broader exploratory updates. contrast, SimpleRI. imposes stricter constraints, resulting more tightly concentrated changes. Notably, maintains near-zero zero divergence at most token distributions. For a more controlled comparison for models fine-tuned on the same dataset, Appendix A.4.2 presents the results for (wwen..5-Math771 trained with DAPO, comparing upper clip settings of 0.28 and 0.2. We see that, analogous to the results of the 32B models, the more restrictive 0.2 upper clip setting results in sparser distributional shifts, as shown by the percentiles corresponding to near-zero divergence (Figure 33p.
in the absence of KL regularization, the DAPO policy
2.3 POSITIONAL CONCENTRATION
Beyond how broadly changes are distributed across token positions, we next ask: where within a generated sequence do distributional shifts tend to occur? Figure 2 plots the mean and median JS divergence as a function of normalized token position (token index divided by response length), with percentile bands, for DAPO and SimpleRL on AIME 2024.
Both models exhibit a clear positional structure: average divergence is consistently highest near the beginning of the response, decreases through the middle, and increases modestly again toward the end. The early concentration aligns with the modification initial high-level branching decisions, while the late increase aligns with adjustments answer formatting and termination behavior. How- ever, high divergence can occur throughout the response, as reflected in the wide percentile spread.
2.4 DIVERGENCE-ENTERR RELATIONSHIP
To further understand the general structure underlying these sparse distributional shifts, we ask: How such shifts related to the model's token-level entropy? We thus examine the relationship between distributional divergence and predictive entropy on the token level. For each token position compute the token-level entropy and analyze how entropy relates to the distributional shifts from the base to RL model. Prior work suggests that RLVR updates may primarily affect high- entropy (uncertain) predictions while leaving low-entropy (confident) predictions largely unchanged
3

---

Published as a conference paper at ICLR 2026
22t-7566 Vade
0.10
0.000
1 Bleeggree
0.005
0.004
0.00
0.003
0.02
Relative
Relative
Position
(a) DAPO
(b) SimpleRL
Figure 2: Mean and median JS divergence by normalized token position, with percentile bands. Both methods concentrate updates the start and, a lesser degree, at the end of responses.
(Wang et al. 2025). We explore this perspective by comparing entropy distributions across low- and bigh-divergence token positions. Specifically, token positions are grouped into low- and high- divergence bins, and we compare the entropy distributions of both the base and RI. models within each bin. Figure I shows these results for DAPO, with corresponding SimpleRI. results provided in Appendix A.4 Figure 16
Base Entropy
Base Entropy
Entrapy
RL Entropy
0.5
Paportion
0.00
0.3
0.04
0.02
0.1
0.00
0.0
0.5
1.0
3.5
2.0
2.5
3.0
3.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Entropy
Entropy
(a) Low JS divergence distributions (< 0.1).
(b) High JS divergence distributions (> 0.1).
Figure 3: Entropy distributions for low and high divergence distributions for DAPO. Low- divergence tokens are generally low-entropy, while high-divergence tokens span both high- and low-entropy regions, indicating that DAPO can modify even initially confident predictions.
The results show that low-divergence token distributions are largely low-entropy, indicating that dis- tributions that preserved are mostly initially low-entropy. High-divergence contexts, however, can span a broad entropy range. In particular, DAPO modifies both initially high- and low-entropy predictions, demonstrating its ability to override trast, SimpleRL concentrates divergence more strongly in higher base entropy regions, reflecting a more conservative update regime. Isolating the effect of clip-higher, Figure 335 illustrates this con- trast more clearly. At high-divergence positions, the higher 0.28 upper clip produces greater pro- portion of distributions with low base entropy, whereas the 0.2 clip concentrates its high-divergence distributions in the higher base entropy regime. Notably, the resulting RL entropy is higher under clip-higher, while for the 0.2 clip it concentrated at lower values, consistent with the entropy col- lapse observed under standard clipping YYu et al. 2025) and the steadily increasing entropy induced by clip-higher.
confident base-model outputs. By con-
2.5 SEMANTIC IDENTITY OF DIVERGENT TOKENS
Given the sparsity and general structure of these shifts, natural next question is: Which types of tokens actually being targeted by RL fufe--ttningg
To investigate this, we examine which types of tokens tend to exhibit high versus low distribu- tional divergence. Figure 4 visualizes representative examples using word clouds, where the size
4

---

Published as a conference paper at ICLR 2026
Since
TinceThe
This
Froe
DEEO
Thus
and
A
$
n
win
equivto
and
Teft
9
bc
6
5
gtt
Frac
we
f that
ssrr
Og
suppliyy
This
AnnA
froe
iis
from
number
iss
need
Let
th
AnAn
times
cont
eet
Now
"and
An
The
O edot
VA 1nins
for
are
(a) Tokens with high IS divergence (JS > 0.1).
(b) Tokens with low JS divergence (JS 0.01).
Figure 4: Word clouds of high and low divergence tokens under DAPO.
of each token is proportional to its frequency. Upon an initial examination, tokens appearing in high-divergence distributions include common function words, reasoning-related terms, and certain equation fragments, whereas those in low-divergenes distributions are dominated by numerals, op- erators, and structural components of mathematical expressions.
However, token identity alone does not determine divergence behavior. Figure shows the full IS divergence distributions for the tokens sampled most frequently from high- and low-divergence distributions, revealing substantial context dependence. For example, the word "the" appears among the most frequent high-divergence tokens, yet its full divergence distribution is overwhelmingly con- centrated in the lower regime. This suggests that token identity alone is insufficient to characterize divergence, and that a contextual perspective is essential, rather than solely by token semantics.
2.6 COMPARISON WITH SUPERVISED FINE- TUNING (SFT)
While the above analyses reveal that RLVR induces sparse distributional shifts, it remains unclear whether this behavior unique to RL fine-tuning. This raises the question: Is such sparsity distinctive property of RLVR, or more general feature fine-uning methods? A natural point of comparison supervised fine-tuming ane-tuning (SFT), which optimizes models imitate target tokens rather than optimizing verifiable rewards on self- eeff-generated trajectories. Appendix (.33dieent.reeestttt a controlled comparison between supervised fine-tuning (SFT) and RI RLVR (DAPO) on Qwen2.5-32B. Under the same JS divergence measurements (Section 2.2). SFT exhibits a substantially larger high-divergence set and a broader divergence distribution than RLVR (Figure S}. This demonstrates that the sparsity of distributional shifts observed under RLVR is not generic consequence of fine-tuning.
PROSS-SAMPLING: FUNCTIONAL IMPORTANCE OF DIVERGENT DISTRIBUTIONS
In the previous section, we showed that, when evaluated along RL-generated trajectories, only small fraction of token distributions exhibit substantial shifts between the base and RL models. This observation motivates a fundamental question: Are these divergent token distributions directly responsible for the performance gains induced by RLVR? More generally, to what extent are the base and RI. policies functionally different their entire sequence distributions?
To answer these questions, we conduct controlled cross-sampling experiments that selectively swap token choices between the base model Thase and the RL-trained model TRL- We consider two com- plementary interventions: forward cross-sampling, which injects RL-sampled tokens into base- model generations, and (ii) reverse cross-sampling, which replaces RL-sampled tokens with base- model tokens during RL generation. Together, these interventions probe both the sufficiency and the necessity of RL-induced loken-leval changes. The general implementation procedure is summarized in Algorithm|] 1 of Appendix A.5
3.1 PROSS-SAMPLING FRAMEWORK
Let (Xt):21 denote the sequence of random variables generated during decoding, and define the stopping time it > X, EOS} A Tmax, where EOS is the end-of-sequence token
5

---

Published as a conference paper at ICLR 2026
and Tmax is the maximum number of tokens generate. The generated response is then the finite sequence X1.-. Let prrim denote the primary policy, which governs generation by default, and let Tint denote the intervention policy, which is used only at selected positions. These policies induce sequence-level distributions prim and Pitt over finite sequences. To model cross-sampling, we introduce a switching rule S V<N + {0, 13, where y<N the set of finite sequences over the vocabulary V. which determines, at each generation step, whether the next token is sampled from the Tint. (St = 1) or Trrm (S, = 0). Given partial sequence Xet, we define the switching variable S. ((Xxt) (0, 1), and the resulting mixed policy governing the law of the token:
X,
(prim.int) (. Xxt) mix
St) T'rrm (: Xxt) Se Tine Xxx)..
The corresponding sequence-level distribution is then denoted by p(prim,int)
mix
In our experiments, to align with the analysis in Section 2 the switching rule S defined as S(X=+) 1 (Dpsimprrm Xxe) || Time Xxt)) > Et, given fixed threshold 2 0, so that cross-sampling intervenes only at high-divergence positions. In Appendix A.6 we provide
simple bounds divergences between the sequential distributions Pmix and Pint under different cross-sampling settings.
Forward Cross-Sampling. forward cross-sampling, the response is generated primarily under the base policy, which serves the primary policy, i.e., Tprim Tbsee The intervention policy is the RL policy, Timt. TRI. This procedure tests whether selectively injecting RL token choices into trajectories that are otherwise generated by the base model is sufficient to recover RL-level performance.
Reverse Cross-Sampling. reverse cross-sampling. the roles of the base and RL policies are reversed. The response generated primarily under the RL policy, which serves the primary policy, i.e., Trrim = TRI. while the intervention policy is the base policy, Tint Thase- This intervention selectively replaces RI-sampled tokens high-divergence positions with base-model choices, allowing us to quantify how rapidly RL performance degrades when those decisions are removed.
25.0
Base Renampping
Resempling
Base 8.23%
Arrurnry 14%%
23%
20.0
17.5
Asston A Aruunsy
15.0
12.5
10.0
0.0
0.5
1.0
1.5
2.0
2.5
5.5
4.0
0.0
0.3
1.0
1.3
2.0
5.0
Average Effectivv Percentage of Resempled Tokens
Average Effective Percentage Resampled Tokens
(a) AIME 2024
(b) AIME 2025
Figure 5: Forward cross-sampling results (Qwen2.5 32B SimpleRL.): injecting RL tokens into base generations progressively recovers RL accuracy.
3.2 RESULTS AND FINDINGS
Figures 5 and 6 show the accuracy curves for forward and reverse cross-sampling on Qwen2.5-32B fine-tuned with SimpleRL., evaluated on AIME 2024. Fach point along the curve corresponds to the Mean @ 32 accuracy obtained by generating responses under the mixed policy Pmix with fixed number of interventions, after which generation is completed under the primary policy. This can be
viewed as enforcing an intervention budget by setting S = for all such that Σ S, 2 k, where k denotes the maximum number of sams-igg interventions. This setup probes whether early, limited interventions are sufficient to induce downstream performance differences when generation subsequently completed by the primary policy.

---

Published as a conference paper at ICLR 2026
Table 1 summarizzs the number and proportion of cross-sampled tokens required to approximately recover RL-level performance (forward) or collapse to base-level performance (reverse) for Qwen2.5 32B SimpleRL and DAPO AIME 2024 and AIME 2025. Additional sropllngmpling results and discussion, including experiments on additional model configurations and datasets, are provided in Appendix A.5
Forward Cross-Sampling: small fraction of RL tokens suffices recover RL-level performance in the base model.
Forward Cross-Sampling: From Figure and Table forward cross-sampling under the mixed (bL) recovers RI.-level accuracy with remarkably few interventions. Injecting fewer policy 7 mix
than 4% RL-sampled tokens per sequence average, corresponding to fewer than 40 effective token substitutions per response, is sufficient to close the performance gap from the base model (approximately 8%) to the RL model (approximately 25%) AIME 2024. On AIME 2025, the effect even more pronounced: using only 1.53% effective tokens, or 13 average token substitutions per response, boosts accuracy from about 5% over 14%. Interestingly, this level of performance (base.RL.) can outperform exceeds that of the RL policy TRL itself, indicating that the mixed policy *mix the standalone RL policy. This potentially occurs because the mixed policy close but not identical to the RL policy (for ε > 0), which may sometimes avoid failures induced by the RL model. Recovering RL-level performance for DAPO requires larger number of interventions, reflecting its substantially stronger fine-tuned performance. Importantly, even though the performance gains substantially larger, the number of critical token-level decisions remains small relative to sequence length.
TE with Bana Rasamoling 2E. Accuracy 25, 23.52% Bace Accurncy: 8.0%
IL with Bave Ranuempling BL Accuraty: 12.11% Base Accuracy. 4%
22.3
Nennto Rccumdry (32 strpplo
20.0
27.5
10
15.0
12.5
10.0
755
Avcrage Effective Percentage Resampted
Average Parentage Resamoled Toiers
(a) AIME 2024
(b) AIME 2025
Figure 6: Reverse cross-sampling results (Qwen2.5 32B SimpleRL): swapping RL tokens with base tokens in RL generations causes near uear-monotonic degradation toward base performance.
Reverse Samplingm Reverting a small portion of RL tokens collapses performance base levels with the RI. model.
Reverse Cross-Sampling: Reverse cross-sampling shows that the RL policy is highly sensitive to a small number of its token-level decisions. From Figure 6 and Table I we observe that for Qwen2.5-32B with SimpleRL, generating under the mixed policy ग. mix tion of basees tokens is enough to rapidly degrade performance. approximately 5% of high-divergence distributions, corresponding to less than 30 effective base- sampled tokens per response, enough to collapse accuracy from RL levels (around 25%) back to base-level performance (around 8%). This phenomenon also holds on AIME 2025: roughly 4.7% of effective base-sampled tokens (around 30 effective tokens per response) reduces accuracy from approximately 12.7% to below 4%, which is below base-level performance. For DAPO, larger number of substitutions is required to crase its gains, consistent with its substantially stronger per- formance. Importantly, even in these cases, the required reversions constitute small fraction
bRL,) only a small frac-
AIME 2024, reverting
7

---

Published as a conference paper at ICLR 2026
of the total generated tokens, reinforcing that RL-level performance, across both SimpleRL and DAPO, depends critically on a sparse set of token-level decisions.
Notably, the substituted base tokens are mostly locally plausible and semantically reasonable (Figure (43, yet they nonetheless progressively derail the reasoning process. Even when two token choices are locally equivalent interchangeable to human reader, they can induce different downstream conditional distributions and lead to diverging reasoning responses, revealing trajectory sensitivity of the model.
Table I: Summary of cross-sampled tokens required to reach approximate RL-level (forward) or base-level performance (reverse) for Qwen2.5-32B on AIME24 and AIME25 with a token gen- eration budget of 8000. Effective token counts/percentages exclude identity swaps during cross- sampling. Token percentages are computed at the sequence level.
Dataset
Method
Eff. Tokens Eff. Tokens
Initial Acc. (%)
Final Acc. (%)
SimpleRL
3.86%
8.23
25
SimpleRL Reverse
5%
29
25.52
< 8.3
AIME24
DAPO
7.8%
280
8,23
> 44
DAPO Reverse
10.1%
173
41.8
< 8.5
SimpleRL
1.53%
13
5.3
> 14
SimpleRL Reverse
4.73%
31
12.71
<4
AIME25
DAPO
6.47%
230
5
33
DAPO Reverse
9.89%
181
32
< 4.5
FINE-GRAINED MECHANICS OF DISTRIBUTION SHIFTS
Having established the general aspects of RLVR-induced distribution shifts, namely their sparsity, positional concentration, relationship to entropy, and importance to downstream task performance, natural next question is: At token positions where substantial changes occur, how novel are these updates? Do they introduce (effectively) new candidate tokens, or primarily redistribute probability mass among existing ones?
This fine-grained analysis reveals that, even at positions with substantial divergence, current RLVR methods mostly do not fundamentally change the candidate space of predictions. Instead, they primarily reorder and selectively amplify tokens that are already plausible under the base model, with limited promotion of low-probability tokens. We study this behavior through multiple lenses: (i overlap in top-k candidate sets and token rank reordering, (ii) low-probability token behavior, and (iii the evolution over the course training. We further validate these findings across addi- tional datasets, model architectures, and RLVR hyperparameter settings Appendix A.4 demonstrat- ing their robustness beyond the primary experimental configuration.
4.1 TOP-k OVERLAP AND RANK REORDERING
We first investigate whether RLVR changes which tokens are considered plausible, mainly changes how they are prioritized. Concretely, we examine (1) the overlap between the base and RL models' top-k candidate sets, and (2) how the relative ranking of shared candidates shifts.
Figure Sesurerrepen reports the fraction of shared tokens between the top-k sets of the base and RL fine-tuned models, restricted high-divergence token distributions. Despite only considering digerdivereence positions, overlap remains high once k > 2. SimpleRL. exhibits over 80% average overlap (often exceeding 85%), while DAPO shows slightly lower but still substantial overlap. Both methods display sharp increase in overlap from & 1 to & 2, suggesting that while the top-1 token often changes, the replacement was typically already among the base model's top-3.
This observation is further clarified in Figure 17 which shows where the RL model's top-3 tokens appear in the base model's 's ranking, among high divergence positions. Around 30% of RL top-1 tokens already ranked first under the base model, and over 80% (DAPO) and 90% (SimpleRL) fall within the base 3op-3. RL t tokens typically lie within the base top-3-4, with SimpleRL exhibiting consistently stronger alignment.
8

---

Published as a conference paper at ICLR 2026
0.5
Averge Cvottig R200
Average Overtip Rone
0.5
0.6-
0.6
0.-
10
Tap
(a) DAPO: Top-k overlap across thresholds.
(b) SimpleRI.: Top-k overlap across thresholds.
Figure 7: Top-k token overlap between base and RL models at divergent positions (JS > 0.1). Com- puted as the size of the intersection divided by k. High overlap for & 2 2 shows that distributional shifts occur mostly within shared candidate sets.
4.2 LOW-PROBABILITY BEHAVIIO:: DOES RL INVENT OR SELECT?
We next examine whether RLVR promotes tokens that were highly unlikely under the base model, or instead amplifies alternatives that were already plausible but underweighted. For each high- divergence position, we take RL model's top-1 token and record its probability under the base distribution. We then compute the fraction of such tokens whose base probability falls below a given threshold among high-divergence positions. Figure 18lrree8ssww that under DAPO, only about 5% of di- vergent top-1 tokens have base probability below 0.0I, while under SimpleRL this fraction is nearly zero (Figure 20) Thus, even for DAPO, which encourages broader exploration through its clip- higher mechanism and lack of KL regularization, RLVR rarely elevates tokens that were unlikely in the base model. Comparing DAPO variants (Figure 34), we observe that the clip-higher mech- anism substantially increases the fraction of RL top-1 tokens (among high-divergence positions) whose base-model probability was initially very small, relative to the variant without clip-higher. This aligns with earlier observations and supports the interpretation that clip-higher enables greater exploration, allowing tokens that were low probability in the base model distribution to be promoted more frequently. Importantly, although such low-pri pwwbabbbbility promotions remain rare overall, they may still be consequential and important for improved reasoning performance.
4.3 EVOLUTION ACROSS TRAINING
Finally, we analyze how the distributional shifts develop over training. Using intermediate check- points from DAPO training Qwen2.5-Math-77.
track token-level distributions while condi- tioning on the final model's outputs, allowing us to follow the evolution of divergence for a fixed set of token sequences. Figure 29 shows that JS divergence increases monotonically throughout train- ing, with higher percentiles (c.g., 95th and 99th) growing faster than lower ones. This widening gap indicates that distributional change becomes increasingly concentrated in a small subset of tokens, while the majority remain relatively stable. Consistent with this perspective, the Jaccard overlap between each checkpoint's divergent-token set and the final increases gradually before rising sharply near the end of training (Figure 29h).
5 RELATED WORK
RL fine-tuning in LLMs. Reinforcement learning has become an important component of LLM fine-tuning, stemming from reinforcement learning with human feedback (RLHF) used align LLM behavior to human preferences (Christiano et al. 2023 Ouyang al.] 2022). Recently, RLVR has emerged as a central paradigm for improving reasoning by optimizing with verifiable reward signals of generated responses (Lambert et al. 2024). A number of RLVR methods build on policy- gradient variants, including Group Relative Policy Optimization (GRPO) (Sao et al. (024). as well extensions such Dr.GRPO (Liu et al. 2025b), DAPO (Yu et al., 2025), and sequence-level variants such as Group Sequence Policy Optimization (GSPO) (Zheng et al. (025). Beyond these core methods, several works propose complementary improvements target impactful updates and

---

Published as a conference paper at ICLR 2026
stabilize training. including entropy-based perspectives (Wang al. 2025
Cheng al. 2025
clipping/KL regularization strategies (Cui et al.] 2025), reweighting based on token probability, perplexity, or position (Yang al. 2025 Deng et al. (025). well as analyses of different reward designs (Shao et al.
2025). Additional works study the reasoning boundaries of RIVR and how to
expand it (uue et al.
2025 Wen et al. 2025 Liu et al. 2025a).
Understanding RLVR and its differences with SFT. A growing body of work argues that RL, fine-tuning often acts as scalpel rather than a hammer, amplifying existing capabilities through localized changes. This is supported through the perspective of evaluations on different domains/- tasks, catastrophic forgetting, parameter changes, and overall KL divergence (Rajani et al. [025] Chu et al. 2025) Shenfeld et al., 2025 Huan et al. 2025). Recent work also analyze locality from the parameter perspective: for example, Mukherjee et al. (2025) report that RL fine-tuning concen- trates effective updates into small subnetworks, while' Zhu et al. (2025) provide theoretical insight into RLVR learning dynamics and the structure of these parameter-sparse updates. Our paper com- plements these perspectives by focusing on quantifying the changes induced by RLVR at the level of token distributions. We show that RL not only induces smaller aggregate divergence than SFT, but that its changes are substantially sparser at the token level.
Toven-Level analyses of RLVR. Several works seek to understand RLVR through token-level lens. Waan et al. (2025) attribute a substantial portion of RL gains to high-entropy minority tokens, while (Cheng et al. (2025) connect such tokens to exploratory reasoning steps; related work also highlights entropy collapse risks and token-level regularization mechanisms (Cui et al. 2025). Other reguullaraa
studies emphasize the disproportionate role of specific tokens sampling decisions (Vassoyan et al.
Lin et al. 2025; Karan & Du 2025). and Huan et al. (2025) analyzes RL-induced changes using teken-level KL divergence and token rank shifts. Recent work by Chen al., (2026) shows that RL training modifies a sparse subset of tokens when viewed through rank-shift statistics, and develops theoretical analysis based on reasoning patterns. Our contributions are complementary but distinct: we conduct systematic empirical study of RLVR-induced token-level changes through the lens of quantities such divergence, entropy, and probability mass redistribution, and connect these shifts to sequence-level reasoning via ross-sampling interventions that establish their impact on reasoning performance.
2025
6 CONCLUSION
Our study reveals that reinforcement learning with verifiable rewards (RLVR) reshapes LLMs in a manner that is sparse, targeted, and structured rather than uniformly diffused across tokens. By analyzing token-level distributional shifts, we show that only small subset of tokens undergo meaningful divergence, and that these divergences carry disproportionate functional importance: cross-sampling interventions confirm that performance gains hinge on precisely these positions. Moreover, our fine-grained analyses suggest that, even at high-divergence positions, RLVR typi- cally refines behavior by reallocating probability mass within an existing candidate set rather than introducing fundamentally new tokens. At the same time, the comparatively rare cases of substantial re-ranking and promotion of initially low-probability tokens may still be important for the observed improvements in reasoning. To complement these analyses, we explored seivergence-wiighee vantage, a simple modification that scales token-level advantages by per-token divergence. These results suggest that such weighting strategies can influence learning dynamics, though stabilizing performance may require model-specific choices and further investigation.
ad-
Together, these findings advance a token-level understanding of RL fine-tuning. They highlight that the essence of RLVR's success lies not in widespread distributional changes, but in selective refine- ments aligned with varying entropy levels. Taken together, our findings suggest that RLVR operates not as global policy shift, but as sparse intervention that steer generation trajectories largely accessible to the base model. Beyond clarifying the me- chanics of existing methods, our work offers perspective for designing future RL objectives and diagnostics that explicitly incorporate distributional structure, opening avenues for more effective, interpretable, and controllable LLM post-training.
small set of high-impact decision points
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Xingwu Chen, Tianle Li, and Difan Zou. Reshaping reasoning in LLMs: A theoretical analy- of RL training dynamics through pattern selection. In The Fourteenth International Confer- ence on Learning Representations, 2026. URL http:ss//oppneeeeee....eee www.ueeee.ooruummmmmm
200399hRD6
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective on reinforcement learning for Ilms, 2025,
14758
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. URL
http//axxvvvvoreeeee www.:oorgaaa.. /
770600374)
Tianzhe Chu, Shengbang Tong, Jihan Yang, Tianzhe Chu, Yuexiang Zhai, Yi Ma, Saining Xie, Dale Schuurmans, Quoe V. Le, and Sergey Levine. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025. Jtttssss:://axxi org/ abs/ http://aaaa....raaaa//0/........ 17161
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan. Huayu Chen, Weize Chen, Zhiyuan Liu, Hao Peng, Lei Bai, Wanli Ouyang. Yu Cheng, Bowen Zhou, and Ning Ding. The entropy mechanism of reinforcement learning for reasoning language models, 2025. http::ttt///aaae
www.orrgggaao........... abs/ 2bbp///...26117777
Jia Deng, Jie Chen, Zhipeng Chen, Daixuan Cheng, Fei Bai, Beichen Zhang, Yinqian Min, Yanzipeng Gao, Wayne Xin Zhao, and Ji- Ji-Rong Wen. From trial-and-error to improvement: A systematic analysis of Ilm exploration mechanisms rlvr, 2025.
abs 07534
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2020. URL
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue. Does math reasoning improve general 11m capabilities? un- derstanding transferability of Ilm reasoning, 2025, URL
00432
Aayush Karan and Yilun Du Reasoning with sampling: Your base model is smarter than you think, 2025.
Schoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder, 2023. URL https:
//arxiv.org/ababbbb httpaaa/2220200.....33
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T)"" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. 2023. URL https: www.i.iiiiv.///a.//aaaaaaaaaaava org/ abs/ 22112111111 17192
Zicheng Lin. Tian Liang, Jiahao Xu, Qiuzhi Lin, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li. Yujiu Yang. and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhances 11m's 's reasoning capability, 2025. URLphtpss //arxi v. www.aas.///...oooooo..o........e /411 19943
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models, 2025a. / / wttpaawwwwww../////////.....

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and
Min Lin. Understanding rl-zero-lit like training: critical perspective, 2025b. URL https:
MistralAI. Mistral small
3, January 2025.
URL attps://mistral.all
mistral- pestral-small-- small-3
Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur and Hao Peng. Reinforcementteaamiiee finetunes small subnetworks in large language models, 2025. URL./httpe::/arrriv.or//////////
71711
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022. URL httphtt/////xxi..cooom w/tp://aavvvv.vo org/ abs/ 2203. 02155
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang. Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. abs 21.42.111111
URL https:
Neel Rajani, Aryo Pradipta Gema, Seraphina
and Ivan Titov, Scalpel vs. hammer: Grpo amplifies existing capabilities, sft replaces them, 2025. URL https://arxxi> org/
Abs/2507.10615
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A grae google-proof q&a benchmark, 2023. URL // /awxiviivv.. httggaab//331..222
John Schulman. Approximating kl divergence. html March 2020. Blog post.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, abs/1707.06344 2017. doi: 10.48550/rrXiv.1707.6664/ URL
https::/arxivv org/ abs// 711.117770000 0001///00/00003343777
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi. Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr, 2025. URL
https: //tp:i//aaiivvvvaa wwt.//rg/////ooooooo.oo/o.e 2tloo//2066.19900
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Y. K. Li, Yu Wu, Daya Guo, and Mingchuan Zhang. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL URLtttts:://aatt arxiv org http://b//2442..333
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RI's razor: Why online reinforcement learning
forgets less, 2025. http:t::: www.aaivvv.oo. orgg abs 2509.640. 04259
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint 2024.
arXiv:
Jean Vassoyan, Nathanaël Beau, and Roman Plaud. Ignore the kl penalty! boosting exploration on critical tokens to enhance rl fine-tuning, 2025. URL
06533
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang. Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning, 2025. URL https:/ //tp///aavvvv//////////ea/.wwv/v org/oog//aa
25066 01939
12

---

Published as a conference paper at ICLR 2026
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base 1lms, 2025. URL https://arxiv: org/abs/ 2506. 14245
Taiqiang Wu, Runming Yang, Jiayi Li, Pengfei Hu, Ngai Wong, and Yujiu Yang. Shadow-ft: Tuning instruct via base, 2025. URL htt.t: http://www.v...//////.......00..
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical ex- pert model via self- saff-iiproveeen,, 2024. URL https: //apiivv///xvv...// org/ abs 2409. 12122
Zhihe Yang. Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. Do not let low- probbaibiyy tokens over-dominate in for llms, 2025. URL https: //apxiv///rxvv.. org/aba/2505. 12929
Qiying Yu, Zheng Zhang. Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya- Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL httphtt:: http://xxvvvv..r/ wrg..bs///ab.......
14476
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model?, 2025, URL httts::/ /ttp:://aaxiv..o org/ /250. 250.//5353.11.7
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models the wild, 2025.
URL httphhtpp///aaxttt wwwprxiwv.v.rr.e.. http://.rrrrbbbb///bbeb abs/ 1015500.1888221 18992
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization, 2025. URL
org/ aaaaab/5/111 18071
Hanqing Zhu, Zhenyu Zhang, Hanxian Huang, DiJia Su, Zechun Liu, Jiawei Zhao, Igor Fedorov, Hamed Pirsiavash, Zhizhou Sha, Jinwon Lee, David Z. Pan, Zhangyang Wang, Yuandong Tian, and Kai Sheng Tai. The path not taken: Rivr provably learns off the principals, 2025. URL
https://arxiv orggabs/25111 08567
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 EXPLORATORY INVESTIGATION, DIVERGENCE- WEIIEEEDD ADVANTAGES
Our earlier analyses reveal that RL refinements are sparse and targeted, with only a small subset of tokens exhibiting meaningful distributional change. Moreover, cross-sampling experiments demon- strate that these nigh-divergence tokens are functionally critical, with performance gains hinging on precisely these positions. This raises a natural question: only small fraction tokens drive improvements, can training be more effectively guided by modulating loken-level learning signals according to these divergences? To investigate this possibility, we conduct a preliminary exploration of divergence-weiigheed advantages as diagnostic intervention, where ad advantages are reweighted by token-level distributional divergence. We explore two different approaches: high-KL. boost, which concentrates updates towards token distributions that are already changing substantially, and low- KL boost, which focuses updates on distributions that have changed less, potentially encouraging updates in previously stable regions.
A.1.1 BRSODBASEDD METHODS
GRPO in brief. GRPO (hao al. 2024) samples G responses {o:) from a policy Te ( g)
for prompt q with ground-truth answer assigns sequence-level rewards { )= and computes a group-normalicn advantage for each sample. GRPO then applies a PPO-style (Schulman et al.) 2017) clipped surrogate objective at the token level, typically with explicit KL penalty to refer- ence model.
DAPO. DAPO Yu et al. 2025) modifies GRPO with an asymmetric clip-higher mechanism, dy- namic sampling of correct/incorrect completions, token-level averaging, and removal of the explicit KL penalty term. Its objective is given by
(0) = [(q,))D, (
G lod
min (r::(0) A clip(rua(0), flow: 1 thight A.:.)
(I)
0 iml t=1
s.t. 0
Ho: is.equivalont((.o)) G,
with
To(oi. g, Oi,ce) To (Oi,e q, Oc,ct))
R; - mean( moan((R)) (R)) A
(0)
stal (1
A.1.2 UIVERGENCE-EIIGHTDD ADVANTAGE
Standard RLVR objectives treat all tokens within a sequence uniformly in terms of their advantages (though the importance sampling ratios are defined on the token-level). Motivated by our observation that distributional shifts sparse and concentrated, advantages according to divergence magnitude can help improve or control aspects of training. We explore modifications where advantages are rescaled depending on the per-token divergences.
investigate whether modulating token-level
General formulation. We define divergence- advantage:
A We A
where Å, denotes the standard group-normalized advantage and we is per-token weight based divergence. To ensure that the introduced divergence weight influences only the weighting, diver- gences are detached from the computation graph.
Choice of divergence. For ease of compatibility with standard frameworks, employ KL. diver- gence with respect to the old policy as our primary divergence quantity:
KT,cle old
DKL. ( ( siie)) || no( (((e))
14

---

Published as a conference paper at ICLR 2026
where Te denotes the policy from the previous update iteration, in PPO/GRPO. This old-policy KL quantifies the magnitude of recent policy updates at each token position, serving as a proxy for the extent of local distributional change. For computational efficiency and compatibility with existing training frameworks such as verl (Sheng et al., 2024). we estimate these quantities using KI estimators computed over sampled tokens only, which may not capture the full distributional structure, In particular, we use the K3 estimator (Schulman 2020) defined by
To xee)) )
TTp-1 xx())
TT(: Tel)
TT(- x(() Ti
KT KT (To To) Ra kg
log
To
xes)
Fett
Alternative divergences signals, including the divergence to the base model (capturing global shifts), discussed in Appendix A.7
Weighting scheme. We adopt simple sigmoid weighting scheme (to ensure bounded weights), which transforms divergence into weights through:
W/ 1 (o(a KT.,) 2), (xx)
Ite-
The parameter a controls the direction and magnitude of emphasis: >> amplifies high-divergence tokens, whereas a < 0 emphasizes low-divergence ones. The sigmoid function provides smooth, bounded nonlinear transformation that enables selective focus on either high- or low-divergence regions depending on the sign of a. This formulation allows us to investigate whether concentrating the learning signal on regions that have already changed or those that remain unchanged yields more effective training dynamics. Alternative weighting schemes, including linear relative weighting, dicussed Appendix A.7
Evaluation. We train with divergence-weighteed advantages using the DAPO training recipe and data on 0wen2.5-Math-77 evaluating on AIME 2024, AIME 2025, and AMC. Results are pre- sented in Table Detailed training hyperparameters and implementation details are documented in Appendix A.2.3
Table 2: Accuracy (%) under divergence-weighteed configurations on Qwen2.5-Mathe Results shown for KL divergence with Tew and sigmoid weighting scheme across AIME 2024, AIME 2025, and AMC datasets. The results displayed are the Mean @ 32 scores (or the pass @ scores computed using 32 samples). Results are each averaged over 3 runs.
Configuration
AIME24
AIME25
AMC
Overall Avg
Baseline DAPO
33.61
18.75
75.08
42.48 = 1.35
Low-KL WLW boost
35.90
19.90
78.97
44.92 = 0.05
High-KI. boost
36.74
20.00
78.40
45.05 + 0.79
These results demonstrate that weighting token-level updates by divergence can amplify perfor- mance gains, providing empirical support for the hypothesis that targeted tokens disproportionately drive improvements. Both low-KL and high-KL boost configurations yield improvements over the baseline, suggesting that different divergence weighting strategies can be effective. However, the op- timal choice between these approaches, and indeed whether divergence weighting provides benefits at all, may depend on the specific models and training methods used. Effective divergence weighting across training configurations may require model-specific paradigms or adaptive scheduling mech- complementary diagnostic
anisms to stabilize learning dynamics. We present this approach tool that may inform future refinements of token-level training strategies. These include approaches that aggregate information from token-level divergences, more effectively promote the rare actions discussed in Section and directly modify the clipping mechanism.
A.2 EXPERIMENTAL DETAILS
A.2.1 TOKEN ANALYSIS
Models and Datasets. Our primary analysis focuses on Qwen2.5-32B (Qwen et al., (0225) as the base model, with RLVR variants trained using DAPO Yu et al. 2025 and GRPO, the latter paired
15

---

Published as a conference paper at ICLR 2026
with the corresponding SimpleRL model ZZng al. 2025
For evaluation AIME 2024 and ALME 2025, we sample 32 responses per problem for robustness. We further extend the analysis to (Yang et al., 2024) with two variants corresponding to differ-
additional models (Qwen2.5-Math- ent upper clip settings, and Mistral-Small-241 (MistralA] 2025)), datasets (AIME25, GPQA (Rein al., 2023). and the models' respective fine-tuning datasets), and comparisons with supervised fine-tuning (SFT). These extensions, reported in Appendix [A.3 and Appendix A.4 confirm that our findings generalize across models, datasets, and training paradigms.
We run model inference using vl 1m (Koon et al. 2023). On AIME, we apply nucleus sampling For divergence calculations on AIME,
(Holtzman et al. 2020) with topp 0.7 and temperature we use the top-p truncated distribution to reflect the effective sampling distribution, to provide more accurate estimate for our voss-sampling experiments. We also examine the distribution of IS divergence values without truncation (and on other topp values) to ensure the main results are not impacted by the truncation. For experiments on the fine-training data, we use topp reflect the training sampling distribution.
A.2.2
For cross-sampling experiments, we use the same inference setup as token analysis. Cross-sampling experiments selectively swap tokens between base and RL models positions where JS divergence exceeds a threshold, allowing us to measure the functional importance of divergent token distribu- tions.
We perform forward and reverse cross-sampling experiments on the following model-dataset combi- nations. For forward cross-samplle sampling, we inject Ra-pledeed tokens into base generations at positions where JS divergence exceeds the specified threshold. For reverse cross-sampling, we replace RL to- kens with base tokens at high-divergence positions on the RL generations. The divergence thresholds used for each configuration are follows:
Qwen2,5-32B + SimpleRL:
AIME 2024: Forward threshold E = 0.03, Reverse threshold E = 0.05
AIME 2025: Forward threshold E = 0.05, Reverse threshold E = 0.05
Qwen2.5-32B DAPO:
AIME 2024: Forward threshold 8 0.08, Reverse threshold & 0.06
-AMME 2025: Forward threshold E 0.1, Reverse threshold 0.08
Mistral-Small-24B + SimpleRL:
AIME 2024: Forward threshold e = 0.002, Reverse threshold 0.02
A.2.3 ADDITIONAL TRAINING DETAILS
We implement RLVR training experiments using verl (Sheng al. 2024) with the standard DAPO recipe YYu et al. 2025).
Qwen2.5-Math-7B DAPO Training. We follow the public DAPO recipe, namely with clip ratios Cjow = 0.2 and Chigh = 0.28. However, for token analysis, we also train a variant with eiih = 0.2 for comparison. We optimize with learning rate 1 x 10-6, 10-step warmup using Adam W, and no explicit reference-KL penalty. Each RLVR step processes 512 prompts with 16 sampled responses per prompt; these are split into mini- batchbschess of 32 prompts, yielding 16 gradient updates per RLVR step. Maximum generation length and the perrlong-penalle threshold are set to 8k and 4k tokens.
Supervised Pine-Tuning (SFT) Training. For the SFT model based on Qwen2.5 32B, we sampled 42k instances from the AM--eepSSee-RR1-1 derwent full parameter fine-tuning for 5 epochs, employing DeepSpeed ZeRO-3 optimization.
1ed-11 4M dataset. The model un-
Divergence-weighted Training For the divergence-wwighhee Math-7B, under the high-KL setting we use 0.3 and set to increase linearly from 0 to 50 starting at step 100. In the low-KL setting, we use = 0.3 and set a to increase linearly from 0 to 50, which we linearly increase beginning step 150.
advantage experiments on Qwen2.5-
16

---

Published as a conference paper at ICLR 2026
For Qwen2.5-7B. in the high-Ku KL relative setting set 4. In the configuration with addi- tional scheduler, we initialize a = 2 and linearly increase it to 3 from step 80 onward.
A.3 RLVR Vs. SUPERVISED FINE-TUNING: CONTRASTING DISTRIBUTIONAL PATTERNS
A natural question is whether the sparse, targeted distributional shifts we observe are specific to RLVR, if they also characterize other fine-tuning approaches. To address this, compare RLVR- trained models with models refined through supervised fine-tuning (SFT). We analyze Qwen2.5-32B trained with SFT alongside Qwen2.5-32B DAPO.
Figure shows JS divergence distributions for both approaches. SFT produces a noticeably larger high-divergence set, whereas RLVR concentrates almost all token distributions below very small values. This directly reflects RLVR's extreme selectivity and the broader edits introduced by SFT. The top-k overlap analysis (Figure[I) highlights that SFT consistently achieves lower overlap with the base model, indicating more aggressive re-ranking, while RLVR largely stays within the base model's existing candidate set. The rank reordering analysis (Figure [2) further shows that SFT promotes many more tokens far outside the base model's 's top-3, whereas RLVR mainly promotes candidates that were already high-ranked.
0.7
10 5000
Reporioon
50h 4343
0.2
Divergence
(a) SFT: Histogram
(b) SFT: Percentiles
10
0.7
2339 01822
0.6
20730 2.623
10
10
FCe 0050
15 Divergance
(c) RLVR (DAPO): Histogram
(d) RLVR (DAPO): Percentiles
Figure 8: JS divergence distributions comparing SFT and RLVR on AIME 2024. RLVR exhibits even sparser distributional shifts than SFT, suggesting more targeted refinement.
Positional analysis further shows that SFT induces elevated divergence across the entire response, while still exhibiting increased divergence near the start of the sequence (Figure I0), mirroring, the carly-position effects seen in RLVR. Finally, under the divergence-entropy analysis (Section SFT's divergent tokens concentrate more strongly in regions of high base-model entropy (compared with DAPO). While this concentration may be partially influenced by SFT outputs appearing more uncertain when evaluated under the base model, the resulting fine-tuned entropy values are never- theless substantially lower than those of the base model (Figure 13 These results are consistent with SFT's objective of directly learning target outputs, leading to globally broader and sharper distributional updates.
Taken together, the metrics highlight that SFT diverges from RLVR along several axes. The SFT model exhibits higher JS divergence overall as well as a larger mass high-divergence tokens (Figure 8), and attains lower top-k overlap with the base model (Figure IT] alongside larger rank shifts (Figure 12). These differences reinforce that RLVR acts as a targeted editor, while SFT drives broader, less selective reshaping of the distribution.
17

---

Published as a conference paper at ICLR 2026
Pertettgge Ohegper Teers 183
40
Precetor of Oivegen Sowens Pb)
10
40
20
10
10
10
Merarau Ibas Pobebiiity (lnalad))
3arrance (box Prububility Thalldl)
(a) Qwen2.5-32B SFT
(b) Qwen2.5-32B DAPO
Figure 9: Percentage divergent tokens whose RL top-1 choice had base probability below a given threshold comparing SFT and models on AIME 2024.
0.1
0.200
Reange S Divergence
0.175
0.150
0.125
0.56
0.100
0.075
0.24
0.050
0:2
0.00
20
60
103
Sequence
Rcladiee Sequaice Postdon (H))
(a) Qwen2.5-328 SFT
(b) Qwen2.5-32B DAPO
Figure 10: Mean JS divergence by normalized token position comparing SFT and RLVR- trained models on AIME 2024.
Aegogg Operrig 500
0.50
0veraat Oxerdi R00
0.4
0.35
10
Tpkk
Tep-
(a) Qwen2.5-32B SET
(b) Qwen2.5-32B DAPO
Figure 11: Top-k token overlap between base and refined models at divergent positions (JS; > 0.1) comparing SFT and RLVR-trained models on AIME 2024.
18

---

Published as a conference paper at ICLR 2026
Ar be: of
028
(a) Qwen2.5-32B SFT
(b) Qwen2.5-32B DAPO
Figure 12: Distribution of base-model ranks for refined models' top-3 tokens high-divergence
positions (JS >> 0.1) comparing SFT and RLVR-trained models on AIME 2024.
0.7
Base Entropy
0.5
0.5
Proporion
0.4
0.3
0.3
0.2
0.1
0.0
0.0
1.0
2.0
Uase Model Entropy
Entropy Value
(a) Qwen2.5-32B SFT Low IS bin (< 0.1)
(b) Qwen2.5-32B SFT High JS bin (>> 0.1)
Figure 13: Entropy distributions across divergence bins for Qwen2.5-332 5-32BB SET on AIME 2025.
0.7
RMB 10001
10 0.8181
0.3
0.2
0.1
6.0
(a) Qwen2.5-32B SET AIME 2025: Per-
centiles
Figure 14: JS divergence distributions for Qwen2.5-32B SFT on AIME 2025. Consistent patterns with AIME 2024 demonstrate robustness across datasets.
19

---

Published as a conference paper at ICLR 2026
A.4 ADDITIONAL TOKEN DISTRIBUTION ANALYSES
This section provides supplementary and extended token distribution analyses. We first present supplementary figures for the main models (Qwen2.5-32B with DAPO and SimpleRL. AIME 2024), then extend the analysis to additional models and datasets to demonstrate the generalizability of our findings.
A.4.1 SUPPLEMENTARY FIGURES FOR MAIN MODELS
We provide additional figures for Qwen2.5-32B with DAPO and SimpleRL that complement the analyses in the main text.
8:800
10a
10
Proportion
Proportion
10 2
10
10-4
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.0
0.1
0.2
0.3
0.4
J5 Divergence
J5 Divergence
(a) DAPO: Histogram (log y-axis)
(b) SimpleRL: Histogram (log y-axis)
Figure 15: JS divergence distributions for Qwen2.5 32B DAPO and SimpleRL on AIME 2024.
0.08
Entropy
0.6
Ieportion
8.08
0.2
0.00
0.0
0.5
1.5
2.0
2.5
3.0
0.0
0.5
2.0
Sase Modal Entrapy
Entrogy Value
(a) Low JS bin (< 0.1).
(b) High JS bin (> 0.1).
Figure 16: Entropy distributions across divergence bins for SimpleRL. Low-divergence tokens are mostly low-entropy. while high-divergence tokens are concentrated in higher-entropy regions, flecting a more conservative update strategy.
To supplement the positional analysis in the main text, we also examine localized averages of JS divergence near the start of the generation and near the final answer span.
Results on GPQA-Diamond. We extend our analysis to DrQa--Damond to demonstrate the gen- eralizability of our findings across different reasoning benchmarks. Figure 24 shows JS divergence percentile curves and positional concentration for Qwen2.5 with DAPO on GPQA-Diamond, revealing consistent sparsity patterns. Figure 25 shows entropy distributions across divergence bins.
20

---

Published as a conference paper at ICLR 2026
050
koz
Meice Distiro
(a) Qwen2.5 32B DAPO.
(b) Qwen2.5 32B SimpleRL.
Figure 17: Distribution of base-model ranks for RL's top-3 tokens high-divergence positions
(JS > 0.1). Most RL-selected tokens were already highly ranked in the base model, especially under SimpleRL.
Hean: 0.5285
30
1000 togo Overgon biers (861
25-
15
0.02
0.01
15->
10-1
10-1
02
0.4
0.6
Beeracee Bose Probuciity Thrasholi
RL NL.hobblll Bly
(a) Percentage of divergent tokens with low base probability.
(b) Histogram of RL probabilities for low base- probability tokens.
Figure 18: Analysis of tail behavior under DAPO for divergent token distributions (JS > 0.1). (a) shows the percentage of divergent tokens whose RL top-1 choice had base probability below given threshold. (b) shows the distribution of RL probabilities for the subset with base probability < 0.01.
21

---

Published as a conference paper at ICLR 2026
144
244
144
12
01
(a) Frequent high IS tokens.
Frequent low tokens.
Figure 19: Histogram of divergences for frequent high JS tokens and frequent low JS tokens (Qwen2.5 32B with DAPO).
Percentage Divergent Tokend 196)
4-
10
-0
10a1
Probabilty Threshold)
(a) SimpleRL
Figure 20: Percentage of divergent tokens whose RL top-1 choice had base probability below given threshold.
22

---

Published as a conference paper at ICLR 2026
(a) DAPO.
(b) SimpleRL.
Figure 21: Probability differences and ratios for top-3 tokens under DAPO and SimpleRL among divergent distributions (JS > 0.1).
Ciargeeco at Dimaase fewm Goserttoon
[5 Draggaazee ve Seteve free Aooee
6:33
fice Couesto
Deeee lover
wet Feme Conention
(a) Near the start of generation
(b) Near the answer span
Figure 22: Local averages of JS divergence as a function of distance from key regions (prompt beginning and answer) for Qwen2.5-32B models on AIME 2024. Average divergence peaks occur in the same early and late windows highlighted by the positional analysis.
23

---

Published as a conference paper at ICLR 2026
Sequence (f-1743)
(h-
[0V.5.77
20
25
1.5
Divergonce
Sequeence (I-14399)
Sequence (n-16614)
2.5
7.0
0.5
0.5
0.4
0.00
0.04
0.25
Dlvergaace
D.vheegnnce
(a) Qwen2.5-32B DAPO
(b) Qwen2.5-32B SimpleRL
Figure 23: Per-sequence scatter plots relating entropy to JS divergence for Qwen2.5-32B DAPO and SimpleRL on AIME 2024. DAPO exhibits a broader entropy spread among divergent tokens, whereas SimpleRL concentrates divergence higher-entropy regions.
0.14
Average J5 Divergence
0.5
9R 7501 0.6660
0.10
0.4
0.06
0.04
050h 0 143
0.1
0.0
0.00
20
40
60
80
100
20
40
60
80
100
Sequence Position
(a) JS divergence percentiles
(b) Positional concentration
Figure 24: JS divergence analysis for Qwen2.5-32B with DAPO on GPQA-Diamond. The sparsity patterns and positional concentration are consistent with findings AIME datasets.
0.7
Base Entropy
Entropy
0.5
0.05
Proorsion
0.1
0.3
0.03
0.2
0.02
0.0
0.00
1.5 Base Model Entropy
3.0
3.5
Entropy Value
(a) Low JS bin (< 0.1).
(b) High JS bin (> 0.1).
Figure 25: Entropy distributions across divergence bins for Qwen2.5-32B with DAPO on GPQA- Diamond. Patterns consistent with those observed on AIME datasets.
24

---

Published as a conference paper at ICLR 2026
Effeet of Top-p Sampling on JS Divergence. To verify that findings are robust to different top-p sampling settings, we compare JS divergence distributions across different sampling configu- top-p 0.7 for sampling. We also evaluate configurations where
rations. The default setting sampling is performed with top-p = 0.8 and top-p = 0.9. Figure F shows that the sparsity patterns remain consistent across different sampling top-p values, confirming that our results not sensitive to the specific sampling top-p value used.
0.7
07
0.4
0.4
0.3
0.3
11910
Feroent
(a) Sampling top-p 0.8
(b) Sampling top-p 0.9
Figure 26: JS divergence percentile curves for Qwen2.5-32B with DAPO on AIME 2024 under different top-p sampling settings. The sparsity patterns remain consistent across different sampling top-p values, indicating robustness to the specific sampling configuration.
JS Divergence on AIME 2025. Figure 27 shows JS divergence percentile curves for Qwen2.5- 32B with DAPO and SimpleRL AIME 2025, demonstrating consistent sparsity patterns across datasets.
0.5
90 10n 418
3.4
0.4
0.3
0.3
76 100 0330
0.2
0.2
P7 160
500 1000
40
60
50
100
20
40
60
80
100
Percentile
Percent
(a) DAPO: Percentile curve
(b) SimpleRL: Percentile curve
Figure 27: JS divergence distributions for Qwen2.5-32B with DAPO and SimpleRL on AIME 2025. The sparsity patterns consistent with those observed AIME 2024, confirming the robustness of our findings across datasets.
Effect of Top-p Truncation on JS Divergence. To verify that our use of top-p truncated distri- butions (with topp 0.7) does not significantly impact our findings, we compare JS divergence distributions computed using the estimated full distribution (top-p 1) with those using truncated distributions. Figure 28 shows that the patterns remain consistent: distributional shifts are highly sparse regardless truncation, with the vast majority tokens showing near-zero divergence.
25

---

Published as a conference paper at ICLR 2026
3.7
1377 0.323
130 0853
3.6
0.6
Khowngere
29 35th 33343
01
SORR 0000
3.
(a) DAPO: Percentile curve (toppl)
(b) SimpleRL: Percentile curve (toppl)
Figure 28: JS divergence distributions computed using top-p 1 for Qwen2.5-32B with DAPO and SimpleRL on AIME 2025. The sparsity patterns are consistent with those observed using top-p truncated distributions, confirming that truncation does not significantly impact findings.
0.175
90th percentle
0.150
95th percentlle
0.8
percentlee
15 15 gence
0.000
90ch peceriilee
prccentlle
0.
0.075
04
0.050
50
130
25
50
150
175
200
Checkpoim Step
Checkpoint Step
(a) JS divergence percentiles.
(b) Jaccard index with final divergent set (JS, > 0.1).
Figure 29: Distributional shifts grow increasingly focused and stable. Most tokens remain un-
changed; updates concentrate in a sparse set late in training.
26

---

Published as a conference paper at ICLR 2026
A.4.2 COMPARISON OF DAPO VARIANTS: CLIP-HIGHER SETTINGS
DAPO's clip-higher mechanism controls the degree of exploration during training. We compare two Qwen2.5-Mann --tthh-BB models trained with DAPO: one with the default clip-higher setting (0.28) and another with more restrictive setting (0.2). Figure 30 shows their JS divergence distributions on AIME 2024 and AIME 2025, revealing how the clip-higher parameter affects distributional shifts across datasets.
3.7
9.7
3.6
0.6
3.5
0.5
30 70 3330
3.8
40
(a) DAPO (0.28) AIME 2024: Percentiles
(b) DAPO (0.28) AIME 2025: Percentiles
0.6
3.5
05
0.4
D 41338
0.4
PDth 1034
FSh 6643
SCB 0000
80
100
Pe Percentle
(c) DAPO (0.2) AIME 2024: Percentiles
(d) DAPO (0.2) AIME 2025: Percentiles
Figure 30: JS divergence distributions for Swen2.5-Mathh trained with DAPO under different clip-higher settings on AIME 2024 and AIME 2025. The more restrictive dipphighhr=0022 setting leads to sparser distributional shifts compared to the default 0.28 setting across both datasets, with a smaller proportion tokens exhibiting nonnegligible divergence. However, on its divergent token set, the JS values are higher as indicated by the higher upper percentiles.
Figure compares positional concentration patterns on AIME 2024 and AIME 2025, while Fig- 32 and Figure examine top-k overlap and rank reordering, respectively. Figure 34 shows the percentage of divergent tokens whose RL top-1 choice had base probability below a given thresh- old for both DAPO variants across different datasets. Figure 35 shows entropy distributions across divergence bins for both DAPO variants.
Comparing the upper clip variants (Figure 31, both clip settings exhibit larger average diver- gences at the beginning of the sequence, with a smaller increase near the end, consistent with the behavior seen in the 32B models. Interestingly, the 0.2 clip setting shows higher average divergence at the beginning of the sequence compared to the 0.28 setting.
For small clip-higher DAPO yields lower average overlap with the base model's top-k set than the variant without clip- (Figure 32). indicating more frequent changes among the highest-ranked tokens. Interestingly, this trend reverses for larger k, where the model trained without clip-higher exhibits smaller overlap, suggesting that agreement with the base model deteriorates in the lower portion of the candidate set.
A consistent picture emerges from the rank-shift analysis in Figure 33 Without clip-higher, the base-model ranks of the RL model's top-3 tokens are more often preserved, but a non-negligible fraction of promoted tokens originate from much lower base ranks compared to the clip-higher variant. Taken together, these results suggest that clip-higher primarily redistributes probability mass within already plausible candidate set, whereas removing tlip-higher tends to strongly amplify small number of existing top tokens. At the same time, some tokens promoted into the RL top ranks originate from much lower base ranks, particularly without clip-higher; however, given the lower
27

---

Published as a conference paper at ICLR 2026
entropy observed without clip-higher, these tokens may still carry relatively small probability mass overall, with most probability concentrated on the top one or two tokens.
500
(a) DAPO (0.28) AIME
(b) DAPO (0.28) AIME 2025
(c) DAPO (0.2) AIME
(d)
DAPO (0.2) AIME
2024
2024
2025
Figure 31: Mean JS divergence by normalized token position for DAPO variants with different clip- higher settings AIME 2024 and AIME 2025.
Aectogo Dheeht 8300
Aeteagh Oveett 5o0e
018-
0.5
0.5
0.03
0.0
0.55
0.7
0.15
2 x 0:35
Tpp
Thp-
(a) DAPO (0.28) AIME 2024
(b) DAPO (0.28) AIME 2025
050
0.50
B 0.01
5::01
0.55
R
0.50
00000g Overiip Rote
0.75
40000 Ovente
0.75
0.05
0.65
0.00
0.50
0.50
(c) DAPO (0.2) AIME 2024
(d) DAPO (0.2) AIME 2025
Figure 32: Top-k token overlap between base and RL models at divergent positions (JS; > 0.1) for DAPO variants AIME 2024 and AIME 2025.
Fine-tuning Data Results. We also analyze distributional shifts the fine-tuning data to examine how models behave on data they were fine-tuned on. Figure 36 shows JS divergence distributions, while Figures 37 38 show additional analyses.
28

---

Published as a conference paper at ICLR 2026
(a) DAPO (0.28) AIME 2024
(b) DAPO (0.28) AIME 2025
Aub.11h
C2T.41 Dase
1
(c) DAPO (0.2) AIME 2024
(d) DAPO (0.2) AIME 2025
Figure 33: Distribution of base-model ranks for RL's top-3 tokens at high-divergence positions
(JS > 0.1) for DAPO variants AIME 2024 and AIME 2025.
29

---

Published as a conference paper at ICLR 2026
2.35
Temwre
(a) AIME 2024
(b) AIME 2025
(c) Training Data
(d) AIME 2024
(e) AIME 2025
(f) Training Data
Figure 34: Percentage of divergent tokens whose RL top-1 choice had base probability below given threshold for Qwen2.5-Math-77 with DAPO variants. Top row: DAPO (clip-iigher-0.255 bottom row: DAPO (clip-higher=0.2), We further observe a distinction between the two clip-high settings, with the more restrictive setting (0.2) promoting substantially fewer tokens with low base probability.
Rase Entropy
0.06
Entropy
0.5
Properion
0.3
0.03
0.2
0.0
0.00
1.0
1.5
Base Model Entropy
Entropy Value
(a) Low JS bin (< 0.1)
(b) High JS bin (>> 0.1)
0.6
0.0
Entropy
0.5
RiL Entropy
0.4
0.4
0.3
0.2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Model
Entropy
Low IS bin 0.1)
High JS bin (> 0.1)
Figure 35: Entropy distributions across divergence bins for Owen2.55Mathh with DAPO variants on AIME 2025. Top row: DAPO (clip-higher=0225) bottom row: DAPO (clip-higher=0.2), Patterns are consistent with those observed in the main text, confirming the relationship between entropy and divergence across different clip-higher settings.
30

---

Published as a conference paper at ICLR 2026
100% 00237
B.eegerr
Dleegerre
0.4
0.1
0.3
0.3
0.2
0.2
60h 3 1990
0.1
60
50
1.0
20
40
60
50
100
Percentlee
Percentiee le
(a) DAPO (clip-tigher=0.28) Percentiles
(b) DAPO (clip-ligher=0.2)) Percentiles
Figure 36: JS divergence distributions for DAPO variants of 2wwen.55-Mtth71 on fine-tuning data (computed using approximated full distributions instead of truncated ones)
Avarage Divergennc
Blvengence
0.15
0.10
0.00
40
60
80
100
20
40
100
Relatvve Sequence Position (%6
Relative Sequence Positton 196)
(a) DAPO (tipphiigher(.28))
(b) DAPO (tipphhgheer-022)
Figure 37: Mean JS divergence by normalized token position for DAPO variants on fine-tuning data.
0.1
0.55
0.7
Swergge- Oerib R20
0.00
535
6-
0.75
0.5
5-001
0.0
0.5
0.5
10
T5pk
Top
(a) DAPO (clip-higher:0288)
(b) DAPO (clip-higher=0.2)
Figure 38: Top-k token overlap between base and RI. models at divergent positions (JS, > 0.1) for
DAPO variants on fine-tuning data.
31

---

Published as a conference paper at ICLR 2026
A.4.3 DISTRAL-SMAL--24 WITH SIMPLERL
trained with SimpleRL on AIME 2024 and AIME 2025 demon- strate the generalizability of our findings across different model architectures, Figure 39 shows JS divergence percentile curves, revealing consistent sparsity patterns. 40 positional con- centration, Figure 41 shows entropy distributions across divergence bins, and Figure shows tail behavior analysis.
We analyze
3.7
077
2339
2.6
0.6
750 1193
750 13310
0.0
(a) AIME 2024: Percentiles
(b) AIME 2025: Percentiles
Figure 39: JS divergence distributions for Mistral-Small-22 with SimpleRL AIME 2024 and
AIME 2025. Sparse distributional shifts are consistent with findings in the main text across both
datasets.
Avenge 55 Diergence
Average 5 Divespence
0.01
0.01
0.00
0.00
-0.01
(a) AIME 2024
(b) AIME 2025
Figure 40: Mcan JS divergence by normalized token position for Mistral-Small--aa with SimpleRL on AIME 2024 and AIME 2025. Consistent with findings for other models, average divergences are more concentrated the start and end of responses.
Bese Entropy Entropy
0.10
Weporron
0.6
0.4
0.06
0.08
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Base Model Entropy
Entropy Value
(a) Mistral-24B+SimpeeRL Low JS bin (<
(b) Mistral 24BtSSlle2BBS High JS bin 0.1)
0.1)
Figure 41: Entropy distributions across divergence bins using full vocabulary for Mistral-Small-2445 with SimpleRL. on AIME 2024.
32

---

Published as a conference paper at ICLR 2026
20
Porcntag bhergrr (14716
Percetagt of Dherger Tharre
10
10-
Prstabllty
Probability
(a) 24 AIME 2024
(b) Mistral-24B-SimpleRR. AIME 2025
Figure 42:
Percentage of divergent tokens whose RL top-1 choice had base probability below given threshold for Mistral-Small-24B with SimpleRL on AIME 2024 and AIME 2025.
A.5 ADDITIONAL CROSS-SAMPLING RESULTS AND DISCUSSION
This section provides supplementary cross-sampling results and the algorithm used for the cross- sampling experiments.
Algorithm D describes the general procedure for a single prompt (in pratice, we batch prompts for efficiency), which generates response primarily under primary policy and selectively intervenes using an intervention policy at positions where the token-level divergence exceeds a fixed threshold.
Algorithm Crops-Sg for a Single Prompt
Require: Prompt prefix I<1, primary policy Tprim, intervention policy Tint, divergence threshold
eJs, maximum steps T
Ensure: Generated sequence Tiat, number of intervention steps
1: & +-0
2: Initialize prefix I<1
3: fort= 1,..., T do
4: Compute divergence JS,- = Das (ee)) || Tine (- (Ee)))
5: if JS, > eJs then
6:
Sample Tint (- (<ee) k+ k+1
7:
8:
else
9:
Sample 3y ~ (: (ee))
10:
end if
11: Append T, to prefix d<++1
12: if xr EOS then
break 14: end if
13:
15:
end for
16: return generated sequence 3ia and intervention count k
Progressive Steering of Reasoning Trajectories. Across both forward and reverse cross- sampling, reasoning performance varies smoothly with the intervention budget. In the forward direction, accuracy improves steadily as more RL-sampled tokens are introduced, with no sharp threshold, indicating that gains accumulate across multiple sparse decision points rather than requir- ing all RL-induced changes. In reverse cross-sampling, performance degrades in similarly smooth, near sar-moontonnc manner as RL token choices are reverted, showing that RL-level performance de- pends on preserving a sparse set of token-level shifts throughout the trajectory. A key aspect both settings is that interventions are applied sequentially, while decoding otherwise follows single pri- mary policy. One might expect that modifying only a few tokens, especially early ones, would have limited impact. However, this is not the case: injecting just the first few RL tokens already yields measurable gains forward cross-sampling, while reverting the earliest divergent tokens no- ticeably degrades performance in the reverse setting. These effects arise because small local
33

---

Published as a conference paper at ICLR 2026
edits redirect the reasoning trajectory, which is then propagated by subsequent decoding under the primary policy.
Connection to speculative decoding and BiLD. Our cross-sampling framework is related to speculative decoding (Leviathan et al. 2023 Kim et al. 2023) in that generation depends on two next-token distributions. It closer Big Little Decoder (BiLD) (Kim al. 2023), which defines routing policy (rather than an exact sampling scheme that preserves designated target distri- bution). BiLD trades off lateney and quality via fallback and rollback rules, while our approach (prim,,ut) to investigate the role of high-divergence decisions between base
defines a mixed policy "mix and RL models for reasoning performance. BiLD triggers fallback based on small-model confi- dence (e.g., max paobbbbbayyylity threshold) and uses rollback based on discrepancy between smal- Vlarge predictive distributions (with a cross-entropy eatsseptttppbbbeeee quantity), whereas intervene when (T (- Xxe) Fint x)t)) exceeds threshold. BiL.D further proposes prediction align- ment by fine-tuning the small model on large-model outputs to reduce avoidable disagreements, while our setting typically uses models that already agree at most positions (base vs. RL) to isolate fine-tuning effects.
Table 3: Summary of cross- sampled tokens required to reach approximate RL-level performance (forward) base-level performance (reverse) for Qwen2.5-32B AIME 2024 and AIME 2025 with a token budget of 8000. Effective token counts/percentages exclude identity swaps during cross-sampling. Token percentages computed at the sequence level.
Datasct
Method
Eff. % %
ElT. #
#
Initial Acc.
Final Acc.
Tokens Tokens Tokens Tokens
SimpleRL.
3.86%
7.58%
38
75
8.23
> 25
SimpleRL. Rev.
5%
8.3%
51
25.52
< 8.3
AIME24
DAPO
7.8%
11.9%
280
410
8.23
> 44
DAPO Rev.
10.1%
14.9%
173
258
44.8
< 8.5
SimpleRI.
1.53%
2.97%
26
5.3
> 14
SimpleRL Rev.
4.73%
7.87%
31
53
12.71
< 4
AIME25
DAPO
6.47%
9.18%
230
326
4.8
> 33
DAPO Rev.
9.89%
14.19%
181
261
32
< 4.5
34

---

Published as a conference paper at ICLR 2026
No
Cluit
Clui
(a) Forward cross-sampling Qwen2.5-32B Sim- plcRL.
(b) Reverse cross-sampling Qwen2.5-32B Sim- plcRL
300
1000
1250
3500
1430
230
1308
3000
1400
(c) Forward cross-sampling Qwen2,5-32B DAPO
(d) Reverse cross-sampling Qwen2.5-32B DAPO
Figure 43: Cross-sampling token pair histograms.
35

---

Published as a conference paper at ICLR 2026
Avergge Accurery (3 camples per prompe
zamples prompt
40
Base
30
with
Accuracy
Accurccy
23-
Bave fecurwcy 8.2%%
23
Rase Accurscy
20
Average
30
150
Effectivv
Effective Number Resampled Takers
(a) Forward cross-sampling
Reverse cross- cross-nampling
Figure 44: Cross-sampling results (DAPO AIME 2024): injecting RL tokens into base gen- erations progressively recovers accuracy, while reverting RL tokens with base tokens causes near-monotonic degradation toward base performance.
40
Averoge Accurary E2 zampes ser
20
- RL with Sase Resamping --- Accuracy 45%
30
RL winh Base Resamoling RL Accuraty 45%
Accurary 0.44%
Base Accurecy: 8.3%
20
Average
33
100
200
300
500
600
700
800
50
100
150
200
250
320
350
400
Average Number of Ressmpped Takens
Average Number Resempled Takens
(a) Random baseline
(b) DAPO
Comparison of random baseline and DAPO cross-sampling on AIME 2024: average number of tokens (including identity swaps) replaced versus accuracy. The random baseline shows minimal performance improvement, demonstrating that targeted RL token selection critical for performance gains.
Figure 45:
-00
Average Accunsty 132 samotes per proppt
Avcrage samplcs per prompt)
30
100
BI with Rose Benampiig BL Accuracy:: 33%
20
RL with Base Resamoling Accuracy:
Base 4.79%
15
10
50
150
200
230
50
100
150
200
Average Effective Number Resampled Takens
Avol age Effective bumber of Resampled Tokeess
(a) Forward cross-sampling
(b) Reverse cross-sampling
Figure 46: Cross-sampling results (DAPO on AIME 2025): injecting RL tokens into base gen- erations progressively recovers RI. accuracy, while reverting RI. tokens with base tokens causes near-monotonic degradation toward base performance.
36

---

Published as a conference paper at ICLR 2026
Suerge teunsc 13 sameles pe gromo
Rt with Base Resampiinn Accuracyy 6% Accuracy: 0.62%
sampless per prompe)
Recamplin
RL Accuracs: 5. 5.35%
Base Accuracy, 0.6%
Average
Average Effective Number of Resempled tokens
Average Effective Number Resempled Tokens
(a) Forward cross-sampling
(b) Reverse cross-sampling
Figure 47: Cross-sampling results (Mistral-Small-24BB + SimpleRL AIME 2024): injecting RL tokens into base generations progressively recovers RL accuracy, while reverting RL tokens with base tokens causes monooonoo degradation toward base performance.
A.6 SEQUENCF-LEVEL DIVERGENCE BOUNDS FOR CROSS-SAMPLING
Setup. Let (Xt);21 be the decoded tokens (each in V), with stopping time inf 1 X₁ = EOS} A Tmax. We may work on the fixed horizon max by absorbing the EOS token: once EOS generated, the process deterministically outputs EOS thereafter. This yields equivalent distribution over XI.Tmax and ensures all sequences are of length T maax- All results below therefore sum over I.... , maxi terms after т contribute zero since both policies become point masses on EOS.
Let prri be the primary policy and Tint the intervention policy. Given a switching rule S: VEN {0, 1), define St == S(X²() and the mixed policy
Tmix Xxt) (1 -S))) pr Xx1) St Time X
Let Piix and Pint denote the induced sequence-level distributions on XI:Tmax-
We provide bound on the sequence-level divergence between the cross-sampled policy and the target intervention policy, the simpler case of KL divergence with a token-level KL switching rule.
Lemma A.1 (KL decomposition). Let P,Q be distributions on X1 admitting factorizations Tmmx
P(Z1:Tmax)
Te pe(
x22) and ((T1:Tmax)) I[ (1( x.(.). Then
Taxx
(P Q1
Σ
Ex DKu(pa(- X || qe(: [x.))].
P(X) Proof. By definition, DKL(P (P Q1Q)= ExxP log Q(X)
Using the factorizations,
P(XI:Tmax)
Tmnx
Pc (X X (e(x, X
log Q(XI:Tmox)
log
Taking expectation under P and exchanging sum and expectation gives
Tmax
Px Xxt) qe(X,
(KL(P || Q)
=
E Exap
log
Xee)
Excrp Excxp((Xee) log (x, Xxx) qi (Xe Xxe)
1=1
Ex DKu(pi(- Xee) ll ge(:
xx.))..
37

---

Published as a conference paper at ICLR 2026
where the second last equality follows by the definition of conditional expectation (or the law of total expectation) applied to the conditional distribution of X₁ given X<t-
Proposition A.2 (Token-level KL threshold sequence-level KL bound). Assume the switching rule is defined by a KL threshold:
S(z<)) 1 [L<Ki(TTrim(( x<)) Tint (xer)) > e).
Define the number of non-intervention steps trajectory
No Σ (()(x(t)= =0}.
Then
DEL (Piix Pinx) S ExxPaux [Nol.
Proof. We apply Lemma A.1 with P Pinix and Q = Pint For any history h Iets since S(h) E 1).
Tmix h) Tin (- h),
S(h) 1, prrim h),, S(h) O.
Hence
DxL(Tmix( h) || Time In) 1{ 00 (primi) ( h) || Tma [))-
By the definition of S, whenever S(h) = 0 the token-level KL at most e. Therefore, for all h,
DRI(Tmiix( (- h) Tine (1) - a {(((h)) 0}.
Under absorbing EOS, DRI Tmix 1(t S 7} in the sum. Substituting into Lemma A.1 with this yields
X Time Xxt) = 0 for I > so we may insert
Tima
Tamax
(Pixx || Pmn))
E Ex Panx [1 S 00] = =E Σ Ex S 1 0 03}
Note that {t [ {EOS & x depends only on X so 1 S 1) = 0} is measurable with respect o (X Thus, may equivalently write the expectation under the full trajectory
X ~ Piix:
Ex ( -O]- = Ex PA [1 = 0H]
Then,
Timax
Tanax
Ex P 03f Σ
Exap mix
[1( 0Hl
Imax
Xx Pmix
Σ
1|tST] =
Σ 1(S, = 03
e EXx
1=1
EEEx No).
Remark A.3 (Effective KL on non-intervention steps). intervention steps
Define the effective token-level KL on non-
Ex Ex
E 1i5 0} DKL(TTrrim X Tint (. xxt))l
=
(R = EfN)] = 0).
Ex Praxe [No]
Then
DKL mix
Pant) R R NNo S E (No),
and typically R & E at most steps in regimes where the models are already close (eg. in the setting of base model and its RL fine-tuned counterpart).
38

---

Published as a conference paper at ICLR 2026
A.7 ADDITIONAL DIVERGENCE- WEIGHTED ADVANTAGE DISCUSSION/RESULTS
This section presents supplementary results for divergence-weighted advantages, including alterna- tive configurations and evaluations on Qwen2.5-7B. The main experiments focus on Qwen2.5-Math 7B with KL divergence computed with respect to Teau and sigmoid weighting. Here we discuss alternative schemes, along with additional results.
A.7.1 ALTERNATIVE DIVERGENCE CHOICES
Beyond the old-policy KL divergence presented in the main text, one could also use the reference- based KL divergence:
KT DKL (iice)) [eee( [x<e))
where Tre denotes the base reference model. The reference-based KI. quantifies the alignment between the current policy and the original base model, measuring the cumulative divergence from the initial model each token position. This contrasts with the old-policy KL, which captures only the magnitude of recent policy updates within single training iteration. In our experiments we mainly just use the old- uld-policy KL as this does not require additional pass through the base model.
A.7.2 ALTERNATIVE WEIGHTING SCHEMES
In addition the sigmoid weighting scheme presented the main text, we examine linear relative weighting:
Wy 1 + a (KL, (KK));
PKL 7
KL,
The linear relative scheme scales weights linearly with the deviation from the mean KL divergence across the sequence, offering simpler alternative to the sigmoid transformation. As in the sigmoid case, 0 amplifies high-divergence tokens, while emphasizes low-divergence ones.
A.7.3 RESULTS ON ADDITIONAL CONFIGURATIONS
Sable the performance of these alternative configurations, including evaluations using linear relative weighting on Qwen2.5-7B.
Table 4: Accuracy (%) under additional divergence-weighted configurations Qwen2.5-7B. Re- sults shown across AIME 2024, AIME 2025, and AMC datasets. The results displayed are the @v scores.
Configuration
A1ME24
AIME25
AMC
Overall Avg
Baseline DAPO
16.77
8.12
70.78
31.89
High-KL Lin. Rel. (sched.)
19.58
12.40
71.12
34.37
High-KL KL Lin. Rel.
20.00
12.29
73.31
35.20
Table 5: Accuracy (%) for 80/20 clip entropy configuration on Qwen2.5-Math-7B. Results shown across AIME 2024, AIME 2025, and AMC datasets, The results displayed are the avg @ 32 scores.
Configuration 80/20 clip entropy
AIME24
AIME25
AMC
Overall Avg
35.26
17.03
72.68
41.66
A.8 WEIGHT-LEVEL ANALYSIS OF CHANGES
Orthogonal to the analysis done in the main text, we also investigate the degree modifications induced by RLVR at the parameter level. More specifically, employ the relative gap ratio (Wul al. 2025). denoted as a, to quantify the magnitude of weight divergence pre- and post-fine-tuningg This ratio is formulated as:
39

---

Published as a conference paper at ICLR 2026
W original
W unddd
Σ W Vriigiaal
Σ
W runed
where Woriginal and W Wieed represent the model parameters before and after fine-tuning, respectively. lower value signifies greater similarity between the parameter sets, indicating smaller overall modification from the fine-tuning process.
In our experiment, we utilized the Qwen2.5-32B and Owee2.5-Math-7B models as foundations. Each model was independently fine-tuned via two distinct methodologies: RL and Supervised Fine- Tuning (SFT). To ensure controlled and equitable comparison, the training regimen for both meth- ods standardized, employing identical dataset size and the same number of training steps. Subsequently, the a was computed between each original model and its corresponding tuned coun- terparts. The results are presented in the following table.
Table Relative gap ratio (o) after RL and SFT fine-tuning.
Model
Qwen2.5-32B
Qwen2.5-Math-78
a after RL
0.00143
0.00136
a after SFT
0.00347
0.00944
The results presented in the table demonstrate consistent trend across both models: the val- ues corresponding to RL fine-tuning are substantially lower than those from SFT. This quantitative analysis at the parameter level suggests that the cumulative weight modifications induced by RL significantly less extensive than those resulting from SFT. This finding provides empirical sup- port for the hypothesis that RL achieves performance gains through sparse and targeted parameter adjustments, contrasting with the more distributed updates characteristic of SFT.
A.9 LLM USAGE
LLMs were used mainly to assist with minor polishing and organizing of the writing,
40


Reviews:
[{"review_id": "wpu4aPh8an", "reviewer": "Reviewer_MBtv", "summary": "This paper analyzes how Reinforcement Learning with Verifiable Rewards (RLVR) modifies token-level output distributions during RL-tuning. The authors first examine distributional shifts between base and RL-finetuned models using Jensen-Shannon divergence, finding that RL-induced changes are highly sparse, with only a small fraction of tokens exhibiting significant divergence. Furthermore, through the Cross-Sampling experiments, they confirm that the effect of RL-tuning stems from the reranking of high-divergence tokens. Motivated by these findings, the authors propose a divergence-weighted advantage modification to the RL objective, showing potential improvement against existing RL-tuning.", "questions": "1. What is the performance of the proposed method against other benchmarks?\n2. In Table 2, what is the pass@1 performance?\n3. What is the difference between emphasizing high-divergence and low-divergence during RL fine-tuning? In Table 2, both of these methods show similar performance gain. Could you explain this?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The structure of this paper is clear; the authors first provide a thorough analysis of the token-level distribution shift and highlight the importance of the high-divergence tokens during RL fine-tuning. They then proposed an improved divergence-weighted objective based on their findings, making the paper sound and logical.\n2. The analysis of token-level divergence and corresponding ablations is insightful and interesting. Although empirical, they are critical in understanding the operation of RL fine-tuning methods from the token-level aspect.", "weaknesses": "1. It is suggested to proofread the paper, especially the experiments. For example, in Table 2, the configuration details should be presented more clearly.  \n2. The proposed method is evaluated in a rather limited setting; for instance, it is tested only on the AIME-2024 dataset. Moreover, the experiments lack comparisons with related works [1].  \n3. The improvement achieved by the proposed method appears modest. According to the results in Table 2, the gain in Avg@32 is limited.  \n\n[1] Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gZFQnLyjJN", "reviewer": "Reviewer_3wey", "summary": "This paper investigates how Reinforcement Learning with Verifiable Rewards (RLVR) fine-tuning affects large language models (LLMs) at the token level. While RLVR is known to significantly improve LLM reasoning performance, the paper demonstrates that these improvements are not distributed evenly across outputs. Instead, RLVR induces sparse and highly targeted changes: only a small fraction of tokens in model responses are significantly altered by RL. Notably, the paper finds that these high-divergence token changes are not restricted to regions of high model uncertainty, but often override even confident base predictions. The authors conduct detailed empirical analyses to characterize these changes, evaluate their functional importance, and propose new RL objective variants to leverage these insights.", "questions": "1. Can you provide results or analyses on non-math tasks or with other LLM architectures to support generalization claims?\n2. How sensitive are your main findings to the sampling parameters (top-p, temperature, number of samples)? Have you run any ablations?\n3. Can you report statistical significance or variance for the divergence-weighted objective gains in Table 2? Are these improvements robust across seeds or runs?\n4. How does using JS divergence on top-p–truncated distributions affect your results? Can you show that your findings are not artifacts of truncation?\n5. What exact divergence thresholds and implementation details were used in cross-sampling experiments? Can you provide all necessary information for reproducibility?\n6. Have you analyzed what input features or contexts make high-divergence tokens critical? Any qualitative or quantitative results on the causes?\n7. Could measuring divergences on off-policy or mixed trajectories affect the identification of critical tokens? Have you considered or tested for on-policy bias?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a novel token-level analysis of distributional shifts caused by RLVR, revealing sparsity and context-sensitivity in model updates that are not addressed by aggregate or entropy-based approaches in previous work. The authors introduce creative cross-sampling experiments that directly test the functional impact of high-divergence tokens and propose divergence-weighted RL objectives to exploit the observed sparsity.\n\n- The empirical methodology is thorough and rigorous, comprising comprehensive token-level and sequence-level analyses, comparisons across several RLVR algorithms and model scales, and carefully designed interventions. Quantitative findings are supported by clear metrics (JSD, accuracy, percentage of affected tokens) and transparent reporting of results.\n\n- The paper is well-organized and clearly written, with all key concepts (e.g., divergence, entropy, cross-sampling) explicitly defined and illustrated. Figures and tables are used effectively to present the main findings, and the logical structure guides the reader through the motivation, experiments, results, and implications.\n\n- The work offers new insights into the mechanisms of RLVR, showing that performance gains stem from a small set of critical token-level changes. Its findings have practical implications for designing more efficient, interpretable, and targeted RLVR fine-tuning strategies for LLMs.", "weaknesses": "1. **Narrow dataset scope and lack of experimental details.** The experiments focus exclusively on math reasoning (AIME24) with Qwen2.5 models, so it is unclear whether findings about token-level sparsity and cross-sampling generalize to other domains, reasoning tasks, and models. The paper uses a fixed sampling setup (32 samples/problem, top-p=0.7, temperature=1), but omits ablations on these choices. No statistical significance or variance is reported for the divergence-weighted advantage gains in Table 2. JS divergence is calculated on top-p–truncated distributions, which can distort divergence magnitudes, yet the paper provides no discussion of this issue. The experimental setup is described at a high level, but several key implementation details necessary for full reproducibility (e.g., cross-sampling thresholds) are omitted.\n\n2. **Limited causal and interpretability analysis.** While cross-sampling demonstrates the criticality of high-divergence tokens, the paper does not deeply investigate why these tokens emerge, what properties of inputs make them critical, or how specific token changes affect reasoning trajectories and outcomes.\n\n3. **On-policy conditioning bias.** Token divergences are measured along RL trajectories only, which may bias where \"critical\" tokens appear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "74rQLGc7qo", "reviewer": "Reviewer_pdCB", "summary": "This paper examines how token-level distributional properties shift during RLVR fine-tuning and explores the functional significance of these changes.\n\nEmpirically, the authors find that RL fine-tuning induces distributional divergence in only a small subset of tokens, mostly occurring early in the fine-tuning process. Notably, the magnitude of this divergence is not fully predicted by token entropy.\n\nTo assess the functional role of these divergent RL-sampled tokens, the authors inserted some of them into generations produced by the base model. They observed that this intervention caused the base model to perform as well as the RL fine-tuned model. Conversely, as expected, inserting base model tokens into the RL model’s outputs reduced its performance to base model levels.\n\nFurthermore, the authors demonstrate that modifying GRPO/DAPO to weigh these divergent tokens differently—rather than treating all tokens uniformly—during fine-tuning results in improved downstream performance. This highlights a practical value for these divergent tokens.", "questions": "It would be good to verify whether these observations hold not only for the Qwen model family, but also extend to other open-source model families such as Llama or Mistral families.\n\nI am also still unclear about why altering the placement of such a small number of tokens after RL fine-tuning can have such a strong impact on downstream performance. Is this function of the task/context within which these analyses were conducted as that is still unclear to me. Could the authors elaborate on their intuition for why this effect occurs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "I found the article to be generally well written. The motivation behind each analysis was clear, and the findings were explained well. The authors’ analyses were thorough—they not only examined how token distributions change, but also investigated which factors might predict these changes, the functional role of the divergent tokens, and how these insights can be leveraged to improve performance of RL fine-tuning.", "weaknesses": "It remains unclear under which specific sequences of tokens these observations were made, and clarifying this would enhance the paper. Additionally, the motivation for using JS divergence over KL divergence could be explained more thoroughly—currently, it is addressed in only a couple of sentences, but this section could be expanded with a simple illustrative example. Especially since the rest of the paper's relies on this observation. \n\nTo streamline the narrative, consider moving certain experiments, such as the analysis of top-k token overlap, to the appendix, as these largely reinforce points already demonstrated by earlier results. At present, the paper contains too many small results, some of which make similar points using different approaches. This redundancy could be reduced; focusing on a few key analyses, with detailed motivation and deeper exploration of findings, would strengthen the paper.\n\nDoing so, could also free up space for the final analysis, which felt somewhat crammed. I believe this final result was particularly important and deserved more attention than it currently receives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zgVvoNEUkW", "reviewer": "Reviewer_SHpG", "summary": "This paper studies the changes RLVR introduces by analyzing the token-level divergences between pre-trained and RLVR-trained versions of the Qwen2.5-7B model. The authors conclude that changes are concentrated in a sparse set of high divergence tokens, and conduct several experiments to validate and explain this finding: the cross-sampling analysis where high-divergence tokens are sampled from RLVR-trained model and put into the base model, and vice versa; the semantic analysis of changed tokens; the dynamics of changes during training.", "questions": "Q1. In figures 3 and 9, in the low JS bin, am I correct that RL entropy (red) almost zero and therefore is not visible? Please correct me if I misunderstood the section 3.3. Perhaps, the scatterplot or 2D histogram with JS / entropy axes would be effective; it also might be made for each sequence separately, and each point would represent separate token in a sequence.\n\nQ2. Have you conducted an experiment where position would be measured not relative to the end of the sequence, but to specific elements of it (e.g. distance from system prompt, from the instruction, model generation start, from the final answer to the question etc.)? If yes, then what have you noticed? This is partially addressed by section 3.4; since the result in figure 10 suggest that single token might operate in low-divergence and high-divergence regimes, the positional information might reveal when some token would have high entropy and high divergence.\n\nQ3. Is it possible to extend percentage of replaced tokens in the figure 5? Would trend be kept or will it be bounded from above by RL accuracy and from below by base model accuracy?\n\nQ4. Results in section 5.1 are shown for high-divergence positions and therefore we expect differences to be noticeable; what changes if we do not restrict the positions and calculate this ranking results for all tokens? I also suggest to make shared x-axis for columns in figure 11 (b) so it would be easier to compare.\n\nQ5. Section 3.2 about the dependency of entropy shift on position in a sequence is not substantive enough: absolute values of JS divergences shown for both methods are very low, and trend is seen only for DAPO. Also, since for DAPO the values are significantly larger than for GRPO (SimpleRL) as in section 3.1, it suggests that entropy shift might be the consequence of nonzero KL divergence regularization term and similar mechanisms to preserve the generation behaviour. Can you speculate if this is the case, or correct me if I'm wrong?\n\nQ6. Have you gathered more results on advantage reweighting experiments? Currently, improvement seems very small. Have you identified for which scenarios and baseline algorithms this method is meaningfully applicable?\n\nConsidering cross-sampling experiment, you might be interested in checking out [2].\n\n[1] Spurious Rewards: Rethinking Training Signals in RLVR, Shao et al.\n[2] Reasoning with Sampling: Your Base Model is Smarter Than You Think, Karan A., Du Y., 2025.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The results are novel to my knowledge and are presented in a concise manner; analysis in the paper is well-motivated; paper is dense and filled with results that generally support their claims and conclusions. Main claims of the paper are supported, but the execution of experiments might be significantly strenghtened further (see weaknesses).\n\nS2. Experiments with cross-sampling and advantage reweighting are interesting and clever and seem to be a promising analysis toolkit; however, see W3 and W4.", "weaknesses": "W1. The most serious limitation of the work is that it lacks the treatment of other post-training schemes such as SFT, preference optimization or RLHF, so it is not clear if conclusions made in paper are specific to RLVR and not to fine-tuning in general.\n\nW2. It is well-known that Qwen family shows different properties when training with RLVR on mathematics [1], and the paper does not study other families and tasks. Although these results are recent and the authors might be not aware about them when preparing the manuscript, this limits the generalisability of the paper findings and should be acknowledged in future work.\n\nW3. All comparisons between GRPO and DAPO are affected by the different datasets the studied models were trained on, and this factor is not isolated. This is acknowledged in a paper.\n\nW4. For experiment with cross-sampling, an important control baseline is missing - for example, the case when a token is sampled from another model randomly, instead of divergence-based rule, or something of that kind. This will quantify the effect of high-divergence replacements, and if control baseline would give lower final accuracy, then the conclusion made in line 268 \"the improvements from RL fine-tuning are concentrated in a sparse set of high divergence tokens\" would be reliable.\n\nW5. Experiment with advantage reweighting in section 6.2 is underdeveloped, shows minor improvement, and it is unclear if the method allows for better results and stable training; another major shortcoming is that only DAPO baseline is analyzed. Since reweighting algorithm appears to use KL divergence, it is unclear how would this algorithm work if baseline algorithm already have KL divergence penalty (e.g. GRPO). Overall, this section is not very useful for the reader and further research in its current form, but the method seems interesting.\n\nW6. As a minor weakness, although paper has a lot of results presented in a concise manner, it is densely written and it is slightly hard to parse and understand the meaning and interpretation of some results. I think that moving some content from main body to appendix and focusing on more clear presentation would improve the paper, but this is more suited for further iterations of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)