Beyond Magnitude: Leveraging Direction of RLVR Updates for LLM Reasoning
Authors: Kexin Huang
Haoming Meng
Junkang Wu
Jinda Lu
Chiyu Ma
Ziqian Chen
Xue Wang
Bolin Ding
Jiancan Wu
Xiang Wang
Xiangnan He
Guoyin Wang
Jingren Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: RLVR
LLM reasoning
Primary area: reinforcement learning

Abstract: Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models. 
While existing analyses identify that RLVR-induced changes are sparse, they primarily focus on the **magnitude** of these updates, largely overlooking their **direction**. 
In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference $\Delta\log p$ between the base and final RLVR models.
Through statistical analysis and token-replacement interventions, we demonstrate that $\Delta\log p$ more effectively identifies sparse, yet reasoning-critical updates than magnitude-based metrics (e.g., divergence or entropy).
Building on this insight, we propose two practical applications:
(1) a *test-time extrapolation* method that amplifies the policy along the learned $\Delta\log p$ direction to improve reasoning accuracy without further training;
(2) a *training-time reweighting* method that focuses learning on low-probability (corresponding to higher $\Delta\log p$) tokens, which improves reasoning performance across models and benchmarks.
Our work establishes the direction of change as a key principle for analyzing and improving RLVR.

Published as a conference paper at ICLR 2026
BEYOND MAGNITUDE: LEVERAGING DIRECTION OF RLVR UPDATES FOR LLM REASONING
Kexin Huang', Haoming Meng", Junkang Xue Wang', Bolin Ding', Jiancan W Xiang Wang"; Xiangnan He Guoyin Wang", and Jingren Zhou' National Engineering Laboratory for BITA, University of Science and Technology of China University of Toronto DDartmouth College "Independent Researcher huangkxemail wsw.ustcc....c edu.
Lu', Chiyu Ma², Ziqian Chen',
fwujcan, xiangwang1223, xiangnanhe| ggaana com
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has substantially im- proved the reasoning capabilities of large language models. While existing anal- yses identify that RLVR-induced changes are sparse, they primarily focus on the magnitude of these updates, largely overlooking their direction. In this work, we argue that the direction of updates is a more critical lens for understanding RLVR's effects, which can be captured by the signed, token-level log probability difference A log between the base and final RLVR models. Through statistical analysis and token-replacement interventions, demonstrate that A log p more effectively identifies sparse, yet reasoning-critical updates than magnitude-basee based metries (e.g., divergence entropy). Building on this insight, we propose two practical applications: (1) a test-time extrapolation method that amplifies the pol- icy along the learned A logp direction to improve reasoning accuracy without further training; (2) a training- time reweighring method that focuses learning on low-probability (corresponding to higher log tokens, which improves rea- soning performance across models and benchmarks. Our work establishes the direction of change as a key principle for analyzing and improving RLVR.
INTRODUCTION
Recent advances have substantially improved the reasoning capabilities of large language models, giving rise to powerful reasoning-centric models such as OpenAl ol (Jacch et al., 2024), Deepseek R1 (Guo et al., 2025), Gemini 2.5 (Comanici et al., 2025), and Qwen3 (Yang et al., 2025a). A key algorithmic driver of this progress is reinforcement learning with verifiable rewards (RLVR) (Guo al., 2025; Team, 2025; Yang al., 2025a), which fine-tunes model's generation policy using feedback from task-specific verifiers, thereby eliciting and amplifying the reasoning ability.
To elucidate how RLVR confers its gains, natural lens is compare what changes in the final RL- trained model TRL relative to its base counterpart Base (Ren & Sutherland, 2025). Previous analyses have consistently shown that the RLVR-induced changes are sparse, impacting only a small subset of tokens in the output sequence. For example, Wang et al. (2025b) associate these changes with high- entropy tokens, Huan al. (2025) corroborate the sparsity by measuring the KL divergence between TBse and TRL, while Yang et al. (2025b) and Deng et al. (2025) attribute this sparsity to selective gradient updates during RLVR training. However, when studying the difference between base and RLVR models, prior studies primarily emphasize the magnitude of change, but largely overlook the direction in their distributions. As shown in Fig. 1(b), magnitude- based metries (e.g.. entropy, KL divergence) yield nearly identical histograms for the base and final RLVR models, indicating that magnitude alone is insufficient to characterize the transformation from Tuse to TRL-
'Jiancan Wa, Xiang Wang and Xiangnan He the corresponding authors.

---

Published as a conference paper at ICLR 2026
Hotogram Roweno
Divergence Histogram
Performance Varying Replacement Ratios
Entropy Eyy-nt Iaaw I (x.)))
Nueemoo oweriop
$4.50
everlop
KL Divergence
(trre) log Nauee yer)
Nac Onl
25 M0
Logs fiferrnce
divergence
Logp Difference Histogrrmm
Measuring magnitude
UASI Gase hatpoerss
Logo Difference log achieves the most effictent replacement
Bose + RLVR
Logp Difference AlogP log Nas On y-r) Nag (yllt.
Base
RLVR
Measuring the direction: Base RLVR
RIVA Talen Replasement Rarto (44)
(a) Different Metrics
Histograms (density) RL/Base Responses
(c) Impact on Token Replacement Performance
Figure I: (a) Token-level metrics for analyzing RLVR updates. (b) Histograms of each metric on re- sponses generated by base and RLVR models. With a log-scale y-axis, most values concentrate near zero for all metrics, but only A logp shows a directional shift distinguishing RLVR from the base model. (c) token-replacement performance: replacing base tokens with RLVR choices at positions selected by each metric, where A log recovers RLVR performance with the fewest replacements.
To address this gap, we directly quantify directional shifts in the model's distribution using the signed, token-level log- pae-obbabblity difference:
A log p(y (r, Vc+)) Tog Vet) - log Base (Vel2,, (2, Met),
(1)
which captures how RLVR shifts the probability mass on each token, with positive values indicating increased probabilities and negative values vice versa. As shown in Fig. (b), histograms of A log exhibit clear bimodal pattern with two distinct tails, highlighting a clear directional signature absent in magnitude-based metrics. This metric can reveal which token RLVR prioritizes, such as reasoning-crinn crittcal tokens (e.g., those enhancing reasoning correctness) versus irrelevant ones. We further validate its utility via a token replacement intervention (Meng et al., 2026): for each metric, we identify salient positions and replace the base model's tokens with the RLVR model's choices at those positions during generation (ef. Algo. 1). As shown in Fig. I(c), selecting by A log reaches RLVR-level performance with the fewest substitutions, pinpointing tokens where RLVR learns reasoning crriticll updates. These findings underscore a key principle: analyzing the direction of changes, rather than solely their magnitude, provides deeper insights. The signed log- peepobbabylity difference provides a practical and effective handle for this diagnostic analysis.
Building on this principle, we first propose a test-timo augmentation that extrapolates the RLVR policy's 's distribution along the A logp p direction for reasoning crriticll tokens selectively, amplifying reasoning-relate updates and improving accuracy without additional training. Furthermore, we observe that tokens with the largest A log p consistently correspond to low- www-pooaabiltty tokens during RLVR training. Motivated by this, we design a probability- reweighting of policy-gradient advantages, upweighting contributions from low-probability tokens focus learning on reasoning- critical positions as A log indicated. This reweighting yields additional gains over current state- of-the-art art RLVR methods (e.g., DAPO (Yu al., 2025)) across diverse benchmarks and models.
In summary, this work introduces directional diagnostic for analyzing RLVR's 's effects and, based on these findings, develops two practical strategies for reasoning enhancement: a test-time extrapo- lation technique and training-time reweighting method. We hope our work offers new perspective for analyzing and improving RLVR through the lens of update direction.
2 PRELIMINARIES
Group Relative Policy Optimization (GRPO). GRPO (Shao et al., 2024) is a variant of the mile- stone policy gradient algorithm PPO (Schulman et al., 2017). It is adapted for LLM training by eliminating the need for a separate critic model. For each QA pair (x, a) sampled from dataset D, GRPO generates group of G responses {yr] using the old policy Tew.. computes their rewards [Ra 1 and estimates the advantage of each response in group-relative manner:
A = R; - mean ( (R.)
(2)
std({ (R)
2

---

Published as a conference paper at ICLR 2026
Then the policy ne is optimized by maximizing the following objective:
JGRPO (0) = =B
min
(o)(())A..)
(11z)
G
Vi
(3)
-, 3DKL (Te (Teel
where (0) Fen (2,,63)))6 is the importance sampling ratio, is the clipping range for (O))
and DKL (Te (Tre) ( regularizes the policy to stay close to reference policy Tyef.
Dynamic Sampling Policy Optimization (DAPO). DAPO(Yu al., 2025) is state-of-the-art critic-free RLVR algorithm that further refines GRPO. It introduces several techniques, including clip-higher mechanism, dynamic sampling strategy, token-level loss aggregation, overlong punish- ment, and removing the KL penalty, DAPO's objective is defined as:
ludl
(O) = -..
min (O),
E
Vi
1=1
(l)p(r..:(0),) 1 Clows + (high) A:.1)
s.t. O
< Kyi is.equivalent(q,:.)) < G.
(4)
Given its success, adopt DAPO the primary baseline algorithm for empirical analysis.
Token-level metrics for RLVR analysis. To study how RLVR turns a base model into the RL- finetuned counterpart, mainly compare the following token-level metrics for RLVR analysis:
Entropy: Wang et al. (2025b) observed that RLVR-induced changes are sparse and tend to con- centrate high-entropy tokens. This token-level entropy is defined as:
Hr (+1r,,,,, = E Ey a.ves) log क (yrl (00l(,,,<<))) ]et)].
(5)
We calculate this entropy for both the RLVR model H.n.. and the base model H
Divergences: Huan et al. (2025) used KL Divergence to quantify the distributional shift, also finding that the changes sparse. The token-level KL divergence is defined as:
TnL FTne(eel p. M<t) Base (Ve (x, y<(e)
DKL
BBae (11,, (xt) = E (-1.see))) log
(6)
DLL We also include its reversed variant Thasee TRL. and the averaged KL Divergence D&LL
1 (DKI.
2 TRi. + KL.
TRL to avoid asymmetry bias for a comprehensive analysis.
3 DISSECTING THE TOKEN-LEVEL CHANGES INTRODUCED BY RLVR
This section aims to dissect the token-level mechanisms through which RLVR training transforms a base model into its fine-tuned counterpart. First, show that the logp difference (A log Eq. 1) captures directional shifts in probability mass and separates base from RLVR generations, whereas magnitude-only metrics (entropy/divergence) do not. Second, conduct a token replacement ex- periment to validate that A log more precisely identifies sparse, reasoning- crittccc tokens targeted by RLVR. Finally, we explain the sparsity through gradient analysis showing that policy updates concentrate on low-probability tokens of RLVR's 's policy gradient updates.
3.1 STATISTICAL ANALYSIS: DIRECTIONAL VS. MAGNITUDE BASED METRICS
Experimental Setup. We conduct a statistical analysis on outputs from several RLVR-base model pairs (ORZ (Hu et al., 2025a). DAPO (Yu al., 2025), UniReason (Huan et al., 2025)) to compare how different token-level metrics capture RLVR-induced changes. We plot histograms of entropy, divergences, and logp difference of different models' generated tokens the AIME-24 dataset.
Statistical Comparison. Fig. 1(b) shows the distributions of these metrics for the UniReason model pair. Across all metrics, the histograms are sharply peaked near (note the log-scale y-axis),
3

---

Published as a conference paper at ICLR 2026
Performan.ce Varying RLVR Replace Ratios
On ORZ 228 (Qwen2.5)
On DAPO-322 (Qwen2.5)
On Un 148 (Owen3)
Ruyh: 46.15
RDFI 32.50
$456
50
AME24 hxp
AMEE2 Axpg32
AMIE2 Rvp
-0-
Dhetgannce
Divegerce
bitrozy Plas
Entropy Maa
bitrusy Mam Randim
Randam
.*. Random
20
20
5050: 3022
Bode: 6.7
0150: 13:46
10-
20
33
5.0
13.0
123
37.5
Figure 2: Token-replacement performance across metrics and model pairs. While all metrics can recover RLVR-level accuracy, A log p does so with the fewest replacements, demonstrating its pre- cision in isolating the sasoning-critical minor tokens changed by RLVR training.
confirming that RLVR- Lindu-id changes are sparse. However, the entropy and KL divergence dis- tributions are nearly identical for both the base and RLVR model outputs. In contrast, the A log distribution exhibits two distinct tails: a positive tail corresponding to tokens favored by the RLVR model and a negative tail for the base model. This pattern holds across all tested model pairs and for multiple entropy/divergence variants (Appx. E): the distributions of magnitude-baat -baseddedeeedddddddee metrics nearly indistinguishable between tokens generated by the RLVR and base models (Figs. 13-15), whereas A log consistently exhibits clear bimodal patterns (Fig. 12).
This is because magnitude-only metries quantify the size of the distributional change but ignore its direction, i.e., whether a given token is more favored by the RLVR model or the base model. With directional information, A log p reveals a clear difference between the two modes, enabling more precise identification of the sparse, reasoning-enhanciin validate their impact on reasoning performance in the following section.
updates induced by RLVR, and we will
3.2 RECOVERING RLVR PERFORMANCE VIA SELECTIVE TOKEN REPLACEMENT
Token Replacement Setup. To further assess how the minority tokens identified by each met- ric affect reasoning ability, we conduct selec-
Algorithm 1 Selective Token Replacement
Require: Base and RLVR models TBsse TRL prompt x, criterion function f fr(() E {0, Initialize response: t 0, Vço
tive token replacement experiment proposed by Meng et al. (2026). At each decoding step, we sample a token from BBsee then apply a metric- specific criterion f decide whether to replace
1:
2: while y: + "<EOS>" do
3: 1 +++1
the token with one sampled from TR.. (Alg. 1). The threshold is adjusted control replacement rates across metrics, enabling fair comparisons.
4: Sample from base: y/ Base (-122 Mee))
5: if fyeel,, M-ez)= = 1 then
6:
Replace the token: yt TRL (-12, Vel)
7: end
We compare entropy, KL Divergences', and logp difference, with the corresponding replacement criteria functions defined as follows:
8: end while
9: return Vst
Entropy: Following the hypothesis that RLVR updates target high-entropy positions (Wang et al., 2025b), we replace the base model's 's token if its token distribution has entropy exceeding a thresh- old r: (7 Ve.) = (-12, V-cr)= > r).
KL Divergences: Similarly, to target positions where the two models diverges most, we replace the token if the divergence is greater than f3 wx))) (1,,),,,,,)))) > T).
'Wang et al. (2025b) argue that RLVR primarily modifies tokens with high entropy. The observed concen- tration of near-zero-entropy tokens is therefore consistent with sparse updates under their assumptions.
"This follows the cross- sass--aee sample experiment by Meng et al. (2026), which originally employs bidirec- tional token swapping to verify RL's sparsity. We use the term selective loken replacement to better reflect our specific setup: comparing how different metrics select base tokens to be replaced by
We mainly use the averaged KL divergence KLL 3/4 [DD.E
D.LL
TRL) ) token replacement
avoid potential asymmetry bias and include KL.s variants D and KL Rg for ablation study.
4

---

Published as a conference paper at ICLR 2026
Toxen Procasiity within Varying dlog
DATC's Probabllzy & Gradient Norn Diactiuuion
forn Deef: Rale
0.14
3.4
MLYA
2.39
Tea 0.
6.5
Tee Produbiity Cota
6.6
4500% slopp
65.15%
1250265
0hop
(a) Gradient norm and probability
(b) Token probability v.s. A logp (c) RLVR performance top-p
Figure 3: (a) Token probability and gradient norm coefficient - (-)-) (-) of a DAPO step, where the gradient concentrates on rare, low-probability tokens. (b) Token probability within different A log p bins, where higher A log p bins contain lower probability for both base and RLVR models. (c) Effect of top-p filtering on RLVR training performance. Performance declines with more filtering.
Logp Difference: A large negative A log p for a token Y1 indicates that RLVR has learned to penalize it relative to the base model. We exploit this by replacing tokens whose logp difference falls below a threshold r: Flogp (ye Jr M<e)) I[ log p(y yet) < T).
This selective replacement setup, controlled by the metric-speciffc thresholds, allows compare the impact of tokens identified by each metric on reasoning performance at matched replacement rates. Fig. shows results on AIME-24 for three representative metrics H.ggue. D. and A log p. while Fig. 6 in Appx. A.2 provides ablations with additional metrics, including the RLVR model's entropy HTRI. and KL-divergence variants. All metrics are contrasted with a random baseline that uniformly replaces tokens: frand - (= < T). The key observations are as follows:
Observation I: Selectively replacing a minority base models' tokens can recover RLVR per- formance. As shown in Fig. 2, replacing 5-30% of a base model's sampled tokens with different metries suffices to match the final RLVR model's accuracy. In contrast, randomly replacing the to- kens without metric selection produces much slower performance growth. This demonstrates that RLVR-modified tokens are sparsely distributed along the sequence but disproportionately important for reasoning, highlighting the efficacy of the evaluated metrics in identifying these critical tokens.
Observation II: Logp difference > divergence >> entropy identifying RLVR R- reason- ing patterns. Across all model pairs (Fig. 2), A log p-based replacement reaches the RLVR model's accuracy with the fewest substitutions (around 10% of tokens). In comparison, magnitude-one met- rics (e.g., divergence and entropy) require clearly more replacement to match RLVR performance, indicating lower precision in identifying reasoning-crine these two, divergence consistently outperforms entropy, suggesting that RLVR changes may be restricted to high-entropy positions. This ordering log highest, followed by divergence, then entropy- remains stable across different divergence and entropy variants (Fig. 6 in Appx. A.2), further validating the superiority of logp difference in isolating the most influential positions.
changes introduced by RLVR. Between
3.3 GRADIENT- BASED EXPLANATION FOR THE SPARSE UPDATES
Our previous analysis established that the RLVR model differs from its base counterpart small but critical subset of tokens most effectively identified by A logp p. Here, we provide a gradient- based explanation for this sparsity of changes: RLVR's policy gradient inherently concentrates updates on rare, low www-probability tokens, correlating with tokens with high A logp p in the final model.
RLVR's policy gradient sparsely concentrates low-probability tokens. The gradient of the DAPO objective JDAPO for an un-clipped token Vi.t can be written as Wi. Va log we (Vi mo ci, where Wi,: = Ai,, combines the importance sampling ratio and advantage. To analyze the token's ' gradient norm, we have the following lemma (see the proof in Appx. D):
LLM policy with logits vector z for the output token yo the (l-norm of the DAPO objective's 's gradient w.r.t. is given by:
Lemma 3.1. For a ceffimax-parametteee
IiJaooo (Vr,e (1 2 2/wocel (1 #o(vi.eoo, Vi,ct))
5

---

Published as a conference paper at ICLR 2026
This partial gradient's fl-norm directly depends on To(ys 22, Ui, Vi,ce), with larger gradient sizes for lower-proode twwrb-rrbbbility tokens. Furthermore, Yang et al. (2025b) formally proved that the full gradi- ent norm is tightly bounded by the 1 - (-)-) term. Consequently, low-probability tokens, despite their rarity, receive disproportionately large gradient updates. We corroborate this empirically in Fig. 3(a), which plots tokens' probability and their gradient coefficient from intermediate DÁPO training step. Although low-probability tokens are sampled infrequently, they account for most of the total gradient mass. This concentration of gradients explains why RLVR's modifications are sparse: learning is naturally focused on small, high-impace set of low-probability positions.
High A log p tokens are the updated low-probability tokens. To complete the argument, we link the low- wwwprobabillity tokens that dominate training updates to the high-A Looh-loopp p tokens in the final model. Fig. 3(b) analyzes tokens grouped by their A log p values. It reveals two patterns: first, the probability of tokens in high-A log bins increases substantially from the base to the RLVR model; second, these high- A log p tokens have clearly lower probabilities in both models. This confirms that the most significant updates learned by RLVR target those low-probability tokens, and the sparsity of RLVR's changes therefore a direct consequence of sparse, high-man magnitude gradients acting on these critical tokens, which can be effectively identified post-hoc by their large A log p.
Excluding low- low-probability tokens during training impairs performance. To causally verify the importance of these low-probability tokens, we conduct training time intervention experiment to provide direct evidence for our hypothesis. We train the Owen2.5-Math-77 base model (Yang et al., 2024) using DAPO but adopt top-p sampling strategy during rollout to filter low-probability tokens. The results, plotted in Fig. 3(c), are conclusive. Even a mild filter (e.g., top-p=0.95) leads to substantial drop in performance compared to the default setting (top-p=1.0) 1.0). As the filter becomes more aggressive (i.e. with lower top-p thresholds), performance degrades sharply. This experiment demonstrates that these low- www-rrbbbblltyy tokens are not merely correlated with gradient size but are essential for the reasoning improvements achieved by RLVR training.
Takeaway
1. RLVR's gains stem from sparse, high-impact modifications. Our analysis reveals that RLVR's performance gains originate not from a global distribution shift, but from targeted, high-impact changes to a minority of tokens.
2. Logp difference pinpoints these sparse changes. By capturing the direction of prob- ability shifts from base to RLVR, logp difference outperforms magnitude-only metrics like entropy or divergence in isolating the reasoning-critical tokens that RLVR learns.
3. Sparsity originates from RLVR's focus on low-probability tokens. The sparse dif- ference is explained by the inherent concentration of RLVR's gradients on rare, low- probability tokens, making these tokens the focal point for improvement and the source of the sparse, high-4 A log changes we observe.
4 EXPLOITING RLVR'S DIRECTIONAL UPDATES TO BOOST REASONING
Building on Sec. 3, which isolates sparse and directional updates via A log p. we propose two prac- tical strategies to utilize this directional learning: (i) a fest-time selective extrapolation that shifts probability mass further along the learned direction on critical tokens; (ii) a maining-time advan- tage reweighting that prioritizes low- www-oobabbllity tokens implicated by high A Alggp. p. Both methods provide practical ways to boost performance by exploiting the directional mechanisms of RLVR.
4.1 TEST-TIME ENHANCEMENT VIA EXTRAPOLATION
Selective test-time extrapolation along the A log p direction. Our token replacement experiment demonstrated that A log p effectively identifies the reasoning-oritical changes of RLVR. This raises a natural question: Can we move beyond simple replacement and actively amplify these critical changes to surpass the RLVR model's performance? We therefore instantiate token-level extrap- olation: treat A logp = log (.) - log (-ase()) as a learned "reasoning direction" " pointing from base to RLVR distribution. Our strategy is amplify this signal by extrapolating the RLVR model's

---

Published as a conference paper at ICLR 2026
distribution further along this direction. The extrapolated policy = is given by:
log " Exxtra (yee x. M-ct) log TRI (ue Meca) 7 A log Mc+)) + = V-ce)
(7)
= (1 + 7) log TRL (Ve [z, y<e)) - - log a Baso (Ve v + 2(2, Ver),
where y is hyperparameter controlling the extrapolating strength, and z(-) is a log- pert-aattiion func- tion. probability space, this is equivalent to re-weighting the RLVR distribution:
Extra (yr (2, Mcx)) & TRL (y. vcr) exp (7 A log p(ye) x 22,c)))).
This framing connects our method to reward- guidddd decoding literature (Khanov al., 2024; Liu et al., 2024; Xu et al., 2025), where a reward function is used to re-weight the probability distribution. Our A logp p thereby acts as token-level reward that encourages better reasoning in this framework.
Why selective? RLVR's improvements concentrate on a minority of tokens; most positions exhibit negligible A logp. global intervention risks distorting well-calibrated tokens. We therefore apply extrapolation selectively, using Foog to gate positions with large negative A log and sample from the extrapolated policy Textrs only at those positions (substituting WRL. in Algo. 1, Line 6).
Empirical Setup. We evaluate our method on the AIME- 24 benchmark using the ORZ, DAPO, and UniReason model pairs, generating 32 samples per question (see Appx. A.1 for more details). To isolate the impact of our strategy, we compare three approaches: (1) RLVR: The original, non-intervened RLVR model TRL: (2) Selective Replace: Base model with tokens replaced by TRI.; (3) Selective Extrapolate: Base model with tokens replaced by Extra For a controlled comparison, we use the same selection criteria for (2) and (3), with the only difference being the extrapolation.
Extrapolation Performance
58-
Sdectivve Repicce
RLYR
Selective Extrapolate
55.42
04.56
54
$1.7
50
47.50
46
46.15
43.33
0RZ:320 (Qwen2.5)
UniReason- (Qwen3)
(Qwen2.5)
Results. On AIME-24, Selective Extrapolation yields higher Avg @ 32 (average of 32 samples) than TRL across ORZ-32B, DAPO-32B, and UniReason-145 under matched gates (Fig. 4). contrast, Selective Re- place matches but does not surpass the RL baseline under the same criteria. These results indicate that moving beyond TRL along log provides incremental gains in reasoning accuracy.
Figure 4: Extrapolation Performance
Extrapolating on TRL. We also apply selective extrap- olation directly on RRL rather than on Base in Algo. (Line 4). As the threshold 7 in fiog increases, the AIME- 24 performance improves up moderate intervention ratio, after which gains plateau (Table 1). This pattern aligns with the sparsity finding: amplifying limited set of maasoning-critiee tokens is effective, whereas aggres- sive interventions yield diminishing returns.
Table 1: Selective Extrapolate (y = 0.1) on the RLVR model (DAPO-32B) instead of the base model.
Replace Ratio 0.0% 1.8%
5.2%
20.0%
Avg@33 32 Threshold
52.50
53.96
55.31
55.10
N/A -0.5
-0.2
0.0
Theoretical Justification. Following a standard simplification in theoretical analysis for LLM RL training (Munos et al., 2024; Shi et al., 2025; Huang et al., 2025), consider tabular softmax bandit policy: TH TT(y(z)) (z) x exp(6x.y), where the logit is individually parameterized by for each prompt-response pair (ar, y). We assume the policy is trained with Natural Policy Gradient (NPG (Kakade, 2001)) following Cui et al. (2025), since its updates resemble the controlled optimization of PPO (Schulman et al., 2017). The update rule of NPG via backtracking simplifies to: 65+1 6% 1 (2, y), where 17 is the step size and A' is the advantage function (Agarwal al., 2021). this context, our extrapolated policy (Eq. 7) is defined as Tw(0t.y)> where w((t,)) = 8" + 7(0 - 8²). Under these conditions, we have the following theorem (the proof can be found in Appx. D):
Theorem 4.1. For given prompt if tabular softmax policy ze. is updated via natural policy gradient (Kakade, 2001), then the extrapolated policy Tw()".)) satisfies:
37 > O, Eyy ()(()))) [Rayy 2 By (-e) (Rxy).
Equality holds and only if the reward Rg. is constant for all y.
This theorem shows that, in the simplified setting, extrapolating along the learned difference di- rection of A logp can improve the expected reward. Nevertheless, need to note that the proof

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of our reweighting method and DAPO on math reasoning benchmarks.
AIME24
AIME25
AMC
Average
Model
Method
Avg@32
Pass @ 16 Avg@ 32
Pass 16 Avg 32
Pass@ Avg@ 32
Pass @ 16
Base
14.79
47.46
6.67
27.84
40.62
79.25
20.69
51.52
Qwen2.5-
DAPO
35.73
54.09
17.6
30.45
73.04
89.03
42.12
57.86
Math-7B
Ours
39.06
60.58
18.54
36.72
73.64
89.69
43.75
62.33
Base
5.42
30.63
5.73
32.8
27.64
78.09
12.93
47.17
Qwen3-
DAPO
36.98
72.3
26.67
46.76
69.13
88.51
44.26
69.19
8B-Base
Ours
38.13
69.87
31.15
55.38
71.05
92.3
46.78
72.52
relies on the idcalized NPG's update rule, with a monotonic learning process consistently adjusting the logits along the reward's direction. In contrast, our empirical analysis has shown that the up- dates learned by RLVR concentrate only on a minority of tokens, with A log on most tokens being negligible. This disparity motivates our selective extrapolation only on positions with significant difference, which exhibit the consistent, directional updates assumed by the theory.
4.2 TRAINING- TIME ENHANCEMENT VIA ADVANTAGE REWEIGHTING
advantage reweighting, While our test-time approach amplifies the learned reasoning signal post-hoc, our training-time strategy proactively strengthens the model's s reasoning signal during learning. Instead of extrapolating the final logp difference A po we leverage the observed correlation between high A log and low-probability tokens (Fig. 3()),, and propose to amplify the learning signal of these critical low-probability to- kens. Since the parameter update is driven by the advantage term A,, in policy gradient methods, modify the advantage in DAPO (Eq. 4) to prioritize low-probability tokens:
Training-time enhancement via probability
Á = [ + a (1 Te (,i,el,,,,,<))) Bi,c A...
where is hyperparameter controlling the reweighting strength. Such a concentration on low- probability tokens also aligns with our top-p experiment in Fig. 3(c), which finds that low-probability tokens are irreplaceable for RLVR training.
Experimental setup. We modify only the advantage (Eq. 8) in the standard DAPO recipe and keep all other hyperparameters fixed. We evaluate model performance on three math reasoning bench- marks: AIME-24, AIME-25, and AMC. Following DAPO's setup, we use top-p=0.7 7 for sampling during evaluation. We report Avg @ 32 and Pass@ 16 both computed over 32 samples per problem to ensure stable estimate of the pass rates (Chen et al., 2021).
Results: performance gains across models and datasets. We compare reweighting method on two models: Qwen2.5-Math-778 (Yang al., 2024) and Qwen3-8B-Base (Yang et al., 2025a). As shown in Tab. 2, enhanc- ing low-probability tokens' weight consistently improves reasoning accuracy across all tested models and datasets. Notably, this enhanced accuracy (Avg@ 32) doesn't it come at the cost of exploration ability (often measured by Pass @ k) (Yue et al., 2025); in fact, the average Pass @ 16 also increases over the DAPO baseline.
Table 3: Results of various reweighting methods.
Method
PPI.
Dominate
Ours
Avg@ @rg@32 32
35.63
36.35
39.06
AIME24
@ass@ 16
61.95
55.27
60.58
A3
16.46
13.02
18.54
AIME25
Pass @ 16
32.19
20.69
36.72
Avg @ 32
72.06
79.97
73.64
AMC
Pass @ 16
89.1
84.93
89.69
Avg @ 32
41.38
43.11
43.75
Average
Pass 16
61.08
53.63
62.33
Comparison of different reweighting. While
our reweighting method is motivated by the critical role of low-probability tokens, existing work has proposed alternative reweighting strategies that stem from different hypotheses: (1) PPL: Deng et al. (2025) find that RLVR updates favor low-ppl responses, so they reweight advantage to en- hance these responses: Appi = [L - a (pp1((:)] A where (pp) is normalized log-PPL
*With 32 samples, we report the stable Pass 16 instead of Pass 32 for @ evaluation.
8

---

Published as a conference paper at ICLR 2026
AIME24 Auges
Training-- Entropy
Response Length
1400-
6.25
0.30
1200
0.2
Entopy
1100
0.35
Durs PPL
500
030
Dominate DARO
41
Stop
Stop
Stop
Figure 5: Training curves for different reweighting methods on Qwen2.5-Math-
weight. (2) Dominate: Yang et al. (2025b) argue that RLVR training can be over-dominated by low-probability tokens, so they propose to counteract this by upweighting high-probability tokens: Adem = (o--to(vi.)) +1 -=) Ar,-- We implement these methods using their recommended hyperpa- rameters and compare the performance on Qwen2.5-Math-77. As shown in Table 3, our method of directly amplifying low-probability tokens achieves the best overall performance for both Avg 32 and Pass @ 16. The training dynamics in Fig. provide further insight: Our method not only exhibits higher reasoning accuracy but also steady increase in response length. This simultancous increase in performance and length a key pattern in effective reasoning RLVR training (Guo et al., 2025), suggesting the promoted reasoning behavior by our method. Morcover, the training entropy of Adom reweighting clearly lower, since they adopt more restrictive clip-higher ratio of Ehigh 0.24 than the default Ehigh 0.28 in DAPOS. The lower entropy (less exploration) also explains their reduced @ass@k k performance in Tab. 3.
5 RELATED WORK
Reinforcement learning for LLM. Reinforcement learning is a pivotal component of the LLM post-training pipeline. Early applications centered on Reinforcement Learning from Human Feed- back (RLHF) for model alignment (Ouyang et al., 2022; Stiennon et al., 2020), while recent ad- vancements shift their focus to building reasoning models with RL. OpenAI ol (Jaech et al., 2024) is the first reasoning model, and DeepSeek RI (Guo et al., 2025) introduces a detailed RLVR (Lam- bert al., 2024) recipe for building reasoning models with the GRPO algorithm (Shao al., 2024). These seminal works inspired the development of a series of subsequent models, from industrial sys- tems like Kimi(Team, 2025), Qwen3 (Yang et al., 2025a), and Gemini 2.5 (Comanici et al., 2025), to open-source academic algorithms such as Dr. (Liu et al., 2025), Open- Spen-Reaaoner-Zerr (Hu al., 2025a), DAPO (Yu al., 2025), GSPO (Zheng al., 2025), and QAE (Wu et al., 2025), to further improve the reasoning abilities. In this paper, we adopt DAPO as our baseline RLVR algorithm.
Understanding the effects of RLVR. The success of RLVR has prompted a line of research ded- icated understanding its effects. While early work analyzed high-level cognitive behaviors of RLVR-trained models (Gandhi et al., 2025; Hu et al., 2025b; Bogdan et al., 2025), recent studies have deepened the analysis with token-level quantification (Qian et al., 2025; Wang et al., 2025a). Cui et al. (2025) studied the token entropy change during RLVR, Yang et al. (2025b) quantified the gradient norm of specific tokens, and Deng et al. (2025); Meng et al. (2026) used token replacement to measure their impact reasoning performance. A core finding from these analyses is that RLVR induces sparse updates, which have been verified through high-entropy tokens (Wang et al., 2025b), KL. Divergences (Huan al., 2025), and the sparse gradient norm (Yang al., 2025b; Deng al., 2025) or weight update (Yuchen et al., 2026). However, when studying the differences between base and RLVR models, prior studies mainly focus on the magnitude of changes, largely overlooking their direction. While (Yang et al., 2025b) analyzes the update direction (increase or decrease) of probabilities each gradient step, we extend the notion of update direction the full distributional shift from the base model to the RLVR model, and we propose explicitly extrapolating along this learned direction in distribution space.
TThis follows the recommended value in their paper (Yang al., 2025b). We also tested the default ebigh 0.28, but it resulted in unstable training.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
In this work, we introduced directional analysis of RLVR based on the logp difference A logp, shown to be more effective identifying sparse yet reasoning-critical updates than magnitude- based metrics (e.g,, divergence entropy). Building on this, proposed test-time extrapolation to amplify these directional updates and training-time reweighting to focus learning on the low- probability tokens that A log p highlights. Both methods improve reasoning performance across different settings, validating our key principle: diagnose and improve RLVR by its update direction.
Limitations and future work. One primary limitation of extrapolation method is the require- ment of two models; future work could integrate this with parameter-efficion finetuning to reduce computational cost. The extrapolation also introduces additional hyperparameters, and future work can explore combining the selection threshold and extrapolation strength for a more adaptive ex- trapolation. Additionally, our reweighting approach could be evaluated for different model scales or combined with other adaptive training techniques.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility our work, we provide detailed descriptions of our experimental setup, including necessary implementation details and hyperparameter settings in the appendix. Our codes are available in https: //g/ thub. cwm/ Hemn/essee33 RLVR-DE rect iecctiionss inss.
ACKNOWLEDGMENTS
This research is supported by the National Science and Technology Major Project (2023ZD0121 102) and the National Natural Science Foundation China (625B2173, U24B201 80, U25A20445, and 62302321). This research is also supported by the advanced computing resources provided by the Supercomputing Center of the USTC.
REFERENCES
Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021. URL http:// org http://aperss////ee
736. html.
Paul C. Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought anchors: Which 11m rea- soning steps matter?, 2025. URL https: / orgg abs/ 220 19143.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, et al. Evaluating large language models trained on code, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261. 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint vrXiv:2505.22117 2025.
Jia Deng, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, and Ji-Rong Wen. Decomposing the entropy- performance exchange: The missing keys to unlocking effective reinforcement learning, 2025.
Kanishk Gandhi, Ayush K Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cog- nitive behaviors that enable self self-improving reasoners, four habits of highly effective STars. In Second Conference on Language Modeling, 2025. URL https: /pwpereeeee.eeeeee.e forum? derrmm?d=-QGJttxxii..
10

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang. Qihao Zhu, et al. Deepseek-rl incentivizes reasoning in Ilms through reinforcement learning. Nature, 645:633 638, 2025.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zero. An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503. 2025a.
Xiao Hu, Xingyu Lu, Liyuan Mao, YiFan Zhang, Tianke Zhang. Bin Wen, Fan Yang. Tingting Gao, and Guorui Zhou. Why distillation can outperform zero-rl: The role of flexible reasoning. arXiv preprint arXiv:2505.2 20067, 2025b.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Ilm capabilities? understanding transferability of Ilm reasoning. arXiv preprint +:XXi:2507.004322 2025.
Kexin Huang, Junkang Wu, Ziqian Chen, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, and Xiang Wang. Larger or smaller reward margins to select preferences for LLM alignment? In Forty-second International Conference on Machine Learning, 2025. URL https: http////ppeeeeeeeeeeeeeeeeeeeeee net. forum? id=nidwgagrreeeee
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1rXiv:2412.167200 2024.
Sham M Kakade. A natural policy gradient. In T. Dietterich, S. Becker, and Z. Ghahramani (eds.), Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001.
Maxim Khanov, Jirayu Burapacheep. and Yixuan Li. ARGS: Alignment reward-guided search. In The Twelfth International Conference on Learning Representations, 2024. URL https: / /oopnnevieww net//eeeeeeeeeeee wwwp:/eeeeettttteeeeeeeeeee
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint 2xxiv:2411.15122. 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman- Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models. In Proceedings of the 36th International Conference on Neural Information Pro- cessing Systems, NeurlPS '22, Red Hook, NY, USA, 2022.
Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares-López Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding- time realignment of language models. In Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 31015-31031. PMLR, 21-27 Jul 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint 2:Xii:2503.20783.. 2025.
Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma. Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, and Jingren Zhou. Sparse but critical: A token-level analysis of distributional shifts in RLVR fine-tuning of LLMs. In The Fourteenth International Conference Learning Repre- sentations, 2026. URL https:/ httt://s//ggeeeeeeeeeeeeee/eedea
Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. Nash learning from human feedback. In Forty-first International Conference on Machine Learning, 2024.
11

---

Published as a conference paper at ICLR 2026
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems (NeurIPS), 35:27730-27744, 2022.
Chen Qian, Dongrui Liu, Haochen Wen, Zhen Bai, Yong Liu, and Jing Shao. Demystifying reason- ing dynamies with mutual information: Thinking tokens information peaks in IIm reasoning. arXiv preprint arXiv:2506.02867, 2025.
Yi Ren and Danica J. Sutherland. Learning dynamics of LLM finetuning, In The Thirteenth Inter- national Conference on Learning Representations, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint (rXiv:/707006344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03330, 2024.
Ruizhe Shi, Runlong Zhou, and Simon Shaolei Du. The crucial role of samplers in online direct pref- erence optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https: wwwp/eveeweeeeeeeee/ee/ee feret/ooommideee 32
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 33:3008-3021, 2020.
Kimi Team. Kimi k1.5: Scaling reinforcement learning with Ilms. arXiv preprint 2259990..1999.. 2025.
Qwen Team. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2024.
Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hi- erarchical reasoning in Ilms through reinforcement learning. arXiv preprint rrXi:25009006646, 2025a.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.01990 2025b.
Junkang Wu, Kexin Huang, Jiancan Wu, An Zhang, Xiang Wang, and Xiangnan He. Quantile advantage estimation for entropy-safe reasoning. arXiv preprint wtXiv:2509.226111 2025.
Yuancheng Xu, Udari Madhushani Schwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. GenARM: Reward guided generation with autoregressive reward model for test-time alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL pectps://operzeeete
n forum?id=JJ
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li. Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-matt technical report: Toward mathematical ex- pert model via self- eef--imprvvemen,, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint wwXiv:2505.09388 2025a.
Zhihe Yang, Xufang Luo, Zilong Wang, Dongqi Han, Zhiyuan He, Dongsheng Li, and Yunjian Xu. Do not let loooribbyyyy tokens over-dominate in rl for 1lms. arXiv preprint Stt::2505.19299 2025b.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
12

---

Published as a conference paper at ICLR 2026
Cai Yuchen, Ding Cao, Xin Xu, Zijun Yao, Yuqing Huang, Benyi Zhang, Zhenyu Tan, Guiquan Liu, and Junfeng Fang. On predictability of reinforeement learning dynamics for large language models. In The Fourteenth International Conference Learning Representations, 2026. URL https:/ /ppeeeevvvvve ew net/ / forum? d- d-SdHAA
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arX1v:2504.13837. 2025.
Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shao- han Huang, Lei Cui, Qixiang Ye, al. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673. 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025.
13

---

Published as a conference paper at ICLR 2026
A SELECTIVE TOKEN REPLACEMENT & EXTRAPOLATION
A.1 IMPLEMENTATION DETAILS
Models. Our experiments use several publicly available RLVR-trained models and their correspond- ing base models from the Qwen series (Yang et al., 2025a; Team, 2024):
ORZ: The O Reasoner-Zero-322 model (Hu et al., 2025a). finetuned from Qwen2.5-32B base model using the PPO algorithm.
DAPO: The DAPO-Qwen-322 model (Yu et al., 2025), finetuned from the same Qwen2.5-32B
base but with the DAPO algorithm.
UniReason: The UniReason- -oressoon-Qeen-11B--R model (Huan et al., 2025), finetuned from Qwen3- 14B-Base using the GRPO algorithm.
Sampling settings. We utilize the AIME-24 dataset evaluate the replacement performance. We adopt the default chat prompt template from each model, with the user prompt defined follows:
[Question]
Please reasen stop by step, and put your final answer within Abboxed
We the sampling parameters with top-p=0.7, temperature=1.0, max-length=20k, and sample 32 responses for each question. The answer is extracted from the last "boxed" wrapped text and verified using Math--errff.. We report the correctness averaged over 32 samples, i.e., Avg@
Hyperparameters for extrapolation. As described in Algo. 1, the replacement is adopted selec- tively, controlled by the threshold adjusted by the parameter in Extra For the extrapolation results in Fig. 4, the "Selective Extrpo- late" and "Selective Replace" methods share the same hyperparameters for each model, which we summarize as follows:
the criteria function f", while the extrapolation strength is
Table 4: Hyperparameters for the extrapolation results (Fig. 4).
Model
ORZ
UniReason
DAPO
Threshold for fTop
-0.4
-0.35
-0.3
Replaced Ratio
10.1%
7.5%
11.4%
? in 77 Exxtra
0.1
0.1
0.05
A.2 ADDITIONAL EXPERIMENTS
Additional metrics. As described in Sec. 3, primary metrics for token replacement are the base model's 's entropy HBase. KL Divergence DKI, and logp difference log p. For our ablation study, include additional metrics: the RLVR model's entropy HRL and KL-divergence variants: DKLL FRL and DKLL TRL We evaluate these metrics as criteria for the DAPO model's selective replacement. By varying the threshold 7 for each criterion, we control the token replacement fre- quency and plot the performance on AIME-24 against various replacement ratios in Fig. 6. As shown in the figure, although the additional metrics' selected replacements also approach the RLVR model's 's performance, they still require more replacement than A log does. This confirms the per- formance ordering for identifying reasoning-critiee tokens: logp difference divergence > entropy.
Selected Tokens.
To provide an intuitive comparison of the metrics, we analyze the tokens utilized for replacing the base model's choice during DAPO's token replacement of entropy H KI. Di- vergence RL, and logp difference A lo To ensure a fair comparison, we adjust the threshold for each metric to achieve replacement rate of approximately 8%. Fig. 7 illustrates each criterion's top 50 substitution tokens. The figure reveals that entropy- -based selection favors logical transi- tion words (e.g.. Thus, need, can), while the divergence and A log criteria utilize more specific mathematical reasoning tokens, including higher proportion of math symbols. Combined with the inferior performance of the entropy criterion, this suggests that these specific mathematical tokens might be more efficient for improving reasoning performance.
14

---

Published as a conference paper at ICLR 2026
Performance Across Varying RLVR Replace Ratios (on DAPO-32B/Qwen2.5)
52.50
Logp Difference dlogp
AIME24 Avg@32
Forward Divergence D
Reversed KL Divergence D
30
Averaged KL Divergence D
Entropy Haw
RLVR Entropy Ho.
20
Random
Base: 6.67
80
100
RLVR Replace Ratio (%)
Figure 6: Selective token replacement results with additional criteria for DAPO.
Top 50 Replace Tokens by Different Metrics
By Entrapy Hager
By KL. Divergance D
By Logp Difference alogp
105
196
165
115
1862
T16
'/54
1/65
11.0
1100
1.4
3,5
2.4
2.5
3.0
Porcentage Repiace
Parcentage
Takons
Perconiage
Figure 7: Top tokens for replacing the base model's choice under different metrics' selection.
Per-Prohlem Accuracy during Replacement. We also report the per-problem accuracy changes in the token-replacement experiment in Fig. 8, to more finely examine how gradually increasing the replacement ratio affects model performance. We observe that: (1) There exist some problems that inherently diffieult for the model, for which the accuracy remains ratios. (2) For the remaining problems, the overall trend is that accuracy generally increases as the replacement ratio grows, and then begins to fluctuate. This is consistent with the fact that, when only performing token replacement, the performance is ultimately capped by the upper bound of the RLVR model. (3) For a small number of problems, accuracy initially drops when we introduce small amount of replacement, and then begins to improve as the replacement ratio continues to increase (e.g., problem of DAPO). qualitative inspection of these cases suggests that, for some of them, small number of RL-replaced tokens introduce token options that the base model is not familiar with. As result, the base model fails to continue the generation coherently, leading to an initial degradation in accuracy. However, when further increase the replacement ratio. the generation becomes more strongly guided by the RL tokens, and the model's performance on these problems recovers and improves.
across all replacement
15

---

Published as a conference paper at ICLR 2026
1000
(a) Per-problem accuracy on AIME24 DAPO's token replacement experiment
Per-problem accuracy on AIME24 of ORZ's token replacement experiment
(c) Per-problem accuracy AIME24 of UniReason's 's token replacement experiment
Figure 8: Per-problem accuracy changes on AIME24 during each model's selective token replace- ment experiment. We report the results with A log p being the selection criterion.
A.3 HYPERPARAMETER SENSITIVITY ANALYSIS
Our test-time extrapolation distribution 2h Extra introduces a hyperparameter that determines the strength of extrapolation along the learned A log direction. This intervention operates within the token replacement procedure (Algo. 1) and is applied only to tokens selected by the criterion A logp 7. To verify the robustness of the performance gain of extrapolation over simply replac- ing the token from TRL. we perform grid search over both y and the token-selection threshold т. We evaluate 7 E {0.05, 0.1} and vary 7 across different ranges for different models. For DAPO and ORZ, we test E (-0.5. -0.4, -0.3, -0.2, -0.1.. For UniReason, we adopt a denser grid T E (-0.5, 45, 0.4, 00.35. -..3 because relatively few replacements are needed to reach the
RLVR performance level (Fig. 2).
As shown in Tab. 5, across nearly all models and hyperparameter settings, extrapolation consistently outperforms the replace-only variant, demonstrating strong robustness of our method. Notably, once the replacement ratio is sufficiently high to match the RLVR's performance, further increases in replacement provide little no additional benefit, since the performance is bounded by the RLVR model itself. In contrast, a proper test-time extrapolation can further exceed RLVR performance by 1-3 points without any additional training.
16

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameter sensitivity analysis for the selective extrapolation experiment. The "sign marks the reported value for extrapolation results in Fig. 4, while the sign corresponds to the end point in token replacement of Fig. 2.
(a) Hyperparameters and Avg @ performance on AIME24 of DAPO (Avg@ of TRL: 52.60).
Threshold t
-0.5
-0.4
-0.3
-0.2
-0.1
Average Replace ratio
8.8%
10.0%
11.4%
13.4%
16.5%
Replace w/ TRL.
51.981
51.56
51.67
52.71
51.98
Extrapolate w/ y 0.05
51.88
53.02
55.42*
54.06
54.9
Extrapolate w/ 7 = 0.1
54.17
53.33
53.85
53.85
54.27
(b) Hyperparameters and Avg 322 performance AIME24 of ORZ (Avg@32 of mnL: 46.15).
Threshold
-0.5
-0.4
-0.3
-0.2
-0.1
Average Replace ratio
9.5%
10.1%
10.8%
11.6%
12.7%
Replace w/ TRL.
43.65
43.33
46.151
44.90
42.81
Extrapolate w/ y 0.05
47.19
45.52
45.83
46.25
43.44
Extrapolate w/ = 0.1
43.75
47.50
45.52
47.08
45.42
(c) Hyperparameters Avg@32 32 performance on AIME24
UniReason (Avg@32 32 maL: 54.58).
Threshold Average Replace ratio
-0.5
-0.45
-0.4
-0.35
-0.3
5.4%
6.0%
6.8%
7.5%
8.5%
Replace w/ TRL
53.65t
53.33
53.12
54.06
53.54
Extrapolate 7 = 0.05
51.88
54.79
53.54
55.00
54.69
Extrapolate w/ 0.1
54.37
53.75
53.96
55.83*
55.10
B RLVR TRAINING DETAILS
Hyperparameters Setting, We adopt the open-sourced DAPO recipe for RLVR training. Our configuration includes double clip ratios (Clow 0.2 and Chigh = 0.28) and learning rate of 1e-6 with a 10-step warmup. Each RLVR step consists of 512 prompts with sampled responses each, processed in mini-batches of 32 prompts to yield 16 gradient updates per step. Maximum generation length (and overlong penalty thresholds) are set to 8k (4k) for Qwen2.5-Math-77 and 20k (16k) for Qwen3-88 -a-3---bb,se,, respectively.
For reweighting, our parameter a (Eq. 8) is set to 0.2 for Qwen2.5 and 0.1 for Qwen3. Following the recommended values by Deng et al. (2025) and Yang et al. (2025b), we set a to 0.1 for Adom and 0.01 for 7PPL For Ādom specifically, also adjust Chigh 0.24.
AIME24 Avg@32
Training time Entropy
Response Length
0.65
1400
0.35
17:00
21 0.25
Length
0.020
0:30
0.15
Run
0.25
Run2
0.20
500
Rund
150
350
600
Stap
Step
Step
Figure 9: Reproducibility analysis. The learning curves across 4 independent runs on Qwen2.5- Math-7B with reweighting method (Bq. 8) show consistent convergence and performance.
Reproducibility Analysis. To account for random variations in the RL process, also performed four separate training runs on the Qwen2.5-Matte displays the learning curves for these experiments (Run 1-4, where Run is reported run in
backbone for our reweighting method. Fig.
17

---

Published as a conference paper at ICLR 2026
Fig. 5). The results indicate that method is highly reproducible; across all trials, the model reached or surpassed the performance levels presented Tab. 3.
C PERFORMANCE BEYOND PURE-MATH REASONING TASKS
Although our models are primarily trained and evaluated on math-focused datasets, it is important to assess their reasoning ability non-math tasks to evaluate generalization ability. Following prior work (Zhao et al., 2025), we use the Minerva dataset (Lewkowycz et al., 2022), which contains 272 undergraduate-llvel STEM problems spanning diverse subjects such Chemistry and Astronomy".
We begin by benchmarking the RLVR-trained models on Minerva using the same sampling param- eters in other evaluations (e.g.. AIME24). As shown in Tab. 6, models trained with reweight- ing method continue to outperform baselines in reasoning accuracy. Importantly, these gains do not come the expense of exploration ability, as reflected by comparable improved Pass k scores.
We further evaluate test-time extrapolation on Minerva. Because Minerva is substantially larger than AIME24 (around 7 times more questions), report Avg@8 for the evaluated 4-B32B models. As shown in Fig. 10, test-time extrapolation consistently improves over the RLVR model's accuracy, validating its generalization ability beyond pure-math datasets. We also report the hyperparameter grids in Tab. 7, where the extrapolation results also consistently outperform replacing with TrL only.
Table 6: Performance of RLVR-trained models on Minerva.
(a) On Qwen2.5-Math-7B
Method
Base
DAPO
PPL
Dominate
Ours
Avg@32
18.35
46.43
48.68
47.01
49.72
Pass @ 16
61.04
69.44
68.69
64.59
70.37
(b) On Qwen3-8B Baan----BBBeee
Method
Base
DAPO
Ours
Avg@32
29.8 70.43
55.04
56.57
Pass @ 16
76.98
76.78
Extrapolation Performance
Selective Replace
58.27
RLVR
58
Selective Extrapolate
Minerva A
57.17
66.44556.6
56.63
56.42
56.16
56
55
54.1.44.3
53
UniReason-148 (Qwen3)
ORZ-328 (Qwen2.5)
DAPO-328 (Qwen2.5)
Figure 10: Extrapolation results on Minerva.
TThe dataset is named OCWCourses in the paper, which can he found in httpa://openreviii net vet/attachment wwwp://wwwwdttaaeemammaame ssppp emmnta mrmmentay__mterria
18

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameters and Avg@8 performance on Minerva benchmark. The sign marks the tuned value in Fig. 10.
DAPO
ORZ
UniReason
Threshold т
-1.0
-0.9
-1.0
-0.9
-1.0
-0.9
Avg replace ratio
6.5%
7.0%
9.2%
9.6%
1.8%
2.2%
Replace w/ TRL
56.63
56.43
56.41
56.39
54.00
54.14
Extrapolate w/ y 0.05
56.8
57.22
57.17'
57.08
54.50
54.50
Extrapolate w/ 7 0.1
58.27
56.57
55.51
55.28
54.32
56.16*
D PROOFS
Proof of Lemma 3.1. For ease of notation, we omit the context 2, Di,ce here. The derivative of DAPO unclipped token Vi, is:
To( Are
(OJDAPO(W.N) Vo riee())A,, V
Toaia (Ho,e)
(0) Vo log TO (Bi.))
Wie V log (31.))
For the softmax-pareen eeetmax-parameterizeed policy no with logits 2 for yi assuming Vi, corresponds to index of vocabulary V, have:
0
1
a exp(z*)
log To (B)r))
Oz,
To (sk.e)
dz, Σi exp(=1)
exp(=x) En exp(a)) (opn(r))-xppaaaa exp(=x) exp((x)))((xx)
1
(2 exp(x:))
(vr.e)
exp(ex) ) exples)
#
[2 exp():))
To (v()),
j=b
To (V,),
j k
T(j k) - To(V))
So the fl-norm of V (=JDAPO()...) becomes:
V :=TTAPo((M.e))
= |1,0,, V. log To (
|wiael Σ T(j = k:) - To [o()
|warrl (1 To(Ui.e) Σ no((i))
(yr.e = Va)
5tk
|waal 2(1 To(Vi.)))
Proof of Theorem 4.1. Let [(,)) = Eyyat, ()) RR..y and we need to show that for each .r:
3y > 0. ( + 70' 0%)) 2 J(
Denote the extrapolation direction d= = 0 0 this is equivalent to showing the directional derivative of J at 6% along d' is positive.
The directional derivative is given by:
d
0J((2)
Va') Vo. ((2)T
&ey
|
00
19

---

Published as a conference paper at ICLR 2026
For the softmax policy To, (v) exp(0x.u)/ Ey exp(fx.,) its gradient satisfies: TA= (1() - s/) - -oo (1))
@Tg, (v/)
00z.y
So the partial gradient of on y is:
(0a)
@To_(1) (v')
Σ Ra..'
T To, (v) Σ Rayyya To, (v)) To, (v)( =0. Rx). Rz To, (1)-TO,
20z.y
80..y
Note that the advantage is A' y) = Ra,9 The Rg under the bandit setting, the directional deriva- tive thus becomes:
Va J(0') = [dgel
Σ
Tos (v) - Top Rs ddz.
y
Σ
Tos (v) - -A'(x, d'e
[dgll
We now analyze the order of A' y) and d'
Under the assumed bandit setting, the order of A y) is the same as the order of Ra.y. i... A 21) > Axx, v2) if and only if Rz,z1 > R2,22 For d' we can prove that its order is also the same as Rz,v with induction.
At 1, using the update rule of NPG, we have:
d2,, dray = (z, (z, v/)) = 1) (Ray Re
So the order of dz is the same as Re, Assume at iteration 1, the order of d' is the same as R then at iteration t + 1, we have:
2. d' d'as + (A'(2, ('xx, dy dr.yy + + (Re,y R..yr).
So we still have d'+1 > dhty
+ Ro > Rayy Thus by induction, the order of d' is the same
as Ry for all Since the order of A' v) and d' are the same, can apply the Chebyshev inequality to get:
(I Tes (y) d
Σ Tay (v) y
Σ THx (v) Ax(,)) &aw 2 =e (v) (v) A.(,))))))))
with the equality holds if and only if A' v) d'a,y is a constant for all y (i.e., constant reward).
Note that the expectation of advantage Ey Tog (v) A v) O. so we have:
V cc J(0)
Σ Tag (v) A A v) d 0.
|
The equality holds if and only if Re,y is constant for all y.
E STATISTICAL COMPARISON OF DIFFERENT METRICS
Empirical setup. We evaluate three RLVR models: ORZ, DAPO, UniReason, and their base coun- terparts. For each model, we generate 32 responses per question from the AIMB-24 dataset, with a sampling strategy of top-p=0.7 and temperature=1.0. Our analysis focuses on several metrics comparing the model pairs: the base/RLVR model's entropy, KL divergences, and the logp differ- ence. The probability distribution versus different A logp bins in Fig. 3(b) is also measured on the DAPO's generation under this setting.
20

---

Published as a conference paper at ICLR 2026
Statisties of Different Metrics. We compute each metric of the three RLVR model pairs on both the base model and the RLVR model's generation. As shown in Fig. 12, the distribution of logp difference A logp is bimodal, with positive tail for the RLVR's generated text and a negative tail for the base model's generation. In contrast, the distributions of other magnitude-hase metrics are nearly identical regardless of which model generated the output (Fig. 13-15).
Word Clouds of High-A logp Tokens. To gain qualitative insight into the tokens identified as higher A log p. whose probabilities are substantially increased by the RLVR training process, generated word clouds from the top-100 high-A logp tokens for each model (Figure 11). As the figure shows, these tokens correspond words related to problem-solving. They fall into two clear categories: explicit reasoning actions (e.g., combine, break, simplify) and logical transitions (e.g., wait, think, step). The prevalence of this vocabulary suggests that the RLVR model has learned to construct more effective reasoning processes.
F THE USE OF LARGE LANGUAGE MODELS
We utilize LLMs only to polish some of the language of this paper. All content was originally drafted by the authors. The use LLMs was restricted refining some pre-existing text, and any suggested modifications were reviewed by the authors to confirm their accuracy and alignment with original meaning.
21

---

Published as a conference paper at ICLR 2026
Top-300 Tokenss by Log Probability Difference
Factor
after
correspond
Notice
plug
essenti Eally
make
It
Rec
By
expand
Expand
Brtrig
ob
rearr
tain
translatos
Also
scuullt
her
e
After
However
split
Here
separately
break
vectors
determine
constraint
quite
Certain
Combine
key
S
unles
passpassiin
lenofftex
until
Each
Rew
Assistan
talt
Upon
perfor
structure
express
L
e
Calcul
Fact
concept-
Specifically
But
S ste Step
And
Speci
having
recali
involves:.n
Here
those
approa
expais
Lon
cos
a
sys vestematti Semmaiicaln
expands
setup
gebra
Perforn
ano
represent diffe differentiy
Recall PI
Suppose eliminate
(a) Top log tokens DAPO
Top 300 Tokens by
Probability Difference
First actually
time
Alternative
algebra
expre ress
Another
un
unless
understar
a
fially
keep?
Lon3
Instead
needs
Genera
Exp cxsto
what
ques
concept
Wrrtt
Instead
Notice
obvicus
P
Problem
Point
figure
simply
Ass
properly
whoLe
Logical
O
haviing
b
makk
reasoned
unitially
Rew
Express
Visual
Consider der
lingng
interprets
A1
Sumerry
notice
Coordinated
But
Quality
rethink
may
Wes
K
extra
Ri
interprat
Coordinat
dinate
That
Banydhin
Epprooe
Understandin
perfora
enee
explore
yer
verification
on
ral
effectively
doubleg
Notice whether relation
(b) Top A log p tokens of ORZ
Top 100 Takens by Log Probabiltty Difference
sccraing
determinge
ff tond check
on
determine
could
define
Linerreeiigg
spee
Again
spreents
Yet
know
Again
properly
a
LSO
clarify
Pepres
If
Wait
ins
Ass
depands
ref
reflection
indicates
confirm
again
Wa
strategy
veriey
get
g
gives
relate
Conttter
CoUncLde
meaning
general
previousty
by
Unders Undern
implies
symmetre
try
assumption
mose comp
doesn
fix
what
Shour
ex
Fure
Statarent
relative
easier
O
Sub
It
Express
Ste
And
reiteeing
Deternine entire further
still
(c) Top A log tokens of UniReason
Figure 11: Word clouds of top A log p tokens, measured w/ different RLVR-trained models.
22

---

Published as a conference paper at ICLR 2026
Logprob Difference UniReason
generatioos
generations
10'
Wittin (-0.25.0.253 97.5%
WWitii (-2.25,0253 M2.2%
109
105
10a
10
Ouant
10
100
101
1.07
107
10
Diffeorance
Difference
(a) Logp difference of UniReason
Logprob Difference of DAPO Base
On Base generations
On RLVR generations
10a
Weht E (..5.05). 90.5%
Within 1 6.5.0.5) 90.9%
10a
10
10"
10
100
10c
10
107
-20
-15
-10
30
15
20
25
Logprob Difference
Logpesb Difference
(b) Logp difference of DAPO
Logprob Difference of ORZ Base
On Base generations
On RLVR generations
1.05
167
10a
10
107
Ohumi
102
1.0
101
107
-25
-10
Logorob flfrrenc
Logeres Difference
(c) Logp difference of ORZ
Figure 12: Logp Difference histograms of different RLVR models, comparing the RLVR and base model's 's generations.
23

---

Published as a conference paper at ICLR 2026
12
(a) Divergence on UniReason's ' generations
(b) Entropy on UniReason's generations
Kow Box
Unihaaii Bice Rudde brbayy Back
19
(c) Divergence on base's generations
(d) Entropy on base's generations
Figure 13: Divergence and entropy histograms of UniReason and its corresponding hase model measured on UniReason or the base model's generations.
modath Eirrgy Ion pennntirn
12
(a) Divergence on DAPO's generations
(b) Entropy on DAPO's generations
Hode Ertoog saus pantatoua
140
11
0
(c) Divergence on base's generations
(d) Entropy on base's 's generations
Figure 14: Divergence and entropy histograms of DAPO and its corresponding base model measured on DAPO or the base model's generations.
ond Base
0P2 Generdton
(a) Divergence on ORZ's s generations
(b) Entropy on ORZ's generations
14
(c) Divergence on base's 's generations
(d) Entropy on base's 's generations
Figure 15: Divergence and entropy histograms of ORZ and its corresponding base model measured on ORZ or the base model's generations.
24


Reviews:
[{"review_id": "WyGExCriFp", "reviewer": "Reviewer_zikJ", "summary": "The paper analyzes reinforcement learning with verifiable rewards through how probabilities shift between the base and RLVR-trained models. The paper finds that upweight advantages of low-probability tokens during RL training improves reasoning accuracy.", "questions": "1. Can you provide examples of tokens with large ∆log p? What qualitative insights do they reveal?\n2. Will amplifying the penalty of negative tokens also bring benefits as indicated in this work [1]?\n\n[1] The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning. NeurIPS 2025", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed advantage reweighting methods is straightforward to implement. Despite the simplicity, it consistently improves reasoning performance across multiple models (Qwen2.5, Qwen3) and benchmarks (AIME, AMC).", "weaknesses": "1. How sensitive are performance gains to γ and τ remain unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K8yQQwHxaG", "reviewer": "Reviewer_aDRb", "summary": "This paper argues that the direction of token-level updates, measured by the signed log-probability difference between base and RLVR models, is more informative than magnitude-based metrics for understanding RLVR's effect on LLM reasoning. The authors propose and validate two methods, test-time extrapolation and training-time advantage reweighting, which exploit the directional insight to improve reasoning performance.", "questions": "How does the proposed direction-based extrapolation perform in non-mathematical reasoning tasks?\n\nThe token-replacement experiment convincingly shows that \\Delta\\log p identifies critical tokens. However, does this intervention sometimes hurt the performance? \n\nIn which instances does replacing a base model's token with the RLVR model's choice result in an incorrect answer, and what characterizes these tokens?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel and intuitive directional metric that effectively captures sparse, reasoning-critical updates. The metric is supported by rigorous token-replacement experiments and gradient analysis.\n\nThe proposed methods are simple yet effective, with consistent gains across multiple models and benchmarks without the need for additional training data.", "weaknesses": "The test-time extrapolation method requires access to both the base and RLVR models, which may limit its practicality in settings where only the fine-tuned model is available.\n\nThe paper focuses primarily on mathematical reasoning benchmarks (e.g., AIME, AMC); it remains unclear whether the findings generalize to other reasoning domains or more diverse tasks.\n\nTheoretical justification in Theorem 4.1 relies on a simplified tabular softmax bandit setting, which may not fully reflect the complexity of modern LLM training dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1gq1V2An1R", "reviewer": "Reviewer_L9QX", "summary": "This paper emphasizes the direction of RLVR updates via a token-level signed metric Δlog p and shows it better localizes sparse, high-impact tokens than entropy or KL. Building on this, the authors propose (i) selective test-time extrapolation that nudges the policy along Δlog p at a small set of salient tokens, and (ii) a training-time advantage reweighting that emphasizes low-probability, reasoning-critical tokens. On math-reasoning benchmarks, both strategies improve over baselines.", "questions": "1. Direction correctness：Could you show the rate at which positive-advantage tokens are increased and negative-advantage tokens are decreased (signed Δlog p), tracked over training, and compare this to arXiv:2505.12929\n2. Sensitivity to τ and γ (robustness): Could you share the performance across a grid of τ and γ to identify a stable operating region rather than a single tuned point.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.Clear idea. The paper puts the direction of change (signed Δlog p) at the center, not just the size.\n2. Practical methods. The two tools—selective test-time extrapolation and training-time reweighting—are straightforward and easy to try.\n3. Low integration cost. The approach needs only log-probs from a base and an RL model; it’s a small code change with no extra labels.\n4. Token-level evidence. The replacement test shows that a small set of tokens drives most of the gains, making the story concrete.", "weaknesses": "1. The paper asserts prior work focuses on magnitude and neglects direction, which is only partly true. While \"Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning\"（arXiv:2506.01939) is largely magnitude-centric (entropy-based selection), Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs (arXiv:2505.12929) explicitly measures for positive-advantage tokens and proposes methods to reduce direction errors(Fig3). Please revise the positioning to acknowledge signed-direction analysis and clarify that your contribution is to directionality (e.g., Δlog p) more directly.\n2. The comparison in the token‐replacement experiment may misinterpret prior works’ use of entropy or KL divergence, which aimed to guide optimization directions during training rather than to describe post‐training entropy distributions — please clarify this distinction.\n3. Hyperparameter robustness. The method relies on a gate τ (token selection) and an extrapolation strength γ (fixed at 0.1). A lack of sensitivity analysis leaves open whether gains are robust across tasks or temperatures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0B5wUCNgjl", "reviewer": "Reviewer_yncQ", "summary": "The paper identifies the direction of policy updates as a critical yet underexplored factor underlying RLVR’s effects. The authors show that reformulating the sampling policy using $\\Delta \\log p$ between the RL policy and the base policy yields improved performance on math-focused datasets. They also provide a concise theoretical explanation for this phenomenon and introduce a simple, training-based method that further strengthens the model.\n\nOverall, the paper is well organized and easy to follow. I enjoyed reading it and believe it offers useful insights to the community. However, the experimental evaluation is limited to math-related tasks. Expanding the experiments to other domains—such as coding, agentic tasks, or logical reasoning—would substantially strengthen the work.", "questions": "1. Could the authors clarify the y-axis in Figure 1(b)? It appears to be neither raw counts nor frequency (since it exceeds 1 at some points). In addition, how were the tokens collected—how many tokens, how many RL training steps, which tasks, and which algorithm?\n2. In Figure 2, replacement stops when performance reaches the RLVR value. What happens if the RLVR replacement ratio is increased further for the three curves?\n3. What is the effect of applying the test-time method on the model trained with your training-based method? Does the combination yield additional gains?\n4. Could the authors provide results on at least one non-math task, as noted above? Adding another domain would significantly strengthen the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and easy to follow.\n2. The observed phenomenon is important and, to my knowledge, has not been systematically explored; it offers meaningful insight.\n3. The theoretical explanation is concise yet sufficiently direct to convey the key ideas. The references are appropriate, and I did not notice overclaims.\n4. The proposed methodology is simple to implement and appears effective. If the results generalize, it could be broadly useful to practitioners.", "weaknesses": "1. The experimental setup is restricted to math-related tasks. It is unclear whether the gains extend to other tasks such as coding, agent-based evaluation, or logical reasoning.\n2. Minor issues (easy to address):\n   - The “averaged KL divergence” defined in lines 133–134 corresponds to the Jensen–Shannon divergence, which has a standard name.\n   - Figure 1(c) uses “token replacement,” but the corresponding setup is only introduced in Section 3.2. I recommend adding a hyperlink or forward reference around lines 073–077 to improve readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)