e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs
Authors: Amrith Setlur
Matthew Y. R. Yang
Charlie Victor Snell
Jeremiah Greer
Ian Wu
Virginia Smith
Max Simchowitz
Aviral Kumar
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
reasoning
test-time compute
RL
exploration
Primary area: foundation or frontier models, including LLMs
TL;DR: We identify three key ingredients to teach LLMs to explore in-context and improve performance when we extrapolate test-time compute beyond what the LLMs are trained for.

Abstract: Test-time scaling offers a promising path to improve LLM reasoning by utilizing more compute at inference time; however, the true promise of this paradigm lies in extrapolation (i.e., improvement in performance on hard problems as LLMs keep "thinking" for longer, beyond the maximum token budget they were trained on). Surprisingly, we find that most existing reasoning models do not extrapolate well. We show that one way to enable extrapolation is by training the LLM to perform in-context exploration: training the LLM to effectively spend its test time budget by chaining operations (such as generation, verification, refinement, etc.), or testing multiple hypotheses before it commits to an answer. To enable in-context exploration, we identify three key ingredients as part of our recipe e3: (1) chaining skills that the base LLM has asymmetric competence in, e.g., chaining verification (easy) with generation (hard), as a way to implement in-context search; (2) leveraging "negative" gradients from incorrect traces to amplify exploration during RL, resulting in longer search traces that chains additional asymmetries; and (3) coupling task difficulty with training token budget during training via a specifically-designed curriculum to structure in-context exploration. Our recipe e3 produces the best known 1.7B model according to AIME'25 and HMMT'25 scores, and extrapolates to 2x the training token budget. Our e3-1.7B model not only attains high pass@1 scores, but also improves pass@k over the base model.

Published as a conference paper at ICLR 2026
e3: LEARNING TO EXPLORE ENABLES EXTRAPOLATION OF TEST-TIME COMPUTE FOR LLMS
Amrith Setlur?,; Matthew Y. R. Yang Charlie Snell", Jeremy Greer", Ian Wu',
Virginia Smith", Max Simchowitz', Aviral Kumar"
Carnegie Mellon University, PC Berkeley, 9Oumi
ABSTRACT
Test-time scaling offers a promising path to improve LLM reasoning by utilizing more inference compute; however, the true promise of this paradigm lies in extrap- olation (i.e., improvement in performance as LLMs keep "thinking" for longer, far beyond the maximum token budget they were trained on). Surprisingly, we find that most existing, open-source reasoning models do not extrapolate well. We show that one way enable extrapolation by training the LLM to perform in-context exploration: spending token budget on chaining operations (generation, verifica- tion, summarization, etc.) or testing multiple hypotheses before committing to an answer. To enable in-context exploration, we identify three key ingredients as part our recipe e3: (1) chaining skills that the base LLM has asymmetric competence in, e,g. chaining verification (easy) with generation (hard), as a way to imple- ment in-context search; (2) leveraging "negative" gradients from incorrect traces amplify in-context exploration during RL, resulting in longer traces chaining additional asymmetries; and (3) coupling task difficulty with training token budget via peeifiicall-desiinee RL curriculum structure this in-context exploration. Our recipe e. produces one of the best known 1.7B models on AIME/HMMT 25, and extrapolates 2.5 the training token budget. Our e3-1.7B not only performs well, but it can go beyond the problem-solving capabilities of the base model itself.
INTRODUCTION
Test-time scaling boosts large language model (LLM) performance by spending more compute on "thinking" before producing answer. Its ultimate promise lies in enabling models to continue improving performance by scaling test compute at deployment. E.g. if the model can learn to implement "algorithmic procedures" like planning, self-refiection, or backtracking, it can discover more accurate responses as more test compute is used. With this motivation, current recipes post-train LLMs via reinforcement learning (RL) (DeepSeek-AL) et al., 2025; Yu al., 2025) and supervised fine-tuning (SFT) (Team, 2025; Muennighoff et al., 2025) at long context windows. However, it is unclear whether these new models can truly realize the promise of extrapolation: if we scale test compute beyond the maximum training budget, would the LLM be able to solve more problems?
Although performance at very long response lengths may be restricted by other factors like model architecture or context lengths (Li et al., 2024), one can at least expect that an L.M should benefit from test-time scaling within the pre-training and fine-tuning training budget. Mechanistically, this can be realized by implementing algorithmic procedures (e.g., generate-verify-revis best-of-N) within the model's chain thought, in-context (Kumar et al., 2024; Setlur et al., 2025a; Gandhi et al., 2024). However, similar to other empirical studies of reasoning models, note that many open models perform poorly when extrapolating to 2-3x the training budget (Qu et al., 2025b; Hochlehnert al., 2025). Thus, relying current RI/SFT recipes to yield effective extrapolation is mostly futile.
In this paper, we show that the key to enabling extrapolation is learning to explore in-context: if model learns to compute by searching through multiple reasoning paths or implementing algorithmic procedures, it can "guide" the search towards the correct answer, and improve its perfor- mance as more test compute becomes available. Even under the original training compute budget, we expect learning to explore in-context to improve performance on unseen, out-of-distribution problems (Ghosh et al., 2021; Duan et al., 2016). To demonstrate this, we build a recipe e3, which
"Equal contribution. Correspondence to aset 1 ur @andrew.. cmu. edu. This work was done CMU. Project website: httpr//atwwewayyaagggguuhhhhaaoo

---

Published as a conference paper at ICLR 2026
Our recipe: e3 Data & budiget curricula
Asymeetriis
Negetive
Fo: osanpe,
gradient
Ileration 0
Easy
spope
Budget
Meduum
Chained soymmrr
Coneration
Nolfication
Hoopp cortiinuu
Incommeed
Ierrtion
Mediun
Jotternct busgets
for each stage
bemolia more compude
Text Time Computs
Figure I: In-context exploration gnables extrapolation (e3): (i) chaining asymmetric capabilities in the base model, e.g., reliably selffveri responses after generating them; (ii) using negative gradients in RL training penalize incorrectly terminated model responses, lengthening them further with more chained asymmetries, until the correct answer is discovered; and (iii) data & budget curricula for RL training that carefully balances explore-exploit tradeoff by sequentially training models on different datasets and training compute budgets. Qwen3-1.7B fine-tuned with e? extrapolates test-time compute outperforming all <2B models on AIME 25.
trains models to explore in-context so that they can perform well at both the training and extrapolation budgets. At its core, e3 is based on the following three ingredients (see Fig. 1):
1) Asymmetries critical for learning for explore. LLMs can learn to explore only when each segment in the output trace is useful in "guiding" subsequent ones, e.g., if verifying initial segments can lead to more refined answers later on. In the absence of external tools, we show that feedback can emerge from asymmetries, which are differences in the model's competence different procedures that constitute an output trace. In the context of self-verification, this corresponds to a verification- generation (VG) gap, where models are more capable of verifying their answers than generating correct ones. While prior work (Setlur et al., 2025b; Swamy et al., 2025; Song et al., 2024; Kim et al., 2025; Gandhi et al., 2025) has noticed such asymmetries, we show that these are critical for extrapolation, meaning that in their absence, scaling is strikingly hard. We conceptualize this by building& a didactic model of in-context exploration under the generation- verification asymmetry. termed y and prove theoretically that chaining asymmetries can drive exploration.
2) Negative gradient in RL amplifies in-context exploration. If asymmetries are a prerequisite for learning to explore, what mechanism gets the L.M to chain them more and facilitate useful exploration strategies during RL? We show that negative gradients (i.e., gradients on incorrect traces) in RL is a key enabler of in-context exploration, when the base model presents asymmetries. Negative gradients drive exploration (Tajwar et al., ICML 2024; Ren & Sutherland, 2024) by moving the probability mass from shorter failed traces onto longer traces that "chain" " new asymmetries (e.g., LLM verifying a calculation one more time). In contrast, SFT only maximizes likelihood on correct traces in the training data and reinforces the model to end the solution within the length of these traces. Conceptually, SFT only aims to reduce the failure probability at a fixed k in our y model, whereas negative gradients also amplify k which is the number of attempts made in-context.
3) Structured exploration with coupled curriculum. Finally, while negative gradients amplify asymmetries and produce longer responses, running RL training longer budgets suffers from poor training convergence, typically seen in long-horizen RL (Agarwal et al., 2021). While one could train with a smaller budget, we show that training on hard problems short context lengths often disincentivizes exploration altogether since the model is forced to commit to an answer prematurely. To resolve this, we design a coupled curriculum over pairs of (data mixture, training budget) that effectively structures the exploration driven by the negative gradient. Our key insight is that at any stage of the curriculum, we choose the smallest "RL optimization friendly" budget such that the model initialized for RL training can: (i) complete most of its responses within the budget; and (ii) can continue to improve performance as it chains more asymmetries beyond the chosen budget.
The above insights constitute recipe e3, that a training budget of up to 16k output tokens. We build the one of the best performing reasoning models at <2B scale on AIME'25 and HMMT'25 (to our knowledge), and our model consistently improves as we extrapolate compute to 32k (2x training budget). e3-11.B also attains consistent improvements on the pass 32 metric, showing that e3 does more than simply sharpening the base model. e3-..BB also improves on non-math reasoning benchmarks (see App. K), not trained upon.
to post-train the Qwen3-1.77 model with
2 RELATED WORK
Scaling test-time compute via long COT. Recent work scaling test-time compute by training models to generate long chains of thought (CoT) that combine verification (Zhao et al., 2025a), search (Lu
2

---

Published as a conference paper at ICLR 2026
et al., 2025), and self-correction (Kumar et al., 2024) has achieved SOTA performance various reasoning benchmarks (DeepSeek-AL et al., 2025; Team et al., 2025; OpenAI et al., 2024), resulting in widespread open-source efforts (Face, 2025; Yeo et al., 2025; Zeng et al., 2025b; Luo al., 2025b). The true benefit of test-time scaling is performance gains under extrapolation. Prior work prompts models to generate extra tokens when response ends (Muennighoff et al., 2025; Aggarwal & Welleck, 2025), but we show models that can explore in-context well, extrapolate more effectively,
Exploration in test-time scaling. Long CoTs allow models to explore various approaches before committing to a final answer. While prior works show the role of exploration in reasoning (Gandhi al., 2025; Liu et al., 2025b; Krishnamurthy et al., 2024; Nie et al., 2024), we discover the crucial enabling factor is the presence of asymmetries, like verification verification-generttion gap (Setlur et al., 2025a; Song et al., 2024; Swamy et al., 2025), in the model. Moreover, while concurrent work builds techniques to boost exploration during RI. via advantage normalization (Li et al., 2022; Yu et al., 2025; Zeng et al., 2025a) or PPO clipping (Yu et al., 2025), they do not highlight the role of negative gradients which as we show incentivizes the chaining of asymmetries, and scales up in-context exploration. Concurrent works (Wang et al., 2025b; Zhu et al., 2025) remark about the role of negative gradient and discuss entropy trends when running RL. Our study formally investigates the underlying mechanism of negative gradients increasing length and entropy, with theoretical results.
RL training with data and length curricula. Recent works investigate curriculum training over problem difficulty (Team et al., 2025; Xie et al., 2025; Shi et al., 2025), going from easy to hard, and over training budget (Luo al., 2025b; Liu al., 2025a), from short to long, during RL. Their primary motivation is efficiency: avoiding zero-advantage updates (Shi et al., 2025; Yu et al., 2025), efficient optimization (Luo et al., 2025b), or efficiency of using test-time compute (Qu et al., 2025b), While we observe similar trends for each curriculum, our key finding is that coupling data and budget curricula enables in-context exploration and improves extrapolation performance beyond mere compute efficiency. We show that training on hard problems with short budgets often yields terse solutions that fail to generalize, while easy problems with long budgets can produce overly verbose outputs (Sec. 6). See App. C for detailed discussion of related works.
PROBLEM STATEMENT: OPTIMIZING & EXTRAPOLATING TEST COMPUTE
Problem statement. RL and SFT are categories of post-training
Accuracy on AIME 2025
algorithms that refine a pre-trained base LLM m, into reasoning
0.25
model, in particular one that produces long thains-of-thought to succeed. Typical outcome-rewan RL trains LLM π (initial-
ized with Th) to maximize performance on outcome 0/1 reward
0.20
118 OpenTrinker 78
ScapS
r" (x(xy). y), for inputs x ~ p and response y ~ TT ( xx) restricted to an apriori fixed maximum token length or training budget Bur (Yu et al., 2025; Luo et al., 2025b). On the other hand, SFT fine-tunes To on long thinking traces from more capable models or humans to distill their reasoning capabilities (Team, 2025; Muennighoff' et al., 2025), where the maximum length of the expert traces also implicitly induces a training budget B similar to RL. Our goal, train models that can improve performance when we extrap- olate test-compute beyond B Even though the true promise of test-time compute extrapolation performance, we find that current thinking models fall short on extrapolation. We evaluate multiple models on a test budget of 32K, & 11.5-2 Brr across all models, on AIME25 in Fig. (see App. B for detailed comparison) and note that most of the performance gains lie in the training budget, and the gains are minuscule as we test beyond that.
STILL
Thoupht
0.15
Open AS1
4K
8K
16K
32K
Test time compute (token length)
Figure 2: Accuracy on AIME 2025 of various open-source models at dif- ferent test time compute budgets. Per- formance gains diminish as the test- time budget increases, with virtually gains from 16k to 32k.
Negative gradient in RL. A key distinction between SFT and RL is the negative gradient, which corresponds to the part of the policy gradient coming from traces that fail. In Eq. 1 we present a generalized version of the policy gradient adopted by most RL post-training methods. From this we note that on a prompt x, RL training observes two types of gradients: (i) the positive gradient which maximizes the likelihood of correct responses y with a positive advantage A(x, y). and (ii) the negative gradient which pushes down the likelihood of an incorrect response with a negative advantage A(x,y) y). Here, y can be sampled on on-policy = ft or off-policy π # Ā. Thus, SFT as a purely positive gradient method that only maximizes likelihood on correct reasoning traces. In Sec. 5, we show why the negative gradient is largely responsible for driving up response lengths
view
3

---

Published as a conference paper at ICLR 2026
and in-context exploration during RL, resulting in better extrapolation than SFT.
(-ywa(-1xx) [Ar(x,y) V. log rty x)] (general form of policy gradient in RL)
(I)
CHAINING ASYMMETRIES: PRE-REQ. FOR IN-CONTEXT EXPLORATION
How can extrapolating beyond the training budget improve performance? We begin by revisiting why longer traces perform better in general. The conventional wisdom is that longer traces can represent solutions that make multiple attempts, interleaving verification and generation (Setlur et al., 2025b; Nie et al., 2024; Krishnamurthy et al., 2024), to arrive at the final answer. We can think of this as the LLM learning to interleave basic "skills", e.g., verification, summarization, or general instruction following, to perform in-context exploration. But why, or when, should post-trainn pest-training favor such traces over other shorter ones that arrive the answer directly? We demonstrate that when the base model exhibits asymmetric incompetence at different skills, RL post-training prefers to learn solutions that chains asymmetric skills in ways that improve final performance. A formal description is:
Definition 4.1 (Chaining asymmetric capabilities p, q in model т.). Let p, q S + S be functions over token sequences S (e.g.. p can be generation, g can be verification), and detect(f, r) detects number of calls to function f in a token trace т. For a reward r, we say that policy chains asymmetries p, q it benefits from calls the composition q(p(-)), compared only p(-):
Ersx (r(T) detect(g(p())) r) > 0] > Er [r(T) detect(p, 7) > 0],
even though there is an optimal policy क, that never calls q. i.e., Er-z; [detect(q, +)] = O.
We focus on a key special case when the model is more accurate at verifying its own answers than is at generating correct ones; that is, when the model exhibits a verification-generattan generation gap (VG Gap), on a particular problem domain (Song et al., 2024; Setlur et al., 2025b; Swamy et al., 2025). In this section, we show that RL training on problem domains with VG gap (i) encourages chaining asymmetries, (ii) enables in-context exploration that (iii) discovers new solutions, often extrapolating to larger budgets and more difficult problem domains. In App. F, we also discuss another chained asymmetry: (1) generating abstract plans hints, then (2) solving from a chosen hint, rather than jumping straight to a reasoning attempt, analyzing results from a concurrent work (Qu et al., 2025a).
Analysis setup. We validate the role of asymmetries in learn ing to explore by investigating two didactic tasks, on which Llama3.2-3B admits different VG gaps. First, the Countdown game (Yao et al., 2023; Gandhi et al., 2024) (CDOWN) requires converting a set of numbers into an equation that evaluates to the desired target. The base LLM is more effective at verify-
17
0.058
0,380
0.488
detoct(q(p()). r)
0.4
0.044
0.216
0.243
0.3
0.022
0.070
0.081
3
0.012
0.022
0.025
ing whether a proposed equation evaluates to the target than searching over all possible equations to solve the task, and traces with more chained asymmetries are more performant. We con- firm this by measuring our Def. 4.1 in Fig. 3: performance on traces with more chains is higher, and both the number of chains and performance get amplified during RI. Second, we study n-digit multiplication (MULT) in natural language, where the base model exhibits limited verification (see App. E). Addition- ally. we fine-tune Llama3.2-33 2-3B on correct n-digit multiplication traces from Seeen-32B-rl-disiie which contains multiple intermediate steps verifying smaller digit multiplications (see App. E for an example). This fine-tuning is a direct way to encourage more verification attempts (MULT-V) V). Comparison of MULT vs. MULI-V V enables direct evaluation of the benefits of asymmetries in base LLM, all else being held equal. In these results, we detect verification segments by looking for thought blocks that start with phrases like "Wait, "..", "Let me verify ...". (see App. for examples).
0.003
0.004
0.005
30
60
Steps
Figure Measuring asymmetry on CDOWN. Rewards r(7) improve as traces have more chains q(p(-)) in them, and this is amplified during RL on Llama3.2-3B.
Finding 1: Verification-generaie
asymmetry in the hase model improves the performance of RL trained solutions. Fig. 4 (a,b) shows a stark difference in performance and length of output traces as the training budget Bur varies CDOWN and MULT. On CDowN, performance consistently increases as B,c increases from 512 + 2048, accompanied by a very clear increase in response length. On MULT, where the base model has limited propensity to verify, performance increases when Bu increases from 1024 to 2048, but it plateaus thereon. Unlike CDOWN, test-time length is far from saturating budget limits and also oscillates widely across RL. training epochs. Contrast this with
4

---

Published as a conference paper at ICLR 2026
(a) Performance B
(b) Response length
(c) Extrapolation Acc.
[c] Multiply verify
2k
0.85
4kc
Coundoun
B,
512
B,
512
<
B, -12
B,
0.45
B
1024
1024
Length
2048
1k
Ry
0.65
0.5k
0.20
0.45
100
200
300
400
100
200
300
400
2k
4k
8k
16k
100
200
300
400
Multiply w/o verify
Extrapllation Acc.
Training iterations
4k
0.58
1004
Fl
1024
B, 2048
0.7
1024
B, 20:8
B, 1096
B,
3015
2k
4096
0.54
0.6
1k
0.50
0.5
100 200 300
400
100
200
300
400
3k
8k 16k
Training iterations
Training iterations
Test compute (tokens)
Test compute (tokens)
Figure 4: RL training with and without asymmetries in the hase model. When asymmetries such the VG gap are present (e.g., in CDOwN), RL. training amplifies response length by chaining more asymmetries explore in-context, where the probability success improves with higher length on hoth Bu and extrapolation regimes. On the other hand, when VG gap is absent in i. (e.g.. in MULT). increases in length and extrapolation performance are subdued. When we explicitly train on base model fine-tuned to verify MULT (a setting we refer to as the MULI-V) V). again observe upward length and extrapolation trends, consistent with CDOWN.
Fig. 4 (d), where RL training MULI-V, which leverages verification, exhibits longer lengths and stronger extrapolation performance. Overall, this implies that leveraging asymmetries improves performance and length-utilization during RL. Curiously, we observe that models with greater VG gap exhibit less KI. divergence from the base model, implying better generalization (see App. E).
Finding 2: Chaining asymmetries enable extrapolation via in-context exploration. Interleaving verification and generation steps chains together asymmetric skills of the base model; we refer to this special case of skill-chaniing as chaining asymmetries. To measure the benefits of chained asymmetries on CDOWN, we plot the pass @k accuracy of the base LLM, shown in Figure 3, and observe that performance increases as more chained asymmetries arise. In fact, the best strategy is to not simply scale k, but rather to scale both & and the number of chained asymmetries (details in App. E). In Fig. 4 (c), we plot the extrapolation performance of the models trained at two values of Bic. On CDowN the model trained with By 0.5-1k makes steady progress on problems in test budgets that are 8-16x Byy itself. On MULT, we find that Bt, has absolutely no effect on extrapolation performance when the base LLM that does not have VG asymmetry, but it has substantial effect when the asymmetry is present. More importantly, while the base model without VG asymmetry fails to extrapolate and solve unsolved problems, with its accuracy improving by merely < 2% despite 16x test-time compute scaling, the base model with VG asymmetry can still extrapolate well.
Why do asymmetries enable in-context exploration? We explain this using didactic y* mm-mdeel (App. A), where we view the LLM as making guesses dj...., ag. each failing with probability under perfect verification. In this simplified case, failure probability p* decays exponentially, so performance improves by increasing both i and p. But if verification is hard, more guesses give little benefit, and gains come only from lowering p (better first guesses, in MULT).
Takeaways: Asymmetries are a critical pre-requisite for learning explore.
Asymmetries like the VG gap enable the model to verify and refine answers in-context.
RL amplifies chaining of asymmetric skills in base model and produces solutions that learn to explore in-context, benefiting from more extrapolated compute at test-time.
5 NEGATIVE GRADIENTS DURING RL CHAIN MORE ASYMMETRIES
Given that asymmetries in the base model are prerequisite for in-context exploration, we now ask: what enables models to exploit and chain these asymmetries during RL. We show that a crucial ingredient is the negative gradient, the gradient term multiplied by negative advantage in Eq. 1. Negative gradient drives in-context exploration via two mechanisms: (i) incentivizing sampling of unseen token sequences; (ii) chaining asymmetries like VG gap that rapidly drives up response length and in-context verification attempts. Note that while mechanism (i) corresponds to the classical notion of exploration, mechanism (ii) is special in that it corresponds to form of "structured"" exploration over strategies in the base model (corresponds to 'meta exploration" (Liu et al., 2020) in RL terms). Below, we study these empirically, and in App. A we study it theoretically.
5

---

Published as a conference paper at ICLR 2026
(a) Accuracy
(b) Number of verifications
(c) Response length
(d) Entropy
50
GRPO
2k
GRPO
0.15
GRPD
Countdown
0.6
40
mask neg. grad
gred
mask
1.5k
30
0.10
entrapy bonus
GRPO
mask neg. grad
20
0.05
extrapolation
10
0.5k
50
75
0
25
50
75
25
50
75
25
50
75
Omni-MATH MAHH
0.3
60
10k
0.15
0.2
8k
0.10
0.1
30
0.05
6k
100
50 100
150
200
100
200
Figure 5: RL training with and without negative gradients: When the base model admits asymmetries like verification generation gap, negative gradients promote in-context exploration by: i) increasing length (c) and chaining asymmetries, which shows up as more verification attempts (b); and ii) increasing token entropy and thus response diversity (d). This leads to better performance on the training budget and upon extrapolation (a). In (b, c). v denotes the statistic computed on correct responses and X on incorrect responses.
Analysis setup. We analyze the evolution of response length, performance, and the number of chained asymmetries, comparing: (i) standard outcome-reward RI. using GRPO (Liu et al., 2025b); (ii) GRPOMask, which zeros out (i.e. masks) the negative gradient and whilst retaining the positive gradient, thereby resembling an approach close to online STaR (Zelikman et al., 2022) or RFT (Yuan et al., 2023). We conduct experiments on CDOwN and DMATH reasoning (problems sourced from DeepScaleR (Luo et al., 2025b)) that exhibit the VG asymmetry and make the following observations:
Finding I: Negative gradients promote diverse responses during RL training, encouraging exploration at two levels: () within rollout; and (ii) across different rollouts. For
0.17
Comuuatioe unague Stterate 0:15
Fracioo of repealing 125011X
8 2 88
(), we observe that removing the negative gradient results in an entropy collapse over the next-token distribution (Fig. 5 (d)). This curtails diversity and leads to responses with repeating stream of tokens when extrapolating token budget (Fig. 6). For (ii), we measure the cumulative unique attempts on the CDOWN as we train the model (Fig. 6). We separate each rollout into attempts using "\nn" and parse the equations. An attempt is unique when its equations differ from other rollouts across training steps. We find more unique attempts when training with negative gradients, indicating better exploration. While it is not surprising that RL benefits from exploration (Hazan et al., 2019), distinctly from standard RL, this exploration can be particularly effective when extrapolating test budgets (Fig. 5(a)).
0.06
0.04
0.00
Swak neg gmad
Figure 6: Negative gradient encourages distinct responses: increases the cumulative number of unique attempts on CDOWN (left) and reduces responses that end with a repeat- ing stream of tokens on DMATH (right).
Finding 2: Negative gradient increases the number of chained asymmetries, and thereby boosts structured exploration (and extrapolation). Concretely, when training on an incorrect response y with tokens y1, 1/2,.., EOS, the negative gradient reduces the conditional probability of each token y: conditioned on the prefix y1:i-1 appearing in this response, i.e. p(p:lyia-1). This process also reduces the probability of the ECS token: p(EOS| ly), for any incorrect response that ends within the response budget. Where does this probability mass go? Clearly since total probability is conserved, this probability mass must be repurposed to increase the likelihood of other tokens. Fig. 5(b) shows that the probability mass recovered from the negative gradient repurposed to increase the probability of chaining new pairs of asymmetric skills to the current trace (e.g.. "Wait, instead of terminating with EOS). This chaining results in a greater response length (c) and higher overall performance.
When negative gradients are masked (GRPOMask) in CoowNor DMATH, verification attempts and length plateau or drop (Fig. 5(b c)). The effect of masking is more pronounced for incorrect rollouts, but also holds similarly for correct rollouts, i.e., the number of chained asymmetries and length plateaus or declines in the absence of negative gradients. We include more results in App. G (Fig. 16), where we also demonstrate that MULr (which does not exhibit asymmetries) benefits far less from negative gradients. This mechanism for boosting exploration by chaining new asymmetries is different from the typical notions of improving coverage trying novel tokens discussed in Finding 1.
Finding 3: LLMs trained with negative gradients extrapolate better. Longer responses that chain asymmetries are more likely to yield correct answers and thus receive positive reward. Therefore,

---

Published as a conference paper at ICLR 2026
Aoorrey on AIMEEE 25
40
Fix dalaset, vary train budget
Anturacy on AIME 225
Fix train budget, vary dataset
Resquaree on hard
100
Fix distased, vary train budget
B. (BK) (easy)
dy (9k) toung]
8
train budgel, vary dalaset
mace
(wees)
By (101) (eaay)
O (easymne)
20
Bydet) (sass)
20
50
Frequency
10
reaa))
(911
2011
on
By (811 (al)
4k 8k Test-timi compute (lakens)
32k
32k
Tast-lime compute (tokens)
Longth
Longth
(a)
(b)
(c)
(d)
Figure 7: RL training on different data/budgets. (a), (c): Best results come from balancing optimization
difficulty (hetter at shorter budgets) and in-context exploration (hetter at longer budgets). (b), (d): Training on
hard problems at the 8k hudget kills longer traces needed discover solutions for hard problems
the policy gradient update reinforces chaining and improves in-context exploration. and this is reinforced: further training incentivizes more in-context exploration (since the gap between number of verifications with and without negative gradient increases as training progresses in Fig. 5(b)). As discussed in Sec. 1, models that learn explore in-context benefit more from additional test-time compute. Fig. 5(a) confirms this: on hard DMATH problems (we classify a problem as hard if QwQ 32B attains pass @32 performance of zero), doubling the test-time budget amplifies the performance gap when negative gradients are used, compared to the masked variant.
Takeaways: Negative gradient in RL amplifies in-context exploration under large VG gaps
Negative gradients in RL move the probability mass from a short-length incorrect trace onto a longer trace with more chained asymmetries that results in the correct answer.
Negative gradients boost response diversity and thus coverage over correct answers, as noted empirically on CDowN, DMATH, and theoretically in our bi-gram model analysis (App. A).
COUPLED CURRICULUM STRUCTURES EXPLORATION IN LONG LENGTH RL
In the presence of asymmetries, training with negative gradicnts produces models that can extrapolate beyond their training budget. However, of course, training on just any arbitrarily chosen training token budget Bar is not enough: if Bur is too small, then we would not expect any form of in-context exploration to emerge. Perhaps unsurprisingly turns out that a much larger Bú, is also not sufficient. In Fig. 7()), we show that different training budgets By lead to different levels of extrapolation performance. So how should we set the budget Bur to attain strong extrapolation performance? And, what prompts should we be training on for a given budger? We answer these questions below.
Setup. We evaluate extrapolation performance on DMATH and CDOwN after training on different budgets and prompt compositions. We split DMATH evenly across three levels of hardness as measured by the performance of DwvenRR--isttllen the number of terms in the equation. We use the GRPO (Liu et al., 2025b) algorithm to train models on all compute budgets and datasets (see App. for the hyperparameter configurations we use).
For CDOWN, we judge problem difficulty based on
Finding I: Training solely at low or high Bur is not desirable. We train on the easy DMATH problems at different training budgets B,, 4k, 8k, 16k (see Fig. 7(a)). While training at the short budget B,,= 4k attains the best performance at the same test budget, it "kills" in-context exploration since traces with many chained asymmetries are typically longer than the training budget of 4k and only successes within 4k are rewarded. Overall, this hinders length increase and chaining of asymmetries driven by the negative gradient, leading to poor extrapolation (no gains from 8k to 32k). Fig. 7(C) shows that this biases the model stop early and incorrectly. In contrast, training at Ba= 16k introduces significant optimization challenges, stemming from policy gradient variance (Agarwal et al., 2021). This model performs worse on its own training budget of 16k compared to a model trained B 8k and extrapolated to 16k. We find that B 8k attains the best extrapolation scaling.
Training naively on a static data mixture is suboptimal. Having identified a reasonable training budget of 8k, we now turn to studying the effect of data compositions (prompt mixtures). To do so, we compare the naive training data mixture with equal proportions of all difficulties (easy + medium + hard) against easy, easy + medium at B,, 8k. As expected, matching train and test composition is ideal for better in-distribution performance, i.e., when evaluating models at a test budget of Bür (see App. 1). However, perhaps surprisingly, the same is not true for performance on out-of out-off-dstribe diddstibbutiin (OOD) problems, especially when performance is computed at budgets >> B As shown in Fig. 7(b), the model trained on only easy problems obtains the best performance OOD AIME 25 when extrapolating compute to 32k. This is despite the fact that AIME' 25 problems resemble hard ones and few prior AIME problems are also present in the hard subset of DMATH.
Finding 2:
7

---

Published as a conference paper at ICLR 2026
eB va, opert-source models
VS, open souncs models
(wed-hard)
e3 across RL ferrtions
#3 vs. budgget& data curficall Excas
Acc. AIME '25
-10
agantia 1.55
Acc. on HMMT 125
eportiika
Test 850 (yeed
0.7
Test scc. (hard)
30
20
1-323
20
0.5
Fuit ni on-
20
31-008
10
ugettc-1. 222
0.5
taiviin on 18k)
19
188- 18c4 [ell
63.1.0 70
rolu R
5,joks (eway)
8- (186 jusy ieafirett 1hc
5-E1RK, D tnventmndi
4k 8k 16k
24k
32k
4k 8k 18k
24k
32k
250 Iteraiion
600
4k 8K Tast-lmee compute (lokans)
16k
24k
32K
Tast-tima compute (lakens)
Tast-time compute (lokens)
(a)
(b)
(d)
Figure 8: RI. training with coupled curricula. In the above figure, the shaded area indicates the extrapolation regime. (a), (b): e3 achieves state-of-the-arl performance across models; ; 2B. (c): extrapolation gain from
switching to a longer budget during training (d): coupled curriculum outperforms data and budget curricula
Why does this happen? Given a dataset, training on budgets smaller than the length of a typical response for the base model penalizes in-context exploration early in training. This results in overly short solutions (see Fig. 7(d) that are mostly exploitative. When projected to our p model from Sec. 4, this means that at overly short budgets, RL mainly attempts to improve the failure probability p of the best guess response, and does not learn to increase & which corresponds to chaining asymmetries. To increase k it needs to be able to learn to increase the number of attempts and requires large enough budget. But the budget cannot be too large either.
How can we avoid challenges with training on a fixed dataset and length budget? One approach to avoid the above challenges is incorporate a curriculum that varies Bir over training. However, this alone is insufficient because, as shown above, training on hard problems with short budgets suppresses in-context exploration. On the other hand, we can design curriculum over the difficulty level and set Bür to a high enough value. However, this presents optimization challenges (see App. I for a detailed study of this on CDOWN). In a nutshell, a curriculum that only varies the training budget or the dataset composition is insufficient to incentivize in-context exploration. To mitigate this problem, we propose "coupled" curriculum over data composition and training budget.
6.1 OUR RECIPE E3: COUPLED CURRICULUM FOR IN-CONTEXT EXPLORATION
We develop coupled curriculum that varies the training budget B and problem difficulty in a coor- dinated fashion during RL on base model with asymmetries. We refer to our recipe (chained asym- metries, negative gradient, and the coupled curriculum) as e3: exploration enables extrapolation.
Key insight for curriculum design. We simplify curriculum design by fixing the dataset at each stage and progressively increasing task difficulty in stage- manner, from easy to hard. Now, at each curriculum stage i, we define dataset D; and focus on selecting an appropriate token budget B The goal is to choose Bur. such that training with this budget encourages in-context exploration when training on D.. That is, RL should reward successful reasoning traces that successfully chain asymmetries, within budget Bur. This ensures that the resulting policy can extrapolate to longer sequences and provides a strong initialization for the next stage i+1, where the token budget increases to B1r,++1- At the same time, for optimization to be efficient, the budget Bar, should be as small as possible while still accommodating most valid completions from т; Balancing these desiderata, we formalize the choice of B via the following optimization as a thumb rule:
Bi (D.) = arg min B s.t. (T; D,, B) S K J D., B), K R>1 B2B,
(2)
where J(T; D, B) denotes the performance of π at budget B on dataset D, and the budget Bu denotes a reasonable minimal length for ı on dataset D,, e.g., Bo can be the average token length of responses from on D.. In practice, we solve the optimization over B by restricting to a fixed set of training budgets: 4k, 8k, 16k. We find the above strategy of choosing the token budget to be useful heuristic for greedily choosing the budget Bu, at stage of the curriculum in a way that incentivizes in-context exploration. E.g., setting k = 1.2, we find 8k to be the best choice for training on easy problems (observe that the trained model satisfies the condition in Eq. 2 at K: = 1.2 in Fig. 7(a)). Following this, e3 fine-tunes the Qwen3-1.7B base model on easy problems in DMATH at Bur of 8k, and subsequently continues training on medium and hard problems in DMATH with a token budget of 16k. For training medium/hard problems in DMATH, we can also optimize the training budget, as we did for the run on easy problems. From Fig. 7(a), we note that the model trained with a token budget of 8k extrapolates compute to budget of 16k and even 24k on AIME '25, after which the gains start diminishing. We find similar extrapolation performance on medium and hard problems in DMATH. Thus, we can safely train on a budget of 16k or 24k on this set, and due to GPU memory
B)
8

---

Published as a conference paper at ICLR 2026
AIME 2025
HMMT 2025
Model
k=
2
8
16
32
k= 1
2
8
16
32
Qwen3-1.7B
35.5
41.4
47.0
52.4
58.3
65.2
22.2
27.3
33.0
39.5
46.7
54.9
1--dittill-ween-.5555
23.1
29.2
34.5
40.1
46.3
52.5
12.5
19.1
24.3
27.9
36
42.8
Nemotron-Reasoning 1...
33.6
38.5
43.6
48.9
53,8
58.0
17.4
22.5
29.6
35.2
40.7
45.0
e3-1.7B (Ours)
43.8
51.1
56.7
60.8 64.0
67.2
24.7
30.4 37.0 44,1 50.8
56.1
Table Final results with e3 on 21ME//MMT225: We mcasure pass (%) on AIME'25 and 25
for our 1.7B model obtained by post-training the Qwen3-1.78 base model on DMATH with our recipe e3, comparing with strong <2B reasoning models. Following See. 6, we coupled task and budget curriculum during (()) train easy problems =, and then (ii) on medium hhard ones =,-= Note that unlike recent trends (Yue al., 2025) that show RL training improving pass I at the cost of pass@k for a higher we note that e3 trained models improve performance by not just sharpening the base model distribution around high reward traces, but by actually chaining asymmetries and discovering new solutions with longer traces.
constraints, we chose to train on the shorter of the two (16k). Finally, in Fig. 8(c), we show that the model produced by e3 by training on easy problems at the end of the first stage does extrapolate well, which is helpful kickstart RL training when we move from the budget of 8k to 16k. Concretely, we observe 2 10% performance gain with extrapolation. We also show that our coupled curricula outperforms separate curriculums that either only vary the token budget (Bur) or the data mixture (D) in Fig. 8(d). Finally, we illustrate the efficacy of the above curricula design on CDOWN in App. H.
6.2 FINAL RESULTS WITH F.3: A State-of-the-ar <2B MODEL ON AIME/HMMT'25
Extrapolation to 32k with e3 >> sl. In Fig. 8(a,b), we compare
the performance of a Qwen3-1.7B model fine-tuned using e3 with open-source models, including some 7B and 32B models.
Learning extrapolates better prompting for
e3
Acc. on AIME '25
40
extropoiation with wall
As shown, at a test-time token budget of 32k tokens, e3 achieves
state-of tate--ofthheartt thttthr-rt performance a model class of size <2B. We outperform the best model in this class by ~8% show that our model, trained only up to a budget of 16k, extrap- olates better than other models including s1.1-32B (Muennighoff' et al., 2025) and OpenThinker-77 (Team, 2025) when we extrap- olate them to 32k output tokens. In principle, one can simply force the model (trained even with SFT) to use more test-time compute by intervening its output trace with an appended prompt (e.g., by appending "Wait" to an output trace as suggested in sl (Muennighoff et al., 2025)). Interestingly, Fig. 9 shows that compared to budget forcing via "Wait", e3 achieves substantially better scaling, without any form of prompting or budget forcing.
25 and HMMT'25, within
30
s1.1-7B =wiil"
agentica"
AIME'25 25 in terms of peak performance, and
Qwen3-1.778 1.nmm---17--wattt
20
e3-1,78
16k 24k 32k
Test-time compute (tokens) Figure 9: e3 (w/o "wait") is su- perior when extrapolating to larger budgets, compared to budget forcing with "wait" prompt 2/4/6/8 times.
Improving pass with e3. In Tab. we also report the passšk performance, comparing e3 with other models of a similar size. We find that our final model at the end of second stage of training on a budget of 16k outperforms other models on higher values of k. on AIME and HMMT '25. We especially note the comparison against the Nemotron- -eaerrron--esoninngg.. 1.5B model (Liu et al., 2025a) trained with a prolonged RL training recipe on broader dataset, including our training data. This model consistently improves pass@ 1 6 performance during RL training (Liu et al., 2025a). To concretely describe our estimation procedure, we used 128 rollouts per prompt to compute a bootstrapped estimate (Chen et al., 2021) of the pass@k performance for k 1, 2, , 32, beyond which it would require substantially many more rollouts to get reasonable high confidence pass @k k estimate. We conclude that 3 is able improve over pass m 32 of the base model, meaning that does go beyond distilling the pass @32 policy corresponding to the base model into a better policy.
Takeaways: Coupled data & budget curriculum structures exploration during training.
RL with fixed Bu, D hurts extrapolation: (i) short B, curtails chaining on hard DMATH as budget is overrun; (ii) large B,, leads to over-exploratory behaviors on easy ones.
We propose coupled curriculum e3: at each stage, given D, choose smallest B,, such that chaining more asymmetries till a budget of 2 By is positively rewarded at RI. initialization.

---

Published as a conference paper at ICLR 2026
7 DISCUSSION AND CONCLUSION
We show that co-tont exploration is a core capability to enable extrapolation of test-time compute in LLMs. Therefore we build recipe that amplifies in-context exploration. Our recipe e3, leverages (1) asymmetries in the base model, (2) negative gradients during RL training, and (3) coupled curriculum over data and token budget to train model that can perform in-context exploration. Applied to the Qwen3-1.7B model, our method achieves state-of-the-e ofrtt---rttttttt-- performance on the AIME/HMMT'2S benchmarks, with particularly strong gains in the extrapolation regime. We also show e improves pass@k over the course of training, for values of k upto 32 that we evaluate. There are multiple implications of our work and interesting directions that future work can build upon.
Sharpening vs in-context exploration. A number of concurrent results either directly (Yue et al., 2025) or indirectly (Shafayat et al., 2025; Zhao et al., 2025b; Shao et al., 2025; Prabhudesai et al., 2025) argue that RL training on LLMSs sharpens the base model's s distribution. In contrast to this, our study shows that if we can utilize a coupled curriculum on top of a base model that admits asymmetries, RL can actually enable chaining new asymmetries, resulting in increase in length, indicating the presence of structured exploration. This behavior is distinct from traditional sharpening that corresponds to cloning one (or few) of the responses sampled from the base model. In fact, our conceptual study in the pd model in Section A also highlights these two distinct phases during RL: an initial in-context exploration phase where negative gradients lead an increase in response length and the policy learns to utilize test-time compute for better exploration, followed by a phase where it sharpens the best traces found thus far. The design of e enables to operate in the former phase. We believe concurrent works that finds RL largely sharpens the model operate in the second regime by training on data that does not require chaining asymmetries or operating with a very low training budget such that chaining is impossible. As result, models trained purely in the sharpening regime may behave similarly to the base model with an alternate prompt, with RL perhaps offering little more than an implicit prompt tuning effect. But we would not expect this for the chaining regime.
Connection with dense progress rewards. While e3 utilizes a coupled curriculum, this curriculum is closely connected with the use of dense rewards, as prescribed by our prior work (Qu et al., 2025b; Setlur et al., 2024). To see why, note that one can reparameterize coupled curriculum into a single round of training with dense rewards applied to short segments of the output response, perhaps in similar way as Qu et al. (2025b); Qi et al. (2025). Therefore, the success of the coupled curriculum approach in e3 improving performance and not only in reducing total training compute perhaps hints at future success with dense rewards at scale, with initial results showing that dense rewards help larger models already being shown in the community (Wang et al., 2025a). We encourage readers to explore the connection between curriculum and dense rewards further.
Introducing new asymmetries. The conceptual model behind e3 applies with any asymmetry, though most experiments in this paper utilize only the verification-generaite interesting to identify other asymmetries and study methods to imbue base models with these asymmetries. Definition 4.1 in Section 4 provides a starting point to define these asymmetries.
gap. It would be
Is curriculum fundamentally needed? natural question is whether curriculum is fundamentally necessary as we vary model sizes and capabilities. Unlike supervised learning on a fixed dataset, online RL generates its own rollouts. Reinforcing chaining behavior via negative gradients (Sec. 5) requires that such chaining reliably improves performance on training problems much more substan- tially compared to sampling diverse traces that do not chain asymmetries. This likely necessitates specific training configurations regardless of model size with standard aitcome-rewant RL, the use of dense rewards (as discussed above). While larger models may admit simpler curricula, deliberately using currciulum dense rewards as inspiration may be critical.
Explicit exploration bonuses. In our runs, the main issue hindering us from benefits of further scaling of output length during RL is the repetition bias in the base model, where it tends to repeat meeiioully-geneatee segments in its trace beyond a certain output length. This repetition bias inhibits the efficacy of in-context exploration beyond certain output length and result inhibits further test-time scaling. We believe that explicit exploration bonuses that enable the model to search for tokens in this regime would result in even better in-context exploration.
Finally, our study is limited in terms of model scale and domain. Future work should explore how e3 generalizes to larger model scales and other reasoning domains.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Christina Baek, Yuxiao Qu Anikait Singh, Yoonho Lee, Max Sobol Mark, Zheyuan Hu, Seohong Park, Bhavya Agrawalla, Sang Michael Xie, Paria Rashidinejad, and the rest of the AIRe lab at CMU for informative discussions, feedback, input on our results, and a previous version of this paper. We thank Yuxiao Qu for help with debugging implementations and infrastructure. The main large-scale experiments in this paper utilized H100 GPU resources from the Orchard cluster in the FLAME center at CMU for which we especially thank Graham Neubig and Chenyan Xiong for their generous support, and TPUs from Google Cloud. We thank Oumi for providing us with resources that supported the experiments on the Countdown domain. This project supported by funding from the Office of Naval Research under N00014 11001-----1-200- and Schmidt Sciences A12050 Fellowship. AS is supported by JP Morgan PhD fellowship. This paper does not reflect the opinions of employers or other parties.
8 REPRODUCIBILITY STATEMENT
We have taken multiple steps to ensure the reproducibility of our work. A detailed description of the training setup, including datasets (DMATH, CDOWN, MULT, and benchmarks such as AIME'25 and HMMT'25), model architectures, and hyperparameters, is provided in the main text (Sections 3-6) and in the appendices (Appendix E-I). The theoretical results and proofs underpinning our claims about asymmetries and negative gradients are included in App. A and App. F. To aid experimental reproducibility, we describe our evaluation protocols (e.g.. pass@k k computation, bootstrapped confidence intervals) in Sec. 6.2 and App. N. We also provide ablation studies and additional results across different training budgets, curricula, and datasets in Appendix G-K. An anonymous link to source code and scripts for data preprocessing, training, and evaluation is included in the supplementary materials. Together, these resources allow independent researchers to replicate both the theoretical and empirical findings presented in this paper.
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22-98):1-76, 2021.
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697, 2025.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
DeepSeek-Ar AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. 2025. URL https: / //tp///avvv////////////////aaav/ org/ 2tbs/2501.12
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.025 2016.
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: //github. www.co////////o /uuggtuugttaa face open-ri.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint arXiv:2404.03668,, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cognitive behaviors that enable eef-viipooiing reasoners, or, four habits of highly effective stars, 2025. URL
https: /ttp///avvvvv/ve///////e../..avv org/ abs 2503. 01307.
Bofei Gao, Feifan Song, Zhe Yang. Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07988 2024.

---

Published as a conference paper at ICLR 2026
Yang Gao, Christian M Meyer, Mohsen Mesgar, and Iryna Gurevych. Reward learning for efficient reinforcement learning in extractive document summarisation. arXiv preprint arXiv: 12894, 2019.
Dibya Ghosh, Jad Rahme, Aviral Kumar. Amy Zhang, Ryan P. Adams, and Sergey Levine. Why generalization in RL is difficult: Epistemic pomdps and implicit partial observability. CoRR, abs/2107.06277, 2021. URL https: //tp///aaiii/ www.eeiiii.ooraaeeeeeeee 22007.2211...0 06277.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In ICML, 2019. URL https://////. /ates///rxxiie /aaxivivvvv. http://awwwwww.p////////////a//e 812. 02690. pdf.
Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXiv:2504.07086, 2025.
Audrey Huang, Adam Block, Dylan Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jor- dan T Ash, and Akshay Krishnamurthy, Seff-improvement in language models: The sharpening mechanism. arXiv preprint arXiv:2412.0195/ 2024.
Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Kiril Gashteovski, Carolin Lawrence, Julia Hockenmaier, Graham Neubig, et al. Scaling evaluation-time compute with reasoning models as process evaluators. arXiv preprint arXiv:2503. 19877, 2025.
Akshay Krishnamurthy, Keegan Harris, Dylan Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? arXiv preprint arXiv:2403. 15371, 2024.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409. 12917, 2024.
Qiyang Li, Yuexiang Zhai, Yi Ma, and Sergey Levine. Understanding the complexity gains of single-task rl with curriculum. arXiv preprint arXiv:2212.128809 2022.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-contess Ilms struggle with long in-context learning. arXiv preprint arXiv:2404.020660, 2024.
Evan Zheran Liu, Ramtin Keramati, Sudarshan Seshadri, Kelvin Guu, Panupong Pasupat, Emma Brunskill, and Percy Liang. Learning abstract models for strategic exploration and fast reward transfer. arXiv preprint wxXiv:207.008890 2020.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.2488 2025a.
Zichen Liu, Changyu Chen, Wenjun Li. Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding -leero-like training: A critical perspective. arXiv preprint 2rXiv22503.2078, 2025b.
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint, 2024.
Ximing Lu, Seungju Han, David Acuna, Hyunwoo Kim. Jaehun Jung, Shrimai Prabhumoye, Niklas Muennighoff, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, et al. Retro-search: Exploring untaken paths for deeper and efficient reasoning. arXiv preprint arXiv:2504.04383 2025.
Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li. Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open- 14b coder at o3-mini level, 2025a. Notion Blog.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RI., 2025b. Notion Blog.
12

---

Published as a conference paper at ICLR 2026
Sara Vera Marjanovié, Arkil Patel, Vaibhav Adlakha, Milad Aghajohari, Parishad BehnamGhader, Mehar Bhatia, Aditi Khandelwal, Austin Kraft, Benno Krojer, Xing Han Lü, et al. Deepseek-rl thoughtology: Let's think about Ilm reasoning. arXiv preprint arX11v2504.07128, 2025.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schif- ferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504. 16891, 2025.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https: //axivv. orgg aass////////// / 2000/5511111 10551199999 19393.
Allen Nie, Yi Su, Bo Chang, Jonathan N Lee, Ed H Chi, Quoe V Le, and Minmin Chen. Evolve: Evaluating and optimizing Ilms for exploration. arXiv preprint arXiv:2410.06255 2024.
OpenAI, i, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed EI- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. URL
wwt.:iiiw..o..aooooo abs 241 6720.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.2260) 2025.
Martin I. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
Penghui Qi, Zichen Liu, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Optimizing anytime reasoning via budget relative policy optimization. arXiv preprint arXiv:2505. 134388 2025.
Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how self-improve. arXiv preprint arXiv:2407.18229, 2024.
Yuxiao Qu, Anikait Singh, Yoonho Lee, Amrith Setlur, Ruslan Salakhutdinov, Chelsea Finn, and Aviral Kumar. Learning to discover abstractions for LLM reasoning. In ICML 2025 Workshop Programmatic Representations for Agent Learning, 2025a. URL https: /ttpnr/eee/peeeeeee// net / /////oorum//////// id ZWEUOOKT8G.
Yuxiao Qu, Matthew YR Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine- tuning. arXiv preprint arXiv:2503.07572 2025b.
Yi Ren and Danica Sutherland. Learning dynamics of Ilm finetuning. arXiv preprint arXiv:2407.10490, 2024.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar, Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint arXiv:2410.081 2024.
Amrith Setlur, Yuxiao Qu, Matthew Yang, Lunjun Zhang, Virginia Smith, and Aviral Kumar. Optimiz- ing I1m test-time compute involves solving meta-rl problem. https: http///lloggoo/.ooo////./oooo cmu, edu/, 2025a. CMU MLD Blog.
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal. arXiv preprint arXiv:2502.12118, 2025b.
Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train?, 2025. URL https: http://xxxvvv.r../// abs abs/250ss5/55 2505.555.... 21444.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr, 2025. Notion
Blog.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
13

---

Published as a conference paper at ICLR 2026
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning, 2025. URL
www//bs/aaaeooooooo..ee...eeeeee h/t////////////// 2504 05520.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint vrXiv:2408.03314, 2024.
Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-immpovement capabilities of large language models. arXiv preprint arXiv:2412.02677 22.002674 2024.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A mini- maximalist approach to reinforcement learning from human feedback. arXiv:2401.040555 2024.
Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprini arXiv:2503.01067, 2025.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn. and Aviral Kumar. Preference Fine-Tuning of LLMs Should Leverage Suboptimal. On-Policy Data, ICML 2024.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Zhexu Wang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with llms, 2025. URL https: /ttp/aaavvv//////////eae/e/aaava wwwp:xivi.or.gb.ooooooee 2501 12599.
OpenThoughts Team. Open Thoughts. http://open-thoughts.tt February 2025.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint arXiv:2506.0133 01939, 2025a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- ery, and Denny Zhou. Seff-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: / 2022.
Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXhv:2504.2057,, 20256.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning, arXiv preprint arXiv:2502.14766 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305. 10601, 2023.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long ohain-of-thought reasoning in llms, 2025. URL /tttpp:://rxiii org/ aaaaabbs/5/00 22022.50220 03373.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprini 2503 14476. 2025.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.0 01825, 2023.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity Ilms beyond the base model? arXiv preprint arXiv:2504.1 13837, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 154811146854888 2022.
14

---

Published as a conference paper at ICLR 2026
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892 2025a.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild, 2025b.
URL https: arxiv.iiivv. org/ abs//50s..225/ 18892.
Eric Zhao, Pranjal Awasthi, and Sreenivas Gollapudi. Sample, scrutinize and scale: Effective inference-time search by scaling verification. arXiv preprint arXiv:2502.01839 2025a.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards, 2025b. /RL.tttss://axx.v.ooo http::///wwwwv.....aa..oo abs 2505. 19590.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Dangi Chen, and Yu Meng, The surprising effectiveness of negative reinforcement in Ilm reasoning, 2025, URL https: //t////aaa //axxiv.or. orrg/ abs/2506. 01347.
15

---

Published as a conference paper at ICLR 2026
Appendices
A.
Analyzing Negative Gradient Dynamics in the ph Model.
B. Testing Extrapolation of Open Source Models. Additional related work.
C.
D. Perspectives on Future Work.
E. Additional Experiments and Details for Section 4 (Chained Asymmetries).
E. Asymmetries Beyond VG Gap: Summarization vs. Generation
G. Additional Experiments and Details for Section 5 (Negative Gradient).
H. Additional Experiments and Details for Section 6 (Curricula Training).
I. Illustrating the Efficacy of Coupled Curriculum.
Proofs for pe Analysis.
K. Performance on Non- Rea-onngn Benchmarks.
L. Broader Impact Statement.
M. Note on Computational Resources Used for e..
N. Example Traces.
O. Use of Large Language Models.
A ANALYZING NEGATIVE GRADIENT DYNAMICS IN THE De MODEL
In this section, we introduce a didactic pt model, where an LLM samples i independent actions sequentially, verifies them (with perfect accuracy), and terminates immediately after the correct one is produced. In this section, we introduce a didactic setup where verification is perfect (and hence, there is high VG gap), and formalize the intuitions regarding negative gradient from the previous section.
Didactic analysis setup. We consider Markov decision process (MDP) (Puterman, 1994) with action space A = ASstopp) where A = [100] are standard actions and st is an early "stopping" action (like BOS) that terminates the trace. For simplicity, we consider policies parametrized as a softmax bigram model TM (Mr+1 at): in this model, the policy only retains one token in its history and is parameterized by softmax over logits described by bi-grams, i.e., TM (a++1 (at) xx (xp 11+1)). In this bi- gram model, the current state 8 always matches the previous action 1,-1, and a* E A denotes the optimal action. In rollout al. a, the initial action a sampled from fixed ny- For >> 1, learner policy samples an action (-rx(-l) E (AA). The MDP terminates with reward 1 at time / ag = a², and with reward 0 if a, = stop (stops too early), or t Bur (budget is exhausted before correct response). The policy is initialized to one that puts high probability mass on choosing a stop. Details are in App. G.
We say that the model learns to explore in-context if it learns to never play stop for any t (no early stopping), until a* is observed, ie.. increasing in pt. On the other hand, classical exploration amounts to upweighting (aa (12-1) without reducing p(stop), i.e., improving p in p².
Finding I: Negative gradient increases length until p(a") is reasonably high. In Fig. 10(a). standard GRPO (B.- 100) increases average response length from 15 to 45 budget, driven by the drop in the marginal probability of stopping early p(stop) (Fig. 10(c)). After multiple RL iterations with negative gradients, the average number of attempts per trace is sufficiently large, and the learner can sample a" with non-trivial probability in any given trace. Once this happens (Fig. 10(c)), in our simple bigram setup, the model rapidly upweights the likelihood of one-step transitions to a", resulting in a phase transition where reward increases as length drops. In contrast, GRPOMask (Fig 10(b) fails to improve reward or increase length. The first phase is akin to chaining more asymmetries in LLMs and results in longer response length. In our LLM benchmarks, however, we do not see the same phase transition since finding "shortcuts" to correct responses considerably more difficult. Moreover, the LLM is conditioned on an entire history and learns to utilize the history carefully in the first phase. This makes it unlikely for it to quickly learn to reduce length substantially even if it transitions into this second phase some problems.
Finding 2: Negative gradient improves coverage by increasing entropy of TM (. (1:-1). When TM samples highly likely yet incorrect action, the negative gradient computed this sample increases entropy by moving probability mass onto less-seen modes of the distribution, including a.. Note that explicit entropy bonus is applied. We show this formally in Theorem A. where
16

---

Published as a conference paper at ICLR 2026
Std. GRPO
Mask negative grad
Prob, of stop of
Avg. token level entropy
4.6
0.9
rwward
45
0.9
rovard
knggt
45
0.08
0.2
0.06
0.6
30
0.6
30
std. GRFO
0.1 1.12
4.5
GRPO
0.04
2+ mark me gnd
mask reg. grod
0.3
15
0.3
15
0.02
0.0
0.00
0.00
2000
4000
0
2000
4000
0
2000
4000
250
1000
2000
Iterations
Iterations
Iterations
Iterations
Figure 10: Negative gradients in the p mmood.. Negative gradients push down p(s pp) during training (c), increasing length (a) and entropy of the next action distribution (d) to accommodate more in-context exploration, only decreasing them when a" is discovered. In contrast, positive gradients rarely change p(stap) or entropy.
we prove that upon sampling highly likely incorrect action with probability p. GRPO update with a negative gradient results in an entropy increase of x p² when all other actions, including a" are highly unlikely. We note this empirically as well in Fig. 10(d), where conditional entropy increases across states, until a" is discovered, after which it drops sharply as the positive gradient rapidly moves mass onto a within a few iterations.
Theorem A.1 (Negative gradient increases entropy when at is unlikely; formal version in Thm. J.3). At state if the most likely action under ι is =: arg m T[alss) & a", then, for any æ,, negative stochastic gradient step increases the entropy of (-|ss) (s) with prob. 2 =(alss). Additionally, in suitable regime of n, the increase Z (m(a)]s) x (a))s))) where 02 is second most likely after 01. contrast, in the absence of the negative gradient, the entropy is preserved with prob. 1 - -<(aass)
Connecting back to our findings in Sec. 5. From the above analysis in the bi-gram model it is clear that the recovered from the <EOS> token indeed moves to the optimal action under certain conditions. But, more generally, for our experiments on LLMs in Sec. 5, all we can say is that the negative gradient will decrease the probability of <EOS>. So, it unclear why the length increases and doesn't it drop from the negative reinforcement. One explanation for the length increase in Fig. 5 is that when we train LLMs with RL we fine-tune a base model and don't train from scratch. So, the priors in the base model have huge effect where the recovered mass from the <BOS> token lands. If the base model has a prior to chain asymmetries (verification -eriiiotnngneattioon (VG) chains), then reducing mass from the <BOS> token increases mass other likely actions in the base model, including the action of "chaining more". If the base model does not present this prior, e.g. if the base model does not present verification-generaa length increase. For this, please see our experiments on MULT (base model presents no VG gap on this task) vs. MULT- (bbas model presents VG gap) in Fig 5. Here, is clear that the length increase during RL training is more prominent in the task where the VG gap is present. This shows that the base model prior has huge role to play in where the probability mass from the <EOS> token goes.
gap on task, then we would not expect to see
A.1 ADDITIONAL DISCUSSION ON THE DIDACTIC ANALYSIS
First, we comment on exploration and meta-cxplorraion in RL, and how negative gradients in our didactic setting can connnect one to the other in the presence of asymmetries. Next, we introduce some details for the policy parameterization and training.
Negative gradients boost exploration, which in the presence of asymmetries incentivizes in- context exploration. In Sec. 5 we showed how negative gradients can boost exploration in RL, and in the presence of asymmetries in the base model, lead to more chained asymmetries and longer responses- a phenomenon we call in-context exploration. Here, we present theoretical result that explains why negative gradient can incentivize the more "traditional exploration" in RL, in our didactic bi-gram model. Since verification is perfect in our bi-gram model, any policy in our policy class always stops the stop token. Thus, an increase in exploration leads to longer traces, and more chained asymmetries. As a result, in this setting, we can view an improvement in exploration as an improvement in netaaexxloration (or in-context exploration), driven by negative gradients.
Parameterization of the policy class. We parameterize the policy class as a softmax policy, where the probability of next action 0:+1 1, at state current a, (in bi-gram model current state is equivalent
17

---

Published as a conference paper at ICLR 2026
to the previous action) is parameterized with the vector of logits [M(a] a.)aaA ie::
(d sla))
aar(1 a1)
En'ed e (M(a act1 A,a
(3)
where M = [M a)late can be expressed as matrix in ((X+I)XK, Note that the cuurent state can never be the stop action, since stop always terminates the MDP.
Training details. We set the initial distribution to be the uniform distribution over all actions except a", i.c., TT(aa)) = O. For each state s, the policy is first initialized with random values of M(. s) in |-3.0, 3.0). and then we set ostop s) 4.0, M (a" s) = -4.0, which mimics the setting where the probability of sampling the stop action is higher than any random action, and the probability of sampling a is lower than any random action. We train with learning rate of 1e-2 and use stochastic gradient descent to update the policy where a single update samples a random trajectory T, starting from random state sampled from the initial state distribution TT, by running the policy until termination of the MDP. We then compute the policy gradient term, by averaging the policy gradient loss over the tokens in the trajectory T: 1/17 Eielo log TM (1+-1) A(a,, 01:1-1)-
B TESTING EXTRAPOLATION OF OPEN-SOURCE MODELS
AIME 2025 Performance
0.6
test-time budget
Supervised Fine-Tuning
Reinforcement Learning
4096
0.5
8192
16284
0.4
32768
0.2
0.1
DeepSeak-R Distill Owen- 1.58
MinMMadh-R-1.3 1.58
DeepSeek R1 Dietil- Qwen-78
OpenThinkeur11 7B
DeepSeek R1 Distill www.iiillll------- -wa--322
s1 11.1-328
SeepScateR 1.5B-PP Preview
STILL-3-1 1.55--prrent previeww
I-Thought- 1.50ptt-..---- Preview
Open-RS1
Figure 11: Extrapolation of test-time compute: We plot the performance (pass@) @ 1) on AIME 2025 at different test-time compute budgets across multiple open-source models of different sizes, trained with SFT or RL.
Extrapolation on AIME 2025. Extrapolation (i.e. the chaining of generation, verification, refinement, etc.) can potentially extend LLM performance after training, and do so beyond the context length the model was originally trained on. To evaluate this properly, we need sufficiently challenging problems that allow meaningful expressiveness in reasoning beyond small context lengths. The math problems associated with AIME align with this, and our evaluations prioritize AIME 2025 to attempt to mitigate any potential data contamination the models' training sets from previous years of AIME, The goal of the experiment is to measure the extent to which test-time compute influences overall model performance as context length increases, with the expectation that increasing output length allows models to "reason" for longer periods, continuing the extrapolation process, and ultimately arriving at the correct answer more frequently.
Experiment setup. Inference for every open-source model performed using Oumi through data-parallel SGLang. All models had inference run with a max output length of approximately 32k tokens, though some are slightly lower due this exceeding their max context length when combined with the prompt. The exact inference hyperparameters are described in Table 2. After inference, the model responses were truncated from the right side until the number of remaining tokens present was equal to the specified test-time budget. 16 responses were collected for every problem in AIME with the specified inference settings, and the Pass@1 rate was calculated by averaging over these 16 responses. Final answers were extracted using regular expression for the boxed portion of the answer, with correct answers marked as passing and incorrect or incorrectly parsed answers marked as nonpassing. The prompt used is in Box I. I, and the problems were taken from the FVU AIME 2025 dataset on HuggingFace!.
hhttps:/huuggingfaan
IMR_2025
18

---

Published as a conference paper at ICLR 2026
Box B.I: AIME Evaluation Prompt Template
You will be given math problem. Solve the problem step by step. Output your final answer in the form of" \boxed answer). Problem: {problem}
Model
Temp.
Top p
Rollouts
Max New Tokens
Model Max
Length
MiniMath R1-1.5B
0.6
0.95
16
32768
40960
DeepSeek RI-Distill Petisttill-Qee----- 1.5B
0.6
0.95
16
32768
40960
OpenTTiinkrr-7B
0.6
0.95
16
31000
32768
DeepSeek-R--Distill--eeeeee
0.6
0.95
16
32768
40960
sl.1-32B
0.6
0.95
16
31000
32768
DeepSeck-RI -I--stiill
0.6
0.95
16
32768
40960
DeepScaleR-1.555 --reereee
0.6
0.95
16
32768
40960
1TILL-3-1.5B. -.5--previwww
0.6
0.95
16
32768
40960
I--Thought-1.55 P.ee-ree
0.6
0.95
16
32768
40960
Open-RS1
0.6
0.95
16
32768
40960
Table 2: Inference parameters used for generating the extrapolation plots in Figure 2.
Results. The results in Figure show that as the maximum number of output tokens increases, every model capable of "reasoning" is able to attain higher Pass@ rate, with performance generally saturating at 16k tokens with relatively minor improvements at 32k. We do not observe this with MiniMatt-RI-1.5B and suspect this is due its fine-tuning focusing solely smaller math problems trained with supervised fine- tin-ttunin, likely resulting in catastrophic forgetting the ability to continuously extrapolate. Interestingly, behavior among models tuned with reinforcement learning compared to DeepSeek RI R1-Distill-Qwen- 1,5B, which was trained with supervised fine-tuning. We suspect that this is likely due the nature of the distillation data from the RI model, which, if varied sufficiently in length, could avoid the length bias normally learned from supervised fine-tuning, while still teaching the model to perform extrapolation.
do not
strong improvement in extrapolation
In addition our evaluation on AIME2025, we also evaluate extrapolation on the older math benchmark of MATH-500. We show the extrapolation performance of open source models below. also compare the performance with our e311.7B model.
Model Test-time compute (tokens)
4096
8192
16384
24576
32768
agentica- 1.5B
74.65
83.18
84.25
84.25
84.25
OpenThhinker-11.
47,90
71.18
82.58
85.85
86,53
Qwen3-1.70
55.98
75.28
83.53
85.10
85,45
OpenThinker-7BB
63.48
73.35
78.93
79.93
80.10
sl.1-32B
73.40
83.43
86.98
87.93
88.23
e3-1.7B (Ours)
66.53
81.38
86.38
Table 3: MATH500 evaluation. Accuracy (%) across varying test-time compute budgets. Our e3- 1.7B consistently outperforms prior sub-2B models, surpasses the 7B SFT-trained OpenThinker-77, and approaches the performance of s1.1-32B (over 18x larger) at longer token budgets. All models are evaluated with top-p = 0.95, temperature = 0.6, and accuracy averaged across rollouts per prompt.
C EXPANDED DISCUSSION OF RELATED WORK
Scaling test-time compute via long CoT reasoning. Prior work explores number of avenues for scaling test-time compute, including majority voting (Wang et al., 2022), best-of-n sampling. and beam search (Setlur et al., 2024; Snell et al., 2024), well sequential self-correction (Qu et al., 2024; Kumar et al., 2024). More recent results indicate that training models to use test-time compute generate longer chains of thought (CoT) that combine verification, search, and self-correction all
19

---

Published as a conference paper at ICLR 2026
in free-form manner, performs better (DeepSeek-AI et al., 2025; Team et al., 2025; OpenAl et al., 2024), resulting in widespread open-source reproduction efforts (Face, 2025; Yeo et al., 2025; Zeng et al., 2025b; Luo et al., 2025b). We situate our work in the paradigm of long CoT reasoning.
Test-time extrapolation. The true benefit of test-time scaling is consistently improving performance as we extrapolate test compute. While prior work tests the model's performance on budgets longer than the training budget Zeng et al. (2025a); Luo et al. (2025a), they do not explain the relationship between the training recipe and the extrapolation, like we aim do in our work. Other works perform extrapolation by explicitly prompting models to generate more tokens when a response terminates Muennighoff et al. (2025); Aggarwal & Welleck (2025), whereas, we show that models that learn to explore in in-context extrapolate test compute better than prompting-basee -baiingg--eedddd approaches (Fig. 8). In partieular, we study the role played by the base model, training algorithm (RL.), as well as data mixtures and token budgets, on the ability to extrapolate. Furthermore, prior work Setlur et al. (2025b) has investigated scaling when train and test budgets are the same, but we expand the scope of this comparison substantially.
Exploration in test-time scaling, While prior works have shown the importance of the base model's ability to conduct exploration (Gandhi et al., 2025; Liu et al., 2025b), we discover that it is crucial for extrapolation. We show that the negative gradient in RL incentivizes chaining multiple asymmetries and leads to longer response length, and better performance. SFT alone does not provide this kind of chaining or exploration benefits. Our analysis is orthogonal to theoretical works Setlur et al. (2025b); Swamy et al. (2024), which shows that RL performs better than SFT, but from statistical perspective, whereas our argument is more focused on the learning dynamics. Concurrent work builds techniques to boost exploration during RL via advantage normalization (Li et al., 2022; Yu et al., 2025) or PPO clipping (Yu et al., 2025), and these techniques can be combined with e3, but they do not study the role of negative gradients in learning to explore. Finally, Wang et al. (2025b) briefly remarks about the role of policy gradient loss and entropy when running RL with only few examples. Our study investigates the underlying mechanism of negative gradients increasing length and entropy.
Data and length curricula. Recent works have also investigated using curriculum on problem difficulty Team et al. (2025); Xie al. (2025); Shi et al. (2025) and output length Luo et al. (2025b); Liu et al. (2024) during RL training. Their motivation stems primarily from an efficiency standpoint: avoiding zero advantage updates Shi et al. (2025); Yu et al. (2025), efficient optimization Luo efficiency of using test-time compute Qu et al. (2025b). While we do make
et al. (2025b), similar observations regarding each curriculum individually, perhaps our most interesting finding is that carefully coupling both data and budget curricula can lead to much better performance and extrapolation, beyond merely some gains in efficient training. We show that training on hard problems with short budgets often yields terse solutions that fail extrapolate, while easy problems with long budgets can cause optimization issues or verbose outputs. Thus, curricula must be carefully designed to support effective extrapolation. Conceptually, our curricula are most related to dense progress rewards (Qu et al., 2025b; Setlur et al., 2024), in the sense that curricula incentivize different degrees of progress for different questions, at different points in training.
D PERSPECTIVES ON FUTURE WORK
There are a number of implications of our work and a number interesting directions that future work should build upon. We list the main technical implications and open questions below.
Sharpening in-context exploration. A number of concurrent RL results either directly (Yue et al., 2025) or indirectly (Shafayat et al., 2025; Zhao et al., 2025b; Shao et al., 2025; Prabhudesai et al., 2025) argue that RL training on LLMs sharpens the base model's distribution, as also previously studied by Huang et al. (2024). In contrast to this, our study shows that if we can utilize a coupled curriculum on top of a base model that admits asymmetries, RL. can actually enable chaining new asymmetries, resulting in an increase in length, indicating the presence of structured exploration. This behavior is distinct from traditional sharpening that corresponds to cloning one (or few) of the responses sampled from the base model. In fact, our conceptual study in the p² model in Section A also highlights these two distinct phases during RL: an initial in-context exploration phase where negative gradients lead to increase in response length and the policy learns to utilize test-time compute for better exploration, followed by phase where it sharpens to the best traces found thus far. The design of e3 enables it to operate in the former phase. We believe concurrent works that
20

---

Published as a conference paper at ICLR 2026
finds RL largely sharpens the model operate in the second regime by training data that does not require chaining asymmetries or operating with a very low training budget such that chaining is impossible. As result, models trained purely in the sharpening regime may behave similarly to the base model with an alternate prompt, with RL perhaps offering little more than an implicit prompt tuning effect. But we would not expect this for the chaining regime. A detailed study on separating these regimes, and identifying all the factors that draw RL training into these regimes is an interesting direction for both theoretical and empirical research.
Connection with dense progress rewards. While e utilizes a coupled curriculum, this curriculum is closely connected with the use of dense rewards, as prescribed by our prior work (Qu et al., 2025b; Setlur al., 2024). To see why, note that one can reparameterize coupled curriculum into a single round of training with dense rewards applied to short segments of the output response, perhaps in a similar way as Qu et al. (2025b); Qi et al. (2025). Therefore, the success of the coupled curriculum approach in e3 at improving performance and not only in reducing total training compute perhaps hints at future success with dense rewards at scale, with initial results showing that dense rewards help larger models already being shown in the community (Wang et al., 2025a). We encourage readers to explore the connection between curriculum and dense rewards further.
Introducing new asymmetries. The conceptual model behind e applies with any asymmetry. though most experiments in this paper utilize only the inniccattiongneeaattinn gap. It would be interesting to identify other asymmetries and study methods to imbue base models with these asymmetries. Definition 4. I in Section 4 provides starting point to define these asymmetries.
Is curriculum fundamentally needed? A natural question whether curriculum is fundamentally necessary as we vary model sizes and capabilities. Unlike supervised learning on a fixed dataset, online RL generates its own rollouts. Reinforcing chaining behavior via negative gradients (Sec. 5) requires that such chaining reliably improves performance on training problems much more substan- tially compared to sampling diverse traces that do not chain asymmetries. This likely necessitates specific training configurations regardless of model size with standard outcome-reward RI., or the use of dense rewards (as discussed above). While larger models may admit simpler curricula, deliberately using currciulum or dense rewards as inspiration may be critical.
Explicit exploration bonuses. In our runs, the main issue hindering us from benefits of further scaling of output length during RL is the repetition bias in the base model, where it tends to repeat previously- -egnerrte segments in its trace beyond certain output length. This repetition bias inhibits the efficacy of in-context exploration beyond a certain output length and as a result inhibits further test-time scaling. We believe that explicit exploration bonuses that enable the model to search for tokens in this regime would result in even better in-context exploration.
E ADDITIONAL EXPERIMENTS AND DETAILS FOR SECTION 4 (CHAINED ASYMMETRIES)
E DETAILS ON MULT AND MULT-V
Data collection. Both MULT and MULT-V consist of multiplication traces for solving a 5-digit x 5 5-digit multiplication problem. For the MULT task, we use a Llama3.2-2 instruction tuned model where the number of intermediate verification attempts is much lower in a trace when asked to solve a multiplication problem. In fact, it is not hard to see that, in general, for multiplication, generation of a trace may be as hard as verifying generated one, as the only way to verify the entire trace is to re-attempt the multiplication or carry out division with the computed target. We contrast this task with the MULT-V task, where the Alama3.2-3BB models are first finetuned on traces from Qwen-32B R---RI-DD RD-Dstilleed and GPT-4o models. These traces contain multiple verification attempts that verify intermediate steps solving smaller multiplication problems, and the steps are part of an entire trace that attempts solve the main multiplication problem involving two 5-digit numbers. For collecting data we used the prompt in Box 1. In App. N Example 2, we also provide an example multiplication trace with verification attempts sampled by the base model in MULT-V, As we will see in Fig. 16, the absence of asymmetries in MULT leads to lower accuracy and verifications when compared to MULT-V, V, where asymmetries are present.
21

---

Published as a conference paper at ICLR 2026
Box E.I: Prompt for generating MULT-V data
Multiply (numl) } and {num2). Please reason step by step, and put your final answer within (boxed()) (). At each step, try to verify your response if possible and prefix the line with "Check:". ,think,
Hyperparameter
Values
train wetnbaatch.ssee size
256
ppo_mini batchasizz
64
learning_rate
5.0e-6
kl loss_coed coef
0.001
entropy coeff temperature
0.001
1.0
rollout.n
16
ppo lowerclip threshold
0.2
ppo ppo.higherclip threshold
0.2
Table 4: Verl Sheng et al. (2024) hyperparameters used for MULT and MULT-V.
Training details. Hyperparameters for our experiments on MULT and MULT-V V are given in Table 4.
E.2 DETAILS ON CDOWN
Training details. Hyperparameters in CDOwN experiments follow the table below unless otherwise specified. In all of our CDowN experiments, take the fine-tuned Llama3.2-3B base model from Gandhi et al. (2025). For Fig. 4, we trained with Bür 512, 1024, 2048 on problems with 4, 5, candidates. The total number of datapoints used 40000, which were evenly split across the four difficulties.
Hyperparameter
Values
main_batch size
128
ppo.mini. woo.mini..atth.stiee
32
learning rate
1.0e-6
Illoss_coes entropy_coeff
0.001
0
temperature
0.6
rollout.n
8
ppo lowerclip threshold
0.2
ppo hiphheigerrlip threshold
0.2
Table 5: Verl Sheng et al. (2024) hyperparameters used for CDOwN.
Evolution of chained asymmetries at test time. In Fig. 12, show that training progresses, responses with more chained asymmetries enjoy greater improvement. diagonal parallel to the main diagonal from top left to bottom right, we move across constant attempt budget (e.g., moving from 16 chained asymmetries x pass to 8 chained asymmetries x 2 passes). Having sequential chained asymmetries become increasingly better than parallel rollouts training progresses, indicating the exploitation of asymmetries in RL training. See example of chained asymmetry in App. N, Example
we move across any
E.3 IN THE PRESENCE OF ASYMMETRIES, KL DIVERGENCE WITH BASE LLM REDUCES AS TRAINING TOKEN BUDGET INCREASES
In Fig. 13, we interestingly observe that training with higher B,, results a smaller token KL- divergence from me all throughout training countdown. On multiplication in the absence of
22

---

Published as a conference paper at ICLR 2026
# chained asymmetries
Gradient step 0
Gradient step 30
Gradient step 60
16
0.06 0:10
16
0.37
0.65
16
0.55 0.62 0.67 0.69
0.6
Pass@k accuracy
8
0.04 0.07 0.12 0.17
0.22
8
0.19 0.27
0.36
0.44 0.49
8
0.19
0.800.28
0.36 0.47
0.4
4
0.02 0.05
0.04
0.07
0.23
0.05
0.14 0.0
0.2
2
0.01 0.02
0.03
0.05
0.07
2
0.01
0.02
0.03
0.05
0.08
0.01
0.02
0.040.06
0.09
1
2
4
8
16
1
2
8
16
2
4
16
Figure 12: Evolution of axymmetries during training on CDOwN: More chained asymmetries to a greater
improvement in pass @k performance actoss gradient steps.
Countdown
Multiply (No verification)
0.70
0.70
KL Divergence
512
KL Divergence
1024
1024
2048
0.45
2048
0.45
4096
0.20
0.20
100
200
300
400
100
200
300
400
Figure 13: Kl-ddvergence with base LLM on CDOWN and MULT: When running RI. training CoowN and MULT with multiple training budgets (512, 1024, 2048 on CDowN and 1024, 2048, 4096 on MUlT) we note the KI. divergence is lower when running RL. training on higher training budgets, when the base model presents asymmetries (in here the asymmetry is given by the verification generation gap on CDOwN).
asymmetries, the KL-divergence values are roughly similar for all B... This means that when the sefficatioongenerraiionn asymmetry is present, the training process deviates less from т. at each token, but is able "chain" multiple verification and generation attempts together to improve accuracy, by learning to explore over the space of basic skills. Prior work argues that model that deviates less from base pre-trained model generalizes better on unseen prompts (Gao et al., 2019). If we were to apply this argument in case, this means that models that able to asymmetries better should yield better performance on unseen prompts, especially with larger test budgets.
E.4 VERIFICATION GENERATION GAP IN LARGER MODELS.
In Fig. 14(efft), empirically validate the verification-gente -eenrtatoonnneeeaiionnnnooo gap for larger model (Qwen3-88) on hard DMATH problems. As our detect (a(tect(((p()).)))) score increases (indicating traces with more verification- cerifiatioo-ggneraation chains) the pass @ performance consistently improves for l: 21, This monotonic trend shows that instances where the in-context verifier can more reliably distinguish good from bad trajectories are exactly those where more attempts afforded by increased sampling budget translates into larger gains, confirming that the asymmetry captured by Definition 4.1 is not small-model artifact but persists even the 8B scale.
In Fig. 14(right), we empirically validate the verification-generation gap for one of the largest open source reasoning models (DeepSeck-RL) 685B parameters) on a set of 150 problems (8 rollouts per problem) in OpennMahheasoonngg Moshkov et al. (2025). We did not use DMATH for this since R1 solves all problems in the hardest set of DMATH with very high accuracy. Instead, we picked 150 random problems from OpenMathReasooing Moshkov et al. (2025) where the pass @ rate run TIR mode is less than 0.05, find
(out of 32 generations) for that on traces with more perification-generaae chains (detect(g(p(-), 7)) in Definition 4.1) the pass@1 performance higher than on traces with fewer chained asymmetries. This tells us that the asymmetry between verification and generation holds in even of the largest (685B parameters) state-of-the the-art reasoning models, as reflected in the model's 's reliance on exploiting this asymmetry to improve performance.
23

---

Published as a conference paper at ICLR 2026
0.5
17
0.24
0.38
0.59
detectiq(pt 1), T)
0.5
9
0.21
0.35
0.55
0.4
0.4
pass
pass
5
0.13
0.21
0.34
0.3
0.3
3
0.06
0.10
0.14
0.2
0.2
-0.1
0.01
0.02
0.04
0.1
1
2
4
20
40
60
80 100
k for pass@k
detect(q(p( (), r)
Figure 14: Verffication-generation gap in large models: We compute our detect(g(p()), 7)) metric in Definition 4.1 on: (left) traces T sampled from the Qwen3-8B model on hard problems DMATH; and (right) traces sampled from the 685B parameter model DeepSeek-RI (on set of very hard problems in OpenMathReesooning (Moshkov et al., 2025)). Higher values of (g(ecct((((())))) means more verification- generation chains. We that as this metric goes up. the performance (pass@k) increases. This verifies that the perification-generaiio gap (as measured by our Definition 4.1) is present even in largest open source state-of-the-art models.
One might ask what happens if we scale the model capacity even further; would we still see a porficattion-genraatin asymmetry being exploited by the model? We believe that the answer this is conceptually yes as long as the model is pre-trained to imitate human-generatee generated behavior from the Internet. particular, to conceptually understand why asymmetries (Definition 4.1) arise at all in the first place, note that intuitively our definition measures how "easy" it is to get high reward with chaining skills vs. not. For any bounded capacity system, such gap should exist as more "information" is gained by performing more computation via skill chaining on a given test input query as opposed to directly querying the system for an answer, with minimal or no test-time compute. With this intuition, such gap exists for humans because humans don't 't have infinite computational and reasoning capacity and are themselves bounded capacity. Since we train models on human generated data during pre-training, this would naturally yield models that imitate systems of bounded capacity.
Of course, the set of test queries on which this kind a gap is more prevalent will change depending upon the characteristies of the bounded capacity system: for instance, for previous gold-medal winners on the international math olympiad, verification and generation are likely equally complex on problems at the difficulty of AIME, whereas for high-school student without a mathematical background there might still be a trend is also expected be reflected language models, and will be determined by the interaction between the base model size, pre- rre-training data, and the test query. In principle though, unless all test queries are perfectly answered by a language model, such a gap should continue to exist on some test queries. We showed this concretely above using experiments with DeepSeek-RR R1 and Qwen3-8B asymmetry despite being much bigger that
asymmetry on an AIME question. This
models that both of them exhibit perification-generaa our 1.7B model, but of course the set of test problems changed in each case.
F ASYMMETRIES BEYOND VG GAP: SUMMARIZATION VS. GENERATION
In Sec. 4, we present "chained asymmetries" chaining of skills such as verification and generation. However, this is not the only notion of asymmetry. To illustrate a different form, we discuss results from a concurrent work Qu et al. (2025a). In this work, authors train a model generate hints based on summaries of many rollouts on problem, then conditioned on this hint generator they train policy using RI. to generate traces that optimize the typical outcome rewards.
Based on terminnology we introduce in Sec. 4, our interpretation of Qu et al. (2025a) is that they introduce an asymmetry between planning or abstraction prediction and response generation. In particular, they show that an LLM can improve performance on reasoning tasks by chaining two skills: (1) abstractly planning or generating high-level hints/hypotheses, followed by (2) generating the solution conditioned one of the hints, opposed directly generating the answer. Imbuing this structure leads to improved performance, especially in regimes where pass @+ k has saturated at large l for model trained with standard RI. Here, include a subset of results from Qu et al. (2025a).
24

---

Published as a conference paper at ICLR 2026
First, they evaluate GPTT-4omini several classification tasks, both with and without asymmetry (adding hints derived from summarized thinking traces). Performance improves significantly when hints are included.
Model
Breast Cancer
Tweet Hate
Lobbying
Bank Note
GPT-4o-mini
45%
60% 90%
88% 94%
45% 100% %
GPT-4o-mini asymmetry
90% %
Table 6: Impact of chaining hint generation with response generation on classification tasks.
Next, they compare RL-trained models. The baseline is trained to generate final answers directly, while the asymmetry model is trained to chain hint generation and solution generation. For fairness, they allocate the same compute budget: the baseline uses & rollouts, while the asymmetry model uses vk rollouts for hints and vk per hint for solution generation. On AIME 2025, the chained asymmetry again yields consistent improvements.
Model
pass @4
pass@ 16
pass 64
RL baseline (no asymmetry)
51.1
65.2
77.1
(with asymmetry)
59.0
71.1
80,5
Table 7: AIME 2025 results comparing RL baseline with RL using chained asymmetry.
The above results indicate, that there other skills like summarization, retrieval, instruction following in the base pre-trained model, which can be chained with generation to improve final performance. With the right base model, and curricula, RL can amplify all of the "chained asymmetries" that improves training rewards, incentivizing in-context exploration and as a consequence also improve the extrapolation performance.
G ADDITIONAL EXPERIMENTS AND DETAILS FOR SECTION 5 (NEgATIVE GRADIENT)
G.1 DETAILS FOR CDOWN
We trained models for 90 steps problems with 5 candidate numbers with a training budget of 2k.
Cumulative unique attempts plot. Fig. (left) was filtered on incorrect traces on problems with 50% success across gradient steps. We select only incorrect traces capture the ability of the model to explore for the correct trace, rather than to output diverse correct traces once one found. We filter for problems with 50% success across training for GRPO and GRPOMask because otherwise the algorithm with better rewards would see more problems with lower cumulative unique attempts, the correct traces discovered early and subsequently reinforced.
Evolution of the conditional distribution given past attempts in CDowN. We run ablations on the conditional distribution of attempt (sequence of tokens that constitute attempt to plug-in operations so as to match the target CDOWN) given past attempts in three different settings, shown in Fig. 15. In (a), plot log p(az (1 (11k--) 1) log p(as (a1:k-2), which should average roughly 0 if the attempts are independent. As training progresses, this quantity grows, indicating a correlation between attempts, especially with larger (potentially because the new attempt can attend more previous attempts, and thus becomes more dependent on them). In (b), we plot log p(a [01:x=)) log (a
which also grows over time. This indicates that the conditional distribution p[new attempt past attempts) sharpens as the number of past attempts grows, implying that the model gets more confident as it explores more in-context. In (c), we plot log p(ax [ and note that it reduces with more attempts way more the trained model, compared to initialization. This means, that the model has learned not to repeat its previous attempt when it immidiately re-attempts to solve the problem. These three trends jointly tell us that the learned model indeed learns to explore-in-cee in-eentext where it adapts and sharpens the conditional distribution over the next attempt with more previous attempts.
25

---

Published as a conference paper at ICLR 2026
Learning to sample an unseen attemppt
Conditional distribution sharpens
Learnigg not to repeat
0.15
loztpla (txx-s))
(oatola ouu-3)
0.10
of au 1]]
-log(o(a,, (0bg--)))
iter
200
0.00
her
Ree
iter
iter CO
-0.05
iee 400
0.025
0.05
iter 200
-0.10
iter 400
0.00
-0.15
-0.025
-0.20
10
Attempts
Attempts
Attempts
(a)
(b)
(c)
Figure 15; Probing the conditional distributions conditioned on past attempts in CDOwN. (a): New attempts
not independent of past attempts (b): Model becomes more certain of what to try next given more past attempts (c): Model learns not to repeat past attempts.
NULT-V
MULT
NULT-V
HULT
# Verifications
30
30
0.70
0.70
- mask neg. grad
mask neg. grad
Accuracy
mask neg gradk
mask neg. grad
20
GRPO
20
- GRPO
0.55
GRPO
0.55
- GRPO
10
10
0.40
0.40
0
0.25
0.25
100 200
300
400
100
200 300
400
100
200
300
400
100
200 300 400
Figure 16: Negative gradient amplifies verification when VG gap is large. While utilizing the negative gradient amplifies the number of calls to verification in MULT-V, the number of verification calls does not grow over training in MULT. Interestingly. though, we find that when negative gradient masked out on MULT-V V. the number of verification calls is still very low and does not increase, corroborating our findings that exploration driven by negative gradients results in in-tontext exploration only in the presence of asymmetries in the base model. A similar trend is also observed in terms of the raw accuracy.
G.2 ADDITIONAL EXPERIMENTS WITH MULT
In Section 5 saw that training with the negative gradient leads more exploration during RL training, which in turn leads to the amplification of any chained asymmetries that may be present in the base model, e.g., more generation-verificatioo steps. In particular, we noted the increase in the number of verification steps in Fig. 5(b). To see how negative gradients affect the response length and number of chained asymmetries in the absence of a strong VG gap, we compare running GRPO with and without negative gradients on our multiplication task MULT where the VG gap is weaker in the base model.
We plot results in Fig. 16, where we note two trends when running RL training with and without negative gradients on MULT (without VG gap), and MULT-V (with VG gap) using a training budget of 4096 tokens. First, note that the number of verifications is higher when in a setting with a large VG gap. When the VG gap is absent, the number of chained asymmetrics (verification-gennet
negative gradients
steps) are roughly the same with and without masking the negative gradient. Second, we note that the accuracy is much higher with negative gradients in the presence of VG gap (MULT-V), and comparable a where we mask the negative gradients in the setting where the VG gap is poor (MULT). Together, this tells us that the boost in exploration driven by negative gradients leads to more chained asymmetries when the base model presents some of them, like large VG gap.
G.3 GRADIENT NORM OF GRPO VS. GRPO WITH MASKED NEGATIVE GRADIENTS.
In Fig. 17, we compare the le gradient norm during training for standard GRPO and our masked- GRPO variant DMATH. We observe that masking negative gradients produces larger gradient norm over training steps, whereas standard GRPO exhibits a smaller one. This is mainly because only average the positive gradient over the tokens with positive advantage (since they end in a correct response) for masked GRPO, but in the std. GRPO setting the gradient is averaged over all tokens (including the zero-advantage tokens), which can result in lower gradient norm for std.
26

---

Published as a conference paper at ICLR 2026
std. GRPO
gradient horm in 82
0.12
masked .ve
0.10
0.08
0
50
100
150
200
gradient steps
Figure 17: Gradient norm of Masked GRPO vs. GRPO: We plot the gradient norm for our experiment in Sec. 5 where run standard GRPO and the masked version of GRPO on DMAT and find that masked GRPO observes higher gradient norm. This is mainly hecause we only average the positive gradient over the tokens with a positive advantage (since they end in correct response) for masked GRPO, but in the std. GRPO setting the gradient is averaged over all tokens (including the zero-ade advantage tokens), which can result in a lower gradient norm for std. GRPO.
GRPO. This gap in performance between masked-GRPO and GRPO is mainly from the lack of exploration (lower entropy and verification-genee chains) note in Fig. 5 and our theoretical analysis in Appendix A, and has less to do with the magnitude of the gradient update.
G.4
ANALYZING THE PERFORMANCE DROP FROM MASKED NEGATIVE GRADIENT
Suboptimal training budget 8k 4k
Suboptimal training Ir 1e-6
20-7
Val accuracy: budget 8k 1e-6
0.25
0.20
0.30
validation reward
raward
0.15
reward
0.25
0.15
0.10
0.20
0.10
0.05
GRPO
0.15
GRPO
0.05
GRPO
meeked neg. grad
masked neg: grad
masked neg grad
0.00
100
150
50
150
50
100
150
gradient steps
gradient steps
gradient steps
Figure 18: Performance drop from masking the negative gradient is orthogonal to the drop from choosing suboptimal hyperparameters: We plot the performance GRPO and GRPO where we mask the negative gradient in two additional settings (beyond what we plot in Fig. 5). In particular, we change the ppprmpprreettrs for the GRPO run on DMATH to suboptimal configurations: dropping the token budget from 8k to 4k and learning rate from le-6 to 2e-7. With all else being the same, we compare the training rewards of GRPO with masked GRPO. In addition, we also plot the validation performance masked GRPO (run with the config configurations in Fig. 5: hudget of 8k and learning rate of le-6) to show that masked GRPO indeed generalizes. This suggests that when we are masking the gradient, we are not necessarily training the model in ways where is expected to fail.
In Fig. 18 we compare standard GRPO (with both positive and negative gradients) masked GRPO (only positive gradients) on DMATH under two deliberately suboptimal configurations: (i) reducing the training token budget from 8k 4k, and (ii) reducing the learning rate from 1 x 10 6 2 x 10 These settings are expected to degrade the performance of standard GRPO, allowing us to ask whether the additional drop observed with masking can be explained purely by hyperparameter choices. Holding everything else fixed, we find that masked GRPO consistently underperforms standard GRPO and exhibits the same qualitative training dynamics as in Fig. 5: lower reward, lower entropy, and shorter output lengths. We also report validation accuracy for masked GRPO under the original (8k, 1 x 10-6) 9) configuration, confirming that the masked policy still generalizes. Taken together, these results show that (i) masking negative gradients reliably reduces exploration (entropy and trajectory length) and harms performance across a range of hyperparameters, and (ii) the performance drop from masking is orthogonal to the degradation induced by suboptimal hyperparameter choices.
27

---

Published as a conference paper at ICLR 2026
H ILLUSTRATING THE EFFICACY OF EQ. 2 AND COUPLED CURRICULUM
0.42
0.4
0.39
1.02
0.44
Accuracy
0.4
0.33
0.32
0.35
1(m / [
0.26
0.23
0.24
0.24
0.2
0.2
0.17
Test budget
J0n) at test budget
1024 (8,)
0.06
J(mz) after training on Ba
4096 (extrapolation)
0.0
0.0
2
2
2
2
None Budget Coupled Curriculum
Test budget / Btr
Figure 19: Left: Given model Ti, at end of stage 1 of the e3 curriculum on Coowx, we illustrate how choosing budget B.r for stage of the curriculum via our presribed Eq. is optimal. On the x-axis we plot the test budget for evaluating IT. Based on the evaluation of Tu, find that 2 tokens is the smallest token budget with K < 1.2. Based on Eq. 2, we now set 2 as the token budget for stage 2 training of the curriculum, and find that mh trained 1024 tokens is comparable with any л2 trained at higher budget value, and much better than Iı trained at a lower value. (((T) is pass @ 128 performance and J (Tra is pass @ accuracy. Right: e3 on CDOWN: our coupled curriculum performs better than budget currciulum (2 222) or no curriculum at all.
In this section, we show that for a given dataset, picking the training token budget based on Eq. actually results in the best performance of the RL trained model. We demonstrate this CDowN by training on problems of 3, 4, and numbers. In our coupled curriculum, we first train on problems of easier difficulty with and numbers, use Eq. 2 to pick the training budget for stage 2 of the curriculum where we train only on difficult problems (CDOwn problems with 7 numbers). Following Eq. 2 with say H. 1.2, select the budget for the next stage, we examine the performance of this first-stage model on the second-stage dataset consisting of harder problems with 7 numbers. Eq. prescribes that we pick the smallest but reasonable B such that there is only a marginal improvement from extending B to 2B, defined by 1.2. As shown in Fig 19 (left), this corresponds to B (D1) 1024 (where D1 is the second stage training dataset). Indeed, Fig 19 (left) shows that when we train the second stage model at 1 (D, = 1024, we get nearly the best performance of 0.42 on the harder problems (7 numbers). We also note that while B.r = 2048 marginally improves test performance over Ber = 1024, it is unclear apriori if B., 2048 would train stably. Our goal here is to come up with prescriptive thumb rule that can inform the practitioner of the smallest training budget at which they can expect to train a model on a given dataset and hope to see the performance the trained model extrapolate well with additional test-time tokens. We also find in the figure the right, that coupled curriculum outperforms budget curriculum or not training with any curriculum.
relatively smaller token budget of 256 tokens. Next,
Choice of In Fig. 19(eft), if we had picked value of 1.3 would have run RL training a much smaller and suboptimal token budget for stage 2 of the curriculum. At the same time, if we had picked 1.05, we would have picked unnecessarily large training budget.
I ADDITIONAL EXPERIMENTS AND DETAILS FOR SECTION 6 (CURRICULA TRAINING)
For casy-to-see comparisons of e3-1.7B with open other open source models of similar and bigger sizes, we split the plot in Fig. 8(a.b) into four different plots, two each on AIME 2025 and HMMT 2025. In each set, we compare e3-1.7B separately with models of size <2B and >B (Fig.
I.1 TRAINING DETAILS AND IN-DISTRIBUTION PERFORMANCE ON TRAINING BUDGET
We present hyperparameters for e. training runs in Table 8.
Note on in-distribution performance. In Sec. 6 we note that for best extrapolation performance, it is important to vary the mixture of tasks in the dataset, as well as the training budget (max token length) in a a coupled way, over the course of RL training. Here, we note that if we were to only care about in-distribution performance, i.e., performance on fixed task mixture (of equally proportioned
28

---

Published as a conference paper at ICLR 2026
e. vi. x26 operssouree models
<3 ve. >28 spurce models
63 <28 open-source models
open-source models
Ace. AIAEIE '25
40
Acc. on AIME 22
40
conde
Acc. on HMMT 25
Acc. HMMTT
30
sporB/iten75
et-320
20
apentti 121
10
ralve Rl
01 11--02
a3/1.70
43/177
8K
24k
32k
4k
8K 16k
24k
32K
4K 8K 16k
24k
B24
Teel-Sime compute (lokerss)
Test-lime compute (lokems)
Teal-l compute (lokers))
Teal-Sme compute (lokenss)
(a)
(b)
(c)
(d)
Figure 20: Comparing e3 with open source models. In the above figure, the shaded area indicates the extrapolation regime. e3-1.7B trained with our RL recipe achieves state-of-the the-art performance across models
<2B and is comparable or better than some 7B and 32B models trained with supervised fine-tuning.
easy, medium, and hard questions in DMATH), then the best way to train is to match the test token budget and prompt mixture with training. In particular, training only on easy problems and budget of 8k yields a performance of 54.3% on a test dataset consisting of all tasks (from easy, medium and hard splits). But, if we match the test mixture with train, and train on all difficulties, then on the same 8k test budget, we note a performance of 58.9%, averaged over all difficulties. Note that the exptrapolation performance (on hard, out-of- cutoo--distribution AIME '25 questions) of the same models is flipped in Fig. 7, indicating that the curricula design is mainly needed for extrapolation, via in-context exploration, as opposed to best performance at a fixed test Bur.
Hyperparameter
Values (Brr 8k)
Values (B,r 16k)
train batch size
128
64
ppo mini batch size
32
32
learning_rate
1.0e-6
1.0e-6
kllloss_coef
0,001
0.001
entropy.com coefff
0,002
0.001
temperature
0.6
0.6
rollout.n
8
32
ppo loweeelrll threshold ldwerolip.thresholdd ppo higherelip threshold
0.2
0.2
0.5
0.35
Table 8: Verl Sheng et al. (2024) hyperparameters used for e3 runs DMATH.
Hyperparameters for e3. In Tab 8, we report the hyperparameters we used for RL training during different stages of our coupled curriculum runs on DMATH. For the first stage of training on B,,=kk, we used fewer number of rollouts per prompt (16), and increased this to 32 for second stage training when 1,,= We did this to account for the larger outcome-reward variance typically associated with long horizon RL training (Agarwal et al., 2021). In general, we find that successful RL run at a budget of 16k can generally characterized by the following trends: (i) averag per-token entropy increases during training (or at least does not drop during training); and (ii) the number of chained asymmetries (verification attempts) and the response length increases during RL training. To improve token-level entropy we move away from purely on policy RI., and use off-policy data to update the current policy, with the policy density ratio clipping mechanism to avoid aggressively off-policy updates. Consistent with the findings in Yu et al. (2025), we find that when updating the policy on stale off-policy data, using a higher clip ratio for the positive advantage tokens is critical for increasing token-level entropy during RL training. This is mainly to weight the probability of some very low probability and positive ad advantage tokens. But increasing the clip ratio too substantially can also de-stabilize training, as we observed in the 16k training runs, due to which we dropped the clip threshold from 0.5 to 0.35. The rest of the hyperparameters are consistent with the default options in Verl (Sheng et al., 2024).
Evaluation protocol for the "Wait" extrapolation method in Muennighoff et al. (2025). In Fig. 9, we explicitly intervene on the model generations (except for our model e3-1.7B) with the "Wait" extrapolation technique from Muennighoff et al. 2025 and plot the performance against the average length of the extrapolated trace (on the x-axis). Concretely, we first run a 16K-token rollout, stopping at < </think> instead of EOS. To extend it with wait, we repeatedly append rotating prefix from ["Wait", "But wait", "Alternativel"" "But hold on"]. rollout again (stopping at </ </think> or 2048 tokens), and after 2 to 8 such extensions we let the model finish without stopping at </think>.
29

---

Published as a conference paper at ICLR 2026
Classifying problem difficulty on DMATH. On the DMATH task, we classify a problem as easy/hard based on the pass @ performance of QwQ-32B (more capable than our base model Qwen3-1.7B) on given problem. For example, from DeepScaleR (Luo et al., 2025b)) as hard if the pass @ performance (measured on 32 rollouts) of QwQ-32B is O. We classify it as medium if this value is between 0.0 and 0.2 and easy otherwise.
describe in Section 5, we bucket problem in DMATH (sourced
1.2 FIXED TRAIN BUDGET, VARY DATASET CURRICULUM ON CDOwN
In this subsection, we demonstrate that training with data curriculum based on difficulty with a fixed train budget can lead to over-exploratory output traces, the example task of CDOWN. With the data curriculum (i.e., fixed budget, vary data), we train first on CDowN problems with 3 candidate numbers (the "easy" problems) for 60 gradient steps, then those with 6 candidate numbers for 60 gradient steps (the "hard" problems), with a Ik budget across all steps. We compare this with the coupled curriculum in which the first 60 gradient steps trained with budget of 256. As shown in Fig. 21, the latter achieves better reward on "hard problems".
attempt 3
0.4
Curriculum coupled data
Curriculum
Curriculum
coupled
Train reward
coupled
data
reward
data
0.2
0.2
Equations
N
0.0
0.0
0
50
100
100
150
100 Gradient steps
Gradient steps
Gradient steps
Figure 21: Coupled vs. data curriculum on CDowN: training only on easy problems at large budgets leads to overfitting on "over exploratory" traces, failing to balance aaplore-explore tradeoff on harder problems later on. Reward graphs are displayed for hard problems.
Why is data curriculum worse than the coupled curriculum? We can view the learning of correct traces as largely composed of two stages: (i) negative gradients encourage exploration, leading the discovery of correct traces, (ii) positive gradients reinforce correct traces, once discovered.
For (i), we observe that training on easy problems exacerbates a tendency perform over-exploratory in-context exploration "underthinks" (see Example 3 in App. N), restricting the discovery of solutions harder problems. When utilizing coupled curriculum, this bias propagates to a shorter budget when compared to the data curriculum, since easy problems are trained on 256 rather than 1K tokens. As shown Figure/ 21, the average number of equations per attempt (naïvely, with candidate numbers, 2 equations are required to perform complete attempt 5 equations for 6 candidates) increases noticeably for the coupled curriculum in the second stage, but platcaus for the data curriculum, implying overfitting "over-exploratory" traces during the first stage.
Furthermore, for (ii), even when nontrivial positive rewards are obtained as we run the data curriculum on hard problems for 60 additional steps (steps 120 to 180), the training reward curve converges more slowly compared to the coupled curriculum (steps 60 to 120), implying that the data curriculum is also worse at reinforcing correct traces if the behavior over exploratory. While we do not many controlled experiments to identify why this might be the case, we hypothesize that this is because of imperfect and noisy credit assignment on over-exploratory traces with outcome rewards. is unclear which segments of the trace should be reinforced vs which segments might simply confuse the model.
I.3 COMPARING EXTRAPOLATION OF NAIVE RL WITH B
In Fig. 22, we compare 13-1.BB, trained with our coupled data-and- budget curriculum on DMATH (easy problems 8k, followed by medium and hard problems at 16k), to naïve RL baseline trained on the same dataset but only at a fixed 16k-token budget. While the naïve baseline shows only mild gains and quickly saturates
extrapolate beyond its training budget, e3 continues to improve
30

---

Published as a conference paper at ICLR 2026
e3 vs. naive RL
Omni- Conni-MATH (hard)
30
Acc. on
20
10
naive RL e3-1.7B
4k 8k 16k 24k 32k
Test-time compute (tokens)
Figure 22: Extrapolation performanceof e3 vs. naïve RL on hard problems: We plot the performance of e3-1.7B trained with our coupled curriculum (Section 6) on DMXTH, with a model that is trained directly at 16k training token budget on the same dataset (shaded region indicates the extrapolation regime). We find that e3 extrapolates test-time compute better than the naive RL baseline, when evaluated on test set of hard problems from Omni-MATH (Gao et al., 2024) dataset.
increase test-time compute to 24k and 32k tokens, achieving substantially higher accuracy the hard split of Omni-MATH problems: e3-1.7B outperforms the baseline by 0.07 points (gap of 7%) the training budget 16k tokens, and this gap grows 0.1 points (11%) when we evaluate the models on the extrapolated budget of 32k tokens. In Fig. 8, we compare the extrapolation performance of these methods AIME and HMMT 2025. There too we e perform better than the naive RL baseline but the gains are already large on the training budget of 16k tokens. In the above figure, we see the gap widen as we allow a bigger test time token budget on a harder set of 64 problems (sampled IID) from Omni-Math levels 6-8).
J OMITTED PROOFS
In this section, we present the formal version of Theorem 1, and provide detailed proof for it. First, we introduce some notations and provide proof overview.
Notations. We use the shorthand ((M;s) to denote the entropy of the conditional distribution over the next action de+1 given the current state s. We also use ( to refer to the policy parameters (for the softmax policy in Eq. 3) at iteration i of RL training, and use the shorthand (²()) to denote the policy induced by the parameter (a), We use V /fref(MMM to denote the gradient of function (M), with respect to M, evaluated at M = M Finally, we use Mg to denote the row of softmax parameters that model the distribution TM (- s), i.e., the row parameters (- s) in our parameter matrix M.
Proof overview. Without loss of generality, fix an arbitrary state s that is different from stop. Given the parameters (9) current RL iterate i, we do Taylor expansion of H H(M))s) around M(), and then show that the gradient V MO (()()):)) is positively correlated with the policy gradient with high probability over the sampling of the action a ~ TMT s), Le::
M, M; (MM()))) V M log (( s) A(s, a)) 2 0, whp. over sampling of action TMo (a
Before, we prove result that lower bounds the increase in entropy with negative gradients, present derivations of the entropy gradient with respect to the model parameters, as well as the policy gradient, which will simplify some calculations in the proof.
Lemma J.1 (Entropy gradient for the softmax bi-gram conditional). Fix a previous action (because the bi gr-gram state 8, O1-1, conditioning on the state is equivalent to conditioning on the last action) A. Let the (column-wise) logit matrix at time be M E (K(++))XK, and define the
31

---

Published as a conference paper at ICLR 2026
corresponding softmax conditional distribution
(at aa) exp(M((a a)) Z(a) = E exp((M(d)) a)).
[a)).
(5)
Z(a)
a'EA
Let the Shannon entropy of this conditional distribution be M(TMr I a)) or H(M a) Then
V (M a) E RK+1 is given by:
M HM a) == o (log H(m) = [I n. + H())) liEA
(6)
Proof. Write Pat == Tas (at
a) for brevity. By definition of the entropy,
H H Σ
Pat logPat Pat.
(7)
at
Insert the softmax expression:
logPo = M(a + aa) - log Z(a).
(8)
Hence,
H Σ Pat [Mia aa) log [(a]]
(9)
Pat M(a
aa) + log ((a) Σ Pat
(10)
Rearranging yields the following closed form expression:
H log ((a) - Σ Pat Mia la).
(11)
at
Computing the Jacobian of the softmax we get:
BMG a) n. (ō.g - m,);
J == V V diag(=) ा
(12)
Starting from the definition H = = -; Ti log x. and using the chain rule,
OH
Σ
(1 log n:)
Σ (i(ois T;) + log.
(13)
JM(
@M a)
Separating the term i i= from the rest:
0H
TT Ty) log TT)) + t, Σ m((1 + lsg
(14)
DM a)
= T Σ TT( + log Ti) - (1 + log n.
(15)
Because E, TT +lo++ogmm m.) = Σ; Te log n; = =1 H(T), we obtain
ÖH
aM(j a) , H()) - log RT)) v, (log TT + H(m)),
(16)
which gives the stated component-wise form. Writing this for every j simultaneously yields the vector expression with the Hadamard product.
L
Lemma J.2 (Policy gradient for the conditional distribution). For an action a ~ TML- 8), sampled
from policy TM(( s), at state s, the policy gradient is given by: V MM, log (a s) A(s, a), where A(s, a) is the advantage of action a. The expression for the 1ith coordinate of the policy gradient can be written down in closed form as:
M,
M. log mia s) A(s,a)), (1 a) a(a [s)ss)) A(s,
where 1 (.) is indicator function.
32

---

Published as a conference paper at ICLR 2026
Proof. Write Z = Ee exp s) and (b s) exp definition
exp M(b
s)/Z for brevity. By
Ta Mía s) log Z.
(17)
For any coordinate b A.
0z
(b log aas [ = 1(b = a)
Z 0M(b |s)
derivative of (als)
= 1(6 = a) exp M
( = a) Tb.
(18)
Z
Multiplying every coordinate by the common scalar A(s, a) produces the stated expression for g(s, M).
Theorem J.3 (Negative gradient increases (M;;s) when p(a"ls) (s) is low). For any state s, current parameters M suppose the most likely action ā is incorrect, i.e., a" + ā =; arg max, T Ma (b s). is Ta, and the second most likely action has probability ga Then, for small enough learning rate 1j > 0 s.t. with probability 2 Ta. negative gradient produces I 18+1) with entropy (M > (M)))s). Additionally, there exists a universal constant
where the probability of sampling
> O s.t., H(M(++);s)) - H(M();) 2 K Pa) whenever ла 2 E + e -H(M In contrast, without negative gradient the entropy remains same with probability 1 =T(a* |s)
Proof. For simplicity let us denote 7 = -=11...., TK+1) E ((A) be the conditional distribution
produced by bi-gram softmax column x- i.e., the probability of sampling action a at state 8, with model parameters given by the current RL, iterate M()) Let us also denote,
a = arg max TT.
H(M =:
Σ DEA
ka log Ta.
where is the probability of sampling action state Given that the current policy тM samples action a (-
(-1 s), the stochastic policy gradient that updates the parameter is given by:
M M + 1 V M log(z(())) (a s)) A(s,
(19)
where 7j is the learning rate, Note, that the policy parameters would only be updated for the row corresponding to the state s. For simplicity, let us use the notation g for:
g V M log(= (a s)) A(s, a).
(20)
Then, M (++1)
M = 17 g. A gives, for some M on the segment (M M(++))
Taylor expansion of the concave function H/ (M;ss) s)
(++ s) = (M (s) + (V Mo (MM ().
+ (9) Vi, (M:ss) (g).
(21)
Let the least eigenvalue of the Hessian of the conditional entropy (note that the entropy is concave function) with respect to the logits be PM. and |p,r < oc, the moment (0) s) > for all actions a E A. This condition is easily satisfied by any policy in our policy class, with finite values of the parameter matrix M. Thus, whenever (g, V MnD H (M(): s)) > O there exists a small enough learning rate 77.
2(g. V M H(M :))
1 S
(22)
elgl3
such that H(M (MM is strictly positive. Thus, we can continue to reduce learning rate 7) such that we can ignore (1)²)) terms in Eq. 21, get the bound:
H(M -HMM
> " (V M MM HM V M log(n s) a)))
(23)
2
33

---

Published as a conference paper at ICLR 2026
Next, it remains to bound the right hand side of Eq. 23 with high probability over the sampling of the action a. For a single incorrect action draw a ~ л we set A(s, a) to be -1 and for such an incorrect action we define the alignment scalar:
- Mg log (a s) A(s, V Mgo H(M):))
T(a)
(24)
Plugging in the derivation of V Mi9 :(M(ss from Lemma J.1, compute the closed form expression for T(ss) using the following definitions:
0/ =: n: (H(M())) + log m) and, 1 =: Σ Tala
(25)
acd
Thus, one has T(a) satisfy:
T(a)= = 0/
when, a E A, i 8 at.
(26)
Note that is an increasing function in whenever > e--eeee ((M Next, we note that va 2 0.
[A| na 2 c H( since, H(M - log |A
Ta
2a 2 0. (27)
Finally, since v(x) x .HHM + x log x is convex in .::
2a 2
Σ TJU
0a - μ 00 0
(28)
The above two implications in Eq. 27 and Eq. 28, and the fact that a + a", together lead us to deterministic lower bound on T(a). implying that it is always positive:
T(a) 2 0.
(29)
This completes the derivation for the first part of Theorem J.3, which does not assume anything about the conditional distribution x =((0)(- s), directly yielding the following result.
Result (i): Under the conditional distribution 7 (5)) (. s), whenever the most likely action * a", then with probability least Ra. T(a) 2 0, for a x (+) s). and any policy in our class of softmax policies. Finally, we plug this into Eq. 23 to conclude that the policy gradient update with probability aa always increases entropy, for a small enough learning rate.
Next, we lower bound T(a) when the second most likely action under the distribution satisfies an additional condition. For this, let us fix some 2 0, such that for g arg max ( (b s), we have 7g - E. Based on our alignment scalar T(-), we define the function g(z) as follows:
g(x) = x (H(M() + log .r)
0 < x S 1,
(30)
where ((;();s) s) is the conditional entropy we defined previously. Then, given the most probable action a, and the runner up action g. the gap between T T(ā) can be lower bounded down follows when Ty 2 exp(-H((M()) (MM()) 1):
T(a)= g(Ta) ((Ta)= - Σ kg g(b)
byta
(1 - no) g( (1 Ta) g(4) = (1 - Ta) (g((ma) - g((m)),
(31)
where the second equality follows from the fact that g(TT,) Z g(b) for any b + a as soon as Tg 2
s)), which is implied by the condition on Ta, E in Theorem J.3.
By the mean-value form of Taylor's theorem there exists a [my. mal such that
8²
((Ta) = g(a) + gg((4) + 2 (()))
(32)
Because g is convex, gE( 1/5 and the linear term eg' (g) is non negative. The minimum of 1 l/2 on [TT.. Ra] is attained at x = Pa. whence g0({) 2 1/pa- Dropping the positive linear term and using this lower bound on the curvature yields Eq. 33.
2²
g2
g(Ta) g( >
>
K,
(33)
2 n
2
34

---

Published as a conference paper at ICLR 2026
since Taa 1/K+1. Plugging the above result into Eq. 31 we get the follow result.
Result (ii) Under the conditional distribution, (-) (- s) whenever the most likely action @ * a*, and when the second most likely action has probability x 2 exp (M then with probability at least na. T(a) 2 d' (TTa 2 - Ta), for
n (.) s), and universal constant c' >> 0. Finally, we plug this into Eq. 23 to conclude that the policy gradient update with probability Ra always increases entropy by least en Kee ra). for small enough learning rate.
Together, Result (i, ii) complete the proof of Theorem J.3.
K PERFORMANCE ON
BENCHMARKS
In this section, we test the performance of e on benchmarks beyond muthhreasoning, Our goal is to verify if our proposed curricula that improves performance on math benchmarks like AIME2025 and HMMT2025 retains performance non-reasoning tasks, compared to the base model Qwen3-1.7B.
We find that our RL-trained e3 model outperforms other RL/SFT baselines on average, including Open WpenThinner3-1.5 1.55B, which was released only a couple of months ago (at the time of the submission). Notably, despite being trained solely on math prompts, e3 achieves particularly large gains on GPQA-D.
Model
MMLU
MMLU-Pro
GPQA-D
Qwen3-1.7B
64.90 (#: 0.76)
36.60 (# 0.86)
33.70 (#: 0.77)
agentica-1.5B 1.5B (RL)
60.50 (# 0.78)
32.00 (: 0.84)
28.40 (# 0.74)
OpenThinker3-I.s (SFT)
67.80 (#: 0.75)
37.00 (:: 0.86)
30,90 (# 0.76)
e3-1.7B (Ours, RI.)
66.30 (+: 0.76)
36.30 (+ 0.86)
37,80 (+ 0.79)
Table 9: Non-reasoning MCQ benchmark results with 95% confidence intervals. Despite being trained only on math prompts, our RL-trained c3-1.7B shows strong generalization, outperforming both RL and SFT baselines on GPQA-D.
We compute the standard deviation using 15,000 points on MMLU, 12,000 on MMLU-PRO, and using 32 rollouts for each of the 450 problems on GPQA-D (n = 450 x 32 = 14, 400).
L BROADER IMPACT STATEMENT
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here. Our findings deepen our understanding of how to train large language models (LLMs) to reason more effectively under test-time compute constraints, which could influence the design of future AI systems. Our approach introduces a training recipe that encourages models learn structured in-context exploration strategies, improving their ability to solve harder problems as test- time compute increases. While this has the potential to improve Al reasoning and decision-making across domains, it also raises questions around the unequal access to models that can leverage such capabilities, especially in settings with limited compute resources. As with all work that improves model capability, care should be taken to evaluate downstream risks such as misuse unintended consequences from stronger extrapolative reasoning. We encourage future research to assess faireess, interpretability, and safety implications as these systems deployed in real-world environments.
NOTE ON COMPUTATIONAL RESOURCES USED FOR E M
For our experiments with CDOWN, MULT, MULT-V, we used a single node with NVIDIA H100 GPUs. For our experiments DMATH we used single H100 node for training budgets upto 8k. For training budgets of 16k and beyond, we used four nodes of NVIDIA H100, each with GPUs. Alternatively, were able
CDOWN, MULT, MULT-V, and DMATH (upto a training budget of 8k) using TPUs. In particular, we found that v5litepod- 128 TPU, which has 32 nodes, 128 total chips (TPU configuration of 32 4) is sufficient run our experiments. Our experiments on GPUs used the open-source LLM post-training framework from Sheng et al. (2024) that uses FSDP sharding to parallelize data and split model parameters across the multiple cards.
experiments
35

---

Published as a conference paper at ICLR 2026
N EXAMPLE TRACES
Below, we present some examples of chained asymmetries in CDowN and MULT-V. Here, we measure the verification calls by running the model generated trace through regular expression parser which checks how many sentences in the model's trace start with the following set of phrases: "Wait", "But wait", "Alternatively". "Is there another way", "But let me double". "But hold on". We picked these phrases based prior work (Marjanović et al., 2025) that did an extensive analysis on the patterns and behaviors (like self-verification, planning, etc.) present the thinking traces of Deepseek-BR
Example I: Chained Asymmetries in Countdown
<think> Let me analyze this step by step:
1. First, let's try to get close to 466 with simple operations:
65 56 121 (too small) 65 56 (too small) 65 56 way too way too large
large 56
2. Let's try working backwards from 466: 466 is much larger than any of the given numbers, so multiplication
will likely be needed. We need to get from 37 to 466, which means we need to multiply by
12.5. But don't have 12.5, so we' 11 need create it through
division.
3. Let's try: 37 14 = 518 (too large) 37 14 23 (tco small)
37 + 14 = 51 (too small)
4. Let's try different approach: 65 + 61 126 126 + 56 182
182 37 219 (getting closer)
5. Let's try: 65
126 126 56 182 182 37
219 219
14 233 (still too small)
6. Let's try one more approach: 65 56 121 121 61 182 182
37 219 219 +
233 (still not 466)
7. Let's try: 65
121 121
182 182
219 219
14 233 (still not 466)
8. One more attempt: 65 61 = 126 126 + 37 163 163 + 56 219
219 14 - 233 (still not 466)
9. Final try: 65
126 126 56
182
219 219
14 205 (not quite there)
10. Let's try: 65 + 61 = 126 126 + 56 = 182 182 + 37 = 219 219
14 233 (still not 466)
11. New approach: 65 61 = 126 126 56 132 182 37 145 145
+ 14 159 (not quite there)
12. Final attempt: 65 + 61 126 126 + 56 = 182 182 + 37 = 219
219 -44 = 205 (not quite there)
13. Let's try: 65 + 56 = 121 121 + 61 = 182 182 37 = 219 219
14 205 (not quite there)
14. One last try: 65
102 102
163 163 56 219 219
+ 14 233 (not quite there)
15. Final attempt: 56 + 14 = 70 70 65 135 135 + 37 172 172
61 233 (not quite there)
16. New approach: 37
518 518
453 453
514 514
56 458 (not quite there)
17. Final try: 61 14 - 854 854 - 65 - 789 789 56 733 733
37 696 (not quite there)
18. Let's try: 37
518 518 65
453 453
514 514
56 458 (not quite there)
19. New approach: 56 * 37 = 2, 052 2, 052 / 65 approx.
31.5 (not
helpful)
20. Final attempt: 65 14 910 910 61 849 849 56 793
793 37 830 (not quite there)
21. New approach: 61 37 2, 237 2, 237 65 1,172 1,172 56
1,116 1,116 14 1,130 (not quite there)
22. New approach: 37 14 518 518 65 = 583 583 61 = 522 522
+ 36 578 (not quite there)
23. Final attempt: 65
910 910
849 849
905
905 37 868 (not quite there)
36

---

Published as a conference paper at ICLR 2026
24. New approach: 37 14 = 518 518
56 = 462 462
65 = 527 527
61 466 (This works!) </think> <answer> (37 14 56 65) 61
</answer>
Example 2: Multiplication with verification
Okay, so I need to multiply 347 by 583. Hmm, 1et think about how to approach this. guess can use the standard multiplication algorithm where you break it down by each digit. Let me start by writing both numbers down: 347 583
First, I'll multiply 347 by each digit of 583 starting from the rightmost digit, which is in this case. Then move to the next digit (8) and finally the leftmost digit (5), remembering to add the appropriate
for each subsequent miltiplication. After that, I'll sum all the results together. Let me start with the rightmost digit.
Starting with the units place: 3 347. Let me compute that: x 7 21. I write down and carry over 2.
12, plus the carried over makes 14, Write down and carry over 1.
x x 3 9, plus the carried over 1 makes 10. Write down 10. So, 3 x 347 = 1041. Let me check that again:
xx1 21 (correct),
12 2 14 (correct), x that's 1041. Alright.
10 (correct).
Check: 3 x 347 1041. Correct.
Next, the tens place. The digit here is 8, but since it's in the tens place, it's actually 80. So, we'll multiply 347 by 80. To make it easier, end.
multiply 347 by and then add
the
x 347:
8 x 7 = 56. Write down 6, carry over 5.
over 32, plus the carried over makes 37. Write down earry
24, plus the carried over makes 27. write down 27. So, 8 x 347 2776. Then add the zero for the tens place, making it 27760. Let verify:
x 347. 9x300=2400, 8x40=320, 8x7=56. 2400+320=2720,
2720+56=2776. Then adding the zero gives 27760. Correct. Check: 8 x 347 = 2776, so 2776 x 10 27760. Correct. Now, the hundreds place. The digit here is 5, but it's in the hundreds place, so it's 500. So, multiply 347 by 500. Again, multiply 347 by and add two zeros.
347:
5 x 7 35. Write down 5, carry over 3.
5 x 20, plus the carried over makes 23. Write down 3, carry over 2.
3 15, plus the carried over makes 17, Write down 17.
So, x 347 1735. Then add two zeros, making it 173500. Let me verify:
x 347. 5x300-1500, 5x40-200, 5x7-35. 1500+200-1700,
1700+35-1735. Then adding two zeros gives 173500. Correct. 347 1735, so 1735 x 100 173500. Correct.
Check:
Now, add all three results together: 1041 + 27760 + 173500. Let me do this step by step. First, add 1041 and 27760:
1041 + 27760. Iet me linc then up:
1041
+27750
28801
37

---

Published as a conference paper at ICLR 2026
Wait, 1041 27760: 1000 27760 28760, then +41 28801. Let
me check with another method:
20000 7000 27000, 700 60 760, 27000 + 760 27760. Then
27760 + 1000 = 28760, +41 28801. Correct.
Check: 1041 + 27760 28801. Correct.
Now add this result to 173500:
28801 173500.
me line them up:
28801
+173500
202301
Mait, 28000 + 173500 201500, then
801, so total 201500
801 202301. Let me verify another way:
28801 + 173500. Break down 173500 into 170000 + 3500. Then 28801 +
170000 198801. Then 198801 3500 202301. Correct.
Check: 28801 173500 202301. Correct.
so, putting all together, the final result is 202, 301. Let me just cross-veriřy using another method to ensure didn't t make any mistakes in the addition.
</think>
The product of 347 and 583 is 202301
Example Over-exploration Countdown (after training only on easy problems with candidates, each attempt contains most 3 equations when 5 is needed)
<think> Let me try to find a way to get to 66. First, let me look for numbers close to 66 or factors/multiples that might be useful. 66 is close 70, which is of cur numbers. Maybe 66, so mayoe we can create
subtract or add to get to 66. 70 somewhere.
Looking at the other numbers: 77 70 = 7 97 73 = 24 (interesting, 24 576 (too large)
al already have 24) 24 24 48 (not helpful) 24
Let me try a different approach: 31 (not helpful)
77 70 97 73
+
New approach: 97 - 73 24 24
70 = 94 (too far)
Let's try: 77 70 7 24
168
(t00 far)
One more attempt: 70
24
94
-3
77 -231 (too
far)
Final attempt: 97 24 = 73 73 - 70 = 3 3 77 = 231 (toc far)
Let's try: 70 24 = 94 94 - 77 = 17 17 65 82 (toc far)
Wait: 24 13 1748 1748
1671 (too far)
Last try: 77 24 101 101
70
31 97 -66 (not what we
want)
Let's try: 97 -3- = 24 24 + 70 94 94
24 = 70 (not helpful)
Final attempt: 70 24 1680 1680 / 77 21.9 (not valid)
Last try: 73 65 8
192 192 / 97 1.98 (not valid)
what if: 24 3 So: 97 73 24 24 24 -0 11 (not helpful)
need to keep trying different combinations.
Final attempt: 77 97 = -20 -20 24 = -480 4480 + 70 = -410 (too
far)
Last try: 70 24 1680 1680
21.9
so the most promising path scoms to bc: 97 - 73 24 24 70 94
94 77 17 This gets us too far from 66.
Let's try: 77 - 70 = 7
24 168 168
97
171 (toc far)
Final attempt: 97 70 167 167
143 143
73 1.97
Mait: 24 65 1560 1560
20.26
Last try: 77 - 65 - 12 12 24 - 288 288
97 3
need to keep trying different combinations
What it: 97 70
27 21 24 1.125
Mait: 24 97 121 121 13
48 48
125 (to0 far)
Last attempt: 70 24 94 94
- 17
+ 65 82
38

---

Published as a conference paper at ICLR 2026
One final try: 24 65 1560 1560 / 97 16.06
Final solution: 97 24
65
1560
20.26
Let me try: 77
31
+ 65 96 (toc far)
Last attempt: 70 24 1680 1680
97 = 1583 1583 / 77 20.59
need to keep trying.
What if: 97 + 24 121 121
73 48 48
65 113
Mait:
24
118 118
Final try: 24 65 = 1560 1560 + 70 1630 1630 77 21.16
One last attempt: 77 24 1848 1848 / 65 28.46
USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) as an assistive tool primarily for rephrasing arguments more crisply and for generating LaTeX templates (e.g., tables, algorithm boxes, or figure formatting). All research ideas, theoretical developments, experiments, and empirical results were conceived, executed, and validated by the authors. The LLM did not contribute to the scientific content, claims, or findings of this work.
39


Reviews:
[{"review_id": "VELFc0kZS8", "reviewer": "Reviewer_X6nQ", "summary": "The paper argues that most open-source “reasoning” LLMs don’t actually improve when you give them much more test-time compute beyond what they were trained on, and proposes a training recipe, e3, to fix that by teaching models to explore in-context. e3 has three parts: (1) exploit asymmetric skills in the base model (especially a verification-better-than-generation gap) so the model can chain “generate → verify → revise”; (2) keep the negative gradient during RL so failed traces push probability mass toward longer traces that chain more of those asymmetries; and (3) use a coupled curriculum that jointly schedules problem difficulty and token budget so exploration is rewarded and optimization stays stable.", "questions": "1. Fig. 8(c): Is the teal curve measuring accuracy for an 8k-budget model, with the jump caused by switching the same checkpoint to 16k at evaluation? If so, could you clarify exactly which components change across the plotted segments?    \n\n\n2. How is “verification” detected in traces—pure string heuristics (e.g., segments beginning with “Wait, …”, “Let me verify …”), or a more robust parser?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles an important gap: most open-source “reasoning” LLMs fail to extrapolate to longer test-time compute than they were trained on.  \n\n2. It analyzes the verification-generation (VG) asymmetry and argues it is a key driver of RL gains beyond the training length by enabling chains of “generate → verify → revise.”  \n\n3. The coupled curriculum is clearly specified (including a concrete budget-selection rule) and empirically outperforms many SOTA models.", "weaknesses": "1. In Fig. 8(a), Qwen3-1.7B with naïve RL appears to extrapolate similarly to e3 at some budgets; the large headline gains seem to come primarily from better training-length performance (≤16k). The analogous naïve-RL baseline is missing from Fig. 8(b), which makes cross-dataset claims harder to assess.  \n\n\n2. Figures 8(a) and 9 both report AIME’25 accuracy for Qwen3-1.7B/e3 but show different values and ranges; the relationship between the two protocols (SOTA comparison vs. “wait”-prompt budget forcing) should be reconciled to avoid confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6rIVVhYEGv", "reviewer": "Reviewer_1mFo", "summary": "The paper argues that realizing the promise of test‑time scaling requires learning in‑context exploration, and proposes e3 which is a training recipe that teaches models to spend extra tokens on structured generate‑verify‑revise behavior so performance extrapolates beyond the training budget. e3 leverages chained asymmetries in the base model (verification–generation gap), preserves negative RL gradients so failures shift probability mass from premature endings into longer, verification‑heavy traces, and uses a coupled curriculum that pairs data difficulty with token budgets to reward chaining within budget and beyond. On Qwen3‑1.7B trained up to 16k tokens, the resulting e3‑1.7B extrapolates to 32k and achieves state‑of‑the‑art results among sub‑2B models on AIME’25 and HMMT’25, with higher pass@k; good analyses and ablations link negative gradients to longer responses, higher entropy, and more verification.", "questions": "I don't have any more questions, please see weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper offers a curriculum based training recipe chaining asymmetries provides remedy to the problem that many open models fail to extrapolate beyond their training budget. The empirical results are strong on the math benchmarks and the choice of Countdown and Multiplication tasks are illustrative in depicting the VG gap.\n- The ablations and mechanistic analyses are well conducted and support the presented claims, and the theory section also provides good insights. Overall, I believe the paper is well-written with clear contributions and gains.", "weaknesses": "- The paper does not directly measure/report how verification capability or the VG gap evolve during training, which I believe is an important direction to investigate to better understand the verification‑generation training dynamics.\n- As an ablation study, the paper demonstrates masking negative gradients in GRPO Mask baseline. Yet, masking negatives also changes the update magnitude and how GRPO’s advantage calculation & clipping behave. So, a more controlled study could have provided better evidence, as the observed gap could stem from these rather than the gradient sign itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZRfUJlVVU", "reviewer": "Reviewer_EVJh", "summary": "The authors state that the model's reasoning capability does not extrapolate well beyond the maximum budget observed during training. The paper explains why the output increases during RL training, with asymmetries. They argue that constraining the budget would encourage the model to become more precise rather than interleaving \"skills\". They suggest recipe \"e3\", which dynamically adjusts the training budget according to the difficulty of the prompts.", "questions": "- Would enforcing asymmetry lead to improved final performance? For instance, one could explicitly teach the base model on verification tasks before training how to solve the problems.\n- Can e3 be extended to explore more fine-grained training budgets? Would the final performance benefit from that?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The concept of \"chain of asymmetric skills\" offers a compelling explanation for the reasoning trajectory extension during RL. Likewise, the \"verification-generation gap\" (VG Gap) clearly accounts for the reasoning path that appears in the pattern of solution-verification-solution-....\n- The authors trained a state-of-the-art  (<2.0B) model on AIME/HMMT 2025 using the suggested method e3.\n- The paper presents a comprehensive experiments that strongly support the authors' claims.", "weaknesses": "- The proposed method \"e3\" appears to lack novelty. The paper does not suggest a clear approach to determine a proper value of $\\kappa$, nor does it include additional experiments on the impact of different $\\kappa$ values. Also, the budgets it explores (4k, 8k, 16k) are too coarse, leading to suboptimal performance.\n- The definition of problem hardness is somewhat ambiguous. For example, the authors define the difficulty level for each dataset separately.\n- The scalability of e3 remains uncertain. If the asymmetry (VG gap) appears only in the small-scale model (I believe so), the broader implications of the paper would be significantly weakened.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NHdrbDos8Z", "reviewer": "Reviewer_erre", "summary": "The paper studies different aspects of test-time scaling driven by Reinforcement Learning. In particular, the authors study the role of chaining asymmetric capabilities (e.g., generation and verification), negative gradients and training length budget in improving RL performance in-distribution, out-of-distribution and extrapolating to longer test budgets. The authors conclude that exploration in RL is enabled by asymmetric capabilities chaining and that negative gradients are important for driving exploration and extrapolation beyond the training budget. Following this, the authors introduce a new method for improving RL performance, by using a curriculum that increases the training length budget together with the data complexity. This results in a model that achieves SOTA performance under 2B.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the paper is well-written, and the introduction section did a good job of motivating the problem setting.\n- The paper studies a fundamental problem in scaling test-time compute using RL and provides novel insights on the role of different aspects of the training pipeline.\n- The discussion of the role of asymmetric capabilities is interesting and novel, and the authors provide experimental evidence to the important of this chaining.\n- I found the analysis of the effect of varying the training length budget on the performance of RL extrapolation to be very interesting, particularly the fact that smaller training length can sometimes perform better when extrapolating to longer sequence lengths compared to training on a larger length. The \"trade-off\" between training length and problem complexity is also interesting.\n- The curriculum method introduced in the paper seems to be convincingly better than alternatives in the setting it was tested.", "weaknesses": "- One main issue that I have with the results and analysis (especially in Section 4 and Section 5) is the fact that in many cases it seems very hard to attribute the improvement/degradation in performance to the properties identified by the authors. Examples for this:\n  - Section 5 discusses the role of the negative gradients in RL, and concludes that they are important for driving exploration, increase in length and extrapolation. However, the authors test this by \"masking\" negative gradients from the RL algorithm, which results in an overall degradation in performance. The \"masking\" clearly makes the algorithm worse, and then a lot of metrics become worse, but it is hard to tie the negative gradients to particular aspects of this, like the authors try to do. For example, we can make the algorithm worse by e.g. choosing wrong hyper-parameters, under-training or over-fitting. Would this not result in a similar behavior, even though negative gradients are used? An experiment like this, i.e. comparing \"masked\" negative gradients to other modifications to the algorithm that result in the same overall accuracy would help to shed light on this.\n  - Section 4 suffers from similar issues, though I think this section is more convincing. Here the authors claim that generation-verification gap drives performance improvement in RL, but this is mostly shown by measuring the number of \"chaining\" occurrences and showing that it increases with RL steps. But it is known that RL increases length, and that this is correlated with improved performance, so it seems that this just finds another property (number of chained asymmetries) that is correlated with increased length. \n- Another main issue is the apparent disconnect between the different parts of the paper. The role of negative gradient seems unrelated to the \"chaining asymmetries\", and these two seem unrelated to the new method introduced in the paper. While the authors claim that the scientific analysis drives the resulting method, it's not clear how this method relates to the analysis.\nMore minor issues:\n- The authors discuss the asymmetric capabilities chaining in general, but in practice the paper is focused on generation-verification gap. Not clear what this more general characterization is adding.\n- The discussion on the role of negative gradients in relation to the position of the EOS token is unclear. In particular, while negative gradients reduce the probability of the EOS token appearing at a given place, they could also increase its probability for appearing earlier. In general, I think that a more theoretical discussion of the role of negative gradients can be introduced into the main paper (now only in the appendix). This could perhaps clarify things.\n- The method introduced in the paper is relatively simple: training in two phases, starting with a simple problems and short sequence length followed by harder problems and longer sequence length. I think that the method is introduced makes this seem more complex than it actually is. While I don't see any problem with a simple method that improves performance, introducing it in a convoluted way seems to suggest that the method is more sophisticated or general than it actually is.\n- The plots in Figure 8 are very hard to parse, with some of the labels overlapping.\n- Line 333: \"Fig. 5(a) confirms this....\" - I think this is not referring to the right figure.\n- The description of the n-digit multiplication experiment in Section 4 is not clear.\n- Lines 321: \"the policy gradient update reinforces chaining and improves in-context exploration, and this is reinforced\" - unclear what this is saying.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)